Facial recognition with material data extracted from images

Abstract

The present invention relates to a method for authorizing an object of a device to perform at least one operation on the device requiring authentication, the method comprising the following steps: - receiving, from a user interface associated with the device, a request to unlock the device from a user; - triggering, in response to receiving an unlock request, illuminating the object with patterned infrared illumination; - triggering the capturing of at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared illumination; - extracting material data from the at least one image; - allowing or denying the object to perform at least one operation on the device requiring authentication based on the material data.

Description

[Technical field]

[0001] The present invention relates to a method, an apparatus and a computer program for identifying an object, in particular for authenticating an object. [Background technology]

[0002] In the prior art, material detection from images can be performed by filtering techniques. Such techniques analyze the beam profile, require intensive computational resources, and in some cases may not provide the required reliability. Therefore, there is a need to improve material detection in terms of reliability and computing cost. Summary of the Invention [Problem to be solved by the invention]

[0003] Therefore, the object of the present invention is to provide a method and an apparatus for the above-mentioned technical problem. In particular, the object of the present invention is to provide a method and an apparatus for reliable identification of materials in general. Furthermore, it is an object to achieve reliable identification and / or authentication of objects or people, especially in situations where a user of a device is authorized to perform at least one operation on the device that requires authentication. In particular, it is an object to provide a method and an apparatus for reliably identifying at least one material information of an object or the object itself, preferably with low technical effort and low demands in terms of technical resources and costs, even in complex situations, for example, when the object contains or is covered with various materials, possibly unwanted materials. [Means for solving the problem]

[0004] This technical problem is solved by the technical subject matter with the features of the independent patent claims. Advantageous technical embodiments of the invention, which can be realized individually or in combination, are presented in the dependent claims and / or in the embodiments described in the following description and / or with reference to the drawings.

[0005] In a first aspect of the present invention, the technical problem of the present invention is solved by a computer-implemented method for extracting material information of an object from a pattern image of the object, the method comprising: receiving a pattern image showing the object; manipulating the pattern image to generate a partial image; extracting material information of the object from the partial image by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training dataset comprising the partial image and material information; and providing the extracted material information.

[0006] In another aspect, the present disclosure relates to a method for authorizing an object of a device to perform at least one operation on the device requiring authentication, the method comprising the steps of: receiving a request to unlock the device from a user from a user interface associated with the device; triggering illumination of the object with patterned infrared illumination in response to receiving the unlock request; triggering capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared illumination; extracting material data from the at least one image; and permitting or denying the object to perform at least one operation on the device requiring authentication based on the material data.

[0007] In another aspect, the present disclosure relates to a device for authorizing a user of the device to perform at least one operation on the device requiring authentication, the device comprising: - receiving, from a user interface associated with the device, a request to unlock the device from a user; - triggering, in response to receiving an unlock request, illuminating the user with patterned infrared illumination; - triggering the capturing of at least one image of the user using a camera disposed on the device while the user is illuminated with the patterned infrared illumination; - extracting material data from the image; - allowing or denying the user to perform at least one operation on the device that requires authentication based on the material data; The method includes a processor configured to execute the steps of:

[0008] In another aspect, the present disclosure relates to a computer program product for authorizing a user of a device to perform at least one operation on a device that requires authentication. In this regard, the computer program comprises instructions for causing a computer to carry out the steps of the method according to the present disclosure.

[0009] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, in which material information is derived from reflection of patterned illumination.

[0010] Since the material information is extracted from the partial images generated by manipulating the pattern image of the object, the computational resources for identifying the material can be reduced. Moreover, by providing the partial images to the data-driven model, the complete pattern image, especially including the potential background, is not utilized for extracting the material information. Moreover, the part of the training process of the data-driven model that is necessary to train the identification model not to ignore the potentially highly variable background can be avoided, leading to a reduction in the required training data. In particular, the data size of each of the partial images (which are numerous to achieve a well-trained model) can be smaller or equal to the data size of the corresponding pattern image of each of the pattern images (which are numerous to achieve a well-trained model). Thus, accurate identification of the material information of the object and / or identification and / or authentication of the object (and even in complex situations, e.g., when the object contains or is covered with various materials, possibly undesirable materials, and is authorized as a user of the device to perform at least one operation on the device requiring authentication) is provided, which utilizes less training data for training the data-driven model, which reduces the technical effort and requirements, especially in terms of technical resources, costs and computational expenses. Therefore, the data-driven model requires fewer parameters, resulting in, for example, fewer neurons in the first layer. Thus, the size of the layer is reduced. All of this further leads to a reduction in the overfitting problem. All of the described technical effects apply to the method, device, data medium and use thereof of the present invention, since the model is trained to be utilized by the method, device, data medium and use thereof of the present invention after training.

[0011] Moreover, manipulating the pattern image into smaller sections for partial images, for example by cropping, has the further advantage that the data-driven model can avoid basing its calculations too strongly on correlations of features in completely different areas of the image showing the object. Considering only correlations of features that are close to each other on the object, i.e. that can be found in the same area of ​​the object, increases the accuracy of the output result. Furthermore, since the material information is further used for the identification, and in particular authentication, of the object, it is almost impossible to fool the identification, and in particular authentication, of the object if an attacker tries to achieve a fraudulent identification / authentication by using an image, a mask, for example a silicone mask, a replica, for example a latex, silicone, cloth replica, etc., to imitate a part of the (face of) a correct human. The present invention improves the security aspect, since spoofing attacks are advantageously prevented. Moreover, identification, and in particular identification / authentication of the object is advantageously possible, especially if the object is a user of the device that should be approved to perform at least one operation on the device requiring authentication. This is particularly useful when the user of the device is wearing non-skin parts, such as glasses, a beard, a hat, etc., or a (partial) mask for hygiene or health reasons. Such occlusions and / or features can be detected by the material information, and the detection is used for identification / authentication / authorization. The invention is therefore applied in particular for the purposes of identification / authentication / authorization in the process of unlocking the device by the user of the device. As a result, the trust level and the overall reliability of the inventive method, the apparatus, the data medium and their use are increased.

[0012] As a result, the present invention provides higher prediction accuracy, particularly by utilizing less training data and fewer computational resources, for all of the extraction of material information of an object, identification and / or authentication of the object, and authorization of a user of a device to perform at least one operation on the device that requires authentication, by utilizing machine learning identification models that can be trained with less computational cost.

[0013] Thus, the present invention provides high accuracy in extracting material information of an object, identifying and / or authenticating the object. Utilizing a machine learning identification model trained on partial images reduces computational costs and therefore allows for use on less sophisticated computing resources, such as mobile devices such as smartphones or tablets.

[0014] In one embodiment of the invention, the object is for example a human being, preferably the face of a user of the device. In a further preferred embodiment of the invention, the material information relates to human skin, silicone, resin, latex, paper, metal, etc. In a further preferred embodiment of the invention, the material information comprises a material class and / or a material type. The material class may comprise biological or non-biological material, translucent or non-translucent material, metal or non-metal, skin or non-skin, latex or non-latex, silicon or non-silicon, cloth or non-cloth, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. The material type may comprise further specifications of the material class, such as for the biological class the type is human skin, or for the non-biological class the types are plastic, glass and metal, etc. Further subtypes may be assigned to each type. In an embodiment, the material information may represent a material type and / or data derived from the material type. The material information may be suitable for obtaining a material type. The material information may comprise a chemical composition and / or a material type. Examples of material types may include inorganic materials, organic materials, silicon, metals, wood, skin, etc. Data derived from the material type may be data based on the material type. In the exemplary case where skin is a material type, data derived from the material type may include objects that are living organisms. The living organism may be, for example, a human being.

[0015] The inventors have developed a technique to measure the distance of an object (e.g. to a reference point, such as the position of a camera) as well as the material of the object. To do so, standard hardware is used: for example, an IR laser point projector (such as a VCSEL array) can be used that projects a spot pattern onto the object and a camera that records the object under illumination, such as a CMOS. In contrast to the structured light approach established in the prior art, only one camera is required.

[0016] Information about the distance of the object and information about the material of the object are extracted from the shape of the spot reflected by the object. The ratio of the light intensity of the central part of the spot to the light intensity of the outer part of the spot contains information about the distance of the object. Information about the material or material of the object can be further extracted from the intensity distribution of the spot reflected by the object. This is possible, among other things, because each material reflects light differently. For example, (infrared (IR)) light penetrates the skin relatively deeply, which results in a certain degree of broadening of the spot. Further analysis of the material of the object can be performed by applying a series of filters to the reflected image in order to extract various different information of the spot. Thus, for example, skin can be detected as the material of the object. Furthermore, material information or material features regarding the material of the object, such as, for example, beard, glasses, hat, scarf, etc., may be determined. Details will be described later. Thus, a rule-based consistency check can be applied, and thus spoofing attacks can be highly prevented, improving the security level with respect to the detection of material information in general, and in particular, the identification / authentication of the user of the object, e.g., the device.

[0017] Complementarily or alternatively, the advantages of artificial intelligence can be exploited. For example, the prior art teaches training a neural network to detect whether an image contains a desired object. However, training such a neural network requires a large number of images. This often leads to overfitting of the neural network and a decrease in the accuracy of the results. Moreover, even after proper learning, which is often the case, only unreliable recognition is obtained, especially when the object / user is covered / wears / contains parts / occlusions that are not desired by the user (e.g., masks, glasses, replicas, etc.).

[0018] According to the subject matter of claim 1, in particular a pattern image showing an object is received and manipulated to generate partial images that may contain no pattern features or one or more pattern features, which are then provided to a data-driven model (e.g. a neural network or a machine learning based identification model) to extract material information of the object. The subject matter according to the invention therefore provides for the use of neural networks with less training data. Overfitting is thus prevented and at the same time the identification accuracy is improved. In particular, the invention provides a method that allows to accurately identify material information of an object and that allows to utilize less training data to train a data-driven model for identifying the object. Furthermore, a further object of the invention is to provide a training method that can be used in a method for identifying / authenticating material information and / or objects and that allows to provide a data-driven model that can be trained to provide good prediction accuracy and at the same time utilize less training data and less computational resources.

[0019] The method relates to a computer-implemented method and can therefore be performed by a general or dedicated computer or computing device, such as a smartphone, laptop, tablet, etc., adapted to perform the method, for example by executing a respective computer program. Furthermore, the method can also be performed by multiple computers, for example by a computer network or any other kind of distributed computing, in which case the steps of the method can be performed by one or more computing units.

[0020] The method allows for the identification of material information of an object. For example, the material information can be assigned to one or more of a material name, a material group, a material class or a material type, such as biological or non-biological, translucent or non-translucent, metal or non-metal, skin or non-skin, latex or non-latex, silicon or non-silicon, cloth or non-cloth, reflective or non-reflective, specular or non-specular, foam or non-foam, hair or non-hair, roughness group, etc. Thus, the material information can relate to human skin, silicone, resin, latex, paper, materials from which masks can be made.

[0021] Examples of determining material information of an object include, for example, determining whether the object (part) from which an image has been acquired contains skin or fur, i.e., belongs to the class of human or animal, or whether it contains metal or plastic, i.e., belongs to the class of non-living materials such as glasses or masks. For example, the owner of a mobile device can take a picture of the object (part) from which material information of the object (part) is derived. The term object refers to an entity from which material information is derived, a human, (part of) an animal, etc., non-living materials such as glasses, masks, replicas, etc. For example, the object can be a face, a fingerprint, or an equivalent part of a body that can be used specifically for identification purposes.

[0022] The defined method includes receiving a pattern image indicative of the object. For example, the pattern image can be received from a camera unit that takes images while the object is illuminated with a light pattern. Complementarily or alternatively, the pattern image can also be received from a storage unit in which the pattern image is already stored. Complementarily or alternatively, the pattern image can also be received by user input, for example a user input indicating which of a number of images stored in the storage should be used as the pattern image.

[0023] According to a preferred embodiment of the present invention, material information of an object is obtained from the illumination of a pattern or reflection of a light pattern on the object.

[0024] In general, the sensor may be configured to record at least one image of a light beam originating from at least one object, for example via reflection. The term reflected image (or simply denoted pattern image) may refer to an image determined by a sensor element comprising at least one reflection feature (or simply denoted feature). As used herein, the term determining at least one (reflection) feature refers to imaging and / or recording at least one light beam generated by an object in response to illumination by a light beam, in particular illumination with at least one illumination feature (or simply denoted feature). In particular, the sensor element may be configured to determine and / or image and / or record at least a (reflection) image. Preferably, the (reflection) image may comprise at least one pattern comprising at least one (reflection) feature.

[0025] Specifically, a pattern image may refer to an image of an object captured while the object is illuminated or illuminated with a light pattern (also referred to as pattern illumination or illumination pattern). The light pattern may include one or more pattern features. The pattern image may include contributions from the reflection of the light pattern on the object and / or from the texture of the object. Texture relates to surface characteristics such as size, shape, density, composition, proportions, etc. of certain parts of the object.

[0026] The light pattern can be generated by any kind of light pattern generating unit, for example laser light, in particular infrared laser light. In particular, one or more vertical cavity surface emitting lasers (VCSELs) are preferably used to generate the light pattern comprising a plurality of laser light spots. However, other light sources can also be used to generate the light pattern, for example LED light sources of one or more colors.

[0027] Preferably, the pattern image is acquired by a detector, a sensor, etc. The detector may be a fixed or mobile device. Furthermore, the detector may be a standalone device or may form part of another device, such as a computer, a vehicle, or any other device. Furthermore, the detector may be a handheld device. As used herein, the term sensor or sensor element generally refers to a device or a combination of devices configured to sense at least one parameter, and may in particular be part of a detector. In this case, the parameter may in particular be an optical parameter, and the sensor element may in particular be an optical sensor element. The sensor element may be formed as an integral single device or as a combination of several devices. The sensor element may comprise a matrix of optical sensors. The sensor element may comprise at least one CMOS sensor. The matrix may be composed of independent pixels, such as independent optical sensors. Thus, a matrix of inorganic photodiodes may be constituted. Alternatively, commercially available matrices may be used, for example one or more of CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips. Thus, in general, the sensor element may be and / or include at least one CCD and / or CMOS device and / or the photosensor may form or be part of a sensor array, such as the matrix mentioned above. In one embodiment of the invention, one or more of commercially available matrices, e.g. CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips, may be used.

[0028] The detector and / or sensor and / or sensor element can be configured to detect biological tissue, in particular human skin. As used herein, the term biological tissue generally refers to biological material that includes living cells. The detector may be a device for detection, in particular optical detection, of biological tissue, in particular human skin. As used herein, optical sensor generally refers to a light-sensitive device for detecting a light beam, such as for detecting illumination and / or a light spot generated by at least one light beam. As further used herein, a light-sensitive area generally refers to an area of ​​the light sensor that is externally illuminated by at least one light beam and generates at least one sensor signal in response to said illumination. The light-sensitive area can be specifically located on the surface of the respective light sensor. The described sensors or light sensors and / or sensor elements or light sensor elements can have sensitivity in one or more of the ultraviolet, visible or infrared spectral ranges. The light-sensitive area of ​​the sensor can be oriented towards the object such that the sensor can record at least one reflected image of the light beam emanating from the object.

[0029] The defined method comprises manipulating a pattern image to generate a partial image. In an embodiment, the partial image may comprise at least one pattern feature, preferably at least a part of one or more pattern features. In an embodiment, the pattern feature may comprise an intensity distribution that is essentially symmetric, preferably essentially radially symmetric, more preferably essentially normal, Gaussian, Laplace-Gaussian, Cauchy, t-distribution, logistic, etc. In an embodiment, a partial image comprising a pattern feature may be defined as a partial image comprising a part of the image comprising at least one maximum of the intensity distribution. In an embodiment, a partial image not comprising a pattern feature may be defined as a partial image comprising an image that does not comprise a maximum of the intensity distribution, preferably a partial image comprising an up and / or down gradient of the intensity distribution, i.e. positive and / or negative intensities.

[0030] In one embodiment, the pattern feature of the light pattern refers to a part of the light pattern that can be distinguished from other pattern features, for example, due to the unilluminated distance between the pattern features or due to the different arrangement of light in different pattern features. Preferably, the pattern feature refers to one or more light spots arranged in a predefined pattern, and the light pattern is preferably in a repetition of the predefined pattern. In particular, it is preferred that the light pattern refers to a point cloud, where the point refers to a light spot, in which case the pattern feature can refer to one light spot. In this case, the light pattern can refer to, for example, a hexagon or a triclinic lattice of light spots that are substantially similar and include a circle. However, the pattern feature can also refer to a plurality of light spots, for example, a hexagon that includes six light spots, and in this case, for example, the feature pattern, i.e., a hexagon, can be repeated to form a regular light pattern.

[0031] As described above, the light pattern is reflected from the object and a corresponding pattern image indicative of the object is received. Thus, the pattern image includes the reflected pattern features. The reflected pattern features in the pattern image are also referred to as pattern features hereinafter.

[0032] In one embodiment, the pattern features can be selected from the pattern image. For example, in one embodiment, the selection of the pattern features can include first deriving information indicative of the location and extent of the object from the pattern image. In this embodiment, known methods for deriving information from the pattern image can be utilized. In one embodiment, known methods for determining the distance at which the pattern features are reflected from the camera can be utilized to receive information regarding the extent and location of the object. For example, pattern features that are within a predetermined distance range relative to each other can be considered as belonging to the same object and can be selected accordingly. Furthermore, the contour of the object can be determined, for example, by comparing the distance of adjacent pattern features to each other. A contour can be determined if the distance of adjacent pattern features exceeds a predetermined threshold. Furthermore, in additional or alternative embodiments, information indicative of the location and extent of the object in the pattern image can also be derived from the pattern image by deriving a material from the reflection properties of each pattern feature. Again, already known methods for deriving material properties from the properties of reflected light can be utilized, and it can be determined that the pattern features indicative of the material associated with the object are selected as the pattern features located on the object. For example, a pattern feature that is indicative of being reflected by human skin may then be determined as belonging to a human face and therefore may be selected as being located on the object.

[0033] In one embodiment, the pattern features located on the object may be selected from the pattern image, for example, by determining whether the position of the pattern feature is located within the contour of the object. In this context, selecting the pattern features may also include determining the position of each pattern feature in the pattern image. For example, a respective feature detection algorithm may be utilized. Such a feature recognition method may be based on easy rules, since the pattern features may be predefined and furthermore clearly distinguishable from other parts of the image that are not illuminated by the pattern features. For example, it may be determined that a pixel of the pattern image may contain a light intensity above a predefined threshold to be part of the pattern feature. Furthermore, the light intensity of neighboring pixels may also be taken into account to determine the position of the pattern feature, depending on the geometric form of the pattern feature. However, more sophisticated feature extraction methods may also be utilized, or the user may perform the position determination by respective input. The pattern features on the object may then be selected by comparing the position of the pattern feature with the indicated position and range of the object, and by selecting the pattern features that are within the boundary of the object.

[0034] In one embodiment, a partial image including one or more pattern features is generated from the pattern image by manipulation. The pattern image may be manipulated by image augmentation techniques such as cropping, rotating, blurring, etc. In one embodiment, the image augmentation may be randomized. Thus, the data size of the partial image may be equal to or smaller than the data size of the pattern image. Such manipulation may suppress background information, for example having features related to objects, for example facial features. In particular, the pattern or pattern features of the partial image are of interest for the extraction of material information. Thus, the size of data to be stored and / or subsequently processed is reduced.

[0035] In general, cropping of an image may refer to removing all areas of a pattern image outside a cropped pattern image. Cropping of a pattern image to generate one or more partial images (herein denoted cropped image or cropped pattern image) may be performed such that each (or a plurality of partial) cropped images may include a predetermined size and may include none or at least one selected pattern feature. The predetermined size may be determined, for example, in the form of a shape of a predetermined area to be covered by the cropped image, or any other characteristic that may determine, for example, the size of the area (such as a radius in case of a circular area). Preferably, a cropped image may refer to a rectangular image determined by a predetermined height and width. In one embodiment, cropping may be based on identifying a peak of the pattern or pattern feature and cropping a certain size around the peak. Multiple cropped images may be derived such that they do or do not overlap. Cropped images may have, in general, any shape and therefore may be characterized by different size characterizations. For example, the cropped images may be circular and characterized by their respective radii. Preferably, each cropped image may include the same predefined size. However, for different cropped images, different predefined sizes may also be utilized, e.g., for a cropped image at the center of the object, a larger size may be predefined than for cropped images within a predefined distance from the object's contour. Preferably, the cropped images are centered on the selected pattern feature. More preferably, for each selected pattern feature, a cropped image may be generated that includes at least the respective selected pattern feature, preferably the pattern feature centered within the cropped image. Furthermore, the cropped image preferably includes multiple pattern features, e.g., including in addition to the central pattern feature, all or part of adjacent pattern features.

[0036] In a further step, the method includes extracting material information of the object from the partial image by providing the partial image to a data-driven model. The data-driven model may be hereinafter referred to as a machine learning based discriminative model or simply a model. The data-driven model may comprise a neural network (hereinafter abbreviated as NN), in particular a convolutional neural network (hereinafter abbreviated as CNN) or a residual neural network (hereinafter abbreviated as ResNet). In general, the NN recognizes and / or classifies known patterns to unknown data after training the patterns in a training phase.

[0037] The inputs to the model, e.g., a dataset, one or more pattern images, one or more partial images, one or more representations, and / or one or more clusters, etc., may further refer, without further specification, to inputs used during training of the model, i.e., training inputs, e.g., a training dataset, one or more training pattern images, one or more training partial images, one or more training representations, and / or one or more training clusters, etc. The term material information, also denoted as material data, may further refer, without further specification, to inputs used during training of the model, i.e., training material information, also denoted as training material data. All descriptions, specifications, and definitions provided for the dataset, one or more pattern images, one or more partial images, one or more representations, one or more clusters, and / or material information also apply, as appropriate, to the training dataset, one or more training pattern images, one or more training partial images, one or more training representations, one or more training clusters, and / or training material information.

[0038] The data-driven model is parameterized to extract material information from partial images, i.e., the data-driven model is trained to be able to extract or provide material information of an object based on partial images as input. Typically, the model is trained using known partial images and known material information.

[0039] In general, the data-driven model is trained using images generated according to the same rules as the pattern images that are subsequently used to provide material information of the object. The machine learning based identification model can utilize any known machine learning algorithm. Neural networks are particularly suitable for use in the context of image feature and / or material and / or object recognition techniques. Therefore, it is preferred that the data-driven model is based on a neural network algorithm. More preferably, the neural network refers to a convolutional neural network.

[0040] Possible processes for generating material information and / or material data from the pattern image and / or image data, e.g. infrared image and / or infrared image data, can be suitable filtering techniques. To extract material information, a beam profile analysis can be performed by applying image filters, in particular material dependent image filters, to pattern features of the pattern image.

[0041] Such techniques are described, for example, in WO 2020 / 187719 A1, the disclosure of which is incorporated herein by reference.

[0042] In this case, it is particularly preferred to utilize a partial image, e.g. a partial image generated from the pattern image by an operation such as cropping and which is not itself a complete pattern image, as input for the model. This has the technically advantageous effect of preventing, in particular, the neural network algorithm from determining the material determination based too much on the correlation of features in completely different regions of the image. As a result, a higher accuracy is achieved in the output, e.g. the extracted and / or determined material information. Furthermore, false positives are prevented. Due to its training, the model is able to output material information when provided with a partial image as input.

[0043] In one embodiment, the model can be parameterized to output classifiers such as human skin vs. non-human skin, material vs. non-material, etc. The model can also be parameterized to more complex outputs such as human skin, material classes (latex, silicone, cloth, etc.), etc.

[0044] The results, i.e. the output of the model, e.g. the extracted material information, can then be provided and / or output. For example, the results can be provided to an output unit of the device providing a visual or audio output. Furthermore, the output can be further processed, e.g. to identify, in particular to authenticate, the object, as will be described in more detail below. Complementarily or alternatively, the output can be used to unlock a door or device if it is determined that the identity of a potential user is allowed the respective access. Complementarily or alternatively, the output can be used to control a device, not only to provide access to a restricted resource, but also, e.g. to control the movement or position of an automated device or robot based on the identification of the object. For example, if an object in the path of a moving robot is identified as a chair, the robot can be controlled to move around the object.

[0045] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, where the pattern image shows the object while the object is illuminated with pattern illumination.

[0046] The pattern illumination is a regular pattern and / or a periodic pattern and / or a dot pattern and / or a speckle pattern, in particular having at least 2, preferably at least 6, more preferably at least 20 pattern features and / or dots and / or speckles per surface area at a distance of 5-200 cm, preferably 30-60 cm, more preferably 15-80 cm.

[0047] In one embodiment, the pattern of the pattern image is due to the object being illuminated with the pattern illumination and / or the pattern image includes a contribution from the reflection of the pattern illumination and / or a contribution from the texture of the object. Thus, the pattern image may be an image of the object using a camera located on the device while the object is illuminated with patterned infrared illumination. In this context, the background contribution of the pattern image may include a contribution from the texture showing features of the object such as eyes, nose, etc. Further background contribution of the pattern image may include a contribution from background light illumination such as sunlight.

[0048] The light pattern illuminates the object. Preferably, the light pattern refers to a regular light pattern that includes regularly-spaced pattern features. However, in other embodiments, the light pattern can refer to an irregular pattern or even an arbitrary pattern.

[0049] In one embodiment, the pattern image may be an image. The image may include image data. The partial image may be an image. The partial image may include a portion of an image, in particular a pattern image. The pattern image may be manipulated to generate the partial image. The pattern image may show the object while the object is illuminated with a light pattern. The partial image may show at least a portion of the object while the object is illuminated with a light pattern. An image of an object captured while the object is illuminated with a patterned infrared illumination may also be referred to as a pattern image. The pattern image is captured while the object is illuminated with a patterned infrared illumination. Preferably, the pattern image arises as a reflection of the light pattern when the light pattern is reflected from the object. Preferably, the material information is derived from the reflection of the pattern illumination. A pattern feature of the light pattern of illumination or a pattern feature of the pattern image may refer to a portion of the light pattern or pattern image that can be distinguished from other pattern features, as described above. In one embodiment, the image may include a pattern image showing the object while the object is illuminated with illumination and / or a partial image showing at least a portion of the object while the object is illuminated with illumination. The image may include, and may be referred to as, infrared pattern image data.

[0050] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, wherein the operating step further comprises determining a representation of the partial image, the representation being associated with a physical signature embedded in the reflection of the pattern illumination in the partial image.

[0051] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, wherein the step of extracting material information of an object from a partial image further comprises providing a representation of the partial image to a data-driven model and / or providing a cluster of partial images to the data-driven model, the data-driven model being parameterized according to a training data set further comprising one or more representations of the partial image and / or one or more clusters of partial images, and the data-driven model being parameterized to extract the material information.

[0052] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, wherein the partial images are part of a dataset of partial images; and the clusters are derived via clustering of the partial images of the dataset based on a representation, the clusters being associated with the representation.

[0053] In one embodiment, the manipulation of the pattern image to generate a partial image that includes no pattern features or one or more pattern features includes generating the partial image from the pattern image through techniques such as cropping, rotating, blurring, etc., as described below with respect to certain portions of the embodiment described with respect to FIG.

[0054] In one embodiment, the contour of the object, which indicates the position and range of the object based on the flood image, can be determined according to any known feature extraction method for visible light images. Preferably, a flood image is provided which shows the object while the object is illuminated with flood light, e.g. flood illumination. The pattern image shows the object while illuminated with pattern illumination. In particular, automatic algorithms such as machine learning algorithms or simple feature detection algorithms can be used. In general, for all embodiments using multiple images, e.g. multiple pattern images for distance determination of feature patterns or flood images for contour detection, it is preferable that the respective images are taken at the same time or at least in a predetermined time range before and after the time the pattern images are taken, and further by the same camera or a camera having a predetermined distance relative to the camera taking the pattern images. This allows the position of features in one image, e.g. in the flood image, to be directly derived from the position of features in the other pattern image. In possible further embodiments of the present invention, a method that only approximates the position and range of the object can be used, or a less accurate method, e.g. with less computational resources, can be used.

[0055] In one embodiment, information about the contour of the object of interest, for example from the flood image, can be used to indicate the location and extent of the object in the pattern image, which can be used to find pattern features, such as light spots in the pattern image, located on the object. Preferably, for each such pattern feature, the pattern image is cropped such that the cropped pattern image, i.e. the partial image, displays the central pattern feature and a predefined amount of the neighborhood of the pattern feature. Preferably, the cropped partial image can include not only the central pattern feature, but also some of the neighborhood features. In one embodiment, the cropped partial image can be absent. The selected crop size is a compromise between overfitting of the neural network if too large an image is cropped, and too low accuracy of object recognition if too small an image is cropped. Thus, the size of the cropped image can be selected based on the requirements of a particular use case.

[0056] In one embodiment, the patterned infrared illumination can include an at least partially regular or irregular pattern.

[0057] In one embodiment, the object may be a user and / or a spoof object, which may be suitable for spoofing an authentication or authentication process.

[0058] In one embodiment, the trigger for capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared radiation can include capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared radiation. The trigger for capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared radiation can occur by capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared radiation. In one embodiment, the trigger for capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared radiation can be followed by capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared radiation.

[0059] In an embodiment, triggering illumination of the object with the patterned infrared illumination may include illuminating the object with the patterned infrared illumination. Triggering illumination of the object with the patterned infrared illumination may result in illuminating the object with the patterned infrared illumination. Triggering illumination of the object with the patterned infrared illumination may be followed by illuminating the object with the patterned infrared illumination.

[0060] In one embodiment, material data extraction.

[0061] In one embodiment, authentication may include or refer to an authentication process. Authentication may include or refer to facial recognition authentication, in particular a facial recognition authentication process.

[0062] In one embodiment, the method may be a computer-implemented method.

[0063] In an embodiment, extracting material data from an image may include generating material types and / or data derived from material types. Preferably, extracting material data may be based on an image, more preferably a pattern image, and most preferably a part image. The material data may be extracted by using a model. Extracting material data may include providing the image to the model and / or receiving material data from the model. Providing the image to the model may include or be followed by receiving the image at an input layer of the model or via a loss function of the model. The model may be a data-driven model. The data-driven model may include an encoder-decoder structure such as a convolutional neural network and / or an autoencoder. Other examples for generating the representation may be FFT, wavelet, deep learning such as cnn, energy-based models, regularization flows, GAN, vision transformers, or transformers used in natural language processing, autoregressive image modeling, GAN, autoregressive image modeling, regularization flows, deep autoencoders, deep energy-based models, vision transformers. Supervised or unsupervised schemes may be applied to generate the representations, as well as embeddings such as Euclidean spaces. The data-driven model may be parameterized according to a training data set including at least one image and material data, preferably at least one pattern image and material data. In another embodiment, extracting the material data may include providing an image to a model and / or receiving material data from a model. In another embodiment, the data-driven model may be trained according to a training data set including at least one image and material data. In another embodiment, the data-driven model may be parameterized according to a training data set including at least one image and material data. The data-driven model may be parameterized according to a training data set for receiving an image and providing material data based on the received image.The data-driven model may be trained according to a training dataset to receive an image and provide material data as an output based on the received image. The training dataset may include at least one image and material data, preferably material data associated with at least one image. The image may include a representation of the image. The representation may be a low-dimensional representation of the image. The representation may include at least a portion of the data or information associated with the image. The representation of the image may include a feature vector. In one embodiment, determining the representation, in particular the low-dimensional representation, may be based on Principal Component Analysis (PCA) mapping or Radial Basis Function (RBF) mapping. Determining the representation may also be referred to as generating the representation. The generation of the representation based on the PCA mapping may include clustering based on features of the pattern image and / or the sub-image. Additionally or alternatively, generating the representation may be based on a neural network structure suitable for reducing dimensionality. The neural network structure suitable for reducing dimensionality may include an encoder. In one example, the neural network structure may be an autoencoder. In one example, the neural network structure may include a convolutional neural network (CNN). The CNN may include at least one convolutional layer and / or at least one pooling layer. The CNN may reduce the dimension of the sub-image and / or the image, for example by applying convolution based on the convolutional layer and / or by pooling. Applying convolution may be suitable for selecting features related to material information of the sub-image.

[0064] In an embodiment, extracting material data from an image using a data-driven model may include providing the image to the data-driven model. Additionally or alternatively, extracting material data from an image using a data-driven model may include generating an embedding associated with the image based on the data-driven model. The embedding may refer to a low-dimensional representation associated with the image, such as a feature vector. The feature vector may be suitable for suppressing the background while maintaining a material signature indicative of the material data. In this context, the background may refer to information independent of the material signature and / or the material data. Furthermore, the background may refer to information related to biometric features, such as facial features. The material data may be determined in the data-driven model based on the embedding associated with the image. Additionally or alternatively, extracting material data from an image by providing the image to a data-driven model may include converting the image into material data, in particular into a material feature vector indicative of the material data. Thus, the material data may further include the material feature vector, and / or the material feature vector may be used to determine the material data.

[0065] In one embodiment, material data of an object, such as a user, can be extracted from at least one image.

[0066] In one embodiment, the data-driven model may be an ensemble model. The ensemble model may include a machine learning architecture trained by ensemble learning. Typically, data-driven models exhibit high variance and are highly dependent on a small number of features. In order to reduce the low accuracy associated with the sensitivity of the data-driven model to the input data, ensemble modeling is performed, which may result in improved accuracy of the trained model. Ensemble learning refers to training multiple algorithms to solve a problem. In the context of this application, multiple data-driven models may be trained and / or used to extract material data. In one embodiment, the ensemble model may be trained based on multiple data-driven models, so-called weak learners. The weak learners may be trained to solve a part of the problem. The data-driven model trained based on the weak learners may be referred to as a meta-model. Thus, the data-driven model may be a meta-model. As an example, multiple weak learners may be trained based on multiple materials, and the meta-model may output material data based on the weak learners. For example, the meta-model may combine the output of the weak learners and / or the meta-model may be trained to combine the output of the weak learners. Additionally or alternatively, the ensemble learning may include training multiple machine learning architectures and inputting the error of at least one machine learning architecture into another machine learning architecture. The training of the multiple machine learning architectures may be sequential, parallel, or a combination thereof. Examples of ensemble learning techniques may include bagging, boosting, stacking, blending, etc.

[0067] In one embodiment, allowing or denying the object to perform at least one operation on the device requiring authentication based on the material data may include allowing the object to perform at least one operation on the device requiring authentication if the material data matches the desired material data and / or authentication may be successful. The desired material data may be pre-determined material data. Authentication may be successful if the image may match an image template. The template may be stored to match the captured image of the object. Furthermore, allowing or denying the object to perform at least one operation on the device requiring authentication based on the material data may include denying the object to perform at least one operation on the device requiring authentication if the material data does not match the desired material data and / or authentication may fail. Authentication may fail if the image does not match the image template. In one embodiment, the object may be rejected if the material data may not match the desired material data, regardless of the authentication of the object. The at least one operation on the device requiring authentication may be access to the device, such as unlocking the device, and / or access to an application, preferably associated with the device, and / or access to a part of an application, preferably associated with the device. In one embodiment, allowing a user to access a resource may include allowing the user to perform at least one operation on a device and / or a system. The resource may be a device, a system, a function of a device, a function of a system, and / or an entity. Additionally and / or alternatively, allowing a user to access a resource may include allowing the user to access an entity. The entity may be a physical entity and / or a virtual entity. The virtual entity may be, for example, a database. The physical entity may be an area with restricted access. The area with restricted access may be one of the following: a security area, a room, an apartment, a vehicle, some of the examples above, etc.The device and / or system may be locked. The device and / or system may only be unlocked by an authorized user.

[0068] In one embodiment, a method for authorizing an object on a device to perform at least one operation on the device requiring authentication, the method includes: receiving a request from a user to unlock the device from a user, from a user interface associated with the device; triggering illumination of the object with patterned infrared illumination in response to receiving the unlock request; triggering capturing at least one image of the object using a camera disposed on the device while the object is illuminated with patterned infrared illumination; extracting material data from the at least one image; and permitting or denying the object to perform at least one operation on the device requiring authentication based on the material data.

[0069] In one embodiment, capturing the at least one image may include capturing at least one image of the object using a camera disposed on the device while the object is illuminated with the patterned infrared illumination.

[0070] In one embodiment, authentication may use a facial recognition authentication process that operates on the image and / or extracted material data.

[0071] In one embodiment, the authentication process can be verified based on the extracted material data.

[0072] In one embodiment, verifying based on the extracted material data may include determining whether the extracted material data matches the desired material data. Determining whether the extracted material data matches the desired material data may be referred to as verifying. Allowing or denying the object to perform at least one operation on the device requiring authentication based on the material data may include verifying the authentication or authentication process. The verification may be based on the material data and / or the image. Determining whether the extracted material data corresponds to the desired material data may include determining a similarity between the extracted material data and the desired material data. Determining a similarity between the extracted material data and the desired material data may include comparing the extracted material data and the desired material data. The desired material data may refer to pre-determined material data. As an example, the desired material data may be skin. It may be determined whether the material data corresponds to the desired material data. In an example, the material data may be a non-skin material or silicon. Determining whether the material data corresponds to the desired material data may include comparing the material data to the desired material data. Comparing the material data to the desired material data will allow or deny the object to perform at least one operation requiring authentication. In this example, skin as the desired material data may be compared to non-skin material or silicone as material data, and the result may be a rejection since silicone or non-skin material may be different from skin.

[0073] In one embodiment, the authentication process or verification thereof may include generating at least one feature vector from the material data and matching the material feature vector with an associated reference template vector for the material.

[0074] In one embodiment, the patterned infrared illumination may be a regular and / or periodic pattern having pattern features, in particular at least 2, preferably at least 6, more preferably at least 20 pattern features per surface area at a distance of 5-200 cm, preferably 30-60 cm, more preferably 15-80 cm.

[0075] In an embodiment, extracting the material data may comprise performing a beam profile analysis by applying an image filter, in particular a material dependent image filter, to the pattern features of the image and / or by providing the image to a data-driven model, which may be parameterized according to a training data set comprising the image and the material information; In one embodiment, an operation requiring authentication may include unlocking the device.

[0076] In one embodiment, extracting material data may include manipulating an image to generate a partial image and extracting material data of the object from the partial image by providing the partial image to a data-driven model, which may be parameterized according to a training dataset including the partial image and the material data.

[0077] In one embodiment, manipulating the image includes: - suppressing texture information from objects located in the region of interest, in particular the user's face, and / or - Segmenting the image and / or the image data; and / or - Cropping the image into a sub-image, and / or - determining a bounding box of a region of interest in an image and cropping the image to extract the region of interest; may include.

[0078] In one embodiment, the material information may include a material class and / or a material type.

[0079] In an embodiment, the model may be suitable for determining an output based on an input. In particular, the model may be suitable for determining material data based on an image as an input. The model may be a deterministic model, a data-driven model, or a hybrid model. The deterministic model preferably reflects a physical phenomenon in a mathematical form, including, for example, a first-principles model. The deterministic model may include a set of equations that describe the interaction between the material and the patterned electromagnetic radiation, thereby resulting in a condition measurement, a vital sign measurement, or the like. The data-driven model may be a classification model. The hybrid model may be a classification model that includes at least one machine learning architecture with deterministic or statistical adaptations and model parameters. Statistical or deterministic adaptations can be introduced to improve the quality of the results, as they provide a systematic relationship between empiricism and theory. In an embodiment, the data-driven model may be a classification model. The classification model may include at least one machine learning architecture and model parameters. For example, the machine learning architecture may be or include one or more of linear regression, logistic regression, random forest, piecewise linear, nonlinear classifier, support vector machine, naive Bayes classification, nearest neighbor classification, neural network, convolutional neural network, generative adversarial network, support vector machine, or gradient boosting algorithm, etc. In the case of a neural network, the model may be, but is not limited to, a multi-scale neural network, or a recurrent neural network (RNN), such as a gated recurrent unit (GRU) recurrent neural network or a long short-term memory (LSTM) recurrent neural network. The data-driven model may be parameterized according to a training dataset. The data-driven model may be trained based on the training dataset. Training the model may include parameterizing the model. The term training may be written as learning.The term may refer, without being limited thereto, to the process of building a classification model, and in particular the process of determining and / or updating parameters of the classification model. Updating parameters of a classification model may also be referred to as retraining. When training is referred to herein, retraining may be included. In an embodiment, the training data set may include at least one of images and material information.

[0080] In one embodiment, the partial image may be generated by manipulating the pattern image. The partial image may be generated from the pattern image by manipulating the pattern image. The manipulation may include determining a representation of the partial image, the representation being associated with a physical signature embedded in the reflection of the partial image. Manipulation of the pattern image to generate the partial image may in particular include generating the partial image from the pattern image and generating a representation of the partial image, preferably a low-dimensional representation of the partial image. Generating a representation of the partial image may be referred to as determining a representation of the partial image. The generated partial image can be further manipulated to generate a representation of the partial image. The representation may be a low-dimensional representation of the partial image. Thus, manipulating the pattern image may include determining a low-dimensional representation of the partial image. In one embodiment, the partial image may include a representation of the partial image. The representation may be associated with a physical signature of the object. The representation of the partial image may be a low-dimensional representation of the partial image. The partial image, in particular the representation of the partial image, may be independent of at least a portion of the background of the pattern image. Independence of at least a portion of the background of the pattern image may be referred to as suppressing and / or removing at least a portion of the background. Background may refer to information independent of material information. In one embodiment, background signatures of the pattern image and / or the part image can be suppressed by generating, i.e. deriving, one or more of the following representations, as described with respect to Fig. 7a and Fig. 7b, particularly for certain parts of the embodiment: In one embodiment, the representation is a mapping of discrete variables to vectors of continuous numbers. For example, in the context of neural networks, the representation is a low-dimensional, learned continuous vector representation of the discrete variables. The use and / or generation of at least one representation in the present invention achieves a reduction in the dimensionality of categorical variables. The representation overcomes the limitations of traditional encoding methods and can be used for purposes such as finding nearest neighbors, input to other models, and visualization. As a result, the generation and / or use of at least one representation provides technically advantageous effects, for example, that the background is suppressed and thus the size of the model is reduced.Therefore, the storage and / or processing of the model on the device is significantly reduced. The partial image and / or the pattern image contain more features than the representation of the partial image. Therefore, the amount of data associated with the partial image may be greater than the representation of the partial image.

[0081] In one embodiment, identifying an object, particularly authenticating an object based on material information, may include determining a matching score of a template and an image of the object, particularly a user. Identifying an object may refer to allowing a user to access a resource. In particular, an authenticated user may be allowed to access a resource. Identification may be part of authorizing a user. Identification may refer to proving a claim of an object, a user. The claim may relate to the identity of the object. Identification may include presentation and / or verification of biometric information, a certificate, and the like. The biometric information associated with the object may be presented with an image of the object. The biometric information may be information related to the appearance of the user. The biometric information may be associated with information related to fingers, face, hands, irises, lips, and the like. An authentication document may be a document that has been authenticated by a government agency, an educational institution, a government agency, and the like. A user may prove his / her identity by providing the biometric information. In an embodiment, the identification may be at least one of face recognition, fingerprint recognition, and iris recognition.

[0082] In one embodiment, identifying an object, in particular authenticating an object based on material information, comprises the following steps: - generating a low-level representation of the image; - determining a matching score between the low-level representation of the image and the low-level representation template; It may include at least one of the following:

[0083] Additionally or alternatively, identifying the object, and in particular authenticating the object based on the material information, may comprise verifying the authenticity of the object based on the material data.

[0084] In one embodiment, identifying the object, and in particular authenticating the object based on material information, may include authenticating the object and verifying the authentication of the object based on the material data.

[0085] In one embodiment, the authentication result may be provided based on a matching score. In one embodiment, the matching score may indicate a similarity between the captured image and a template. The template may include an image of at least a portion of an authorized user. The template may include a low-level representation associated with the authorized user. A registered user may be an authorized user. An authorized user may be a user that is allowed to access a resource. Preferably, the matching score may indicate a similarity between the low-level representation of the image and the low-level representation template. The low-level representation may include at least one feature vector. The feature vector may indicate features associated with the image. The template vector may indicate features associated with the template, in particular the template of a registered user. The low-level representation template may include at least one template vector. The feature vector and / or the template vector may include an n-dimensional vector. The vector may include at least one numerical value, preferably n numerical values. The matching score may indicate a distance between the feature vector and the template vector. In one example, the matching score may be obtained by determining a vector product of the feature vector and the template vector. The vector product may be a dot product.

[0086] In an embodiment, background signatures in the pattern image and / or partial images may be suppressed by generating, i.e. deriving, one or more representations and / or applying clustering to generate one or more clusters, as described below with respect to certain parts of the embodiment, in particular as described with respect to Fig. 7a and Fig. 7b. In general, clustering may involve grouping one or more pattern images and / or one or more partial images and / or one or more representations in whole or in part, such that those in a first cluster (also denoted as group) are more similar to each other with respect to certain properties, e.g. brightness, translucency and / or background, than those included in other (second, third, ...) clusters. The described clustering provides in particular the technically advantageous effect that the background of an object (shown in the image) is irrelevant, e.g. for the purpose of determining material information and / or identification, in particular authentication of the object.

[0087] In an embodiment, the clustering may be based on one or more representations of one or more sub-images and / or one or more representations of one or more pattern images. In a more preferred embodiment of the invention, the clustering and / or the one or more clusters are associated with the one or more representations, e.g., in a one-to-one correspondence.

[0088] In a further embodiment of the present invention, the technical problem of the present invention is solved by an apparatus for extracting material information of an object from a pattern image of the object, the apparatus comprising: a receiving unit configured to receive a pattern image showing the object; an operation unit configured to operate the pattern image to generate a partial image; an extraction unit configured to extract material information of the object from the partial image by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training dataset comprising the partial image and the material information; and a providing unit configured to provide the extracted material information.

[0089] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method for training a data-driven model suitable for extracting material information of an object from a pattern image of the object, the method comprising: receiving a pattern image showing the object; manipulating the pattern image to generate a partial image; training the data-driven model to determine material information of the object from the partial image, the training comprising parameterizing the data-driven model according to a training dataset comprising the partial image and the material information; and providing the trained data-driven model.

[0090] In a further embodiment of the present invention, the technical problem of the present invention is solved by an apparatus for training a data-driven model suitable for extracting material information of an object from a pattern image of the object, the method comprising: a receiving unit configured to receive a pattern image showing the object; an operation unit configured to operate the pattern image to generate a partial image; a training unit configured to train the data-driven model to determine material information of the object from the partial image, where the training comprises parameterizing the data-driven model according to a training dataset comprising the partial image and the material information; and a providing unit configured to provide the trained data-driven model.

[0091] In a further embodiment of the present invention, the technical problem of the present invention is solved by a non-transitory computer-readable data medium storing a computer program comprising instructions for causing a computer to execute the steps of the method of the present invention.

[0092] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer program for extracting material information of an object from a pattern image of the object, the computer program comprising instructions causing a computer to execute the steps of the method according to the present invention.

[0093] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer program for extracting material information of an object from a pattern image of the object, which program comprises program code means for causing the system and / or apparatus and / or device etc. to perform the method according to the present invention when the program is executed on a computer controlling the system and / or apparatus and / or device etc.

[0094] In a further embodiment of the present invention, the technical problem of the present invention is solved by a data structure product comprising manipulated image data for extracting material information from a pattern image, e.g. acquired by an IR sensor, the manipulated image data comprising a partial image; and / or at least one representation associated with a physical signature of the partial image, in particular a physical signature embedded in the reflection of the partial image; and / or a representation labelling pattern features in the pattern image and / or at least one cluster associated with material information.

[0095] In a further embodiment of the present invention, the technical problem of the present invention is solved by using the manipulated data of the present invention and / or the data obtained by the method of the present invention and / or the trained data-driven model obtained by the method of the present invention for authenticating an object, in particular for biometric authentication of the object.

[0096] In a further aspect of the present invention, the technical problem of the present invention is solved by a computer-implemented method for identifying an object, in particular for authenticating an object, comprising: receiving a request to unlock a device; providing illumination in response to receiving the request; capturing at least one image while illumination is provided; extracting material information from the image; identifying the object, in particular authenticating the object based on the material information; Includes.

[0097] In one embodiment, the method steps are performed by a device.

[0098] In one embodiment, the object is a user of a device, where the method is for authorizing the user to perform at least one operation on the device that requires authentication, in particular unlocking the device, where receiving the request includes receiving a request (a request to unlock the device) from a user interface associated with the device, and capturing at least one image includes capturing at least one image of the user using a camera disposed on the device while the user is illuminated with illumination; the method further includes allowing or denying the user to perform the at least one operation on the device that requires authentication based on the material information.

[0099] In one embodiment, the image includes a pattern image, and the authentication uses a facial recognition authentication process operating on the pattern image data and / or the material information, and the facial recognition authentication process is verified based on the material data. Performing a facial recognition authentication process on the image and / or the material data may refer to verifying the authentication process based on the material data. The facial recognition authentication process is performed on the pattern image data and / or the material information.

[0100] Validating the authentication process based on the materials data may include determining whether the materials information corresponds to the desired materials information.

[0101] The term object generally refers to an entity that includes a material that can be identified and / or authenticated and from which material information (also denoted as substance data) can be determined. The object may be a human, an animal, or a non-living entity. In one embodiment, the object may be a user of the device.

[0102] Identifying an object refers to determining the identity of the object. The identity can refer to a general identity, e.g. a class identity indicating that the object is part of a pre-determined object class, or a specific identity referring to determining whether the object refers to a pre-determined unique object. An example of determining the general identity of an object refers to, e.g., determining whether an object on an image is a human, i.e., belongs to the class of humans, or a chair, i.e., belongs to the class of chairs, etc. An example of specific identification can refer to identifying a pre-determined individual, e.g., the owner of a smartphone, identifying a chair of a specific individual, e.g., identifying a specific chair belonging to a specific owner, etc. In one embodiment, the method allows for authentication of the object. Authenticating an object refers specifically to identifying a specific object. Preferably, the method is adapted to authenticate a human being in order to grant access to a locked resource with limited access. In one embodiment, authentication of the human being is utilized to grant or deny access to a computing device.

[0103] The term device generally refers to a technical device capable of providing illumination and / or being triggered to capture an image. The device may include multiple sub-devices, e.g. a camera, a processor, a memory, a display, a module capable of communicating via a (remote) communication (5G, Wi-Fi, Bluetooth, etc.) interface. Preferably, the device may further include and / or be associated with a server, such as a cloud server, a virtual server, a cloud-based server, etc. In one embodiment, the device is a handheld device, such as a smartphone. In one embodiment, the object may be a user of the device.

[0104] The device is configured to allow a user to perform at least one operation on the device. The operation can be, for example, any interaction (input and / or output) with the device with which the device is capable of interacting. In one embodiment, such an operation is only possible after a user of the device has been identified and / or authorized. Thus, unauthorized operation or operation by an attacker using, for example, spoofing techniques, is prevented by improved object (i.e., user) identification and / or authentication. This technical advantageous effect is achieved, inter alia, by the claimed subject matter.

[0105] In particular, since the identification and / or authentication is based on material information, it is almost impossible to spoof the identification and / or authentication using an image, a mask, or a replica. Furthermore, the material of a particular accessory and / or the accessory itself (e.g., beard, glasses, head, scarf, etc.) may be the information on which the identification and / or authentication is based. In particular, the device can distinguish between skin and non-skin materials. Thus, spoofing attacks are well prevented and the overall security level of the device is improved.

[0106] In a further step, the device receives a request to unlock the device. The request can be provided by an object, in particular a user who wishes to interact with the device, for example. In one embodiment, the request can be received from a user interface associated with the device, in particular, for example. The request can be an unlocking request, for example a request to put the device into a state in which a user can, for example, interact with the device, for example, to perform at least one operation on the device.

[0107] In response to receiving the request, the device provides illumination. While illumination is provided, the device captures at least one image. Preferably, the illumination is an illumination of the object. More preferably, the image is an image of the object. In possible embodiments of the invention, the device may provide visual, audible, etc. indications to the object so that the device is positioned such that the illumination illuminates the object, and in one embodiment, the image is captured such that the image shows the object, or a part of the object, for example the user's face. In alternative embodiments, such indications are not necessary and the illumination and the image may be independent of the object. However, if the illumination is sufficient and / or the image does not sufficiently show the object, for example because the illumination is misaligned so that a wrong part of the object is on the image and / or a part of the object on the image is not sufficiently illuminated, the subsequent identification and / or authentication may be rejected, for example, especially if the device determines that the image and / or material information from the image is not suitable for identifying and / or authenticating the object.

[0108] In one embodiment, the illumination is triggered by the device and / or triggers the device to capture an image. Thus, in one embodiment, a camera or the like and / or an illuminator (each of which is described in further detail below, for example) is located on the device and / or associated with the device. In one embodiment, the illumination is infrared illumination. In a further preferred embodiment of the invention, the illumination is flood illumination and / or pattern illumination. In a further preferred embodiment of the invention, the illumination is infrared flood illumination and / or infrared pattern illumination. Thus, in one embodiment, the image is a pattern image, as described above and in further detail below. Thus, in one embodiment, material information is extracted from the infrared pattern image data.

[0109] In a further step, material information is extracted from the image. Preferably, the image comprises a pattern image and / or a part image. Material information is denoted identically as material data within the present invention. In one embodiment, the extraction of material information is performed as described herein above and in further detail below.

[0110] In one embodiment, where the image comprises a pattern image, extracting the material information comprises extracting material information of the object from the pattern image of the object and / or applying an image filter, in particular a material dependent image filter, to the beam profile of the image to generate a depth map.

[0111] In one embodiment, extracting material information of an object from a pattern image of the object includes: receiving a pattern image showing the object; manipulating the pattern image to generate a partial image; extracting material information of the object from the partial image by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training dataset including the partial image and the material information; and providing the extracted material information.

[0112] In one embodiment, the image includes a pattern image. The pattern image may be indicative of an object and may be received. Extracting material information of the object from the pattern image of the object may include: manipulating the pattern image to generate a partial image; and extracting material information of the object from the partial image by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training dataset including the partial image and the material information. The extracted material information may be provided.

[0113] In one embodiment, all possible combinations of the embodiments for extracting material information of an object from a pattern image of the object as described above are applied.

[0114] Preferably, the material information may be derived by using a neural network and / or the depth information may be derived from beam profile analysis via one or more suitable filters. More preferably, in contrast to the prior art that uses triangulation in a structured light approach, in an embodiment a pattern analysis of the pattern, preferably a regular pattern analysis of the pattern, may be used for depth recognition.

[0115] As explained above, one process for generating material data from (infrared) images (data) is via BPA. Thus, the model can include one or more filtering techniques. This approach applies and is particularly relevant to the recognition of, for example, faces based on material information, and thus to extracting material information for verification in authentication and / or identification processes. The BPA described in WO 2020 / 187719 A1 is described below. BPA offers the possibility to generate material information by applying material-dependent image filters to one or more pattern features. The disclosure of WO 2020 / 187719 A1 is incorporated herein by reference.

[0116] In a further step, the object is identified and / or authenticated based on the material information. In a particular embodiment, the data-driven model may be parameterized to map the image to a classifier such as whether it is human skin, whether it is a particular material (metal, silicone, etc.). In an embodiment, the model may also be parameterized to more complex outputs such as human skin, material class, etc. Based on the material information, the authentication and / or identification process may be verified. For example, if no human skin is detected in the image via the material information, the authentication and / or identification process may be stopped. In this case, there is a possibility of spoofing. However, if skin is detected, an authentication process may be performed on the light-cast image, as described in more detail below.

[0117] In one embodiment, the identification and / or authentication may use a facial recognition authentication process operating on the pattern image and / or the infrared pattern image and / or the material information and / or the material data. In a further preferred embodiment, the facial recognition process may be verified based on the material information and / or the material data.

[0118] In one embodiment, generating the material information may include determining a material heat map (described below) indicative of material distribution in the region of interest. The heat map may be a scalar field of material information derived from a data-driven model. In one embodiment, the operation requiring identification and / or authentication may include unlocking the device. In one embodiment, material information may be extracted for a region of interest of an object shown in the image, e.g., a face, or for example, a portion of the face of a user of the device. In one embodiment, the patterned infrared illumination may include a periodic and / or regular pattern, preferably a hexagonal dot pattern having 1000-2000 dots per image, or a pseudo-random structured light. In one embodiment, the method further includes generating a depth map from the infrared image data, e.g., generating a heat map (described below) whose parameters relate to the depth and / or depth distribution of the object shown by the image. In one embodiment, extracting material information from the image may include generating material information and / or material data.

[0119] In one embodiment, generating the material information may include determining a binary classifier that distinguishes, for example, between skin and non-skin and / or between a particular material (e.g., silicon, metal, etc.) and a non-particular material. In a more preferred embodiment, generating the material information may include determining a material classifier that represents the material.

[0120] The extraction of material information can be based on a data-driven model, such as a trained neural network, such as a convolutional neural network. The model may be parameterized to map a partial image or images to a classifier, such as human skin or non-human skin, material or non-material. Alternatively or supplementally, the model can be parameterized to a more complex output, such as a material classifier indicating the material, for example human skin, material class (latex, silicone, cloth), etc. Based on such extraction of material information, an identification and / or authentication process can be performed and / or verified.

[0121] The label may be a binary classifier, for example, distinguishing between material / non-material or human skin / non-skin. Preferably, the label may include more metadata, such as additional annotations for each mask, i.e. mask material, such as silicone, latex mask, mask brand, or one or more facial features, such as beard, glasses, hat, etc.

[0122] During model training, data including images and / or representations and / or labels and / or clusters can be fed into a classifier network, such as a convolutional neural network (CNN), using supervised or unsupervised learning techniques. The trained model can output, and thus extract, material information associated with and / or including one or more labels.

[0123] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, wherein the illumination comprises pattern illumination and / or flood illumination, in particular infrared pattern illumination and / or infrared flood illumination, and the at least one image comprises a series of images captured while the pattern illumination and / or flood illumination is provided, the series of images comprising at least two sets of images and / or at least two composite images.

[0124] According to a preferred embodiment, a flood light image and / or a pattern image can be received. Preferably, the flood light image shows the object while it is illuminated with flood light, e.g. while flood illumination is provided. The pattern image shows the object while it is illuminated with pattern illumination. In an embodiment, instead of the flood light image, a natural light image can also be used that shows the object while it is illuminated by natural light or artificial room light. In an embodiment, the pattern illumination can be infrared pattern illumination and / or the flood illumination can be infrared flood illumination.

[0125] In one embodiment, a device that is an image recording device, such as a mobile device, mobile phone, etc., may be provided with two illuminators, one for emitting flood light, such as an LED, and another for emitting a light pattern, such as a VCSEL array. The camera or the like of the device may capture the object illuminated by the flood light at one time and the object illuminated by the light pattern at another time. In a more preferred embodiment, these images are then passed to a processor that may be configured to execute a neural network. In a further preferred embodiment, the processor may be adapted to perform at least the image processing in a secure environment to avoid external access.

[0126] In one embodiment, the pattern of pattern illumination and / or the pattern image can include pattern features, such as a point cloud, a hexagonal or triangular lattice of spots. In the pattern image, the position of each spot can be determined, for example, by determining a local intensity maximum in the pattern image. In the flood image, objects can be identified by their shape. Each or both determinations can use, for example, a trained convolutional neural network.

[0127] The sequence of images is preferably taken simultaneously or at least over a predefined time range centered around the time the pattern image is taken, and further preferably the sequence of images is taken by the same camera or an additional camera at a predefined distance from the first camera.

[0128] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, wherein the method comprises generating at least one representation from material information and matching the representation with an associated reference template vector of the material.

[0129] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, comprising: Here, the material information is Providing a partial image to a data-driven model, The data-driven model is trained with a training data set of part images and material information; generating material information associated with the partial image; is extracted by

[0130] In a further embodiment, the image may be a partial image including one or more pattern features generated from a pattern image showing at least a portion of the object while the object is illuminated with the light pattern via the operation; Material information is providing the partial image to a data-driven model, the data-driven model being trained with a training data set of the partial image and material information; generating material information associated with the partial image; It is extracted by

[0131] In a further embodiment, the image may be a partial image showing at least a portion of the object while the object is illuminated with the light pattern; Material information is providing the partial image to a data-driven model, the data-driven model being trained with a training data set of the partial image and material information; generating material information associated with the partial image; It is extracted by

[0132] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, where at least one representation is generated by applying a material filter to one or more pattern features of the pattern image, the representation can include a representation of luminance included in the material information and / or a representation of semi-transparency included in the material information.

[0133] In one embodiment, the representation is a mapping of discrete variables to vectors of continuous numbers. For example, in the context of neural networks, the representation is a low-dimensional, learned, continuous vector representation of the discrete variables. The use and / or generation of at least one representation in the present invention provides a reduction in the dimensionality of categorical variables. The representation overcomes the limitations of traditional encoding methods and can be used for purposes such as finding nearest neighbors, input to other models, and visualization. As a result, the generation and / or use of at least one representation provides technically advantageous effects, for example, background is suppressed and thus the size of the model is reduced. Thus, the storage and / or processing of the model on the device is significantly reduced.

[0134] In one embodiment, extracting material information from the image comprises extracting material information from a manipulated image, the manipulated image being provided by manipulating the image, the manipulating the image comprising: suppressing texture information from objects located in the region of interest, in particular the user's face; and / or cropping the image and / or data of the image; and / or cropping the image into a sub-image; and / or determining a bounding box of the region of interest in the image and cropping the image to extract the region of interest.

[0135] The texture information may originate from the object of interest and / or one or more portions of interest of one or more objects, all located within the region of interest. In one embodiment, the manipulation of the image corresponds to the manipulation of a pattern image, as described in further detail herein.

[0136] In further embodiments, extracting material information comprises determining a binary classifier that distinguishes between materials and non-materials, in particular skin and non-skin; and / or determining a material classifier indicative of the materials; and / or determining a material heatmap indicative of material distribution in the region of interest, in particular the user's face.

[0137] In a further embodiment of the present invention, the technical problem of the present invention is solved by a device for identifying an object, in particular for authenticating an object, where the device comprises a processor including: a receiving unit configured to receive a request to unlock the device; a providing unit configured to provide illumination in response to receiving the request; a capturing unit configured to capture at least one image while the illumination is provided; an extraction unit configured to extract material information from the image; and an identification unit, in particular an authentication unit, configured to identify the object, in particular to authenticate the object based on the material information.

[0138] In a further embodiment of the present invention, the technical problem of the present invention is solved by a device, where the object is a user of the device, and the device is for authorizing the user to perform at least one operation on the device requiring authentication, in particular unlocking the device, where receiving the request comprises receiving a request to unlock the device from a user interface associated with the device, and capturing at least one image comprises capturing at least one image of the user using a camera disposed on the device while the user is illuminated with illumination; the processor is configured to perform authorizing or denying the user to perform at least one operation on the device requiring authentication based on the material information.

[0139] In a further embodiment of the present invention, the technical problem of the present invention is solved by a non-transitory computer-readable data medium storing a computer program comprising instructions for causing a computer to execute the steps of the method of the present invention.

[0140] In a further embodiment of the present invention the technical problem is solved by a computer program for identifying an object, in particular for authenticating an object, which computer program comprises instructions causing a computer to carry out the steps of the method according to the present invention.

[0141] In a further embodiment of the present invention the technical problem is solved by a computer program for identifying an object, in particular for authenticating an object, which program comprises program code means for causing the system and / or the apparatus and / or the device etc. to execute the method according to the invention when the program is executed on a computer controlling the system and / or the apparatus and / or the device etc.

[0142] In a further aspect of the present invention, the technical problem of the present invention is solved by a computer-implemented method for authorizing a user of a device to perform at least one operation on a device requiring authentication, the method comprising: receiving a request to unlock the device; receiving at least one image of the user in response to receiving the request; generating pattern image data, in particular infrared pattern image data, of an area of ​​interest from the at least one image; extracting material information from the pattern image data; determining at least one occlusion from the material information; and allowing or denying the user from performing at least one operation on the device requiring authentication based on the material information and / or the at least one occlusion.

[0143] In one embodiment, the method steps are performed by a device.In one embodiment, the user may be the object to be identified and / or authenticated.

[0144] In general, an occlusion may be an object and / or part of no interest, hiding another object of interest and / or another object and / or part to be inspected (e.g., information extraction, identification, authentication, etc.). Occlusion detection may refer to techniques aimed at detecting areas in an object and / or an image of such an object that are occluded or appear to be occluded, i.e., hidden by the presence of an occlusion. Occlusions have been problematic so far because they limit the information available in an object or an image of an object. Occlusions may be transparent, non-transparent, or partially transparent. In general, in the present invention, an occlusion is an object that is not relevant and / or may interfere with the identification and / or authentication of another object to be identified and / or authenticated and / or to recognize the user of the device. An occlusion may in some way hide a part of the object to be identified and / or authenticated and / or recognized. In one embodiment, if the object to be identified and / or authenticated and / or recognized is face-related, the occlusion may be related to random objects such as glasses, scarves, masks, replicas, books, cups, etc.

[0145] A human trying to identify and / or authenticate and / or recognize an object can ignore occlusions and focus on the non-occluded parts or regions of the object. However, even a fully trained neural network has great difficulty in detecting occlusions or performing the process of identifying and / or authenticating and / or recognizing an object in the presence of occlusions. In particular, CNN-based models do not perform well because one or more occlusions lead to high intra-class variability and high inter-class similarity.

[0146] Extracting material information and determining at least one occlusion from the material information as provided by the present invention allows for more reliable and simpler occlusion detection. In particular, the size of the model (e.g., NN or CNN, etc.) can be reduced, and at the same time, the identification and / or authentication and / or recognition results are significantly improved. In particular, according to the present invention, material information such as skin or non-skin, metal, silicone, etc. can be used to detect at least one occlusion, thus improving the identification and / or authentication and / or recognition of faces. Based on the material information, e.g., material distribution, the occlusion can be determined. The material information can also be denoted as material data.

[0147] In one embodiment, an object and / or a user may attempt to perform an operation on the device, and the device may require identification and / or authentication and / or authorization based on material information and / or occlusion and / or occlusion assessment to allow or deny the user to perform at least one operation on the device.

[0148] In one embodiment, the pattern image may include an infrared pattern image.

[0149] In one embodiment, the region of interest relates to the entire image, or one or more portions of the image, which may be discontinuous or overlapping. For example, if the image shows the upper part of the body, the region of interest may be the face. In one embodiment, if the user is wearing a mask, material information of the region of interest, e.g., the face, may be extracted and / or determined. In one embodiment, it may then be determined that, for example, half of the region of interest is, e.g., skin material, and, for example, the other half is, e.g., mask material, e.g., paper, tissue, etc.

[0150] In one embodiment, in response to determining an occlusion, full or partial recognition of regions of interest, faces, etc. may be performed.

[0151] In a further embodiment, receiving the request includes receiving a request from a user interface associated with the device, i.e. a request to unlock the device, and at least one image is captured by a camera positioned on the device while the user is illuminated with illumination, in particular flood illumination and / or pattern illumination, in particular infrared flood illumination and / or infrared pattern illumination.

[0152] In a further embodiment, the image comprises a pattern image and extracting the material information comprises extracting the user's material information from the user's pattern image by a filtering technique. The filtering technique may comprise applying an image filter, in particular a material dependent image filter, to one or more pattern features.

[0153] In a further embodiment, extracting the material information comprises: manipulating the pattern image or one or more partial images to generate a partial image; extracting the user's material information from the partial image by providing the partial image or one or more partial images to a data-driven model, the data-driven model being parameterized according to a training dataset including the partial image and the material information; and providing the extracted material information.

[0154] In a further embodiment, the illumination comprises pattern illumination and / or flood illumination, in particular infrared pattern illumination and / or infrared flood illumination, and the at least one image comprises a series of images captured while the pattern illumination and / or flood illumination is provided. The pattern image may show the object while it is illuminated with the pattern illumination. The pattern illumination may be a regular and / or periodic pattern with pattern features such as dots and / or speckles. The pattern features may comprise at least 2, preferably at least 6, more preferably at least 20 pattern features and / or dots and / or speckles per surface area at a distance of 5-200 cm, preferably 30-60 cm, more preferably 15-80 cm. The pattern illumination, in particular infrared pattern illumination, may have a wavelength of 700 nm to 1100 nm. The series of images may include a set of at least two images and / or at least two composite images; and the pattern illumination, in particular the infrared pattern illumination, includes a periodic and / or regular pattern, preferably a hexagonal dot pattern of 1000-2000 dots per image and / or pseudo-random structured light.

[0155] In a further embodiment, the method of any one of the preceding claims, wherein extracting material information from an image comprises: determining a binary classifier distinguishing between materials and non-materials, in particular skin and non-skin; and / or determining a material classifier indicative of materials; and / or determining a material heatmap indicative of material distribution in a region of interest, in particular the user's face; and / or extracting the material information from a manipulated image, wherein the manipulated image is provided by manipulating an image, where manipulating the image comprises: suppressing texture information from the user, in particular the user's face, located in the region of interest; and / or segmenting the image and / or data of the image; and / or cropping the image into sub-images; and / or determining a bounding box of a region of interest in the image and cropping the image to extract the region of interest.

[0156] In one embodiment, all possible combinations of the embodiments for extracting material information of an object from a pattern image of the object as described above are applied.

[0157] In one embodiment, all possible combinations of the embodiments for identifying an object, in particular for authenticating an object as described above, are applied.

[0158] In a further embodiment of the present invention the technical problem is solved by a computer implemented method further comprising determining at least one material heat map indicative of material distribution in the region of interest.

[0159] In an embodiment, the identification and / or authentication of the object and / or the approval of the user is further based on the material distribution, e.g. in a region of interest, and / or the amount of occlusion and / or at least one material type, for one or more portions of the image. In an embodiment, the amount of occlusion is the amount of occlusion of the object and / or the image showing the object and / or at least one region of interest and / or a face. In a more preferred embodiment of the invention, a heat map is generated for the object and / or at least one region of interest and / or a face in the at least one image.

[0160] In one embodiment, a material heat map provides the possibility to display and / or assign different colors to different parameters, e.g., different material information. Providing a material heat map showing material distribution thus provides a way to visualize parts of an image showing parts of an object or user that contain and / or relate to different material information. In general, a heat map is a visualization of data that includes the visualization of certain parameters, e.g., material information, by the use of different colors.

[0161] In general, a landmark refers to a detected feature, i.e., a feature of an object or user, detected in an image of the object, e.g., a floodlight image. If the object or user is a face, the landmark may be the eyes, nose, mouth, etc. In one embodiment, a model, e.g., a CNN, is trained and / or provided to be capable of detecting one or more landmarks in an image.

[0162] In one embodiment, based on the material information, the results of one or more partial images can be stitched together into a complete image. Thus, the material distribution of the object can be determined. Preferably, based on the material distribution, at least one occlusion can be determined. In a further preferred embodiment, the at least one occlusion is associated with one or more landmarks, such as eyes, mouth, etc., preferably from the floodlight image.

[0163] In one embodiment, a rule set may be used to validate the face recognition process based on the association with one or more landmarks. For example, if a mask or the like is associated with eye landmarks, the process may be stopped. Similar features such as beards, glasses, hats, scarves, etc. may also be subject to rule-based consistency checks. Thus, spoofing attacks are highly prevented and the security level is increased.

[0164] In one embodiment, one or more occlusions may be identified based on material distribution relative to the landmarks. If, for example, the face is not occluded, then full face recognition may be performed. If, for example, the face is occluded, then partial face recognition may be performed. Preferably, for face recognition, the amount of occlusion may be determined from the material heatmap. If the amount is below a threshold and / or verified via a rule set, then partial face recognition may be performed.

[0165] In a further embodiment, in response to the determination of at least one occlusion, full and / or partial recognition is performed, particularly of the user's face.

[0166] In a further embodiment, the method further includes: determining locations of one or more landmarks in the at least one image; and aligning at least one material heat map to the locations of the one or more landmarks.

[0167] In a further embodiment, the method further comprises: determining and / or assessing an amount of occlusion, in particular of the face, from at least one material heat map, where determining and / or assessing the amount further comprises evaluating a rule set and / or one or more occlusion scores from the material heat map and / or providing a response signal for rejecting or performing recognition, in particular full recognition and / or partial recognition.In a further embodiment, the method further comprises: determining and / or assessing an amount of occlusion of the face, where determining and / or assessing the amount further comprises assessing an amount of occlusion of the nose and / or the mouth and / or at least one eye on the face in the at least one image.

[0168] In a further embodiment, a partial and / or complete face recognition process is performed on the at least one image based on the determined and / or assessed amount of facial occlusion in the at least one image.

[0169] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer-implemented method, where the illumination comprises pattern illumination and / or flood illumination, and the at least one image comprises at least one pattern image and / or at least one flood image, where the pattern image is an image captured while the pattern illumination is provided and the flood image is an image captured while the flood illumination is provided; the method further comprises determining at least one occlusion portion from a heat map of the at least one pattern image, cropping the at least one flood image to exclude portions, and performing authentication, in particular facial authentication, on the flood image, in particular based on facial features.

[0170] In one embodiment, the floodlight image can be used in a full and / or partial face authentication process. Preferably, the portion may relate to a portion of the image showing all and / or part of one or more occlusions. In one embodiment, cropping, as described herein in particular for the present invention, can be applied to the portion, such that for example a portion of the image showing the eye area of ​​the face is used for further processing, for example extraction of material information, identification and / or authentication and / or recognition of an object or user.

[0171] In a further embodiment of the present invention, the technical problem of the present invention is solved by a device for authorizing a user of a device to perform at least one operation on a device requiring authentication, the device comprising a processor configured to perform the following steps: receiving a request to unlock the device; receiving at least one image of the user in response to receiving the request; generating pattern image data, in particular infrared pattern image data, of an area of ​​interest from the at least one image; extracting material information from the pattern image data; determining at least one occlusion from the material information; and allowing or denying the user to perform at least one operation on the device requiring authentication based on the material information and the at least one occlusion.

[0172] In a further embodiment of the present invention the technical problem is solved by a non-transitory computer readable data medium storing a computer program comprising instructions for causing a computer to execute the steps of the method according to the present invention.

[0173] In a further embodiment of the present invention, the technical problem of the present invention is solved by a computer program for authorizing a user of a device to perform at least one operation on a device requiring authentication, the computer program comprising instructions causing a computer to perform the steps of the method according to the present invention.

[0174] In a further embodiment of the invention the technical problem of the present invention is solved by a computer program for authorizing a device user to perform at least one operation on a device requiring authentication, the program comprising program code means for causing the system and / or apparatus and / or device etc. to perform the method according to the present invention when the program is executed on a computer controlling the system and / or apparatus and / or device etc.

[0175] In a further embodiment of the present invention, a method is presented for extracting material information from a pattern image of an object, the pattern image including contributions from reflection of the pattern illumination and texture of the object, the method comprising: receiving a pattern image acquired by a sensor from the pattern-illuminated object; manipulating the pattern image to generate a partial image including at least a portion of the pattern feature(s); training a data-driven model to determine material information from the pattern image, the training comprising parameterizing the data-driven model according to a training dataset including the partial image and the material information; and providing the trained model.

[0176] In a further embodiment of the invention, a method for extracting material information from a pattern image of an object, the pattern image including contributions from reflections of the pattern illumination and texture of the object, comprises: receiving a pattern image acquired by a sensor from the pattern-illuminated object; manipulating the pattern image to generate a partial image including at least a portion of the pattern features (two or more); extracting the material information by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training dataset including the partial image and the material information; and providing the extracted material information.

[0177] In a further embodiment of the present invention, an apparatus for processing a pattern image of an object to extract material information, the pattern image including reflections of pattern illumination and contributions from the object's texture, comprises: a receiving unit configured to receive a pattern image acquired by a sensor from a pattern-illuminated object; a manipulation unit configured to manipulate the pattern image to generate a partial image including at least a portion of the pattern feature(s); a training unit configured to train a data-driven model to determine the material information from the pattern image, the training including parameterizing the data-driven model according to a training dataset including the partial image and the material information; and a providing unit configured to provide the trained model.

[0178] In a further embodiment of the present invention, an apparatus for processing a pattern image of an object to extract material information, the pattern image including contributions from reflections of pattern illumination and texture of the object, comprises: a receiving unit configured to receive a pattern image acquired by a sensor from a pattern-illuminated object; a manipulation unit configured to manipulate the pattern image to generate a partial image including at least a portion of the pattern feature(s); an extraction unit configured to extract material information by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training dataset including the partial image and material information; and a providing unit configured to provide the extracted material information.

[0179] In a further embodiment of the present invention, manipulated image data for extracting material information from a pattern image acquired by an IR sensor includes a partial image including at least a portion of a pattern feature present in the pattern image and at least one representation associated with a physical signature embedded in a reflection of the partial image or at least one class associated with the representation.

[0180] A further embodiment of the invention is the use of the trained model for authenticating an object, in particular for biometrically authenticating the object.

[0181] In a further embodiment of the invention there is the use of manipulated image data to extract material information for authenticating an object, in particular for biometrically authenticating the object.

[0182] In a further embodiment of the invention, the pattern of the pattern image results from a pattern-illuminated object.

[0183] In a further embodiment of the invention, the irradiation pattern is a regular or periodic pattern.

[0184] In a further embodiment of the invention, the pattern is a dot or speckle pattern, preferably having at least 20 dots or speckles per surface area at a distance of 20-60 cm or 15-80 cm.

[0185] In a further embodiment of the invention, the object is a face.

[0186] In a further embodiment of the invention, material information is obtained from reflection of the patterned illumination.

[0187] In a further embodiment of the invention, the material information relates to human skin, silicone, resin, latex, paper and / or mask materials.

[0188] In a further embodiment of the invention, the manipulation comprises determining a representation of the sub-image, the representation being associated with a physical signature embedded in the reflection of the sub-image.

[0189] In a further embodiment of the invention, the operation comprises clustering based on a representation of the sub-images and / or the method comprises clustering based on a representation of the sub-images, the determined clusters being associated with the representations.

[0190] In a further embodiment of the present invention, the clustering comprises sorting and / or grouping of similar images due to luminance, translucency and / or background, such that the material information is background independent.

[0191] In a further embodiment of the present invention, the partial images, the representations of the corresponding partial images, and the clusters of the corresponding partial images are provided to a model trained to extract material information from the pattern image.

[0192] In a further embodiment of the present invention, the model is a data-driven model, the model is parameterized based on partial image training data, representations of the corresponding partial images, and clusters of the corresponding partial images, and the model is parameterized for extracting material information from the partial images.

[0193] In a further embodiment of the present invention, the material information includes a material class or type.

[0194] In a further embodiment of the present invention, a method is presented for authorizing a user of a device to perform at least one operation on a device that requires authentication, the method comprising the following steps: receiving a request to unlock the device from the user from a user interface associated with the device; triggering illumination of the user with flood infrared illumination and patterned infrared illumination in response to receiving the unlock request; triggering capturing at least one image of the user using a camera disposed on the device while the user is illuminated with the patterned infrared illumination; extracting material data from the infrared pattern image data; and allowing or denying the user from performing at least one operation on the device that requires authentication based on the material data.

[0195] In a further embodiment of the present invention, a device for authorizing a user of the device to perform at least one operation on a device that requires authentication, the device comprising a processor configured to perform the following steps: receiving a request from the user to unlock the device from a user interface associated with the device; triggering illumination of the user with flood infrared illumination and patterned infrared illumination in response to receiving the unlock request; triggering capturing at least one image of the user using a camera disposed on the device while the user is illuminated with the patterned infrared illumination; extracting material data from the infrared pattern image data; and allowing or denying the user from performing at least one operation on the device that requires authentication based on the material data.

[0196] In a further embodiment of the invention, a user may be illuminated with flood infrared illumination and a series of images may be captured while the user is exposed to the pattern illumination and the flood illumination, which may include image pairs or a composite image.

[0197] In a further embodiment of the invention, authentication uses a facial recognition authentication process that operates on infrared pattern image data and material data.

[0198] In a further embodiment of the present invention, the face recognition process is verified based on material information.

[0199] In a further embodiment of the present invention, the facial recognition authentication process or verification thereof comprises generating at least one feature vector (or representation) from the material data and matching the material feature vector (or representation) with an associated reference template vector for the material.

[0200] In a further embodiment of the invention, at least one material feature vector (or representation) is generated from providing material data to a convolutional neural network trained on a training dataset of the material data, generating output, and converting the output into a material feature vector.

[0201] In a further embodiment of the present invention, at least one material feature vector (or representation) is generated by applying a material filter to the material data.

[0202] In a further embodiment of the present invention, the material feature vector (or representation) includes a representation of the luminance or translucency contained in the material data.

[0203] In a further embodiment of the present invention, generating the materials data includes manipulating the infrared image data to suppress texture information from an object of interest located within the region of interest.

[0204] In a further embodiment of the present invention, the manipulation includes splitting and cropping the infrared image data into sub-images.

[0205] In a further embodiment of the present invention, generating the material data includes determining a binary classifier that distinguishes between skin and non-skin.

[0206] In a further embodiment of the present invention, generating the materials data includes determining a materials classifier indicative of the material.

[0207] In a further embodiment of the invention, generating the material data includes determining a material heat map indicative of material distribution in the region of interest.

[0208] In a further embodiment of the present invention, the operation requiring authentication includes unlocking the device.

[0209] In a further embodiment of the invention, the area of ​​interest is the user's face.

[0210] In a further embodiment of the present invention, generating the infrared image data includes determining a bounding box of a region of interest in the image and cropping the image to extract the region of interest.

[0211] In a further embodiment of the invention, the patterned infrared illumination comprises a periodic or regular pattern, preferably a hexagonal dot pattern of 1000-2000 dots per image, or pseudo-random structured light.

[0212] In a further embodiment of the invention, the method includes generating a depth map from the infrared image data.

[0213] In a further embodiment of the present invention, a method is presented for authorizing a user of a device to perform at least one operation on a device requiring authentication, the method comprising: in response to receiving an unlock request, receiving at least one image of the user using a camera disposed on the device while the user is illuminated with flood infrared illumination and / or patterned infrared illumination; generating infrared pattern image data of an area of ​​interest from the at least one image; extracting material data from the infrared pattern image data and determining occlusion from the material data; and permitting or denying the user from performing at least one operation on the device requiring authentication based on the material data and the occlusion evaluation.

[0214] In a further embodiment of the present invention, a device for authorizing a user of the device to perform at least one operation on a device requiring authentication, the device comprising a processor configured to perform the following steps: in response to receiving an unlock request, receiving at least one image of the user using a camera disposed on the device while the user is illuminated with flood infrared illumination and / or patterned infrared illumination; generating infrared pattern image data of an area of ​​interest from the at least one image; extracting material data from the infrared pattern image data and determining occlusion from the material data; and allowing or denying the user from performing at least one operation on the device requiring authentication based on the material data and the occlusion assessment.

[0215] In a further embodiment of the invention, generating the material data includes determining a material heat map indicative of material distribution in the region of interest.

[0216] In a further embodiment of the present invention, full and partial face recognition is performed in response to a determination of occlusion.

[0217] In a further embodiment of the present invention, the amount of occlusion is determined from a material heat map.

[0218] In a further embodiment of the invention, evaluating the amount of occlusion of the face includes evaluating a rule set or one or more occlusion scores from a material heat map generated for the face in the at least one image.

[0219] In a further embodiment of the present invention, a complete face recognition process is performed on the at least one image based on the amount of facial occlusion assessed in the at least one image.

[0220] In a further embodiment of the invention, the method comprises the further step of determining the location of one or more landmarks in at least one image and aligning at least one material heat map to the locations of the landmarks.

[0221] In a further embodiment of the invention, assessing the amount of facial occlusion includes assessing an amount of occlusion of the nose and mouth on the face in the at least one image.

[0222] In a further embodiment of the invention, assessing the amount of facial occlusion includes determining a type of material.

[0223] In a further embodiment of the invention, a user may be illuminated with flood infrared illumination and a series of images may be captured while the user is exposed to the pattern illumination and the flood illumination, the series of images may include image pairs or composite images, and facial recognition is performed on the flood images based on facial features.

[0224] In a further embodiment of the present invention, at least one flood image for the partial face recognition process is cropped to exclude occluded portions determined from a corresponding heat map of the at least one pattern image (preferably to the eye area of ​​the face).

[0225] In a further embodiment, the material information may be extracted by evaluation of at least one beam profile of the pattern image. The at least one material information may be determined by applying at least one first material-dependent image filter to the pattern image. The material-dependent image filter may be at least one filter selected from the group consisting of: an intensity filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or a median filter; a gray level generation-based contrast filter; a gray level generation-based energy filter; a gray level generation-based uniformity filter; a gray level generation-based dissimilarity filter; a low energy filter; a threshold area filter; or a linear combination thereof; or a second material-dependent image filter that may be correlated to one or more of an intensity filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation-based energy filter, a gray level generation-based uniformity filter, a gray level generation-based dissimilarity filter, a low energy filter, or a threshold area filter, or a linear combination thereof.

[0226] In one embodiment, distance information may be extracted from the pattern image. At least one distance feature may be determined by applying at least one first distance-dependent image filter to the pattern image. The first distance-dependent image filter may be at least one filter selected from the group consisting of a depth filter from photon ratio; a depth filter from defocus; a linear combination thereof; or a second distance-dependent image filter. The second distance-dependent image filter may be correlated to the depth filter from photon ratio and / or the depth filter from defocus, or a linear combination thereof. In a further embodiment, a third image filter and a fourth image filter may be applied. The third image filter may be one of a depth filter from photon ratio or a depth filter from defocus, or a linear combination thereof. The fourth image filter may be a brightness filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation-based energy filter, a gray level generation-based uniformity filter, a gray level generation-based dissimilarity filter, a low energy filter, or a threshold area filter, or a linear combination thereof.

[0227] In one embodiment, distance information, for example vertical coordinate, and / or material information may be determined based on applying a filter to one or more pattern features of the pattern image.

[0228] In one embodiment, a method for authorizing a user may include generating a heat map of distance and / or material information that may provide a depth map to avoid scamming a simple image based authorization process.

[0229] In further embodiments of the present invention, the at least one material information may be a property selected from the group consisting of scattering coefficient, translucency, transparency, deviation from Lambertian surface reflectance, speckle, and the like.

[0230] In a further embodiment of the present invention, the distance dependent image filter and the material dependent image filter are applied simultaneously to the reflected image.

[0231] In a further embodiment of the present invention, the sensor element may include at least one CMOS sensor.

[0232] In a further embodiment of the present invention, the detector may include at least one illumination source. The illumination source may be configured to generate at least one illumination pattern for illumination of the object. The illumination pattern may include at least one pattern selected from the group consisting of: at least one dot pattern, in particular a pseudorandom dot pattern; a random dot pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including a convex uniform tiling; at least one line pattern including at least one line; at least one line pattern including at least two lines, such as parallel or intersecting lines.

[0233] In a further embodiment of the invention, the illumination light source can include at least one laser light source and at least one diffractive optical element. It is to be understood that the method as described above, the device as described above and the computer-readable data medium as described above have similar and / or identical preferred embodiments, in particular as defined in the dependent claims. Furthermore, the training method as described above, the training device as described above and the training computer-readable data medium as described above have similar and / or identical preferred embodiments, in particular as defined in the dependent claims.

[0234] It is to be understood that a preferred embodiment of the invention can also be any combination of the dependent claims or the above embodiments with the respective independent claim.

[0235] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. [Brief description of the drawings]

[0236] In the diagram below: [Figure 1] FIG. 1 illustrates a schematic and exemplary embodiment of a device comprising a processor, a memory and a camera. [Diagram 2] FIG. 1 illustrates a schematic and exemplary embodiment of a camera. [Diagram 3] FIG. 2 illustrates a schematic and exemplary embodiment of a processor associated with a camera for processing a camera signal; [Figure 4] 4a-c are schematic and exemplary diagrams illustrating an embodiment of a flow chart of an embodiment of a biometric authentication process using facial features. [Diagram 5] Figure 5a shows a schematic and exemplary diagram of an embodiment of cropping an image into sub-images when the peak is in the center of the image, Figure 5b shows a schematic and exemplary diagram of an embodiment of cropping an image into sub-images when the cropping is random with respect to the position of the peak, and Figure 5c shows a schematic and exemplary diagram of a neural network for sub-images. [Figure 6] 1A-1C are schematic and exemplary diagrams illustrating various examples of clusters; [Figure 7] 7a, b show schematic and exemplary embodiments of generating a representation and / or a clustering. [Figure 8] FIG. 2 shows a schematic and exemplary diagram of an embodiment of a training step of a data-driven model for extracting skin features. [Figure 9]Fig. 9a is a schematic and exemplary diagram of an embodiment of a data-driven model for extracting skin features, and Fig. 9b is a schematic and exemplary diagram of training an ensemble. [Figure 10] 10a, b are schematic and exemplary diagrams illustrating one embodiment of a cluster of sub-images and their skin / non-skin material vectors. [Figure 11] FIG. 2 shows a schematic and exemplary situation in which a user is wearing a mask and the face is partially occluded. [Figure 12] 12a, b are schematic and exemplary diagrams illustrating one embodiment of the generation of a heat map of material distribution. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0237] Detailed Description of the Preferred Embodiments Fig. 1 shows, in a schematic and exemplary manner, one embodiment of a device 100 comprising a processor 101, a memory 102, and a camera 103. In particular, Fig. 1 shows, in a schematic and exemplary manner, a general hardware configuration.

[0238] The device 100 may include a camera 103, a processor 101, a memory 102, and a display 104. The processor 101 is described in more detail with respect to FIG. 3. The device 100 may be a computing device such as a handheld device. The device 100 may be any computing device that is mobile or portable. The device 100 may be a mobile device having wireless or RF communication capabilities (e.g., WLAN, Wi-Fi, cellular, and / or Bluetooth). Examples of mobile devices include mobile phones or smartphones, tablet computers, laptop computers, portable gaming devices, portable Internet devices, and other handheld devices, as well as wearable devices such as smart watches, smart glasses, headphones, pendants, earpieces, etc. The display 104 may be, for example, an LCD screen or a touch screen configured for interactive input from a user. The camera 103 may be configured to capture an image of the device's external environment. The camera may be positioned to capture an image in front of the display. The camera may be positioned to capture an image of the user (e.g., the user's face) while the user interacts with the display. Camera 103 is described in further detail with respect to FIG. 2. Memory 102 may be a data memory. Memory 102 may be a physical system memory, which may be volatile and / or non-volatile. Memory 102 may comprise any computer-readable storage medium, such as non-volatile mass storage. When a computing system is distributed, processing and / or memory capabilities may also be distributed.

[0239] As further explained with respect to FIG. 2, the device 100 may include at least one projector, which is an illumination source. The illumination source may include at least one light source. The illumination source may include a plurality of light sources. The illumination source may include an artificial illumination source, in particular at least one laser source, and / or at least one incandescent lamp, and / or at least one semiconductor light source, for example at least one light emitting diode, in particular an organic and / or inorganic light emitting diode. By way of example, the light emitted by the illumination source may have a wavelength of 300 to 1100 nm, in particular 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as in the range of 780 nm to 3.0 μm, may be used. In particular, light in the near infrared range part, in particular in the range of 700 nm to 1100 nm, where silicon photodiodes are applicable, may be used. Using light in the near infrared range allows light to be undetectable or only weakly detectable by the human eye, but detectable by silicon sensors, in particular standard silicon sensors. The illumination light source may be adapted to emit light of a single wavelength. In other embodiments, the illumination may be adapted to emit light having multiple wavelengths allowing additional measurements in other wavelength channels. The light source may be or include at least one multi-beam light source. For example, the light source may include at least one laser light source and one or more diffractive optical elements (DOEs).

[0240] The illumination source may be configured to generate at least one illumination pattern for illuminating the object. The illumination pattern may include at least one pattern selected from the group consisting of: at least one dot pattern, in particular a pseudo-random dot pattern; a random dot pattern or a quasi-random pattern; at least one Sobol pattern; at least one quasi-periodic pattern; at least one pattern including at least one known feature; at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern; at least one pattern including a convex uniform tiling; at least one line pattern including at least one line; at least one line pattern including at least two lines, such as parallel or intersecting lines. As used herein, the term "pattern" refers to any known or predetermined arrangement including at least one feature of any shape. A pattern may include at least one feature, such as a dot or a symbol. A pattern may include multiple features. A pattern may include a periodic or non-periodic arrangement of features. As used herein, the term "at least one illumination pattern" refers to at least one any pattern including at least one illumination feature adapted to illuminate at least a portion of an object. As used herein, the term "illumination feature" (also written as "pattern feature") refers to at least one feature that spans at least a portion of a pattern. An illumination pattern may include a single illumination feature. An illumination pattern may include multiple illumination features. For example, an illumination pattern may include at least one line pattern. For example, an illumination pattern may include at least one stripe pattern. For example, an illumination pattern may include at least one checkerboard pattern. For example, an illumination pattern may include at least one pattern including a periodic or non-periodic arrangement of features. An illumination pattern may include a regular and / or regular and / or periodic pattern, such as a triangular pattern, a rectangular pattern, a hexagonal pattern, or even a pattern including a convex tiling.The illumination pattern may exhibit at least one illumination feature selected from the group consisting of at least one point; at least one line; at least two lines, such as parallel or intersecting lines; at least one point and one line; at least one arrangement of periodic or non-periodic features; at least one feature of any shape. For example, the illumination source can be adapted to generate and / or project a point cloud. The distance between two features of the illumination pattern and / or the area of ​​the at least one illumination feature may depend on the circle of confusion in the image. The illumination source may include at least one light source configured to generate the at least one illumination pattern. In particular, to generate and project the illumination pattern, the illumination source may include at least one laser source and at least one diffractive optical element (DOE). The detector may include at least one laser source and at least one point projector, such as a DOE, adapted to project the at least one point pattern. As further used herein, the term "projecting at least one illumination pattern" refers to providing at least one illumination pattern for illuminating at least one object. When only a single illumination feature, such as a single point, is present, the projected illumination pattern may be sparse. To increase reliability, the illumination pattern may include multiple illumination features, such as multiple points. When the pattern is sparse, a single image may be used to simultaneously detect biological tissue and identify faces.

[0241] As used herein, the term "object" refers to a point or area that emits at least one light beam, in particular at least one reflection pattern. For example, the object may be at least one object selected from the group consisting of a scene, a part of a human body (such as a face or a part of a face), a human being such as a person, wood, a carpet, a foam, an animal such as a cow, a plant, a piece of tissue, a metal, a toy, a metal object, a beverage, a food such as a fruit, meat, fish, a dish, a cosmetic product, an applied cosmetic product, a fabric, a fur, a maintenance product, a cream, an oil, a powder, a carpet, a juice, a suspension, a paint, a plant, a body, a body part, an organic material, an inorganic material, a reflective material, a screen, a display, a wall, a paper such as a photograph. The object may include at least one surface onto which the illumination pattern is projected. The surface may be adapted to at least partially reflect the illumination pattern back towards the detector. For example, without wishing to be bound by this theory, human skin may have a reflected beam profile, also called a backscattering profile, that includes a portion generated by the back reflection of the surface, called the surface reflection, and a portion generated by very diffuse reflection from the light penetrating the skin, called the diffuse portion of the back reflection. For the reflectance profile of human skin, see "Lasertechnikinder Medizin: Grundlagen, Systeme, Anwendungen", "Wirkung von Laserstrahlung auf Gewebe" 1991, pages 171-266, Juergen Eichler, Theo Seiler, Springer Verlag, ISBN 0939-0979. The surface reflectance of skin may increase as the wavelength increases towards the near infrared. Furthermore, the penetration depth may increase as the wavelength increases from visible light to the near infrared. The diffuse portion of the back reflection may increase with the increase in the penetration depth of the light. These material properties can be used to distinguish skin from other materials, particularly by analyzing the backscattering profile.

[0242] Fig. 2 shows, in a schematic and exemplary manner, an embodiment of a camera 203. The camera 203 may correspond to the camera 103 of Fig. 1. In particular, Fig. 2 shows, in a schematic and exemplary manner, a typical hardware configuration.

[0243] The present invention relates to splitting a pattern image into partial pattern images. For this purpose, at least two projectors can be provided: a flood illuminator 206, e.g., LED, for projecting flood light, and a dot projector 207, e.g., VCSEL array, for projecting the light pattern. A vertical cavity surface emitting laser (VCSEL) is a type of semiconductor laser diode that emits a laser beam vertically from the top surface, as opposed to a conventional edge-emitting semiconductor laser (also called an in-plane laser), which emits from a surface formed by cutting individual chips from a wafer.

[0244] The dot projector 207 can generate a light pattern on the object. The light pattern may refer to a predefined light pattern. The light pattern may include multiple pattern features. For example, light spots as pattern features may be arranged in a triangular, cubic or hexagonal pattern. The camera 203 can then capture the object illuminated by the flood light at one time and the object illuminated by the light pattern at another time. That is, the camera 203 can receive the light reflected by the object when illuminated by the light of the flood illuminator 206 and / or the dot projector 207, and generate an image of the object and / or a pattern image from the reflected light. The images, i.e., the pattern image and the flood light image, are then passed to an image processor, as described in particular in FIG. 3, which can be configured to execute a neural network.

[0245] The camera may include one or more image sensors 205 (also simply referred to as sensors 205) for capturing digital images. The image sensor 205 may be an array of sensors. The sensors of the sensor array may include, without limitation, charge-coupled element (CCD) and / or complementary metal oxide semiconductor (CMOS) sensor elements for capturing infrared images (IR) or other non-visible electromagnetic radiation. The term (image) sensor or (image) sensor element generally refers to a device or a combination of devices configured to sense at least one parameter. The parameter may be an optical parameter and the sensor may be an optical sensor. The sensor may be formed as an integral single device or as a combination of several devices. The sensor may include a matrix of optical sensors. The sensor may include at least one CMOS sensor. The matrix may be composed of independent pixels, such as independent optical sensors. Thus, a matrix of inorganic photodiodes may be constituted. Alternatively or additionally, commercially available matrices may be used, for example one or more CCD detectors, such as CCD detector chips, and / or CMOS detectors, such as CMOS detector chips. Thus, in general, the sensor may be and / or include at least one CCD and / or CMOS (device) and / or the photosensor may form a sensor array or be part of a sensor array such as the matrix described above. The camera 203 may include multiple image sensors 205 to capture multiple types of images. For example, the camera 203 may include both an IR sensor and an RGB (red, green, blue) sensor, such as an RGB camera 205a.

[0246] The sensor may be or may include at least an inorganic photodiode having sensitivity in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. In particular, the sensor may have sensitivity in the part of the near infrared range where silicon photodiodes are applicable, particularly in the range of 700 nm to 1100 nm. Alternatively and / or additionally, the sensor may have sensitivity in one or more of the ultraviolet, visible or infrared spectral ranges. In particular, the sensor may have sensitivity in the visible spectral range of 500 nm to 780 nm, most preferably 650 nm to 750 nm, or 690 nm to 700 nm. In particular, the optical sensor may have sensitivity in the near infrared range. In particular, the sensor may have sensitivity in the part of the near infrared range where silicon photodiodes are applicable, particularly in the range of 700 nm to 1000 nm. The sensor may have sensitivity in particular in the infrared spectral range, particularly in the range of 780 nm to 3.0 μm.

[0247] The camera 203 includes an illuminator for illuminating the subject with different types of light detected by the image sensor. For example, the camera may include an illuminator for visible light (e.g., a "flash illuminator"), an illuminator for RGB light, and / or an illuminator for infrared light. In particular, the camera 203 may include a flood IR illuminator 206 and a pattern illuminator 207. In certain embodiments, the illuminator includes, but is not limited to, an array of light sources such as VCSELs (Vertical Cavity Surface Emitting Lasers). The image sensor and the illuminator may be included in a single or separate chip packages. In some embodiments, the image sensor and the illuminator are located in separate chip packages.

[0248] In certain embodiments, image sensor 205 is an IR image sensor, and the image sensor is used to capture infrared images used for face detection, face recognition authentication, material detection, and / or depth detection. For face detection, illuminator 206 can provide flood IR illumination to flood a subject with IR illumination, and image sensor 205 can capture an image of the flood IR illuminated subject. The flood IR illuminated image can be, for example, a two-dimensional image of the subject illuminated by IR light.

[0249] For depth or material detection, the illuminator 207 can provide patterned IR illumination. The pattern can be a pattern of light projected onto the object, with a known and controllable configuration and pattern. The pattern can be regularly or irregularly arranged in a structured pattern of light patterns. In a particular embodiment, the pattern is a speckle pattern. Patterns include, but are not limited to, dots, speckles, stripes, dashes, nodes, edges, and combinations thereof.

[0250] The images captured by the camera 203 include images having the user's face (e.g., the user's face is included in the image). An image having the user's face may include any digital image in which at least a portion of the user's face is shown within the frame of the image. Such an image may include only the user's face, or may include the user's face in a smaller or partial portion of the image. The user's face may be captured with sufficient resolution in the image to allow image processing of one or more features of the user's face for facial recognition.

[0251] 3 shows diagrammatically and exemplarily an embodiment of a processor 301 associated with a camera 303 for processing input data in general, and for processing a camera signal in particular. The camera 303 may correspond to the camera 103 of FIG. 1. The processor 301 may correspond to the processor 101 of FIG. 1.

[0252] The processor 301 may be adapted to, for example, select pattern features of the pattern image to be located on the image of the object from the pattern image by executing respective computer control signals, and / or generate a partial image of the pattern image based on the selected pattern features, for example via cropping, and / or extract material information of the object from the partial image, and / or identify the object based on the material information, in particular authenticate the object, for example by providing the partial image to the neural network module 310.

[0253] In particular, images captured by the camera 303 may be processed by the processor 301. The processor may include circuitry configured to execute instructions defined in an instruction set architecture implemented by the processor. The processor may execute applications that run on the device.

[0254] The processor may include an image signal processor 308 (IP), also referred to as an image processor 308 (IP). The IP 308 may include circuitry suitable for processing images received from the camera. The IP 308 may include any hardware and / or software (e.g., program instructions) capable of processing or analyzing images captured by the camera.

[0255] According to a preferred embodiment with special features in the context of the authentication process, for sensitive operations the processor may include a secure enclave processor 309 (SP), also denoted as secure processor 309 (SP). The SP 309 may be a secure circuit configured to authenticate an active user. The SP 309 may be a circuit that protects isolated internal resources from direct access by external circuits. The internal resources may be memory that stores sensitive data such as biometric information, encryption keys, etc.

[0256] The SP 309 may perform a facial recognition authentication process based on images captured by the camera 303 and processed by the IP 308. The functions of the IP 308 and the SP 309 may be performed by a single processor. The processor may perform an enrollment process. During the enrollment process, the camera 303 captures or collects images and / or image data from an authorized user and then authenticates the user using a facial recognition authentication process. Templates may be generated from the images of the enrollment process and stored in the SP 309.

[0257] The facial recognition camera 303 and / or camera module can communicate image data to the SP 309 over a secure channel. The secure channel can be, for example, either a dedicated path for communicating data (i.e., a path shared only with the intended participants) or a dedicated path for communicating data encrypted using an encryption key known only to the intended participants. The SP 309 can run one or more machine learning models.

[0258] To operate the machine learning model, one or more neural network modules 310 can be used. The neural network modules 310 can be located in the IP 308 and / or the SP 309. The SP 309 can compare image features to templates stored for each type of image and generate an authentication score according to a matching score or other ranking of matches between the user in the captured image and in the stored templates. The authentication scores of images such as flooded IR images and patterned illumination can be combined to determine the identity of the user and, if authenticated, allow the user to use the device (e.g., unlock the device).

[0259] 4 shows a schematic and exemplary flow chart of one embodiment of a biometric authentication process using facial features, including steps 401 capturing image data, 402 providing image data from a camera, 403 pre-processing image data, 404 recognizing and verifying human skin or material, 405 verifying, 406 providing other unlocking mechanisms, e.g., a user pin, 407 authenticating via a floodlight image based authentication process, 408 matching facial features, 406 providing other unlocking mechanisms, e.g., a user pin, and 410 generating operational parameters such as unlocking application, device, etc.

[0260] One or more images are captured by a camera in step 401 and provided, for example, to a processor in step 402. The images may be captured in response to a prompt by a user. For example, an unlock attempt image may be captured when a user attempts to access the device. A sequence of multiple images of the user's face may be taken over a short period of time (e.g., one second or less). The sequence of multiple images of the user's face may include flooded IR image and patterned illumination image pairs. Additionally or alternatively, the images may include images captured while an object is illuminated with visible light.

[0261] The images may include flood IR images and / or patterned illumination images, preferably in conjunction with physical randomization embodiments such that background is suppressed. Flood IR image data may include image data of a user captured when the user is illuminated by flood IR illumination. Patterned illumination image data may include image data of a user generated from an image of a user captured when the user is illuminated by patterned illumination. The patterned illumination used to generate the patterned illumination images may include patterns of any pattern features and pattern densities. Patterns may be irregular (structured light for triangulation), regular or periodic arrangements or pattern features (triclinics, hexagons, etc.).

[0262] Image capture may be configured to provide manipulated images of the same scene (e.g., a user's face). For example, partial images can be generated by capturing different parts of the scene or by different camera settings, such as different spot patterns. In this way, image data of the same scene can be manipulated at the time of capture. For example, a non-digitized physical randomizer can capture 20 ms 5 frames as a time series with variation-corrected SNR, or augment image capture by brightness, distance, etc.

[0263] 4a-b are illustrations of images including a patterned illumination image (also denoted as speckle projection, e.g., a human face), e.g., a two-dimensional image of a human face. The pattern features can be dots, spots, speckles, triangles, etc. FIG. 4a shows a hexagonal dot pattern. When applying the BPA technology underlying the present invention, as will be described in particular with respect to FIG. 6 below, each dot 420 of the dot pattern contains information about the distance and material of the object. This reduces the number of dots required per area. In particular, 1000-2000 dots per image are sufficient for accurate 3D mapping and / or material detection and / or identification, in particular authentication of objects based on material information. Furthermore, as a result, the distance 421 between the dots 420 can be increased without losing functionality and / or accuracy. In addition, as a result, even if the number of dots 420 is small, e.g., within an eye-safe pyramid 422, the power per dot 420 can be high. For example, the laser power in the BPA can be 0.5 mW per dot. Figure 4b shows an irregular structured light dot pattern when applying structured light technology, for example. Figure 4c shows a cross-section of the detected intensity distribution showing the background intensity contribution and the pattern contribution for 0.5 mW per spot in a face image taken at 100 klux and 70 cm object distance. As can be seen, at 8-bit grayscale, the contrast between the background intensity contribution and the pattern contribution is in the range of 20-40 orders of magnitude, which is sufficient for reliable BPA for human skin or material recognition and / or authentication.

[0264] In step 403, the image is provided to a processor for pre-processing. The processor may be a secure processor (SP or secure enclave on a smartphone) or a regular IP, as described with respect to FIG. 3. The combination of flood IR image data and patterned illumination data allows the SP to authenticate the user through face recognition. The camera module and / or camera may communicate the image data to the SP via a secure channel. The secure channel may be a dedicated path for communicating data (i.e., a path shared only by the intended participants) or a dedicated path for communicating data encrypted using an encryption key known only to the intended participants. The camera module and / or camera and / or IP may perform various processing operations on the image data before providing it to the SP to facilitate biometric authentication based on user-specific biometric features performed by the SP.

[0265] Image pre-processing can include detecting a bounding box around the face in the floodlight image. Such a bounding box may be detected via a model such as a neural network or via commonly known techniques for bounding box determination in face recognition. Additionally, landmarks can be determined. The bounding box and landmarks can be associated with corresponding pixels in the pattern image data. In particular, the pattern image data associated with the face and lying within the bounding box from the floodlight image can be used, for example, to crop the pattern image to a region of interest.

[0266] The images may be pre-processed to extract the material information, preferably with digital randomization applied to suppress the background. One pre-processing can include digitally manipulating the image data. In particular, the images may be manipulated by image augmentation techniques such as cropping, rotating, blurring, etc. Such manipulations can suppress background information with facial features. Patterns are important for extracting material information. Therefore, models for extracting material information can be trained on manipulated image data, for example image data after the described manipulations. In this way, the size of the model can be reduced. For example, this method can significantly reduce the storage and processing of models on a smartphone, for example, in the case of face recognition using CNN, from more than 20,000,000 parameters to less than 10,000,000, preferably less than 8,000,000, or between 3,000,000 and 7,000,000, preferably about 4,000,000 parameters to recognize partial images.

[0267] One option for pre-processing is cropping the image into sub-images, as shown diagrammatically and exemplarily in Fig. 5a and Fig. 5b. Fig. 5c visualizes a neural network 503 for cropping an image 504 into multiple sub-images 505. The cropping may be based on identifying a peak of the pattern and cropping a constant size around and / or next to the peak. In one embodiment, the peak 501a may be in the center of a particular sub-image 502a (see Fig. 5a). The sub-images may have a constant size and may not overlap. In another embodiment, the cropping may be random with respect to the position of the peak (Fig. 5b). The sub-images 502b may have any size. The sub-images may overlap. The sub-images may contain at least one pattern feature 501b (intensity peak) or more pattern features. The sub-images may contain parts of the pattern features or outlier signatures of the pattern features. In the preferred embodiment based on neural networks, another option is single shot detection (SSD; region-based neural network or mask RCNN) to provide a bounding box for partial image segmentation. Thus, the operation can be based on segmentation via anchor points based on pattern features of the pattern image (segmentation or SSD or RCNN).

[0268] From the partial image data, materials may be recognized (step 404) and preferably the material recognition may be used for the verification step 405. Such recognition may be based on a data-driven model, such as a trained neural network, such as a convolutional neural network. The training of such a network is described in more detail below. The model may be parameterized to map the partial image to a classifier, such as human skin or non-skin, material or non-material. The model can also be parameterized to more complex outputs, such as human skin, material classes (latex, silicone, cloth), etc. Based on such recognition, the authentication process may be verified (step 405). For example, if no human skin is detected from the image (verification 405: No), the authentication process is stopped and a notification is provided to the user. In such a case, authentication is not possible or there is a high probability of spoofing (anti-spoofing). Alternatively or additionally, other unlocking mechanisms may be applied, such as a user pin. However, if skin or the like is detected, the verification may lead to further processing, such as providing the materials present in the image. Such material detection may be associated with landmarks, such as those detected in the floodlight image, for consistency checks. The consistency checks may include rule sets. For example, if latex is detected to be associated with eyes, the authentication process may be stopped. Material recognition and verification steps may also be performed after face recognition.

[0269] For authentication, the flood image data and the partial image data can be processed to provide material recognition and face recognition. In particular, if verification 405 is "yes," for authentication, the flood image can be analyzed for facial features (step 407). The analyzed facial features can be compared to templates. The templates in the template space can be provided to obtain a matching score. In certain embodiments, the template space may include templates of enrollment profiles of authorized users on the device (e.g., templates generated during the enrollment process). The matching score can be a difference score between the facial features and the corresponding features in the template space (e.g., feature vectors of authorized users generated during the enrollment process). The matching score can be higher the closer (e.g., the shorter the distance or the smaller the difference) the feature vector is to the feature vector in the template space.

[0270] Comparing the feature vector from the template space with the template to obtain a corresponding matching score may include using one or more classifiers or classifiable networks to classify and evaluate the differences between the generated feature vector and the feature vector from the template space. Examples of different classifiers that may be used include, but are not limited to, linear, piecewise linear, non-linear classifiers, support vector machines, and neural network classifiers. In some embodiments, the matching score may be evaluated using a distance score between the feature vector and the template from the template space.

[0271] For authentication, the matching score may be compared to an unlock threshold of the device (step 408). The unlock threshold may represent a minimum difference in feature vectors between an approved user's face according to the template and the user's face in an unlock attempt to unlock the device. For example, the unlock threshold may be a threshold that determines whether the unlock feature vector is sufficiently close to the template vector associated with the approved user's face.

[0272] If the matching score is above the unlock threshold, e.g., "Yes" at step 408, the user's face in the image captured for unlocking matches the face of an authorized user. If the user's face "matches" a template, enrollment profile, or authorized user, the user in the captured image is authenticated as an authorized user of the enrollment profile on the device, and the device is unlocked (step 410). The unlocking may allow the user access to use the device and / or the user to access selected features of the device (e.g., unlocking features of applications running on the device, payment systems or making payments, accessing personal data, enlarging notifications, etc.).

[0273] If the matching score falls below the unlock threshold, e.g., "Yes" at step 408, the device remains locked and the facial recognition process may stop or other unlock options may be requested (step 409). For example, the unlock options may include unlocking the device using another authentication protocol. The unlock options may include presenting the user with one or more options to proceed with unlocking the device or accessing the functionality with a different type of authentication (e.g., presenting the user with an option to proceed with a second authentication protocol). Presenting the options may include, for example, displaying one or more options on a display of the device and prompting the user through audio and / or visual communication to select one of the displayed options to proceed with unlocking the device or accessing the functionality on the device. The user may then proceed to unlock / access the device using the selected option. After successful authentication using the selected option, the user's initial request to unlock / access the device may be granted. The unlock options may include unlocking the device using a passcode, password, pattern entry, a different form of biometric authentication, or another authentication protocol.

[0274] Another criterion could be unlock attempts that can be compared to a threshold, which could be the maximum number of unlock attempts allowed or the maximum time allotted for an unlock attempt.

[0275] If the unlock attempts exceed the threshold of other unlock mechanisms, the device may be locked from further facial authentication attempts for a specified period of time and / or until other authentication protocols are used to unlock the device.

[0276] If the unlock attempts are below a threshold (e.g., if the number of unlock attempts is below a maximum number of attempts allowed), a new image of the user may be captured. In some embodiments, the device may prompt the user to provide input to restart the facial recognition process. For example, the user may be prompted to acknowledge or otherwise confirm (visually and / or audibly) the attempt to restart the facial recognition process.

[0277] Fig. 6 shows, in a schematic and exemplary manner, different examples of clusters, namely a first cluster 601, a second cluster 602 and a third cluster 603. Clustering, which is the generation of (low dimensional) representations of partial images and the assignment of the representations to clusters, is explained with respect to Figs. 7a-7c. As explained below, brightness and possible background can play a role in clustering. This is illustrated in Fig. 6 by plotting partial images associated with one representation that are part of the same cluster 601, 602 or 603.

[0278] A possible process for generating material information and / or material data from the image and / or image data can be a suitable filtering technique. The material information, also denoted as material data, is generated by applying a material-dependent image filter to one or more pattern features of the pattern image. Such a technique can be beam profile analysis (BPA), as described in WO 2020 / 187719 A1. The disclosure of WO 2020 / 187719 A1 is incorporated herein by reference.

[0279] In particular, the sensor (element) can be configured to record a beam profile of at least one (reflection feature) of the (reflection) image. The evaluation device may be configured for determining and / or selecting a first area of ​​the beam profile and a second area of ​​the beam profile. The first area of ​​the beam profile may include substantially edge information of the beam profile and the second area of ​​the beam profile may include substantially central information of the beam profile. The beam profile may have a center, i.e. a maximum of the beam profile and / or a central point of the plateau of the beam profile and / or a geometric center of the light spot, and a falling edge extending from the center. The second area may include an inner area of ​​the cross section and the first area may include an outer area of ​​the cross section. As used herein, the term "substantially central information" generally refers to a low percentage of edge information, i.e. a low percentage of the intensity distribution corresponding to the edge, compared to the percentage of central information, i.e. a percentage of the intensity distribution corresponding to the center. Preferably, the central information has a percentage of edge information of less than 10%, more preferably less than 5%, and most preferably the central information does not include edge content. As used herein, the term "substantially edge information" generally refers to a low percentage of central information compared to the percentage of edge information. The edge information may include information of the entire beam profile, particularly information from the central and edge regions. The edge information has a percentage of central information less than 10%, preferably less than 5%, and more preferably the edge information does not include central information. If the beam profile is close to or around the center and includes substantially central information, at least one area of ​​the beam profile may be determined and / or selected as the second area of ​​the beam profile. If the beam profile includes at least a portion of the falling edge of the cross section, at least one area of ​​the beam profile may be determined and / or selected as the first area of ​​the beam profile. For example, the entire area of ​​the cross section may be determined as the first area. The first area of ​​the beam profile may be area A2 and the second area of ​​the beam profile may be area A1.

[0280] The edge information may include information on the number of photons in a first area of ​​the beam profile, and the center information may include information on the number of photons in a second area of ​​the beam profile. The evaluation device may be adapted to determine a surface integral of the beam profile. The evaluation device may be adapted to determine the edge information by integration and / or summation of the first area. The evaluation device may be configured to determine the center information by integration and / or summation of the second area. For example, the beam profile may be a trapezoidal beam profile and the evaluation device may be adapted to determine the integral of the trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge and center signals may be replaced by an equivalent evaluation utilizing the properties of the trapezoidal beam profile, such as, for example, determining the slope and position of the edges and the height of the central plateau, as well as deriving the edge and center signals by geometrical considerations.

[0281] Additionally or alternatively, the evaluation device may be adapted to determine one or both of center and edge information from at least one slice or cut of the light spot. This can be achieved, for example, by replacing the area integral of the combined signal Q with a line integral along the slice or cut. To improve accuracy, multiple slices or cuts through the light spot may be used for averaging. In case of an elliptical spot profile, improved distance information may be obtained by averaging over multiple slices or cuts.

[0282] The evaluation device may be configured to derive the combined signal Q by one or more of dividing the edge information and the center information, dividing a multiple of the edge information and the center information, dividing a linear combination of the edge information and the center information. Thus, in effect, a photon ratio may be used as the physical basis of the method.

[0283] The sensor element may be configured to record a beam profile of at least one reflection feature of the reflection image. The detector comprises at least one evaluation device. The evaluation device may be configured to identify and / or select at least one reflection feature, in particular at least one light spot, of the reflection image provided by the sensor element. The evaluation device may be configured to perform at least one image analysis and / or image processing to identify the reflection feature. The image analysis and / or image processing may use at least one feature detection algorithm. The image analysis and / or image processing may include: filtering; selecting at least one region of interest; forming a difference image between an image produced by the sensor signal and at least one offset; inverting the sensor signal by inverting the image produced by the sensor signal; forming a difference image between images produced by the sensor signal at different times; background correction; decomposition into color channels; decomposition into hue; saturation; luminance channel; frequency decomposition; singular value decomposition; applying a blob detector; applying a corner detector; applying a determinant of a Hessian filter; applying principal curvature based region detector; applying maximum stable extremum region detector; applying a generalized Hough transform; applying a ridge detector; applying an affine invariant feature detector; applying an affine adaptive interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale invariant feature transform; The methods may include one or more of: applying a scale space extremum detector; applying a local feature detector; applying a fast robust feature algorithm; applying a gradient position and orientation histogram algorithm; applying a histogram of oriented gradient descriptors; applying a Deriche edge detector; applying a differential edge detector; applying a space-time interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplace operator of a Gaussian filter; applying a differential Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Schall operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high pass filter; applying a low pass filter; applying a Fourier transform; applying a Radon transform; applying a Hough transform; applying a wavelet transform; thresholding; and generating a binary image.In particular, evaluating the reflected image includes selecting a region of interest in the reflected image. The region of interest may be determined manually by a user or automatically, such as by recognizing an object in the image produced by the sensor elements. For example, in the case of a spot-like reflective feature, the region of interest may be selected as the area around the spot profile.

[0284] The evaluation device 132 may be configured to determine the material property m by evaluating the beam profile of the reflected image. The beam profile of the reflected image may be selected from the group consisting of a trapezoidal beam profile; a triangular beam profile; a linear combination of a conical beam profile and a Gaussian beam profile. The evaluation device may be configured to apply at least one distance-dependent image filter and at least one material-dependent image filter to the beam profile and / or to at least one specific region of the beam profile. In particular, the image filter Φ maps the image f or a region of interest in the image to a real number Φ(f(x,y))=φ, where φ denotes a feature, in particular a distance feature in the case of a distance-dependent image filter or a material feature in the case of a material-dependent image filter. Images may be affected by noise, and the same is true for the features. Thus, the features may be random variables. The features may follow a normal distribution. If the features do not follow a normal distribution, they may be transformed to follow a normal distribution, for example by a Box-Cox-Transformation.

[0285] The evaluation device determines at least one distance feature φ by applying at least one distance-dependent image filter Φ1 to the reflected image. 1z The distance feature may be or may include at least one measurement of the distance of the object, a distance value, a longitudinal coordinate of the object, or other information regarding the distance of the object. The distance dependent image filter may be a depth filter from photon ratio; a depth filter from defocus; or a linear combination thereof; or |ρ Ф1other , Фz |≧0.40(Фz is a depth filter from the photon ratio or a depth filter from the defocus or a linear combination thereof) by a further distance-dependent image filter Φ, which is correlated to the depth filter from the photon ratio and / or the depth filter from the defocus or a linear combination thereof 1other , . . . The further distance-dependent image filter Φ 1other is the distance-dependent image filter Φ z and |ρ Ф1other , Фz |≧0.60, preferably |ρ Ф1other , Фz Two image filters Φ may be correlated by | ≥ 0.80. i and Φ j The similarity of may be evaluated by the correlation of their features, specifically by calculating the Pearson correlation coefficient.

[0286]

number

[0287] where μ and σ are the mean and standard deviation of the obtained features. The test of the correlation of the filters can be performed with a set of random test images, specifically a matrix filled with random numbers. The number of random test images can be selected so that the result of the correlation test is statistically significant. The correlation coefficient takes values ​​between −1 and 1, where 0 means that there is no linear correlation. The correlation coefficient is suitable to determine whether two filters are similar or equivalent. To measure whether the features of the filters are correlated with a given property, for example distance, the test images can be selected such that the correlation filter actually yields that property. As an example, to measure whether the features of the filters are correlated with distance, beam profiles recorded at different distances can be used as test images. To obtain a comparable, transferable and transparent evaluation, a fixed test set of test images can be defined.

[0288] For example, the distance dependent image filter may be a depth from photon ratio filter. The depth from photon ratio filter may include evaluating a combined signal Q from at least two sensor signals of the sensor elements. The evaluation device determines a distance characteristic φ by evaluating the combined signal Q. 1z The distance characteristic determined by evaluation of the combined signal Q may correspond directly to the longitudinal coordinate of the object. The combined signal Q may be determined using various means. By way of example, software means for deriving the combined signal, hardware means for deriving the combined signal, or both may be used and may be implemented in the evaluation device. Thus, the evaluation device may by way of example include at least one divider, where the divider is configured to derive a quotient signal. The divider may be embodied fully or partially as one or both of a software divider or a hardware divider.

[0289] The evaluation device may be configured to derive the combined signal Q by one or more of dividing the sensor signals, dividing a multiple of the sensor signals, or dividing a linear combination of the sensor signals. The evaluation device may be configured to derive the combined signal Q by one or more of dividing the sensor signals, dividing a multiple of the sensor signals, or dividing a linear combination of the sensor signals. 1z , and the distance feature φ 1z may be configured to determine:

[0290]

number

[0291] where x and y are lateral coordinates, A1 and A2 are different areas of at least one beam profile of a light beam propagating from an object to a detector at the sensor location, and E(x, y, z o ) is the object distance z orepresents a beam profile given by: Area A1 and Area A2 may be different. In particular, A1 and A2 are not congruent. Thus, A1 and A2 may differ in one or more of shape or content. The beam profile may be the cross-section of a light beam. The beam profile may be selected from the group consisting of a trapezoidal beam profile; a triangular beam profile; a linear combination of a conical beam profile and a Gaussian beam profile. In general, a beam profile is a function of the intensity L(z o ) and beam shape S(x,y;z o ) in the image. The combined signal can thus be derived to determine a vertical coordinate that is independent of the luminance. Furthermore, the combined signal can be used to determine the distance z0 independently of the size of the object. In this way, the combined signal allows the distance z0 to be determined independently of the material and / or reflecting and / or scattering properties of the object and independently of changes in the light source due to, for example, manufacturing precision, heat, moisture, dirt, lens damage, etc. As an example, the distance-dependent feature φ 1z is a function of the combined signal Q, φ 1z =φ 1z (Q), where the function may be a first order polynomial, a second order polynomial, or a higher order polynomial of Q. Further, as an example, the object distance z0 may be expressed as a function of the distance-dependent feature φ 1z function, z0=z0(φ 1z ), where the function φ 1z Thus, the object distance z0 may be a function of the coupled signal Q, z0=z0(Q), where the function may be a first order polynomial, a second order polynomial, or a higher order polynomial in Q.

[0292] The light sensitive areas of the at least two light sensors may be arranged such that the first sensor signal comprises information of a first area of ​​the beam profile and the second sensor signal comprises information of a second area of ​​the beam profile, the first area of ​​the beam profile and the second area of ​​the beam profile being either adjacent or overlapping or both.

[0293] The evaluation device may be configured to determine and / or select a first area of ​​the beam profile and a second area of ​​the beam profile. The first area of ​​the beam profile may include substantially edge information of the beam profile and the second area of ​​the beam profile may include substantially center information of the beam profile. The beam profile may have a center, i.e. a maximum of the beam profile and / or a center point of the plateau of the beam profile and / or a geometric center of the light spot, and a falling edge extending from the center. The second area may include an inner area of ​​the cross section and the first area may include an outer area of ​​the cross section. Preferably, the center information has a percentage of edge information of less than 10%, more preferably less than 5%, and most preferably the center information does not include edge content. The edge information may include information of the entire beam profile, in particular information from the center and edge areas. The edge information has a percentage of center information of less than 10%, preferably less than 5%, and more preferably the edge information does not include center information. At least one area of ​​the beam profile may be determined and / or selected as the second area of ​​the beam profile if the beam profile is close to or around the center and includes substantially center information. If the beam profile includes at least a portion of the falling edge of the cross section, at least one area of ​​the beam profile may be determined and / or selected as the first area of ​​the beam profile. For example, the entire area of ​​the cross section may be determined as the first area. The area of ​​the first area of ​​the beam profile may be area A2, and the second area of ​​the beam profile may be area A1.

[0294] The edge information may include information on the number of photons in a first area of ​​the beam profile, and the center information may include information on the number of photons in a second area of ​​the beam profile. The evaluation device may be adapted to determine an area integral of the beam profile. The evaluation device may be adapted to determine the edge information by integration and / or summation of the first area. The evaluation device may be adapted to determine the center information by integration and / or summation of the second area. For example, the beam profile may be a trapezoidal beam profile and the evaluation device may be adapted to determine the integral value of the trapezoid. Furthermore, if a trapezoidal beam profile is assumed, the determination of the edge and center signals may be replaced by an equivalent evaluation utilizing the properties of the trapezoidal beam profile, such as, for example, determining the slope and position of the edges and the height of the central plateau, as well as deriving the edge and center signals by geometrical considerations.

[0295] Additionally or alternatively, the evaluation device may be adapted to determine one or both of center and edge information from at least one slice or cut of the light spot. This may be achieved, for example, by replacing the surface integral of the combined signal Q with a line integral along the slice or cut. To improve accuracy, several slices or cuts through the light spot may be used for averaging. In case of an elliptical spot profile, averaging over several slices or cuts may provide improved distance information.

[0296] The evaluation device may be configured to derive the combined signal Q by one or more of dividing the edge information and the center information, dividing a multiple of the edge information and the center information, dividing a linear combination of the edge information and the center information. Thus, in effect, a photon ratio may be used as the physical basis of the method.

[0297] The evaluation device may be configured to derive the combined signal Q, in particular by dividing the first and second sensor signals, by dividing a multiple of the first and second sensor signals, or by dividing a linear combination of the first and second sensor signals. As an example, Q may be simply determined as Q=s1 / s2 or Q=s2 / s1, where s1 denotes the first sensor signal and s2 denotes the second sensor signal. Additionally or alternatively, Q may be determined as Q=a·s1 / b·s2 or Q=b·s2 / a·s1, where a and b are, by way of example, predefined or determinable real numbers. Additionally or alternatively, Q may be determined as Q=(a·s1+b·s2) / (c·s1+d·s2), where a, b, c and d are, by way of example, predefined or determinable real numbers. As a simple example of the latter, Q may be determined as Q=s1 / (s1+s2). Other combination or quotient signals are possible.

[0298] For further details and embodiments of the evaluation of the combined signal Q and the determination of the vertical coordinate z, reference is made to WO 2018 / 091640, WO 2018 / 091649 A1 and WO 2018 / 091638 A2, the contents of which are hereby incorporated by reference.

[0299] For example, the distance dependent image filter may be a depth from defocus filter. As outlined above, the evaluation device may be configured to determine at least one image of the region of interest from the sensor signal. The evaluation device may be configured to determine at least one blurring function f a By optimizing the distance feature φ 1z The determined distance feature φ 1z may directly correspond to the vertical coordinate of the object. 1zmay be determined using at least one convolution-based algorithm, such as a depth from defocus algorithm. To obtain the distance from the image, the depth from defocus algorithm estimates the defocus of the object. For this estimation, a blurring function is assumed. In particular, the blurring function models the blur of the defocused object. At least one blurring function f a may be a function or a composite function composed of at least one function from the group consisting of Gaussian, sinc function, pillbox function, square function, Lorentz function, radial function, polynomial, Hermite polynomial, Zernike polynomial, Legendre polynomial.

[0300] The blurring function can be optimized by varying at least one parameter of the blurring function. The reflected image is then compared to the blurred image i b The evaluation device may be b and the blurring function f a from the distance feature φ 1z The distance feature φ may be reconstructed. 1z by varying the parameter σ of the blurring function, b Using this, blurred image i b and the blurring function f a may be determined by minimizing the difference between the convolution of

[0301]

number

[0302] σ(z) is a set of distance-dependent blurring parameters. The further images may be blurred or sharp. At least one further image is convolved with a known blurring function to produce a blurred image i b Thus, the depth from defocus algorithm can be generated from the distance feature φ 1zcan be used to obtain

[0303] The evaluation device measures the distance feature φ determined by applying a depth filter from the photon ratio. 1z and the distance feature φ determined by applying the depth from defocus filter. 1z The combined distance information z may be configured to determine at least one combined distance information z based on a distance feature φ determined by applying a depth filter from a photon ratio. 1z and the distance feature φ determined by applying the depth from defocus filter. 1z The joint distance information z may be a real function that depends on and . The joint distance information z may also be a distance feature φ determined by applying a depth filter from the photon ratio. 1z and the distance feature φ determined by applying the depth from defocus filter. 1z The depth from defocus distance measurement can be a rational or irrational polynomial with respect to the photon ratio. Depth from defocus is a complementary method to depth from photon ratio, but uses a similar hardware configuration. Furthermore, the depth from defocus distance measurement can have similar accuracy. Combining both techniques can produce advantageous distance measurement results with high accuracy.

[0304] For example, the distance-dependent image filter may be a structured light filter combined with a depth-from-photon-ratio filter and / or a depth-from-defocus image filter. The detector may include at least two sensor elements, each with a matrix of light sensors. The at least one first sensor element and the at least one second sensor element may be arranged at different spatial positions. The relative distance between the first sensor element and the second element may be fixed. The at least one first sensor element may be adapted to determine at least one first reflection pattern, in particular at least one first reflection feature, and the at least one second sensor element may be adapted to determine at least one second reflection pattern, in particular at least one second reflection feature. The evaluation device may be configured to select at least one image determined by the first or second sensor element as a reflection image and to select at least one image determined by the other one of the first or second sensor elements as a reference image. The reference image may be determined by one or more of recording at least one reference feature, imaging at least one reference feature, calculating a reference image. The reference image and the reflected image may be images of the object determined at different spatial positions having a certain distance. The distance may be a relative distance, also called a baseline. The evaluation device may be adapted to select at least one reflected feature in the reflected image and to determine a distance feature φ determined by applying a depth image filter from photon ratio and / or a depth image filter from defocus. 1z and adapted to determine a distance estimate of at least one of the selected reflectance features of the reflectance image given by an error interval ±ε.

[0305] The evaluation device may be adapted to determine at least one reference feature of the at least one reference image that corresponds to the at least one reflection feature. The evaluation device may be adapted to perform an image analysis and to identify features of the reflection image. The evaluation device may be adapted to identify at least one reference feature in the reference image that has substantially the same vertical coordinate as the selected reflection feature. The evaluation device may be adapted to determine an epipolar line in the reference image. The relative positions of the reference image and the reflection image may be known. For example, the relative positions of the reference image and the reflection image may be stored in at least one storage unit of the evaluation device. The evaluation device may be adapted to determine a straight line extending from the selected reflection feature of the reflection image. The straight line may include a possible object feature that corresponds to the selected feature. The straight line and the baseline develop an epipolar plane. Since the reference image is determined at a different relative position than the reflection image, the corresponding possible object feature may be imaged on a straight line called an epipolar line in the reference image. Thus, the feature of the reference image that corresponds to the selected feature of the reflection image is located on the epipolar line. Due to image distortions or changes in system parameters due to aging, temperature changes, mechanical stress, etc., the epipolar lines may cross or be very close to each other and / or the correspondence between reference and reflected features may be unclear. Furthermore, each known location or object in the real world may be projected onto the reference image and vice versa. The projection may be known by detector calibration, while the calibration corresponds to teaching-in the epipolar geometry of a particular camera.

[0306] The evaluation device may be configured to determine at least one displacement area in the reference image corresponding to the distance estimate. In particular, the displacement area can be an area in the reference image where a reference feature corresponding to the selected reflection feature is expected to be located in the reference image. Depending on the distance to the object, an image position of the reference feature corresponding to the reflection feature can be displaced in the reference image compared to an image position of the reflection feature in the reflection image. The displacement area can include only one reference feature. The displacement area can also include multiple reference features. The displacement area can include an epipolar line or a part of an epipolar line. The displacement area can include multiple epipolar lines or multiple parts of multiple epipolar lines. The displacement area can extend along the epipolar line, or orthogonally to the epipolar line, or both. The evaluation device may be adapted to determine the reference feature along the epipolar line corresponding to the distance feature and to determine an extent of the displacement area along the epipolar line or orthogonally to the epipolar line corresponding to an error interval ±ε. The measurement uncertainty of the distance measurement may result in a displacement region that is non-circular since the measurement uncertainty may vary with direction. In particular, the measurement uncertainty along the epipolar line or lines may be larger than the measurement uncertainty in an orthogonal direction with respect to the epipolar line or lines. The displacement region may include a region that extends in an orthogonal direction with respect to the epipolar line or lines. The evaluation device may determine the displacement region around the image location of the reflective feature. The evaluation device may be configured to determine a distance estimate and to estimate φ 1z It can be adapted to determine the displacement region along the epipolar line corresponding to ±ε.

[0307] The evaluation device may be configured to match selected features of the reflected pattern with at least one feature of the reference pattern in the displacement region. The evaluation device may be configured to match selected features of the reflected image with reference features in the displacement region by using at least one evaluation algorithm taking into account the determined distance estimate. The evaluation algorithm may be a linear scaling algorithm. The evaluation device may be adapted to determine an epipolar line closest to and / or within the displacement region. The evaluation device may be adapted to determine an epipolar line closest to an image position of the reflected feature. The extent of the displacement region along the epipolar line may be larger than the extent of the displacement region orthogonal to the epipolar line. The evaluation device may be adapted to determine the epipolar line before determining the corresponding reference feature. The evaluation device may determine a displacement region around an image position of each reflected feature. The evaluation device may be adapted to assign an epipolar line to each displacement region of each image position of the reflection feature, such as by assigning an epipolar line closest to the displacement region and / or an epipolar line within the displacement region and / or an epipolar line closest to the displacement region along a direction orthogonal to the epipolar line. The evaluation device may be adapted to determine a reference feature corresponding to the image position of the reflection feature, such as by determining a reference feature closest to the assigned displacement region and / or a reference feature within the assigned displacement region and / or a reference feature closest to the assigned displacement region along the assigned epipolar line and / or a reference feature within the assigned displacement region along the assigned epipolar line.

[0308] The evaluation device may be configured to determine the displacement of the matched reference feature and the selected reflection feature. The evaluation device may be configured to determine the longitudinal information of the matched feature using a predefined relationship between the longitudinal coordinate and the displacement. For example, the longitudinal information may be a distance value. The predefined relationship may be one or more of an empirical relationship, a semi-empirical relationship, and an analytically derived relationship. The evaluation device may comprise at least one data storage device for storing the predefined relationship, such as, for example, a look-up list or a look-up table. The evaluation device may be adapted to determine the predefined relationship by using a triangulation method. If the position of the selected reflection feature in the reflection image and the position of the matched reference feature and / or the relative displacement of the selected reflection feature and the matched reference feature are known, the longitudinal coordinate of the corresponding object feature may be determined by triangulation. Thus, the evaluation device may be adapted to select reflection features, for example sequentially and / or row by row, and to determine the corresponding distance value using triangulation for each potentially possible position of the reference feature. The displacement and the corresponding distance value may be stored in at least one storage device of the evaluation device 132.

[0309] Additionally or alternatively, the evaluation device may include the following steps: - determining the displacement area for the image location of each reflection feature; - assigning an epipolar line to the displacement region of each reflection feature, such as by assigning an epipolar line closest to the displacement region, and / or an epipolar line within the displacement region, and / or an epipolar line closest to the displacement region along a direction orthogonal to the epipolar line; - assigning and / or determining at least one reference feature to each reflection feature, such as by assigning a reference feature closest to the assigned displacement area and / or a reference feature within the assigned displacement area and / or a reference feature closest to the assigned displacement area along the assigned epipolar line and / or a reference feature within the assigned displacement area along the assigned epipolar line; The method may be configured to perform the following steps:

[0310] The evaluation device evaluates at least one material characteristic φ by applying at least one material-dependent image filter Φ2 to the reflected image. 2m The material characteristic may be or may include at least one information regarding at least one material property of the object.

[0311] The material dependent image filter may be a brightness filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or a median filter; a gray level generation based contrast filter; a gray level generation based energy filter; a gray level generation based uniformity filter; a gray level generation based dissimilarity filter; a rho energy filter; a threshold domain filter; or a linear combination thereof; or |ρ Ф2other,Фm |≧0.40 by (Ф m is one of a brightness filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation based energy filter, a gray level generation based uniformity filter, a gray level generation based dissimilarity filter, a rho's energy filter, or a threshold domain filter, or a linear combination thereof), a further material dependent image filter Φ that is correlated to a brightness filter, a spot shape filter, a squared norm gradient, a standard deviation, a smoothness filter, a gray level generation based energy filter, a gray level generation based uniformity filter, a gray level generation based dissimilarity filter, a rho's energy filter, or a threshold domain filter, or a linear combination thereof 2other , . . . The further material-dependent image filter Φ 2other is the material-dependent image filter Φ m and |ρ Ф2other,Фm |≧0.60, preferably |ρ Ф2othe,Фm may be correlated by |≧0.80.

[0312] For a description of exemplary material-dependent image filters, see the descriptions above of the brightness filter, spot shape filter, smoothness filter such as squared norm gradient, standard deviation, Gaussian filter or median filter, gray level generation based contrast filter, gray level generation based energy filter, gray level generation based uniformity filter, gray level generation based dissimilarity filter, low energy filter, and threshold area filter.

[0313] The material-dependent image filter may be at least one arbitrary filter Φ that passes hypothesis validation. The hypothesis validation may include validating the material dependency of the image filter by applying the image filter to a predefined data set. The data set may include a plurality of beam profile images. The beam profile images may include N B is given by a sum of Gaussian radial basis functions,

number

number

number

number

[0314] The values ​​of x,y, are

number

[0315] An exemplary dataset is given in WO 2020 / 187719 A1. The parameter table in WO 2020 / 187719 A1 lists all the images f k All Gaussian functions of g lk Center position x lk =x l ,y lk =y l is stated.

[0316] The parameter table of WO 2020 / 187719 A1 includes, inter alia, material classifiers for a white Teflon target (denoted as control), fabric, dark skin (denoted as dark_skin), pale skin (denoted as pale_skin), and highly transparent skin (denoted as trans), as well as a parameter table for each image f referenced by image index k. k The distance z to the

[0317] 7a and 7b show a schematic and exemplary process for generating an output 703, representations and / or clustering from a partial image 701 via a neural network 702. In particular, for the partial image, a low-level representation is generated via an autoencoder and / or a principal component analysis (PCA). Furthermore, the representations may be clustered. Thus, an embodiment for training for further suppression of background and enhancement of physical information signatures by representation generation and clustering is visualized.

[0318] After cropping, the signal signature of the object's texture (background light + facial features) is already randomized. However, such randomization may not be enough to extract material information. Especially for approaches that rely on data-driven models, attention in the training process needs to focus on the pattern signature (material) rather than the background (facial features). Various approaches can be applied to further suppress the background signature in the image.

[0319] One process can rely on PCA (Principal Component Analysis) and / or RBF (Radial basis function (RBF) neural networks). In general, large data sets can be structured by using principal components via PCA. This allows a simplified illustration of the data set by approximating a large number of statistical variables with a smaller number of linear combinations, i.e. principal components. By projecting the data points onto only the first few principal components, a lower dimensional data is obtained while preserving the variation of the data as much as possible. In an embodiment of the invention, the PCA and / or RBF neural networks generate a representation of the sub-image. Such a representation is an operation in the sense of reducing the dimensionality of the sub-image. Thus, the sub-image is mapped to a lower dimensional representation of the image. This representation is associated with the physical signatures embedded in the reflections of the sub-image and also suppresses background signatures.

[0320] Based on PCA or RBF mapping, representations of the sub-images are generated. These are associated with the corresponding images. Once the representations are generated, they are assigned to PCA clusters or clustered via an algorithm. As explained, the clustering of patches may avoid overfitting since textures are suppressed and / or physical features may not be taken into account. In particular, for example, masking and dilation of patches reduces texture information. A two-dimensional PCA 710 (two-component PCA) is shown diagrammatically and exemplarily in Fig. 7c. The PCA 710 includes a first principal component 711, a second principal component 712, and three clusters 713a-c, namely a first cluster 713a, a second cluster 713b, and a third cluster 713c.

[0321] From the representations and the latent clustering, manipulated image data is generated for extracting material information from the pattern image acquired by the IR sensor, the manipulated image data including a sub-image having at least a portion or one or more of the pattern features present in the pattern image and at least one representation associated with a physical signature embedded in a reflection of the sub-image or at least one class associated with the representation.

[0322] A data-driven model can be trained on the manipulated image data in this way. By manipulating the data in this way, texture or background signatures can be suppressed. This is in contrast to face recognition, where the texture of the object contains the facial features that the model is trained on. In material classification, the model needs to learn the relationship between patterns and material signatures.

[0323] FIG. 7b shows, in a schematic and exemplary manner, another process for generating a representation and / or a clustering. In particular, an embodiment for training for further suppression of the background and enhancement of the physical information signature by generating a representation and / or a clustering is visualized. FIG. 7b visualizes an autoencoder, i.e. an encoder-decoder neural network consisting of an encoder neural network 704 and a decoder neural network 705 trained to predict and generate an output. In particular, the encoder neural network 704 and / or the decoder neural network 705 may be convolutional neural networks. The manipulation of the partial images is performed via a neural network structure including an encoder and a decoder. The neural network may be a CNN suitable for image processing. The encoder part 704 of the CNN generates a representation provided to the latent space representation 706, and the decoder part 705 generates the partial images. The representation is trained by comparing the partial images of the input layer of the encoder network with the partial images of the output layer of the decoder network. After such training, each partial image is associated with one representation trained by a network that further suppresses the background signature by dimensionality reduction. Once the representations are generated, they can be clustered by a clustering algorithm such as k-means or elbow curve, although other clustering methods are possible.

[0324] Other options for constructing low-level representations of sub-images are FFT, wavelets, deep learning like cnns, energy models, regularized flows, GANs, vision transformers or transformers used in natural language processing, autoregressive image modeling, GANs, autoregressive image modeling, regularized flows, deep autoencoders, deep energy-based models, vision transformers. Supervised or unsupervised schemes can be applied to generate the representations, as well as embeddings such as cosine or Euclidean metrics in ML languages.

[0325] FIG. 8 shows, in a schematic and exemplary manner, steps of training a data-driven model for the extraction of skin features in one embodiment based on supervised or self-supervised learning, in particular the training embodiment of a classifier network. Attention in the training process is focused on the pattern signature (material) and not on the background (facial features). Thus, the model is trained based on physical information and not on, for example, facial features. Specifically, the flow chart includes the following steps: A raw signal is provided from a speckle-based measurement (step 801). Thereby, a grating structure may be produced either by a laser array or by a grating. Even for pseudo-random gratings, material information may be determined. The raw data is then corrected based on a dynamic mask provided in step 803 (step 802). The patch size may relate to the camera resolution, distance regime, object type, field of view, etc. The mask may be dynamically generated or predefined. The dynamic adaptation may be based on face recognition from infrared or RGB image projections and / or landmark determination. For example, by detecting landmarks and using the corresponding masks, glasses, beard, etc. may be detected.

[0326] In step 804, the operations described above generate partial image data and representations. The partial images are then grouped according to the representations (step 805). The patches may be further augmented (e.g., randomizer, physical randomizer without digital, etc.). Finally, in step 806, one or more partial images and / or one or more clusters and / or one or more representations are provided for training a classification network. The clusters may include additional annotations, e.g., per silicone of mask, latex mask + brand, external context from metadata, glasses, beard and / or other facial features as image context.

[0327] The manipulated data can include known data of the sub-images. For example, a training dataset of pattern images may be labeled as skin / non-skin. Such labels can be preserved through manipulation of the sub-images.

[0328] The manipulated data including the representations, labels, and / or clusters can be fed into a classifier network, such as a CNN, using supervised or self-supervised learning techniques. For example, Resnet, triplet loss, npx loss, npair loss can be used. The representations and / or clusters can be used as inputs for the input layer of the network or via a loss function of the model, for example via a contrast divergence loss function. In this way, the manipulated data including the representations and / or clusters become part of the training process of the model. Additionally, cost functions, error functions, objective functions, key representations, and / or clusters may be included in the training.

[0329] The method can include constructing a contrastive self-supervised representation by learning the differences or similarities between the sub-images. Contrastive learning is a classification algorithm that classifies data based on similarity and dissimilarity. In the contrastive learning algorithm, the inner workings can be completed by learning an encoder f such that Score(f(x),f(x+)) >> score(f(x),f(x-)). Thus, x+ can be considered as a positive sample that is similar to x, and x- as a negative sample that is not similar to x. Score is a function that measures the similarity between two samples.

[0330] An embedding or representation is a mapping of discrete (categorical) variables to continuous vectors. In the context of neural networks, an embedding is a low-dimensional, learned continuous vector representation of a discrete variable. Embeddings in neural networks are useful because they reduce the dimensionality of categorical variables and provide a meaningful representation of the categories in the transformed space.

[0331] Neural network embeddings are trained low-dimensional representations of discrete data as continuous vectors. These embeddings overcome the limitations of traditional encoding methods and can be used for purposes such as finding nearest neighbors, inputting other models, and visualization.

[0332] The embeddings can be used to project expressions or embeddings onto human skin (without skin).

[0333] The material information extracted from the trained model may be associated with a label. If the label is a binary classifier that distinguishes between material and non-material, e.g., material to be classified and negative control material, or human skin / no human skin, the network classifies the subimage accordingly. If the label includes more metadata, such as additional annotations per mask, mask material, such as silicone, latex mask, or mask brand, the additional metadata may include external context, such as facial features, such as beard, glasses, hats, etc.

[0334] FIG. 9a illustrates diagrammatically and exemplarily one embodiment of a data-driven model 901, i.e. a (classification) network to be trained, for extracting one or more features, such as skin features.

[0335] The model 901 may be a NN or a CNN. An input 902, such as the above-mentioned patches, is provided to the model 901. The model 901 notably includes multiple layers 903 (e.g., an input layer, a hidden layer, an output layer, etc.) and outputs an output 904. In one embodiment, the output layer includes a classifier and / or a heatmap, etc.

[0336] The training of the model 901 is performed as described with respect to FIG. 8. Thus, the training dataset (i.e., input 902) may include one or more (manipulated) images, one or more representations, one or more clusters, and / or one or more labels. The training dataset preferably refers to historical data. The information extracted from the trained model (i.e., output 904) may relate to labels, i.e., the model 901 classifies according to one or more labels. The labels may be binary classifiers distinguishing between material and non-material, or human skin / no human skin, etc. The one or more labels may include metadata, as described with respect to FIG. 8. In an embodiment, the training may be based on supervised or self-supervised learning.

[0337] In general, the same rules are applied to the training of the model to output one or more labels that are later applied during the application of the model. In one embodiment, one or more (manipulated) images, one or more representations, and / or one or more clusters in the training dataset can be generated according to the principles and rules described above. In particular, any known training method for training a respective machine learning based algorithm, such as a neural network or a convolutional neural network, can be utilized. After the training process, the model 901 is adapted to identify one or more labels when provided with the described input data. The trained model can then be provided to a device, such as a mobile (computing) device. Since the model 901 can be trained on manipulated image data, for example image data after the described manipulations, the size of the model can be reduced. As a result, the storage and processing of the model on the mobile (computing) device can be significantly reduced.

[0338] FIG. 9b shows the training of an ensemble in a schematic and exemplary manner. A stack of ensembles may be trained for one or more clusters. That is, multiple classifier networks 901a, 901b, and 901c as described for the model 901 in FIG. 9a are trained for each cluster and / or for different sets of clusters. The multiple classifier networks 901a, 901b, and 901c may be identical to some extent or may be different from each other, as long as all of them can process equivalent input data and produce equivalent output. Although FIG. 9b shows three classifier networks 901a, 901b, and 901c in a schematic and exemplary manner, any positive number of classifier networks may be applied to the ensemble learning.

[0339] In general, NNs or CNNs may have high variance. In particular, the variance of such (NN or CNN) models can be reduced by using one or more ensembles and applying ensemble learning. That is, instead of a single model, multiple models can be trained and the respective predictions from these models can be combined. This solves the technical problem of reducing the variance of the predicted value and also results in a better predictive value. In general, ensemble learning involves training multiple models on the same dataset and combining the predictions of each model in some way to make a final result or prediction, which may be the most stable and best prediction.

[0340] In particular, a collection of models may be trained with the same configuration, with different initial random weights learned on the same dataset. Each model may then be used to make a prediction, and all such predictions may be averaged, or the prediction of the model with the highest confidence level may be selected and become the final output prediction of the ensemble. In various embodiments, variations in training data (i.e., the selection of data used to train each model in the ensemble may vary), ensemble models (i.e., the selection of each model used in the ensemble may vary), and / or combinations (the method of how results from ensemble members are combined may vary) may be applied.

[0341] For combination variations, the following embodiments are possible: based on the test data set, the confidence level of each trained model is compared and the model with the highest confidence level is selected, or the average of the predicted values ​​from several, possibly all, ensemble members is calculated. In an embodiment, an average weight may be applied, i.e. the predictions from each model may be weighted. This provides a weighted average ensemble. Furthermore, the weights may be optimized using a hold-out validation data set. Additionally or alternatively, stacking may be applied, i.e. a new model may be used to learn how to best combine the predictions from each ensemble member. This new model may be a linear model or a non-linear model that further considers certain input samples in addition to the predictions provided by each ensemble member. Additionally or alternatively, boosting may be applied, i.e. one or more new ensemble members may be added to correct the errors of the previous model. As a result, variations in the combinations may involve ensemble techniques of NNs or CNNs, including model averaging ensembles, weighted averaging ensembles, stacking ensembles, boosting ensembles, and / or model weighted averaging ensembles.

[0342] 10a and 10b show a schematic and exemplary illustration of a cluster-wise classification, i.e. clusters 1001, 1002, 1003 of sub-images and their classification via a binary classifier, e.g. whether their material vectors are human skin 1001a or no human skin 1001a.

[0343] FIG. 11 shows, diagrammatically and exemplarily, a situation in which a user 1101 is wearing a mask 1103 such that the user's face is partially obscured while being illuminated with a pattern 1102 .

[0344] 12a and 12b show the generation of heat map of material distribution and its use in face recognition process for verification. The flow chart includes step 1201 of capturing image data of flood image and / or pattern image, step 1202 of providing image data from camera, step 1203 of pre-processing image data, step 1204 of authentication processing based on flood image and / or pattern image, and step 1205 of generating heat map of material distribution. If face matching and authentication based on material features is "no" in step 1206, other unlocking mechanisms, such as user pin, can be applied in step 1207. If step 1206 is "yes", unlocking application, device or other operating parameters are generated in step 1208. With reference to FIG. 12b, if occlusion is detected in step 1209, full face recognition is performed in step 1210. On the other hand, if occlusion is detected in step 1209, partial face recognition is performed in step 1211.

[0345] State-of-the-art technologies, such as EP3623995A1 and US2019080149A1, require the use of complex algorithms and decisions to detect occlusion.

[0346] These drawbacks are overcome by the present invention, whereby material information is extracted and identification and / or authentication is performed based on the material information. Furthermore, the extraction of material information allows for more reliable and easier occlusion detection. In particular, a consistency check based on the extraction of material information can be applied.

[0347] Based on the material detection, the results of the partial images can be stitched back together into a complete image. In this way, the material distribution of the face can be determined and a corresponding heat map of the material distribution can be generated.

[0348] Based on such material distribution, possible occlusions, such as masks, may be determined. These may be associated with landmarks, such as eyes and mouths, from the floodlight image.

[0349] Based on such material distribution associated with landmarks, a rule set can be used to validate the face recognition process. For example, if a mask is associated with eye landmarks, the process can be stopped. If a mask is associated with a mouth landmark, the process can be validated. Similarly, features such as beards, glasses, hats, scarves, etc. can also be subject to rule-based consistency checks to prevent spoofing attacks and increase the security level.

[0350] Further, occlusion may be designated based on such material distribution associated with the landmarks. If the face is not occluded, full face recognition may be performed. If the face is occluded, partial face recognition may be performed. To perform such face recognition, an amount of occlusion may be determined from the material heat map. If it is below a threshold and verified via a rule set, partial face recognition may be performed. One or more occlusion scores from the material heat map may be generated for the face in the at least one image. Aligning the at least one material heat map with the landmarks may evaluate the amount of face occlusion. In particular, the analysis may include evaluating the amount of occlusion of the nose and mouth of the face in the at least one image. Further, the analysis may include evaluating the amount of face occlusion including determining a material type, such as a mask type. Based on such determination, the occlusion may be cut out from the at least one illuminated image for partial face recognition processing.

[0351] Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0352] For the processes and methods disclosed herein, the operations performed in the processes and methods may be performed in differing orders. Furthermore, the schematic operations are provided only as examples, and some of the operations may be optional, may be combined into fewer steps and operations, may be supplemented with additional operations, or may be extended to additional operations without departing from the essence of the disclosed embodiments.

[0353] In the claims, the word "comprising" does not exclude other elements or steps and the indefinite article "a" or "an" does not exclude a plurality.

[0354] A single unit or device may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0355] The procedures performed by one or more units or devices, such as receiving a pattern image, selecting a feature pattern, cropping the pattern image, determining the identity of an object, etc., can be performed by any other number of units or devices. These procedures can be implemented as program code means of a computer program and / or as dedicated hardware.

[0356] The computer program product may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, may be supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0357] The units described herein may be processing units that are part of a classical computing system. The processing unit may include a general-purpose processor, or may include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other specialized circuit. The memory may be a physical system memory, which may be volatile, non-volatile, or a combination of the two. The term "memory" may include a computer readable storage medium, such as a non-volatile mass storage. If the computing system is distributed, the processing and / or memory capabilities may also be distributed. The computing system may include multiple structures as "executable components." The term "executable components" is a structure well understood in the computing arts as a structure that may be software, hardware, or a combination thereof. For example, when implemented in software, one skilled in the art will understand that the structure of the executable components may include software objects, routines, methods, etc. that may be executed on the computing system. This may include both executable components in the heap of the computing system or executable components on a computer readable storage medium. The structure of the executable components may reside on a computer-readable medium and, when interpreted by one or more processors, e.g., processor threads, of a computing system, cause the computing system to perform functions. Such structure may be directly computer readable by a processor, e.g., as where the executable components were binary, or may be structured to be interpretable and / or compilable to generate such binary directly interpretable by a processor, e.g., whether in a single stage or multiple stages. In other examples, the structure may be hard-coded or hard-wired logic gates that are implemented exclusively or nearly exclusively in hardware, such as in a field programmable gate array (FPGA), application specific integrated circuit (ASIC), or other specialized circuitry.Thus, the term "executable component" is a term that describes a structure well understood by one of ordinary skill in the art of computing, whether implemented in software, hardware, or a combination thereof. Any embodiment herein is described with reference to acts performed by one or more processing units of a computing system. When such acts are implemented in software, one or more processors direct the operation of the computing system in response to executing the computer-executable instructions that make up the executable component. A computing system may also include communication channels that enable the computing system to communicate with other computing systems, for example, over a network. A "network" is defined as one or more data links that enable the transmission of electronic data between computing systems and / or modules and / or other electronic devices. When information is transferred or provided to a computing system over a network or other communications connection (e.g., either wired, wireless, or a combination of wired or wireless), the computing system properly regards the connection as a transmission medium. A transmission medium may include a network and / or data links that may be used to transmit desired program code means in the form of computer-executable instructions or data structures and that may be accessed by a general-purpose or special-purpose computing system or combination thereof. Although not all computing systems require a user interface, in some embodiments a computing system includes a user interface system for use in interfacing with a user. The user interface serves as an input or output mechanism to a user, for example via a display.

[0358] Those skilled in the art will appreciate that at least a portion of the present invention may be implemented in a networked computing environment having many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cell phones, PDAs, pagers, routers, switches, data centers, wearables such as glasses, etc. The present invention may also be implemented in a distributed system environment where both local and remote computing systems perform tasks that are linked, for example, through a network, by either wired data links, wireless data links, or a combination of wired and wireless data links. In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0359] Those skilled in the art will also appreciate that at least a portion of the present invention may be implemented in a cloud computing environment. A cloud computing environment may be distributed, but this is not required. If distributed, a cloud computing environment may be distributed internationally within an organization and / or may have components owned across multiple organizations. For purposes of this specification and the claims that follow, "cloud computing" is defined as a model that enables on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the many other advantages that such a model may derive when deployed. The computing system of the figure includes various components or functional blocks that may implement various embodiments disclosed herein, as described. The various components or functional blocks may be implemented on a local computing system or may be implemented on a distributed computing system that includes elements that reside in the cloud or that implement aspects of cloud computing. The various components or functional blocks may be implemented as software, hardware, or a combination of software and hardware. The computing system shown in the figure may have more or fewer components than those shown in the figure and may combine some of the components as appropriate.

[0360] Any reference signs in the claims shall not be construed as limiting the scope.

[0361] As used herein, "determining" also includes "initiating or causing a decision to be determined," "generating" also includes "initiating generation and / or causing generation," and "providing" also includes "initiating or causing to be initiated determining, generating, selecting, transmitting, and / or receiving." "Initiating or causing to be performed an action" includes any processing signal that triggers a computing node or device to perform the respective action.

[0362] In the claims and the description, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0363] The disclosures and embodiments described herein relate to the above methods, systems, devices, computer program elements, and vice versa. Advantageously, advantages provided by any of the embodiments and examples apply equally to all other embodiments and examples, and vice versa.

Claims

1. 1. A method for authorizing an object of a device to perform at least one operation on the device that requires authentication, the method comprising the steps of: receiving, from a user, a request to unlock said device from a user interface associated with the device; - triggering illumination of the object with patterned infrared illumination in response to receiving the unlock request; - triggering the capture of at least one image of said object using a camera located on said device while said object is illuminated with patterned infrared illumination; - extracting material data from at least one image; - allowing or denying said object to perform at least one operation on said device that requires authentication, based on said material data; A method comprising:

2. 2. The method of claim 1, wherein capturing the at least one image comprises capturing at least one image of the object using a camera disposed on the device while the object is illuminated with patterned infrared illumination.

3. 3. The method of claim 1 or 2, wherein the authentication uses a facial recognition authentication process operating on the image and / or the extracted material data.

4. The method according to claim 1 or 2, wherein the authentication process is verified based on the extracted material data.

5. The method of claim 4 , wherein verifying based on the extracted materials data comprises determining whether the extracted materials data corresponds to desired materials data.

6. 3. The method of claim 1 or 2, wherein the authentication process or verification thereof comprises generating at least one feature vector from the material data and matching the material feature vector with an associated reference template vector for the material.

7. The method of claim 6 , wherein the material feature vector includes a representation of luminance or translucency contained in the material data.

8. 3. The method according to claim 1 or 2, wherein the patterned infrared radiation is a regular and / or periodic pattern having pattern features, in particular at least 2, preferably at least 6, more preferably at least 20 pattern features per surface area at a distance of 5 to 200 cm, preferably 30 to 60 cm, more preferably 15 to 80 cm.

9. 3. The method according to claim 1 or 2, wherein extracting the material data comprises performing a beam profile analysis by applying an image filter, in particular a material dependent image filter, to pattern features of an image and / or by providing the image to a data-driven model, the data-driven model being parameterized according to a training data set comprising the image and material information.

10. The method of claim 1 or 2, wherein an operation requiring authentication includes unlocking the device.

11. 3. The method of claim 1 or 2, wherein extracting the materials data comprises manipulating an image to generate a partial image and extracting materials data of the object from the partial image by providing the partial image to a data-driven model, the data-driven model being parameterized according to a training data set comprising the partial image and the materials data.

12. Manipulating the image includes: suppressing texture information from objects located in the region of interest, in particular the user's face, and / or - segmenting said image and / or data of said image, and / or - Cropping the image into a sub-image, and / or - determining a bounding box of a region of interest in said image and cropping said image to extract said region of interest; The method of claim 1 or 2, comprising:

13. The method of claim 1 or 2, wherein the material information comprises a material class and / or type.

14. 1. A device for authorizing a user of a device to perform at least one operation on the device that requires authentication, the device comprising the steps of: receiving, from a user, a request to unlock said device from a user interface associated with the device; - triggering illumination of the user with patterned infrared illumination in response to receiving the unlock request; - triggering the capture of at least one image of the user using a camera disposed on the device while the user is illuminated with patterned infrared illumination; - extracting material data from said image; - authorizing or denying said user to perform at least one operation on said device that requires authentication based on said material data; 16. A device comprising: a processor configured to execute

15. 3. A computer program for authorizing a device user to perform at least one operation on a device requiring authentication, the computer program comprising instructions for causing a computer to perform the steps of the method according to claim 1 or 2.

Images