Improved biometric authentication method

Abstract

The invention relates to the field of biometric authentication, in particular to the field of anti-spoofing. The present disclosure relates to methods, apparatus, devices, material information and computer elements for generating biometric authentication information.

Description

Technical Field

[0001] This invention relates to the field of biometric authentication, and more specifically to the field of anti-spoofing. This disclosure relates to methods, apparatus, devices, material information, and computer elements for generating biometric authentication information. Background Technology

[0002] Biometric authentication processes require security measures to mitigate the risk of spoofing. This is particularly important when using mobile phones for payments or other security-sensitive operations. Such algorithms rely on image processing and neural networks for image detection. Neural networks depend on complex convolutional operations trained on labeled image data. For complex tasks such as facial authentication, the number of features and their impact on the process can be difficult to assess in terms of accuracy. Therefore, improvements to the performance of neural networks are needed. Summary of the Invention

[0003] In one aspect, a method for generating biometric authentication information is disclosed, the method comprising:

[0004] - Provide one or more patterned light images, wherein the patterned light images(s) include an image of a user under illumination by at least one infrared pattern illuminator of the device;

[0005] - Generate one or more material properties from the one or more patterned light images;

[0006] - Generate biometric authentication information by providing one or more generated material properties to at least one layer of a neural network, and / or provide one or more generated material properties to at least one layer of a neural network, for example, trained or configured to generate biometric authentication information.

[0007] On the other hand, an apparatus for generating biometric authentication information is disclosed, the apparatus comprising:

[0008] - An image providing interface configured to provide one or more patterned light images, wherein the patterned light images include an image of a user under illumination by at least one infrared pattern illuminator of the device;

[0009] - A material generator configured to generate one or more material properties from the one or more patterned light images;

[0010] - A material property provider configured to: provide one or more generated material properties to at least one layer of a neural network configured to generate biometric authentication information; and / or generate biometric authentication information by providing one or more generated material properties to at least one layer of a neural network, for example, trained or configured to generate biometric authentication information.

[0011] On the other hand, a method is disclosed for an authorized device user to perform at least one operation requiring authentication, the method comprising:

[0012] - In response to receiving an unlock request, trigger the use of a camera located on the device to capture one or more patterned light images of the user, wherein the one or more patterned light images include images of the user illuminated by at least one infrared pattern illuminator located on or belonging to the device, optionally wherein trigger the use of a camera located on the device to capture one or more floodlight images of the user;

[0013] - Generate one or more material properties from the one or more patterned light images;

[0014] - Generate biometric authentication information by providing one or more generated material properties to at least one layer of a neural network, for example, trained or configured to generate biometric authentication information, and / or provide one or more generated material properties to at least one layer of a neural network configured to generate biometric authentication information, optionally wherein the one or more floodlight images are provided to the neural network used to generate biometric authentication information.

[0015] - Authorize the user to perform at least one operation that requires authentication based on the generated biometric authentication information.

[0016] On the other hand, an apparatus is disclosed for an authorized device user to perform at least one operation requiring authentication, the apparatus comprising:

[0017] - A trigger interface, such as a software interface, hardware interface, processor interface, communication interface, device-to-device interface, or human-machine interface, is configured to trigger the use of a camera located on the device to capture one or more patterned light images of the user in response to receiving an unlock request, wherein the one or more patterned light images include images of the user illuminated by at least one infrared pattern illuminator located on or belonging to the device, optionally wherein the trigger uses a camera located on the device to capture one or more floodlight images of the user;

[0018] - A material property generator, such as a processor, image processing unit, CPU, ISP and / or neural network processing unit, which is configured to generate one or more material properties from the one or more patterned light images;

[0019] - An authentication information generator, such as a processor, image processing unit, CPU, ISP, and / or neural network processing unit, is configured to: generate biometric authentication information by providing one or more generated material properties to at least one layer of a neural network, for example, trained or configured to generate biometric authentication information; and / or provide one or more generated material properties to at least one layer of a neural network configured to generate biometric authentication information, optionally wherein the one or more floodlight images are provided to the neural network used to generate biometric authentication information;

[0020] - An authorization unit, such as a processor, image processing unit, CPU, ISP and / or neural network processing unit, is configured to authorize the user to perform at least one operation requiring authentication based on the generated biometric authentication information.

[0021] In another aspect, an apparatus is disclosed that is configured to generate biometric authentication information according to the method disclosed herein or by means of the apparatus disclosed herein. In yet another aspect, an apparatus is disclosed that is configured to perform the method disclosed herein or include the apparatus disclosed herein.

[0022] In another aspect, a material property extracted from at least one patterned light image for generating biometric authentication information according to the method or apparatus disclosed herein is disclosed. In yet another aspect, the use of the material property extracted from at least one patterned light image for generating biometric authentication information according to the method or apparatus disclosed herein is disclosed.

[0023] On the other hand, this disclosure relates to a computer element having instructions, such as a computer program or a computer-readable medium, which, when executed on one or more processors (such as multiple image processing units and / or multiple neural network processing units, or multiple devices), are configured to perform steps of the methods disclosed herein or are configured to be executed by the means disclosed herein.

[0024] Any disclosures, embodiments, and examples described herein relate to the methods, apparatus, devices, material information, and computer elements listed above and below. Advantageously, the benefits provided by any embodiment and example also apply to all other embodiments and examples. Example

[0025] Reliable authentication and anti-spoofing protection are crucial for biometric authentication. Neural networks are commonly used in this process, but determining the attributes of captured images can be challenging due to the diversity of biometric scanning scenarios, such as facial, fingertip, or retinal scans. Particularly in facial authentication, changes in appearance, such as wearing masks, glasses, or wearing a beard, can slow down the network or reduce accuracy. Providing the neural network with further information for authentication allows for a more reliable process. Specifically, improving the authentication process can be achieved by providing material information or attributes extracted from patterned light images to at least one layer of a neural network configured to generate biometric authentication information. Extracting material information from patterned light images allows for the identification of materials present in floodlight images for authentication. Furthermore, improving the authentication process can also be achieved by providing biometric attributes (such as the pose of the biometric object) extracted from floodlight images to at least one layer of a neural network configured to generate biometric authentication information. This assists the authentication process by using a more “intelligent” neural network and focusing the authentication emphasis on biometric features (such as the eyes or mouth on the face) rather than material features (such as a mask covering the mouth or glasses in front of the eyes).

[0026] The following examples will outline embodiments of this disclosure. It should be understood that this disclosure is not limited to the embodiments and / or examples described.

[0027] These methods and / or devices can be used to authorize a user of a device to perform at least one operation requiring authentication on the device. The device can be any device configured to authorize a user to perform at least one operation on, associated with, and / or triggered by the device requiring authentication. The device can be a mobile phone, tablet computer, automobile, autonomous vehicle, in-vehicle entertainment system, door, or any other electronic system or component requiring authentication, or a part thereof. A user can register for one or more devices and / or operations. The device can store such user registrations for use of the device and / or specific operations running on, associated with, and / or triggered by the device.

[0028] User authentication may include biometric authentication. Biometric authentication may be based on biometric authentication information. Biometric authentication information may involve the user's biometric features and / or template features. Biometric authentication information may involve a matching score between at least one biometric vector generated from an image (e.g., a floodlight image) captured when an event triggers the user to perform at least one action on the authorized device, and at least one template vector stored for the user authorized to perform at least one action. Biometric features and / or template features may be extracted from the user's(s) images(e.g., multiple floodlight images). The biometric vector and template vector may be obtained from the biometric features and template features, respectively.

[0029] Biometrics may include a face, a portion of the face, a retina, a fingerprint, or any other biometric feature that can be detected via image processing or from an image taken by the user. Authentication may involve a process of verifying the identity of a user requesting to perform at least one action. Authentication may involve a process of verifying that the user is a registered user of the device and / or the at least one action. Authentication may involve a process of verifying the user when or before performing at least one action. Authentication may involve a process of verifying the user and performing at least one action based on such verification.

[0030] Authorizing a user to perform at least one action on a device requiring authentication, that is associated with, and / or triggered by, that device may involve approving the user requesting to perform the action. Authorization may be tied to a user and / or an action. For example, a device may register more than one user. A user may register for one or more actions. For authorization, a user's identity may be verified via biometric authentication. Further, to authorize a user to perform a specific action requested by the user, the user's registration for that specific action may be checked. Registration may include generating a template vector stored for a user authorized to perform at least one action. Registration may include generating a template vector stored associated with the user and at least one authorized action.

[0031] Manipulating multiple patterned light images may include randomizing image information of a region of interest (particularly a region of interest including at least one biometric feature) or image information included in that region of interest. Manipulating multiple patterned light images may include randomizing image information of a region of interest (particularly a region of interest including at least one biometric feature) or image information included in that region of interest while maintaining at least a portion of the multiple patterned features intact. Manipulating multiple patterned light images may include randomizing image information of at least one biometric feature or image information including that at least one biometric feature while maintaining at least a portion of the multiple patterned features intact. Manipulating multiple patterned light images may include generating one or more image cuts of portions of image information including the region of interest, or can be implemented by generating such one or more image cuts, while maintaining at least a portion of the multiple patterned features intact. Manipulating one or more patterned light images to suppress background texture information in the region of interest may include manipulating the patterned light images by randomizing the background texture information. Manipulating multiple patterned light images may include generating one or more image cuts of portions of image information including the region of interest, or can be implemented by generating such one or more image cuts. The region of interest may be a region including the user's biometric information. The region of interest may include at least a portion of the user's face or a portion thereof. Manipulating the patterned light image(s) to suppress background texture information in the region of interest and / or by randomizing the image information of the region of interest may include generating one or more image cuts that include a portion of the image information of the region of interest, or may be implemented by generating such one or more image cuts. This allows the data-driven model to focus on patterned reflections, and thus on the material signature embedded in the patterned light image signal. Manipulating one or more patterned light images to suppress background texture information in the region of interest may include performing image enhancement techniques on the patterned light image, or may enhance the image. Enhancement or manipulation may include at least one of scaling, cropping, rotating, blurring, distorting, shearing, resizing, folding, changing contrast, changing brightness, adding noise, multiplying by at least a portion of pixel values, filtering, adjusting color, applying convolution, imprinting, sharpening, flipping, averaging pixel values, etc. Enhancement or manipulation may include cropping the patterned light image(s) to generate a partial image.

[0032] Material properties can relate to the patterned light image or the properties of manipulating the patterned light image. Material properties can relate to properties obtainable from the patterned light image. Material properties can relate to or include material information extracted from the patterned image(s). Material properties or information can relate to the type of material reflecting the patterned light. Material properties can relate to at least one material class, such as organic, inorganic, silicon, plastic, etc. Material properties can be associated with the user's skin. Material properties can be associated with a type of organic or inorganic material. Material properties can relate to the manipulated patterned light image. Material properties can relate to the manipulation techniques of the patterned light image.

[0033] A neural network configured to generate biometric authentication information may include a network architecture having at least one input layer, one or more hidden layers, and at least one output layer. The neural network may be based on sequential and / or parallel neural network architectures. One or more network architectures may include an input layer, one or more hidden layers, and an output layer, which may be connected to execute sequentially and / or in parallel. At least one layer of the neural network may include either an input layer or channel, one or more hidden layers or channels, and / or an output layer or channel. The neural network may include parameters trained on a training dataset, such as kernels, weights, biases, functional relationships, operators, constraints, etc. The neural network configured to generate biometric authentication information may receive one or more images (e.g., multiple floodlight images) at the input layer and generate biometric vectors based on which biometric authentication information can be determined. Additionally, the neural network configured to generate biometric authentication information can receive one or more material properties, such as those extracted from multiple patterned light images, and / or one or more biometric attributes, such as those extracted from multiple floodlight images (e.g., biometric attributes related to the location of multiple biometric objects or features in the images), at the input, hidden, and / or output layers of the neural network, and generate biometric vectors based on which biometric authentication information can be determined. One or more material properties and / or one or more biometric attributes can be provided to at least one layer of the neural network configured to generate biometric authentication information in a combined and / or sequential manner. One or more material properties and / or one or more biometric attributes can be weighted and / or provided to at least one layer of the neural network configured to generate biometric authentication information in a combined and / or sequential manner.

[0034] In one embodiment, one or more material properties from the one or more patterned light images are generated by manipulating the patterned light images(s) and extracting one or more material properties from the patterned light images(s). Generating one or more material properties may include manipulating the patterned light images(s) and generating at least one material feature vector. The at least one material feature vector may be generated by feeding the manipulated patterned light images(s)(s) to at least one neural network (e.g., a convolutional neural network), which is trained on a training dataset consisting of the manipulated patterned light images(s)(s) and corresponding one or more material properties associated with the at least one material feature vector. The at least one material feature vector may be generated by feeding the manipulated patterned light images(s)(s) to at least one neural network (e.g., a convolutional neural network) trained to generate at least one material feature vector associated with one or more material properties.

[0035] In another embodiment, one or more patterned light images are manipulated to suppress texture information in the region of interest. One or more material properties can be generated from one or more patterned light images. Manipulating the patterned light images may include generating partial images of at least one or more portions having one or more patterned features. Using one or more partial images allows for a reduction in the number of channels on the input layer of a neural network used for material extraction or authentication, and also a reduction in the number of channels in the network architecture used for analyzing the image in the hidden layers. This reduces the scale in terms of the number of parameters (neurons) per layer and / or the number of layers (network depth). By using partial images to reduce the number of parameters and the scale of the neural network, the performance of the neural network operation can be improved. Manipulating the patterned light images may include changing the distance between at least two patterned features, representing an object as a two-dimensional plane, deleting at least a portion of an image, rearranging at least a portion of an image, generating partial images, and / or any combination thereof. The manipulation of the patterned light images may be performed by one or more central processing units (CPUs). The patterned light images can be manipulated by partitioning the patterned images. Generating multiple partial images from multiple patterned light images may include cropping the multiple patterned light images into at least one or more portions that include one or more pattern features and suppressing background texture information.

[0036] Manipulating (multiple) patterned light images can include one or more image enhancement techniques.

[0037] Manipulating multiple patterned light images can include at least one of scaling, cropping, rotating, blurring, distorting, shearing, resizing, folding, changing contrast, changing brightness, adding noise, multiplying by at least a portion of pixel values, filtering, adjusting color, applying convolution, imprinting, sharpening, flipping, and averaging pixel values. Manipulation of multiple patterned light images can be performed to change and / or remove at least a portion of background texture information. For example, cropping a patterned light image can change and / or remove at least a portion of background texture information. Manipulations such as cropping a patterned light image can, for example, result in changing the distance between at least two background texture features and / or changing at least one background texture feature. Therefore, performing manipulations using at least one image enhancement technique can result in changing and / or removing at least a portion of the texture information of the patterned light image.

[0038] In another embodiment, manipulating the patterned light images includes generating a segmented image including a region of interest for each patterned light image, and generating manipulated partial images, such as multiple partial images, for each segmented image having at least one or more portions of one or more patterned features. Generating the segmented images may include determining the bounding box of the region of interest in the image and cropping the image to extract the region of interest.

[0039] In another embodiment, the manipulated patterned light images(s) are associated with pixel location information related to a region of interest. The pixel location information may refer to a sub-region of the region of interest. The pixel location information may refer to a sub-region of the region of interest. The pixel location information may refer to a sub-region associated with one or more biometric features or biometric authentication information. Manipulated, such as, multiple partial images, may be associated with pixel location information representing the location of a partial image within the manipulated, such as, multiple, patterned light images(s). Based on the extracted material information and location information of each manipulated, such as partial image(s), a material heatmap representing the material distribution in the region of interest (e.g., a user's face) can be generated.

[0040] In another embodiment, manipulating the patterned light images includes generating a segmented image including a region of interest for each patterned light image, and generating manipulated, such as multiple, partial images for each segmented image, having at least one or more portions of one or more patterned features.

[0041] In another embodiment, generating one or more material properties includes: providing one or more manipulated patterned light images to at least one data-driven model trained to extract one or more material properties from the manipulated patterned light images(s); and / or extracting one or more material properties from the manipulated patterned light images(s). The data-driven model is parameterized based on a training dataset comprising the manipulated patterned light images and the associated one or more material properties.

[0042] A data-driven model trained to extract material information from manipulated patterned light images can include at least one neural network. The neural network can include a network architecture having at least one input layer, one or more hidden layers, and at least one output layer. The neural network can be based on sequential and / or parallel neural networks. One or more network architectures including an input layer, one or more hidden layers, and an output layer can be connected to execute sequentially and / or in parallel. The neural network can include parameters trained on a training dataset, such as kernels, weights, biases, constraints, etc. The data-driven model can be a neural network. A neural network configured to be trained to extract material information from a data-driven model can receive one or more images (e.g., manipulated images generated from (multiple) patterned light images) at the input layer and generate material information.

[0043] In another embodiment, a data-driven model that generates one or more material properties generates a binary skin classifier that distinguishes between skin and non-skin. The data-driven model that generates one or more material properties can generate a material classifier that distinguishes between one or more material types. Generating one or more material properties may include generating at least one material feature vector (or representation) from one or more manipulated, for example, partial images and / or matching that material feature vector (or representation) with one or more reference template vectors of different material types. At least one material feature vector (or representation) can be generated by feeding a manipulated, for example, partial patterned light image to a convolutional neural network trained on a training dataset consisting of the manipulated, for example, partial patterned light image and one or more associated material properties, thereby generating at least one material feature vector (e.g., generating an output and transforming that output into a material feature vector). The material feature vector (or representation) may include representations of brightness and / or translucency included in the patterned light image.

[0044] In another embodiment, one or more material properties relate to at least one characteristic of the manipulation of at least one material type and / or (or multiple) patterned light images (e.g., the generation of one or more manipulated partial images). One or more material properties may be or include characteristics of the manipulation of material type and / or (or multiple) patterned light images (e.g., the generation of one or more partial images). In another embodiment, one or more material properties relate to or include, but may be limited to, or may include at least one of the following: at least one feature vector of at least one (or multiple) manipulated patterned light images, the size of (or multiple) manipulated patterned light images, the number of light spot patterns of (or multiple) manipulated patterned light images, facial regions of (or multiple) manipulated patterned light images, texture, skin type, distance from the sensor detecting the patterned light image to the object reflecting the patterned light image, translucency of (or multiple) manipulated patterned light images, and / or luminosity of (or multiple) manipulated patterned light images. One or more material properties may relate to or include skin, organic materials, inorganic materials, organic material classes, and / or inorganic material classes. In another embodiment, one or more material properties relate to at least one manipulated, such as partial image, the size of the manipulated, such as partial image, the number of light spot patterns in the manipulated, such as partial image, the facial region of the manipulated, such as partial image, texture, skin type, the distance from the sensor that detects the patterned light image to the object reflecting the patterned light image, and / or the luminosity of the manipulated, such as partial image.

[0045] In another embodiment, one or more floodlight images are provided. The floodlight images may include an image of a user illuminated by at least one infrared floodlight illuminator on the device. The one or more floodlight images may be provided as input to a neural network configured to generate biometric authentication information. For example, in response to receiving an unlock request, a camera located on the device may be triggered to capture one or more floodlight images and / or (multiple) patterned light images of the user. The one or more floodlight images may include images captured when the user in the image is illuminated by at least one infrared patterned light illuminator located on the device. The floodlight images may include images captured when the user in the image is illuminated by at least one infrared floodlight illuminator located on the device. Authentication of the user may be based on the one or more floodlight images. The one or more floodlight images may include background texture information from the patterned light images. While biometric authentication uses this information from the one or more floodlight images, material detection is based on patterned light reflection and the background texture information is suppressed through manipulation.

[0046] The device may include at least one image processing unit and at least one neural network processing unit. The at least one image processing unit may include one or more central processing units (CPUs) and / or one or more image signal processors (ISPs). The image signal processors may include a dedicated processor architecture configured to perform image processing operations. The CPU may include a general-purpose processor architecture configured to perform operations. The CPU may include multiple cores, for example, 4 to 64 cores, such as 6, 8, or 16 cores. The cores may be identical or different from each other; for example, some cores may have higher data throughput while others may have lower power consumption. Each core may include an algorithm and logic unit (ALU), a control unit, registers, and cache memory. Cache memory may be shared between different cores, particularly a Level 3 cache. The ALU can perform a wide range of data manipulation operations, including: basic arithmetic operations such as addition, subtraction, multiplication, and division; bitwise operations such as AND, NAND, OR, NOT, and XOR; logical operations such as comparing two values ​​to determine if one is greater than, equal to, or less than the other; bit-level operations such as bit shifting and rotation, setting and clearing individual bits; and basic trigonometric functions such as sine, cosine, and tangent. The ALU can perform one operation at a time. The control unit can be configured to perform tasks such as fetching instructions from memory, instruction decoding, instruction execution coordination (including sending control signals to other components such as the ALU and registers), data movement between different CPU components and memory, control flow of instructions and data within the CPU (especially for out-of-order (OOO) execution), timing and synchronization, and exception handling.

[0047] The CPU can have a clock frequency of 2.5 GHz to 5 GHz, that is, the frequency at which the CPU's clock generator generates pulses to synchronize its components. The CPU's context switching latency (i.e., the time spent saving the current state of a process and loading the state of another process) can be from 50 ns to 500 ns, such as 100 ns or 125 ns. The CPU can have a 64-bit bus width, that is, the number of bits that can be transferred simultaneously between the processor and its memory. The CPU can have a memory bandwidth of 50 GB / s to 250 GB / s (e.g., 100 GB / s), that is, the rate at which data can be read from or written to system memory.

[0048] At least one neural network processing unit may include one or more graphics processing units (GPUs). A GPU may include multiple cores, for example, 100 to 10,000 cores, such as 1536 cores, 2304 cores, 4608 cores, or 8704 cores. Each core may include an arithmetic logic unit (ALU), which performs fewer operations than a CPU. The ALU can be configured to perform operations required to process the neural network, such as floating-point addition, multiplication, or exponentiation. The GPU may be optimized for matrix multiplication or addition and convolution, where these operations can be performed in parallel. Such a GPU may also be referred to as a neural engine or a tensor processing unit (TPU).

[0049] The GPU can have a clock frequency of 0.5 GHz to 2 GHz (e.g., 1.0 GHz or 1.5 GHz). The context switching latency of the GPU or TPU can be 1 µs to 200 µs, e.g., 1.5 µs or 17 µs. The GPU or TPU can have a bus width of 128 bits to 1024 bits (e.g., 128 bits, 192 bits, 256 bits, 320 bits, 512 bits, or 1024 bits). The GPU can have a memory bandwidth of 300 GB / s to 1000 GB / s (e.g., 400 GB / s or 800 GB / s). At least one neural network processing unit can be configured to perform at least convolution, pooling, subsampling, and / or flattening operations associated with neural networks (e.g., convolutional neural networks (CNNs)). Such operations, or a portion thereof, can be part of its instruction set architecture (ISA). Such operations, or a portion thereof, can be implemented in hardware, for example, in the GPU's ALU.

[0050] At least one image processing unit may include one or more central processing units (CPUs), wherein multiple patterned light images are provided to the CPU, which is configured to manipulate the multiple patterned light images and / or be configured to extract material information based on processing of at least one data-driven model provided with the multiple manipulated patterned light images. The CPU may be configured to manipulate the one or more patterned light images, for example, by generating multiple partial images. The CPU may be configured to extract material information by executing a data-driven model based on the generated manipulated, for example, multiple partial images. The CPU may generate a task list for executing at least one data-driven model by at least one neural network processing unit. At least one neural network processing unit may be configured to execute the data-driven model and extract and / or provide material information based on the multiple manipulated patterned light images provided to the at least one neural network processing unit.

[0051] One or more floodlight images can be provided to at least one image processing unit. The at least one image processing unit may include one or more image signal processors (ISPs). One or more floodlight images can be provided to one or more ISPs to prepare for biometric authentication. One or more floodlight images and one or more patterned light images can be captured. One or more floodlight images can be provided to at least one image processing unit, particularly one or more ISPs, for example, to prepare one or more floodlight images to generate biometric authentication information. One or more patterned light images can be provided to at least one image processing unit, particularly one or more CPUs, for example, to manipulate one or more patterned light images to generate material information. Because of the lightweight performance of image verification based on material information as proposed herein for authentication, the CPUs can be used to improve the performance of the authentication process and reduce the bandwidth of the ISPs used for biometric authentication based on one or more floodlight images.

[0052] Floodlight and patterned light images can be captured simultaneously. The floodlight and patterned light images can include images captured when a user in an image is illuminated by at least one infrared floodlight illuminator and at least one infrared patterned light illuminator located on the device. In this embodiment, the floodlight and patterned light images can overlap. The floodlight and patterned light images can be provided as components of at least one image processing unit, respectively.

[0053] The prepared floodlight images(s) can be provided to at least one neural network processing unit configured to execute at least one data-driven model, which is trained to generate biometric authentication information from the floodlight images(s). At least one image processing unit (e.g., one or more central processing units (CPUs) and / or one or more image signal processors (ISPs)) can generate a task list for the at least one neural network processing unit to execute the at least one data-driven model, in order to provide biometric authentication information based on the provided floodlight images(s) provided to the neural network processing unit. The task list can be provided to a task manager for the at least one neural network processing unit to execute the at least one data-driven model, in order to provide biometric authentication information based on the provided floodlight images(s). Biometric authentication information can be provided to at least one image processing unit.

[0054] In another embodiment, one or more biometric attributes are generated from the floodlight images(s). Generating the one or more biometric attributes may include generating at least one biometric vector. The at least one biometric vector can be generated by providing the floodlight images(s) to a data-driven model (e.g., a neural network, such as a convolutional neural network) trained on a training dataset consisting of the floodlight images(s) and corresponding one or more biometric attributes associated with the at least one biometric vector. The at least one biometric vector can be generated by providing the floodlight images(s) to a data-driven model (e.g., a neural network, such as a convolutional neural network) trained to generate biometric attributes. The data-driven model (e.g., a neural network) trained to generate biometric attributes may differ from a neural network trained to generate biometric authentication information. One or more biometric attributes and / or one or more material properties may be provided to at least one layer of a neural network configured to generate biometric authentication information. One or more floodlight images may be provided to at least one image processing unit, particularly one or more central processing units (CPUs), for example, for generating biometric attributes. One or more floodlight images may be provided to at least one neural network processing unit for executing at least one data-driven model to generate biometric attributes.

[0055] In another embodiment, providing one or more material properties and / or biometric properties to at least one layer of a neural network includes applying biases, weights, functional relationships, and / or operators to at least one of the input channels, hidden layers, and / or output channels of the neural network. The generated one or more material properties and / or biometric properties may be weighted, or may include weights for providing to at least one layer of the neural network. Providing the generated one or more material properties to at least one layer of a neural network configured to generate biometric authentication information may include applying at least one of biases, weights, functional relationships, constraints, and / or operators to at least one layer of the neural network (particularly at least one of the input channels, hidden layers, and / or output channels of the neural network). Providing the generated one or more material properties to at least one layer of a neural network configured to generate biometric authentication information may include applying at least one of biases, weights, functional relationships, constraints, and / or operators to at least one layer of the neural network (particularly at least one of the input channels, hidden layers, and / or output channels of the neural network). At least one of the biases, weights, functional relationships, constraints, and / or operators applied to the input channels, neural network layers, and / or output channels of at least one layer of a neural network may be part of the training process or parameterized during the training of the neural network configured to generate biometric authentication information. At least one of the biases, weights, functional relationships, constraints, and / or operators applied to at least one layer of a neural network (particularly at least one of the input channels, hidden layers, and / or output channels of the neural network) may be part of the training process or parameterized during the training of the neural network configured to generate biometric authentication information. One or more material properties generated may be weighted when provided to at least one layer of the neural network configured to generate biometric authentication information. The weighting of one or more material properties generated may be part of the training process or parameterized during the training of the neural network configured to generate biometric authentication information. One or more biometric properties generated may be weighted when provided to at least one layer of the neural network configured to generate biometric authentication information. The weighting of one or more biometric properties generated may be part of the training process or parameterized during the training of the neural network configured to generate biometric authentication information.

[0056] In another embodiment, the neural network configured to generate biometric authentication information generates at least one biometric vector based on a floodlight image of the user of the device. The at least one biometric vector can be compared with at least one template vector associated with the user to be authenticated. In another embodiment, the neural network configured to generate biometric authentication includes convolutional layers with biases, weights, functional relationships, and / or operators.

[0057] In another embodiment, one or more biometric attributes relate to the positional attributes of biometric objects and / or biometrics(s) on an image (e.g., multiple floodlight images or patterned light images). At least one biometric vector may relate to multiple biometrics(s) suitable for authentication on the floodlight image. In addition to material properties, biometric attributes may also relate to other attributes that may be processed prior to processing biometric authentication information and may be provided to at least one layer of a neural network configured or trained to generate biometric authentication information. Biometric attributes may relate to attributes used to generate biometric authentication information by providing biometric attributes to at least one neural network. One or more biometric attributes may include facial attributes, such as, but not limited to, gaze, landmarks, and pose (including, for example, pitch, yaw, and / or roll). The biometric object may be a face, include a face, or relate to a face. One or more biometric attributes may include or relate to at least one facial positional attribute, which includes, but is not limited to, gaze, landmarks, and / or pose (e.g., pitch, yaw, and / or roll). One or more biometric attributes may be provided to at least one layer of a neural network configured to generate biometric authentication information. Biometric attributes can relate to properties of a user's image (e.g., multiple floodlight or patterned light images). For example, for a face, biometrics can include gaze, facial pose-related features (such as pitch, yaw, or roll), bounding boxes, etc. Biometric attribute generation can be performed by one or more central processing units (CPUs) and / or neural network processing units. Material property generation can be performed by one or more CPUs and / or neural network processing units.

[0058] Operations requiring authentication may include unlocking the device and / or one or more components of the device and / or triggering or performing one or more functions or operations by the device. Functions or operations may involve, but are not limited to, electronic payment operations, password manager access and / or use, in-app purchase functions, etc.

[0059] The region of interest may include at least one biometric feature. The region of interest may include, or be, the user's face or one or more parts of the face. Patterned infrared illumination may include periodic or regular patterns, preferably hexagonal dot patterns or pseudo-random structured light with 500 to 5000, 800 to 2500, or 1000 to 2000 dots per image. The methods and apparatus disclosed herein may further include the step of generating a depth map from the patterned light image. The depth map can provide a depth reconstruction of the patterned light image. For this reconstruction, triangulation can be used. The patterned light image used to generate material information may differ from the patterned light image used to generate biometric authentication information. The infrared light pattern may include at least one regular and / or constant and / or periodic pattern, such as a triangular pattern, rectangular pattern, hexagonal pattern, or a pattern including further convex tessellation. For example, the infrared light pattern may be a hexagonal pattern, preferably a hexagonal infrared light pattern. The illumination pattern may include multiple rows on which illumination features are arranged at equidistant positions at a distance d. These rows may be orthogonal to the epipolar lines. The distance between rows may be constant. Different offsets can be applied to each row in the same direction. This offset may cause a shift in the illumination characteristics of the row. The offset δ can be δ = a / b, where a and b are positive integers, such that the illumination pattern is a periodic pattern. For example, δ can be 1 / 3 or 2 / 5. Using a periodic pattern with said offset allows for the differentiation of artifacts from the available signal. The light pattern may include fewer than 4000 spots, for example, fewer than 3000 spots, or fewer than 2000 spots, or fewer than 1500 spots, or fewer than 1000 spots. The light pattern may include patterned coherent infrared light with fewer than 4000 spots, or fewer than 3000 spots, or fewer than 2000 spots, or fewer than 1500 spots, or fewer than 1000 spots. Attached Figure Description

[0060] The following description, with reference to the accompanying drawings, will further illustrate this disclosure:

[0061] Figure 1 The device showcased included a processor, memory, camera, and display.

[0062] Figure 2 An example of the camera is shown.

[0063] Figure 3 An example of a processing module associated with (multiple) image sensors and configured to process image signals is shown.

[0064] Figure 4 Another example is shown of a processing module associated with (multiple) image sensors and configured to process image signals.

[0065] Figure 5 A flowchart illustrating an embodiment of a biometric authentication process that uses facial features and includes skin detection is provided.

[0066] Figure 6 The image shows a patterned light image and the signal generated from the patterned light image.

[0067] Figure 7 An example flowchart is shown for a method of preprocessing patterned light images.

[0068] Figures 8a to 8c An example of a method for generating a partial image from a pattern image is shown.

[0069] Figure 9 An example of a method for training a data-driven model configured to detect materials based on partial images is shown.

[0070] Figure 10 An example block diagram is shown, featuring an image processing unit configured to perform image preprocessing and a neural network processing unit configured to perform material detection.

[0071] Figure 11 An example block diagram is shown, featuring an image processing unit configured to perform image preprocessing and a neural network processing unit configured to perform material and biometric authentication detection.

[0072] Figure 12 An example of the training process for a neural network to extract image attributes to be provided to a biometric authentication detector is shown.

[0073] Figure 13 An example of providing extracted image attributes to a biometric authentication detector is shown.

[0074] Figure 14 An example of providing extracted image attributes to a biometric authentication detector is shown.

[0075] Figure 15 An example of providing extracted image attributes to a biometric authentication detector is shown. Detailed Implementation

[0076] The following embodiments are merely examples for implementing the methods, apparatus, systems or applications disclosed herein and should not be considered limiting.

[0077] Figure 1 A device 100 with a processor 102, a memory 104, a camera 106, and a display 108 is shown.

[0078] Device 100 can be any mobile or portable computing device. The device can be a handheld device. The device can be a mobile device with wireless or RF communication capabilities (e.g., WLAN, Wi-Fi, cellular, and / or Bluetooth). Examples of mobile devices include mobile phones or smartphones, tablets, laptops, portable gaming devices, portable internet devices and other handheld devices, as well as wearable devices (such as smartwatches, smart glasses, headphones, pendants, earbuds, etc.).

[0079] Display 108 may include an LCD screen or touchscreen configured for interactive input from a user. Camera 106 may be configured to capture images of the external environment of device 100. Camera 100 may be positioned to capture images in front of display 108. When a user interacts with display 108, camera 106 may be positioned to capture images of the user (e.g., the user's face).

[0080] Figure 2 An embodiment of camera 106 is shown.

[0081] Camera 106 may include one or more image sensors 200 for capturing digital images. The image sensors 200 may include multiple infrared (IR) sensors. These image sensors may include, but are not limited to, charge-coupled device (CCD) and / or complementary metal-oxide-semiconductor (CMOS) sensor elements for capturing infrared (IR) images or other invisible electromagnetic radiation. Camera 106 may include more than one image sensor 200 to capture multiple types of images. For example, camera 106 may include both an IR sensor 202 and an RGB (red, green, and blue) sensor 204.

[0082] Camera 106 may include one or more illuminators 206 for illuminating a subject using different types of light detected by image sensor 200. For example, camera 106 may include multiple visible light illuminators 208 (e.g., "flash illuminators"), multiple RGB light illuminators 210, and / or multiple infrared light illuminators 212, 214. Camera 106 may include a floodlight IR illuminator 212 and a patterned IR illuminator 214. In some embodiments, the patterned IR illuminator 214 may include an array of light sources, such as, but not limited to, VCSELs (vertical-cavity surface-emitting lasers). The multiple image sensors 200 and the multiple illuminators 206 may be included in a single or separate chip package.

[0083] In some embodiments, the image sensor is an IR image sensor, and this image sensor is used to capture infrared images for face detection, face recognition, face authentication, material detection, and / or depth detection. For face detection, recognition, and / or authentication, illuminator 206 can provide flood IR illumination to illuminate the subject with IR light, and the image sensor can capture an image of the subject illuminated by the flood IR light. The flood IR illuminated image can be, for example, a two-dimensional image of the subject illuminated by IR light.

[0084] For depth and / or material inspection, irradiator 206 can provide patterned IR irradiation.

[0085] As an example, IR irradiation can have wavelengths from 300 nm to 1100 nm, particularly from 500 nm to 1100 nm. Additionally or alternatively, light in the infrared spectral range, such as light in the range of 780 nm to 3.0 µm, can be used. Specifically, IR irradiation in a portion of the near-infrared region suitable for silicon photodiodes (specifically in the range of 700 nm to 1100 nm) can be used. The pattern can be a light pattern projected onto the subject with a known and / or controllable configuration and pattern. The pattern can also be a light pattern projected onto the subject with a random, unknown, and / or dynamic configuration and pattern. The pattern can be regularly arranged (e.g., triangular patterns, rectangular patterns; hexagonal patterns, or patterns including additional convex tessellation) or irregularly arranged, forming a structured light pattern. In some embodiments, the pattern is a speckled pattern. The pattern can include, but is not limited to, dots, spots, stripes, dashes, nodes, edges, and combinations thereof. The pattern can be generated by a VCSEL (Vertical-Cavity Surface-Emitting Laser) array. One option is to use a hexagonal pattern, which has the advantage of solving the correspondence problem.

[0086] Images captured by a camera may include images with a user's face (e.g., the user's face is included in the image). An image with a user's face may include any digital image that shows at least a portion of the user's face within an image frame. Such an image may include only the user's face, or it may include the user's face in a smaller portion or block of the image.

[0087] The user's face can be captured in an image at a sufficient resolution to allow image processing of one or more features of the user's face, thereby enabling facial authentication.

[0088] Figure 3 An example of a processing module 300 associated with (multiple) image sensors 200 and configured to process image signals is shown. Figure 4 Another example is shown of a processing module 300 associated with (multiple) image sensors 200 and configured to process image signals.

[0089] The processing module 300 can be embedded as follows: Figure 1 and Figure 2 Of the 100 devices shown.

[0090] Processing module 300 may be a component of a system-on-chip (SoC) integrated circuit (IC). Processing module 200 may include an image processing unit 302 and a neural network processing unit 304. Image processing unit 302 may perform various stages of an image processing pipeline. Image processing unit 302 may include an image processing circuit system configured for digital image processing, such as processing raw image data, de-mosaicing the raw image data, and image manipulation operations (such as image panning, horizontal and vertical scaling, lens correction, color space conversion, and / or image stabilization transformation). In some embodiments, image processing unit 302 may receive raw image data from image sensor 200 and process the raw image data into a form usable by other sub-components, such as neural network processing unit 304.

[0091] The neural network processing unit 304 may include a neural processor circuit system configured to process neural network operations. The neural network processor circuit may include multiple neural engine circuits configured to perform neural network operations such as convolution, spatial pooling, and local response normalization. The neural processor circuit may perform various machine learning operations based on computations including multiplication, addition, and accumulation. Such computations may be arranged to perform, for example, convolution of input data and kernel data. The image or neural processor circuit may include configurable circuitry that performs these operations in a fast and energy-efficient manner, while mitigating the resource-intensive operations associated with CPU image processing or neural network operations.

[0092] When using dedicated circuit systems (such as those configured for or configurable for image or neural network processing), operation and interaction control or planning are handled by the CPU. In other words, the CPU processes the task list of different dedicated circuits and the interactions with such circuits.

[0093] Specifically, facial authentication and its additional security requirements, or other image-driven detection processes (such as material inspection), may involve various processing-intensive operations, thereby reducing performance and increasing battery consumption. Therefore, while mitigating CPU-intensive operations, the use of configurations or configurable circuitry tailored to specific operational tasks can be managed to improve performance and reduce battery consumption. One way to achieve this is by using lightweight operations—operations with lower processing intensity compared to others—which is particularly important for operations involving image processing, such as facial authentication or material inspection.

[0094] Figure 5 A flowchart illustrating an embodiment of a biometric authentication process that uses facial features and includes skin detection is provided.

[0095] Image capture can be triggered by requesting authorization or authentication of the user of device 100 (e.g., unlocking device 100). The image can be captured by camera 106. For example, one or more images (floodlight images) can be captured when the user's face is illuminated by (multiple) IR floodlight illuminators 210. Further, for example, one or more images (patterned light images) can be captured when the user's face is illuminated by (multiple) IR patterned light illuminators 212.

[0096] Furthermore, for example, one or more images (floodlight image and patterned light image) can be captured when the user's face is illuminated with an IR floodlight illuminator 210 and an IR patterned light illuminator 212. Alternatively, one or more images (RGB images) can be captured when the user's face is illuminated with (multiple) RGB or visible light illuminators 208, 210.

[0097] The captured images can be provided as raw image data to the processing module 300 for user authorization, including biometric authentication and / or skin detection. The image processing unit 302 may include an image signal processor (ISP). The ISP may include circuitry suitable for processing images received from the camera. The ISP may include any hardware and / or software (e.g., program instructions) capable of processing or analyzing images captured by the camera.

[0098] The image processing unit 302 may include a central processing unit (CPU). The CPU may use any suitable instruction set architecture. The CPU may be configured to execute instructions defined in that instruction set architecture. The CPU may be a general-purpose or embedded processor. It may use any of a variety of instruction set architectures (ISAs) (such as x86, PowerPC, SPARC, RISC, ARM, or MIPS ISA) or any other suitable ISA. The processing module 300 may include multiple CPUs. In a multiprocessor system, each of these CPUs may implement the same ISA.

[0099] The captured image can be provided as raw image data to the image processing unit 302, particularly to the ISP and / or CPU. If the raw image data is provided to the ISP and / or CPU, it may depend on the event that triggered the image capture, such as an event that triggered the image capture to unlock the device. The set of instructions or operations to be executed may depend on the event that triggered the image capture, such as an unlock event. The set of instructions or operations to be executed may include one or more instructions or operations to be performed by the ISP and / or CPU. The CPU, as the central processing unit, can provide a task list based on the triggering event, which includes one or more instructions or operations to be executed and the resources to perform such operations.

[0100] The raw image data can be preprocessed, including image manipulation performed by the image processing unit 302. After image capture, the CPU can trigger one or more preprocessing operations. For example, an unlock event can trigger image manipulation as a preprocessing operation. Image manipulation can depend on the operation that triggered image capture (such as an unlock event) and the image type (such as a floodlight image, a patterned light image, or an RGB image).

[0101] Flood light images can be provided to an ISP for correction manipulation and / or segmentation, for example. Patterned light images can be provided to a CPU for partitioning, masking, and / or segmentation, for example. In particular, patterned light images can be segmented to extract facial regions and / or landmarks. Further, in particular, facial segments of the patterned light image can be partitioned into partial images, such as those shown in the image. Figure 7 as well as Figures 8a to 8c More detailed description in the context.

[0102] Manipulated image data can be provided to the neural network processing unit 304 for authentication and / or skin detection. The neural network processing unit 304 can receive the manipulated image data, perform multiplication-accumulation operations (e.g., convolution) on the manipulated image data based on stored kernel data, perform further post-processing operations on the result of the multiplication-accumulation operations, and generate output data. Specifically, a portion of the image obtained from a patterned light image can be provided to the neural network processing unit 304 to execute a data-driven model configured for skin detection. The data-driven model can be instantiated and executed on the neural network processing unit. Specifically, multiple processed floodlight images can be provided to the neural network processing unit 304 to execute a data-driven model configured for template-based authentication. The data-driven model can be instantiated and executed on the neural network processing unit 304.

[0103] For instantiation and execution, manipulated image data can be broken down into smaller data units for parallel processing at multiple neural engines included in the neural network processing unit. Typically, multiple operation loops are executed to generate output for the task associated with the neural network. A compiler executed by the CPU can analyze the hierarchy and nodes of the neural network and determine how to break down the manipulated image data based on the hardware constraints of the neural network processing unit 304. One function of the compiler may be to determine how to break down the manipulated image data into smaller data units for processing at the neural engines of the neural network processing unit 304, and how to iterate this processing to produce the task result.

[0104] Another function could be to determine the task list for the neural network processing unit 304.

[0105] The neural network to be executed may include network layers or sublayers that are instantiated or implemented as a series of tasks to be performed by the neural network processing module.

[0106] For example, a neural network may include network layers (or sublayers), including convolutional layers with sublayers and pooling layers. The neural network can be instantiated by a neural network processing module 304. To do this, the neural network is converted into a task list to become executable by the neural network processing module 304. The CPU converts the neural network into a task list. The task list includes a linear linked list defining a sequence of tasks, which includes tasks for the individual convolutional layers, sublayers, and / or pooling layers. Each task may be associated with a task descriptor that defines the configuration for the neural network processing module 304 to execute the task. Each task may correspond to a single network layer of the neural network, a portion of the network layers of the neural network, or multiple network layers of the neural network. Based on the task list generated by the CPU and provided to the neural network processing module 304, the neural network processing module 304 instantiates the neural network by executing the tasks in the task list under the control of a neural task manager.

[0107] The neural task manager can receive a list of tasks from a compiler executed by the CPU, store tasks in its task queues, select tasks to be executed, and send instructions to other components of the neural processor circuitry to execute the selected tasks. The neural task manager 310 may include one or more task queues 1004. Each task queue 1004 is coupled to the CPU 208 and the task arbitrator 1002. Each task queue 1004 receives from the CPU 208 a reference to a task list 904 for a task that instantiates the neural network 900 when executed by the neural processor circuitry 218. The references stored in each task queue 1004 may include a set of pointers and counters pointing to the task list 904 in the task descriptor 1012 in system memory 230. Each task queue 1004 may further be associated with a priority parameter that defines the relative priority of the task queue 1004. The task descriptor 1012 of a task specifies the configuration for the neural processor circuitry 218 to execute the task.

[0108] Specifically, a portion of the image obtained from the patterned light image can be provided to the neural network processing unit 304 to execute a data-driven model configured for skin detection.

[0109] The data-driven model to be instantiated and executed on the neural network processing unit 304 is described in more detail in the context of Figure 8. The results of skin detection can be provided to the image processing unit 302 to verify the image data used in the authentication process. If no skin is detected in the patterned light image using the data-driven skin detection mechanism disclosed herein, operation is stopped and authentication is not triggered, for example, the device is not unlocked or an alternative unlocking mechanism is triggered. If skin is detected in the patterned light image using the data-driven skin detection mechanism disclosed herein, further operation is performed and authentication is triggered.

[0110] For such sensitive operations (such as authentication based on a user's biometric data), the processing module 300 may include multiple security zones. These security zones may encapsulate certain resources configured to authenticate users and process sensitive biometric information in an encrypted environment. The security zones may protect isolated internal resources from direct access by external circuits. These internal resources may be storage devices that store sensitive data (such as biometric information, encryption keys, etc.).

[0111] The facial recognition process can operate within a secure area based on images captured by camera 106 and processed by image processing unit 302. The functions of the recognition process can be performed within the secure area. One example could be a registration process. During the registration process, camera 106 can capture or collect images and / or image data from the user to be authorized, so that the user can be subsequently authenticated using a facial recognition authentication process. Templates can be generated from the images of the registration process and stored in the secure area storage device. Another example could be an authentication process based on floodlight images. For facial authentication, camera 106 can transmit image data to the processing unit with a secure area via a secure channel. This secure channel can be, for example, a dedicated path for transmitting data (i.e., a path shared only by the intended participant), or a dedicated path for transmitting encrypted data using a cryptographic key known only to the intended participant. The secure area processing unit can operate one or more machine learning models. One or more neural network modules can be used to operate the machine learning models. The neural network modules can reside within the secure area. The secure area can compare image characteristics with templates of each type of stored image to generate an authentication score based on a matching score or other matching ranking between the user in the captured image and the user in the stored template. The system can combine authentication scores from images such as floodlight IR and patterned illumination images to determine a user's identity. If the user's identity is authenticated, they are allowed to use the device, for example, to unlock it. If authentication is successful, operational parameters such as unlocking the application or the device are generated. If authentication fails, the authentication is unsuccessful; for example, the device is not unlocked or an alternative unlocking mechanism is triggered.

[0112] For authentication, floodlight image data can be processed to provide material identification. For authentication, floodlight image data can be processed to provide facial recognition and authentication. Facial feature analysis can be performed on floodlight images(s) by a neural network configured to generate at least one biometric authentication information. For example, the neural network can generate a facial feature vector. The feature vector can be compared with a template feature vector. The template feature vector can be provided from a template storage device (such as a secure area storage device) to generate a matching score. The template feature vector can be generated based on the authorized user's registration on the device (e.g., a template generated during the registration process). The matching score can be a score of the difference between the facial feature vector and the corresponding template vector (e.g., the feature vector of the authorized user generated during the registration process). The closer the feature vector is to the template feature vector (e.g., the smaller the distance or the smaller the difference), the higher the matching score may be.

[0113] Comparing a feature vector with a template vector to obtain a corresponding matching score may include using one or more classifiers or networks that support classification to classify and evaluate the differences between the generated feature vector and the feature vector from the template.

[0114] Examples of different classifiers that can be used include, but are not limited to, linear classifiers, piecewise linear classifiers, nonlinear classifiers, support vector machines, and neural network classifiers. In some embodiments, the matching score can be evaluated using the distance score between the feature vector and the template.

[0115] For authentication, a matching score can be compared to a device unlock threshold. The unlock threshold can be represented as the minimum difference between the feature vector of the authorized user's face based on the template vector and the facial feature vector when the user attempts to unlock the device. For example, the unlock threshold could be a threshold used to determine whether the unlocked facial feature vector is sufficiently close to the template vector associated with the authorized user's face.

[0116] Figure 6 The image shows a patterned light image and the signal generated from the patterned light image.

[0117] The patterned light image displayed can be captured when a user's face is illuminated by (multiple) IR patterned light illuminators 212. The signal generated from this image data is displayed along a patterned reflection line received by an image sensor (e.g., a CMOS or CCD-based sensor 200). In this case, the y-axis represents the grayscale value at the image grayscale level. The x-axis represents the position along the line displayed in the image. The reflected light pattern signal can be characterized by the maximum value at a specific location above the background texture signal information. The reflected light pattern can be further characterized by the width of the light distribution. The reflected light pattern essentially includes the physical properties of the objects reflecting these patterns. In particular, luminosity, reflectivity, and light distribution are properties of the material surface, depending on the source and distance of the reflection. These properties of the objects reflecting the light can be extracted from the patterned light image using filtering techniques known from beam profile analysis. This technique is described in WO 2020187719A1, which is incorporated herein by reference. These techniques rely on data-driven techniques, which will, for example, in Figures 7 to 9 Further illustrative descriptions will be provided within the context of this document.

[0118] Figure 7 An example flowchart is shown for a method of preprocessing patterned light images.

[0119] Image preprocessing may include detecting bounding boxes around a face in a flood illumination image. These bounding boxes can be detected using models such as neural networks or using commonly known techniques for bounding box determination in face recognition. Furthermore, landmarks can be determined. Bounding boxes and landmarks can be associated with corresponding pixels in the patterned light image data. Specifically, patterned light image data associated with a face and located within a bounding box in the flood illumination image can be used to crop the patterned image into a region of interest. In other embodiments, predefined bounding boxes can be used. These boxes can be displayed to the user when the image is captured.

[0120] Images can be preprocessed to extract material information. One type of preprocessing can include digital manipulation of image data. In particular, images can be manipulated using image enhancement techniques such as cropping, rotation, and blurring. This manipulation can suppress background information with facial features. Patterns are key to extracting material information. Therefore, any model for extracting material information can be trained on manipulated image data. This reduces the size of the model, significantly reducing the storage and processing requirements of the model on a smartphone.

[0121] One option to do this is to crop the image into partial images. Figures 8a to 8c ). Figures 8a to 8c An example of a method for generating a partial image from a pattern image is shown.

[0122] Cropping can be based on identifying peaks in the pattern and cropping by a specific size around those peaks. In this embodiment, the peaks can be located at the center of the image (see [link to image]). Figure 8a Parts of the image can have a fixed size and may not overlap. In another embodiment, the cropping can be random in terms of the location of the peaks. Figure 8b Partial images can be of arbitrary size. Partial images can overlap. Partial images can include at least one pattern feature (intensity peak) or more pattern features. Partial images can include portions of pattern features or anomalous signatures of pattern features. Other options for partitioning manipulation include object detection algorithms, such as single-shot detection (SSD or region-based CNN for region-based neural networks), to provide bounding boxes for partial image cropping. This manipulation can be based on cropping (cropping or SSD or RCNN) via anchor points of pattern features in the patterned image. Figure 8c ).

[0123] Materials can be identified from partial images. This identification can be based on data-driven models, such as trained neural networks (e.g., convolutional neural networks). The training of such networks will be described in more detail below. The model can be parameterized to map partial images to classifiers, such as human skin or non-human skin, materials or no material or specific material classes. The model can also be parameterized to produce more complex outputs, such as human skin, material classes (latex, silicon, fabric), etc.

[0124] Based on this identification, the authentication process can be verified. For example, if non-human skin is detected in an image, the authentication process can be stopped, and the user can be notified. In this case, authentication cannot be performed or spoofing may occur (anti-spoofing). If skin is detected, verification may lead to further processing, such as providing materials present in the image. This material detection may be associated with landmarks, such as landmarks detected in a floodlight image for consistency checks. Consistency checks may include a set of rules. For example, if latex is detected as associated with the eyes, the authentication process can be stopped. Material identification and verification steps can also be performed after facial recognition.

[0125] Figure 9 An example of a method for training a data-driven model configured to detect materials based on partial images is shown.

[0126] Raw image signals from spot-based measurements that generate patterned light images can be provided. The raw image data can be corrected based on dynamic masking to segment regions of interest (e.g., faces). Patterned light images labeled with material information (e.g., skin / non-skin, material / no material, or a specific material class) can be provided as training data.

[0127] Patterned light image data can be manipulated to generate partial images, representations of each partial image, and clustered partial images.

[0128] It can be as follows Figures 8a to 8c The training data is cropped as described in the context. After cropping, the signal signature of the object's background texture (background lighting plus facial features) has been randomized. However, this randomization may not be sufficient to extract material information. In particular, for data-driven model-dependent methods, attention during training can be focused on the pattern signature (material) rather than the background (facial features). Different methods can be applied to further suppress the background signature in the image.

[0129] A process can rely on PCA (Principal Component Analysis) or RBF (Radial Basis Function (RBF) Neural Network) to generate representations of partial images. This representation is a manipulation in the sense of reducing the dimensionality of the partial image. Therefore, the partial image is mapped to a low-dimensional representation of the image. This representation is associated with a physical signature embedded in the reflection of the partial image and further suppresses background signatures. Representations of partial images can be generated based on PCA or RBF mappings. These representations can be associated with corresponding images.

[0130] Another process for generating representations may include manipulating portions of an image via a neural network architecture comprising an encoder and a decoder. This neural network may be a CNN adapted for image processing. The encoder portion of the CNN generates a representation, and the decoder portion generates a portion of the image. This representation can be trained by comparing a portion of the image at the input layer of the encoder network with a portion of the image at the output layer of the decoder network. Through this training, each portion of the image can be associated with a representation trained by the network, thereby further suppressing background signatures through dimensionality reduction.

[0131] Other options for constructing low-level representations of partial images include FFT, wavelets, deep learning (such as CNNs), energy models, normalized flow, GANs, visual transformers, or transformers for natural language processing, autoregressive image modeling, GANs, deep autoencoders, deep energy-based models, and visual transformers. Supervised or unsupervised schemes can be applied to generate representations (and also to generating embeddings in ML languages, such as cosine or Euclidean metrics).

[0132] Once the representations are generated, the images and representations can be grouped. Grouping can include clustering the images into groups using clustering algorithms such as k-means or elbow curves. Other clustering methods are also possible. Once the representations are generated, they can be assigned to PCA clustering or clustered using algorithms.

[0133] This representation, and potentially through clustering, can generate manipulated image data for extracting material information from patterned images acquired by an IR sensor. The manipulated image data may include: partial images having at least a portion of the patterned features present in the patterned image; and at least one representation or at least one class associated with a physical signature embedded in the reflections of the partial images.

[0134] Based on this manipulated image data, a data-driven model can be trained. By manipulating the data in this way, texture or background signatures can be suppressed. This contrasts with facial recognition, where the object's texture includes facial features trained on the model. In material classification, patterns and their relationship to material signatures are embedded in the training process.

[0135] To train a data-driven model for extracting skin features, supervised or self-supervised learning can be used. Manipulated data can include known data from portions of the images. For example, a training dataset of pattern images can be labeled with skin / non-skin tags. Such labels can be achieved by manipulating portions of the images. Manipulation can be performed as described above.

[0136] Manipulated data (including representations, labels, and clusters) can be fed into a classifier network (such as a CNN) using supervised or self-supervised learning techniques. For example, ResNet, triplet loss, npx loss, or npair loss can be used. The representations and clusters can be used as input to the network's input layer or via a model loss function (e.g., via a contrastive divergence loss function). Other options are cost functions, error functions, and objective functions. Thus, the manipulated data, including representations and clusters, can be part of the model training process.

[0137] Neural network embeddings can include learned low-dimensional representations of discrete data as continuous vectors. These embeddings overcome the limitations of traditional encoding methods and can be used for purposes such as finding nearest neighbors, inputting to another model, and visualization. Embeddings can be used to project representations or embeddings onto human skin-non-skin binary classifiers.

[0138] Material information extracted from the trained model can be correlated with labels. If the label is a binary classifier distinguishing between material / no material or human skin / non-human skin, the network will classify a portion of the image accordingly. If the label includes more metadata, such as additional annotations for each mask, the material of the mask (e.g., silicone, latex mask), or the brand of the mask, the network will classify a portion of the image accordingly. Additional metadata can include external background, such as facial features like beards, glasses, hats, etc.

[0139] Such a trained data-driven model can be provided to devices for extracting material information, for example, through classification based on partial images generated from patterned light images.

[0140] Figure 10 An example block diagram is shown, featuring an image processing unit configured to perform image preprocessing and a neural engine configured to perform material detection.

[0141] Image processing unit 302 may include a CPU and an ISP. The CPU may preprocess (multiple) patterned light images to extract material information from partial images by a neural network processing unit. Preprocessing may include lightweight algorithms that crop the patterned light images to partial patternsed light images or otherwise manipulate the patterned light images. The CPU may generate a task list for the generated manipulated, such as partial images, to provide to the neural network processing unit. The ISP may preprocess (multiple) floodlight images to extract biometric authentication information from the floodlight images by the neural network processing unit. The CPU or ISP may generate a task list for the generated floodlight images to provide to the neural network processing unit. In some embodiments, the CPU may be configured to extract material information based on the preprocessed patterned light images, for example, as in... Figures 5 to 9 The context described herein. In other embodiments, the neural network processing unit 304 may be configured to extract material information based on a preprocessed patterned light image, for example, as described in Figures 5 to 9 Described in the context of.

[0142] The ISP can preprocess (multiple) floodlight images to extract biometric authentication information from (multiple) floodlight images by a neural network processing unit. The CPU or ISP can generate a task list for the generated floodlight images to provide to the neural network processing unit.

[0143] The neural network processing unit 304 can be configured to extract material information based on a preprocessed patterned light image. The neural network processing unit 304 can also be configured to extract biometric identification information about the floodlight image. The material information and biometric authentication or identification information can be provided to the CPU to trigger the device to perform a requested operation. For example, the device can be unlocked upon successful user authentication. In other embodiments, the CPU can be configured to perform material detection. The CPU can be configured to extract material information based on a preprocessed patterned light image. (As in...) Figures 5 to 9 The lightweight operation for material inspection described in the context allows the use of a CPU, while the ISP and neural network processing unit 304 are used for face authentication / recognition processing with higher performance requirements.

[0144] Figure 11 An example block diagram is shown, featuring an image processing unit configured to perform image preprocessing and a neural network processing unit configured to perform material and biometric authentication detection.

[0145] The image processing unit 302 and the neural network processing unit 304 may be components of the system-on-chip (SoC) processing module 300. The image processing unit 302 may include a central processing unit (CPU) configured to perform general-purpose operations and an image signal processor (ISP) configured to perform image-specific operations. The neural network processing engine 304 may include one or more SoC components configured to perform neural network-specific operations (such as convolution, pooling, or fully connected layers). Depending on the architecture of the neural network, the neural network processing engine 304 may include one or more components configured to process different neural network-specific operations for each component.

[0146] The neural network processing engine 304 can be configured to perform material detection and / or biometric authentication based on a neural network, which is trained and configured to perform such operations. The neural network processing engine 304 can be configured to perform material detection on one component and biometric authentication on another component, depending on the neural network architecture and operations required to perform such operations. Material detection can be, for example, in... Figures 4 to 10 As described in the context, it is implemented based on a data-driven model (including neural networks) configured to extract material information. Biometric authentication detection can be, for example, in... Figures 4 to 10 As described in the context, it is implemented based on a data-driven model (including neural networks) configured to extract biometric authentication information.

[0147] Figure 12 An example of the training process for a neural network to extract image attributes (such as material properties and / or biometric attributes) to be provided to a biometric authentication detector is shown.

[0148] Neural networks can be based on an encoder-decoder architecture. A neural network may include at least one encoder and at least one decoder. The encoder and decoder may be implemented on the SoC architecture described herein. A neural network processing unit may be configured to train the encoder and / or decoder and / or execute the trained encoder and / or trained decoder. The neural network processing unit may include a GPU-enabled computer processor. Training a neural network on a GPU-enabled computer processor may output weights or kernels described using floating-point numbers. In this embodiment, floating-point operation parameters may be converted to integer representations to be used on the neural network processing unit.

[0149] The encoder and / or decoder may include at least one convolutional neural network (CNN). The encoder and / or decoder may include other types of networks or functions, such as FFT, wavelets, deep learning (e.g., CNN), energy models, normalized flow, recurrent neural networks (RNN), gated recurrent units (GRU), long short-term memory (LSTM) recurrent neural networks, visual transformers, or transformers for natural language processing, autoregressive image modeling, autoregressive image modeling, normalized flow, deep autoencoders, deep energy-based models, visual transformers, etc. Supervised or unsupervised schemes can be applied to generate representations. The encoder and / or decoder may include at least one multi-scale convolutional neural network. Using a multi-scale convolutional neural network, the encoder can generate a representation of the input image based on feature vectors in the feature space. For example, the encoder process can generate a 32 × 32 grid representation from an input image that may have a higher resolution (e.g., a 256 × 256 image), where each region (cell) of the grid contains a feature vector. The encoder and decoder may be based on the same or different networks or functions. The encoder architecture may differ from the decoder architecture. The encoder and decoder may have the same architecture.

[0150] The training process for the encoder-decoder architecture may include providing training datasets(s) that include image data and associated metadata, particularly image attributes such as biometric attributes or material properties. Image data may include(s) floodlight images, RGB images, and / or patterned light images of the user under appropriate lighting conditions. The training process for the encoder-decoder architecture may include providing training data, including image data and metadata, as input to the encoder-decoder. Image data may include images such as floodlight images, RGB images, and / or patterned light images. Image data may include manipulated images, such as partial images, generated from patterned light images. Image data may include labeled images that include metadata associated with image attributes, such as facial attributes (e.g., eyes, nose, mouth, pose, pitch, yaw, and / or roll) or material attributes (e.g., material type, skin, organic material, inorganic material, material class), manipulated image representations (e.g., partial image representations or, for example, through...). Figures 8a to 8c (Feature vectors generated by the method described in the context), such as the size of the manipulated portion of the image, the number of light spots in the manipulated portion of the image, the facial region of the manipulated image, texture, skin type, distance to the image or portion of the image, semi-transparency and / or luminosity.

[0151] Material properties can involve, for example, in Figure 5 or Figure 9The extracted material information is described in the context of the image. Material properties may relate to material types, such as, but not limited to, skin, organic materials, inorganic materials, material classes, textures, and / or skin types. Material properties may relate to material information extraction processes, such as the process of generating material information from (multiple) manipulated patterned light images or extracting material information from (multiple) patterned light images, such as, but not limited to, manipulated image representations (e.g., partial image representations or, for example, through...) Figures 8a to 8c The input to the encoder-decoder architecture can include multiple training images with different users and / or faces. The faces in the images can have different attributes, such as facial attributes, material attributes, and / or material detection attributes. The attributes of the faces and / or materials present in the training images can be known. Known information about these attributes can be provided as metadata during training. Therefore, training images can be augmented using metadata including facial attributes, material attributes, and / or material detection attributes.

[0152] Operations on images trained by an encoder-decoder architecture can include, but are not limited to, face detection, facial feature detection (e.g., landmark (eyes, nose, mouth, etc.) detection, gaze estimation, smile estimation, etc.), facial pose estimation, occlusion detection, attention detection, facial recognition authentication (e.g., face matching), material detection, or any other process involving neural network modules operating on images with faces to evaluate facial attributes or features.

[0153] For training, the input can be provided to the encoder. The encoder can define the features in the image as feature vectors in the feature space. For example, the encoder can define the facial features of a flood image of a face and / or the material features of a pattern image of a face as feature vectors in the feature space, for example, as in... Figure 5 or Figure 9 Described in the context of [the above]. Feature vectors may include feature vectors representing user facial features and / or material features in a feature space. The feature space may be an n-dimensional feature space. Feature vectors may be n-dimensional numerical vectors that define features of the image for regions in the feature space corresponding to regions in the image. The number of dimensions may depend on the embodiment. For example, in the case of material features, the feature vector may refer to a partial representation of the image, such as, for example, in [the context of the above]. Figure 5 or Figure 9 Described in the context of.

[0154] Representations and / or feature vectors generated by the encoder can be provided to the decoder. The decoder can decode these representations or feature vectors. The decoder can include neural networks, such as recurrent neural networks (RNNs), gated recurrent units (GRUs), long short-term memory (LSTM) recurrent neural networks, or any of the above architectures. Decoding the representations or feature vectors can include mapping the feature vectors via a regression network to determine (e.g., extract) output data from the input. Decoding the representations or feature vectors can also include mapping the feature vectors to one or more classifiers via a classification network to determine output data based on the image input. The output data from the image input can relate to facial attributes, material attributes, and / or material detection attributes.

[0155] The output generated by the decoder may include information or attributes, such as facial attributes, material attributes, and / or material detection attributes of the image input. For training purposes, the attributes of the image input may be known and may be correlated with the decoded feature vectors. When the decoder operates on the feature vectors in the feature space, it may provide one or more predictions of the attributes of the image input based on the correlation between known attributes and feature vectors. The evaluated image attributes may include, but are not limited to, biometric attributes (such as facial attributes in the image, such as facial location (e.g., facial bounding box), facial pose (e.g., facial pitch, yaw, and / or roll), distance between the face and the camera, landmark location, facial gaze estimation, facial smile estimation, occlusion detection) and / or material attributes (e.g., material type, number of light spots, manipulated image size, facial region, texture (skin type), distance from the sensor to the imaged object, translucency, and / or luminosity).

[0156] Through training, image attributes can be determined by associating decoded feature vectors with metadata. For example, metadata can provide known attributes of (multiple) faces in the image input, where these known attributes define the attributes evaluated through the decoder process. Similarly, metadata can provide known attributes of (multiple) materials in the image input, where these known attributes define the attributes evaluated through the decoder process. Associating decoded feature vectors with metadata can include the decoder evaluating the differences between the decoded feature vectors and the metadata. Therefore, the encoder-decoder architecture can perform error function analysis on the differences between decoded feature vectors given metadata and refine the feature vector decoding process until the feature vector decoding process accurately determines the metadata. Thus, the decoder-encoder architecture can be trained with training images and metadata to accurately determine image attributes, such as biometric attributes and / or material attributes.

[0157] Figure 13 An example of providing extracted image attributes to a biometric authentication detector is shown.

[0158] Based on, for example, in Figure 12 The trained encoder-decoder structure described in the context of or based on, as in Figures 5 to 9 The model trained for material extraction, as described in the context, can obtain image attributes such as biometric attributes and / or material properties and provide them to downstream processes, such as biometric authentication detectors or neural networks configured to generate at least one biometric authentication information. The biometric authentication information may include a matching score between at least one facial feature vector generated from an image captured at the time of an unlocking event and at least one template vector stored for an authorized user. The generation of the biometric authentication information may include image attributes, such as biometric attributes and / or material properties.

[0159] For authentication, floodlight image data can be processed to provide facial recognition and authentication. Facial feature analysis can be performed on floodlight images(s) by a neural network configured to generate at least one biometric authentication information. The neural network configured to generate at least one biometric authentication information can be trained on the floodlight image data to generate facial feature vectors. For example, the neural network can generate facial feature vectors. The feature vectors can be compared with template feature vectors. The template feature vectors can be provided from a template storage device (such as a secure area storage device) to generate a matching score. The template feature vectors can be generated based on the authorized user's registration on the device (e.g., a template generated during the registration process). The template feature vectors can be generated by a neural network configured to generate at least one biometric authentication information based on the authorized user's registration on the device. The matching score can be a score of the difference between the facial feature vector and the corresponding template vector (e.g., the feature vector of the authorized user generated during the registration process). The closer the feature vector is to the template feature vector (e.g., the smaller the distance or the smaller the difference), the higher the matching score may be.

[0160] Comparing a feature vector to a template vector to obtain a matching score may include using one or more classifiers or supporting networks to classify and evaluate the differences between the generated feature vector and the feature vector from the template. Examples of different classifiers that can be used include, but are not limited to, linear classifiers, piecewise linear classifiers, nonlinear classifiers, support vector machines, and neural network classifiers. In some embodiments, the matching score may be evaluated using the distance score between the feature vector and the template.

[0161] For authentication, a matching score can be compared to a device unlock threshold. The unlock threshold can be represented as the minimum difference between the feature vector of the authorized user's face based on the template vector and the facial feature vector when the user attempts to unlock the device. For example, the unlock threshold could be a threshold used to determine whether the unlocked facial feature vector is sufficiently close to the template vector associated with the authorized user's face.

[0162] Other non-limiting examples of downstream processes related to biometric authentication information that use previously acquired attributes may include: a facial recognition authentication process that uses pose information (e.g., pitch, yaw, and roll data) to improve the accuracy of assessing the match between a face and a registered user; or a landmark detection process that uses occlusion information to more quickly and accurately mask occluded areas to generate landmark locations.

[0163] To improve the generation of at least one biometric authentication information, biometric attributes and / or material properties can be provided to the neural network that generates the biometric authentication information. Biometric attribute generation can be performed by one or more image processing units (e.g., central processing units (CPUs) via, for example, in... Figure 12 or Figure 13 The material property generation can be performed within the network architecture described in the context of [the specific network architecture described]. Material property generation can be performed by one or more central processing units (CPUs), for example via [a specific network architecture]. Figure 12 or Figure 13 The biometric attribute generation is performed within the context of the network architecture described. Biometric attribute generation can be performed by one or more image processing units (such as (multiple) central processing units (CPUs) or (multiple) image signal processors) or (multiple) neural network processing units, for example via... Figure 12 or Figure 13 The generation of biometric attributes can be performed within the context of the network architecture described. This can be accomplished by one or more neural network processing units, for example, via... Figure 12 or Figure 13 The generation of material properties can be performed within the context of the network architecture described. This can be accomplished by one or more neural network processing units, for example, via [the network architecture described in the original text]. Figure 12 or Figure 13 ,or Figures 5 to 9 The material property generation can be performed within the context of the network architecture described. Material property generation can be performed by one or more image processing units, particularly one or more central processing units (CPUs), for example via... Figure 12 or Figure 13 ,or Figures 5 to 9 The network architecture described in the context is used for execution. Material property generation can include, for example, in... Figure 12 or Figure 13 ,or Figures 5 to 9 The method described in the context.

[0164] Figure 14This demonstrates another example of providing extracted image attributes to a biometric authentication detector.

[0165] As demonstrated in this embodiment, different attributes extracted from an image, including material properties, can be provided to at least one layer of a neural network configured to generate authentication information. Material properties can be extracted, for example, as shown in... Figure 12 and Figure 13 The material properties are described in the context of [the relevant information]. Material properties can be extracted material information, for example, as in [the context of...]. Figure 5 or Figure 9 The material properties are described in the context of the image(s). Material properties can be material information related to the image(s)(s). Biological properties can be extracted, for example, as in... Figure 12 and Figure 13 The context describes this. Other attributes (such as biometric attributes) may include facial attributes such as gaze, landmarks, and posture (including, for example, pitch, yaw, and / or roll).

[0166] Neural networks can operate using one or more convolutional layers (e.g., weights × input). Image attributes (such as material attributes and / or biometric attributes) can be applied individually to different layers, or at least some of these attributes can be jointly applied to a single layer or multiple single layers. Image attributes (such as material attributes and / or biometric attributes) can be applied to different layers or the same layer of a neural network configured to generate authentication information. The application scheme of the attributes can be part of the training. Training may include, for example, defining the effects of different attributes to determine the order in which one or more attributes are applied to which layers. Applying these attributes to a neural network configured to generate authentication information can help determine the authentication information. Biometric attributes and / or material attributes can be applied to a neural network configured to generate at least one biometric authentication message and trained on floodlight image data to generate facial feature vectors. Biometric attributes and / or material attributes can be applied to one or more classifiers or (multiple) supporting classification networks to classify and evaluate the differences between the generated feature vectors and feature vectors from a template.

[0167] Attributes (such as material properties and / or biometric attributes) can be provided as scalar values ​​to the neural network configured to generate authentication information (e.g., weights × input + bias). Attributes (such as material properties and / or biometric attributes) can be provided as weights applied in convolutions to the neural network configured to generate authentication information (e.g., weights × input + bias). Attributes (such as material properties and / or biometric attributes) can be provided as weighted attributes as inputs to the neural network configured to generate authentication information. For example, the weighted combination used as input to the network may include a first weight applied to a first attribute, a second weight applied to a second attribute, a third weight applied to a third attribute, and so on. Attributes (such as material properties and / or biometric attributes) can be provided as constraints or boundaries to the neural network configured to generate authentication information. Providing attributes as constraints or boundaries can narrow the search space for the neural network to evaluate and determine authentication information.

[0168] Attributes (such as material properties and / or biometric attributes) can be provided as operators. Operators can be applied to the outputs of one or more network layers. Therefore, operators can update the inputs of the next layer based on the applied operators. Operators can be general operators or functions generated for mapping attribute data. Operators can be masking functions to mask the network layer outputs used as inputs to the next network layer. Masking or closing parts of the input can reduce the search space of the network layers, thereby improving the accuracy and speed of the network layers in determining the output. These attributes can be applied to neural networks configured to generate authentication information using other methods. For example, combinations of the described methods can be used as inputs to the respective network layers.

[0169] Figure 15 This demonstrates another example of providing extracted image attributes to a biometric authentication detector.

[0170] As demonstrated in this embodiment, different attributes extracted from (multiple) patterned light images, including one or more material properties, can be provided to at least one layer of a neural network configured to generate authentication information. One or more material properties may include the representation of a portion of the image, the size of the portion of the image, the number of light spots in the portion of the image, the facial region of the portion of the image, the texture of the portion of the image, the distance from the face to the camera in the portion of the image, luminosity, etc. In this way, material information from the patterned light images can be fused into the generation of biometric authentication information, thereby improving the accuracy and speed of the network layer in determining biometric authentication information. For example, in... Figure 14 These properties, as described in the context, can be applied in different ways.

[0171] This disclosure has also been described in conjunction with various preferred embodiments and examples. However, by studying the accompanying drawings, this disclosure, and the claims, those skilled in the art, as well as those who practice the claimed invention, will understand and implement other variations.

[0172] Any steps presented in this document can be performed in any order. The methods disclosed herein are not limited to a specific order of these steps. Nor is it required that different steps be performed in a particular place or on a particular computing node in a distributed system; that is, each step can be performed on different computing nodes using different devices / data processing.

[0173] This disclosure has also been described in conjunction with various preferred embodiments and examples. However, by studying the accompanying drawings, this disclosure, and the claims, those skilled in the art, as well as those who practice the claimed invention, will understand and implement other variations.

[0174] Any steps presented in this document can be performed in any order. The methods disclosed herein are not limited to a specific order of these steps. Nor is it required that different steps be performed in a particular place or on a particular computing node in a distributed system; that is, each step can be performed on different computing nodes using different devices / data processing.

[0175] As used herein, "determine" also includes "initiating or causing determination," "generate" also includes "initiating and / or causing generation," and "provide" also includes "initiating or causing determination, generation, selection, sending, and / or receiving." "Initiating or causing an action" includes any processing signal that triggers a computing node or device to perform a corresponding action.

[0176] In the claims and specification, the word "comprising" or "including" or similar wording does not exclude other elements or steps and should not be construed as limiting oneself to the listed elements or steps. The indefinite article "a" or "an" does not exclude multiple. A single element or other unit may perform the function of several entities or items recited in the claims. The fact that certain measures are recited only in mutually different dependent claims does not indicate that a combination of these measures cannot be used in advantageous implementations or that additional elements may be included.

[0177] The provision within the scope of this disclosure may include any interface configured to provide data.

[0178] This can include application programming interfaces (APIs), human-machine interfaces (such as displays), and / or software module interfaces. Provision may include transmitting or submitting data to the interface, particularly displaying data to a user or having it used by a receiving entity.

[0179] Various units, circuits, entities, nodes, or other computing components may be described as being “configured to” perform one or more tasks. “Configured to” should be described as meaning “having a circuit system that performs one or more tasks during operation.” Units, circuits, entities, nodes, or other computing components may be configured to perform tasks even when the unit / circuit / component is not operating. Units, circuits, entities, nodes, or other computing components forming a structure corresponding to “configured to” may include hardware circuitry and / or memory storing executable program instructions to perform the operation. For convenience in the description, units, circuits, entities, nodes, or other computing components may be described as performing one or more tasks. This description should be interpreted as including the phrase “configured to.” Any statement of “configured to” is not expressly intended to invoke the interpretation of 35 USC § 112(f).

[0180] Generally, the methods, apparatuses, systems, computer elements, nodes, or other computing components described herein may include memory, software components, and hardware components. Memory may include volatile memory (such as static or dynamic random access memory) and / or non-volatile memory (such as optical or magnetic disk storage devices, flash memory, programmable read-only memory, etc.). Hardware components may include any combination of the following: combinational logic circuit systems, clock storage devices (such as flip-flops, registers, latches, etc.), finite state machines, memory (such as static random access memory or embedded dynamic random access memory), custom-designed circuit systems, programmable logic arrays, etc.

[0181] Any disclosures and embodiments described herein relate to the methods, systems, apparatuses, devices, chemicals, materials, and computer program elements listed above, and vice versa. Advantageously, the benefits provided by any embodiments and examples also apply to all other embodiments and examples, and vice versa.

[0182] All terms and definitions used in this article are to be understood in a broad sense and have their general meaning.

Claims

1. A method for generating biometric authentication information, the method comprising: - Provide one or more patterned light images, wherein the patterned light images include images of a user under illumination by at least one infrared pattern illuminator of the device; - Generate one or more material properties from the one or more patterned light images; - Generate biometric authentication information by providing one or more of the generated material properties to at least one layer of a neural network.

2. The method as described in claim 1, wherein, One or more material properties from the patterned light images are generated by manipulating the patterned light images(s) and extracting one or more material properties from the patterned light images(s).

3. The method as described in any of the preceding claims, wherein, Generating one or more material properties includes manipulating (multiple) patterned light images and generating at least one material feature vector, wherein the at least one material feature vector is generated by feeding (multiple) manipulated patterned light images to at least one neural network trained to generate at least one material feature vector associated with one or more material properties.

4. The method as described in any of the preceding claims, wherein, Generating one or more material properties includes: providing the one or more manipulated patterned light images to at least one data-driven model trained to extract one or more material properties from the manipulated patterned light images(s); and extracting one or more material properties from the manipulated patterned light images(s), wherein the data-driven model is parameterized according to a training dataset, which includes the manipulated patterned light images and the associated one or more material properties.

5. The method as described in any one of the preceding claims, wherein, The one or more material properties relate to at least one material type and / or at least one characteristic of manipulating the patterned light image(s), wherein the one or more material properties relate to skin, organic materials, inorganic materials, organic material classes and / or inorganic material classes.

6. The method as described in any of the preceding claims, wherein, The one or more material properties relate to at least one feature vector of at least one manipulated patterned light image(s), the size of the manipulated patterned light image(s), the number of light spot patterns of the manipulated patterned light image(s), the facial region of the manipulated patterned light image(s), texture, skin type, distance from the sensor that detects the patterned light image to the object reflecting the patterned light image(s), the translucency of the manipulated patterned light image(s), and / or the luminosity of the manipulated patterned light image(s).

7. The method as described in any of the preceding claims, wherein, Generate one or more biometric attributes from (multiple) floodlight images, wherein generating one or more biometric attributes includes generating at least one biometric vector, wherein the at least one biometric vector is generated by feeding (multiple) floodlight images to a convolutional neural network trained on a training dataset consisting of (multiple) floodlight images and corresponding one or more biometric attributes associated with the at least one biometric vector.

8. The method as described in any of the preceding claims, wherein, One or more biometric attributes are generated from multiple floodlight images, wherein the one or more biometric attributes and the one or more material attributes are provided to at least one layer of the neural network configured to generate biometric authentication information.

9. The method as described in any of the preceding claims, wherein, One or more floodlight images are provided, wherein the one or more floodlight images include an image of the user under light illumination from at least one infrared floodlight illuminator of the device, wherein the one or more floodlight images are provided as input to a neural network configured to generate biometric authentication information.

10. The method as described in any of the preceding claims, wherein, Providing the one or more material properties and / or biometric properties to at least one layer of the neural network includes applying biases, weights, functional relationships, and / or operators to at least one of the input channels, hidden layers of the neural network, and / or output channels, wherein the generated one or more material properties and / or biometric properties are weighted to be provided to at least one layer of the neural network.

11. The method as described in any of the preceding claims, wherein, The neural network configured to generate biometric authentication information generates at least one biometric vector based on a floodlight image of the user of the device, wherein the at least one biometric vector is compared with at least one template vector associated with the user to be authenticated.

12. The method as described in any of the preceding claims, wherein, The neural network configured to generate biometric authentication includes at least one convolutional layer with biases, weights, functional relationships, and / or operators.

13. The method as described in any of the preceding claims, wherein, The one or more biometric attributes refer to the location attributes of biometric objects and / or (multiple) biometric features on the image.

14. An apparatus for generating biometric authentication information, the apparatus comprising: - An image providing interface configured to provide one or more patterned light images, wherein the patterned light images include an image of a user under illumination by at least one infrared pattern illuminator of the device; - A material generator configured to generate one or more material properties from the one or more patterned light images; - A material property provider configured to generate biometric authentication information by providing one or more generated material properties to at least one layer of a neural network.

15. An apparatus configured to perform the method according to any one of claims 1 to 13 or including the apparatus according to claim 14.

Images