FINGERPRINT SCANNING SYSTEMS AND METHODS WITH BLURRING DETECTION AND CORRECTION

Abstract

An example device includes a display component comprising a fingerprint sensor configured to scan a fingerprint of a finger. The device includes one or more processors operable to perform operations including detecting movement of the finger during a fingerprint authentication phase. The operations include acquiring, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused smearing of the fingerprint. The operations include reconstructing an unsmeared fingerprint from the fingerprint data based on the movement of the finger and an estimated fingerprint distortion. The reconstructing reduces the smearing of the fingerprint so that it is detectable by a fingerprint matching component.The processes involve recognizing the fingerprint by the fingerprint matching component. Fingerprint recognition involves comparing the reconstructed fingerprint with a stored fingerprint template.

Description

CROSS-REFERENCE TO RELATED REVELATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 476,982, filed December 23, 2022, which is incorporated herein by reference. STATE OF THE ART

[0002] A display component of a computing device can be configured for fingerprint authentication. Various security-related functions can rely on fingerprint authentication. For example, fingerprint authentication can be used to securely access the computing device, such as locking or unlocking the computing device, and / or one or more applications and programs. Finger movement during scanning can result in blurring of the captured image. Fingerprint authentication can become difficult if a scanned fingerprint is blurred. SUMMARY

[0003] The present disclosure generally relates to a display component of a computing device. The display component may be configured to authenticate a fingerprint. Finger movement during scanning may result in blurring of the captured image. A blurred fingerprint may cause the fingerprint recognition system to reject the fingerprint, thereby disabling access to the computing device and / or one or more applications and programs. A fingerprint sensor may capture multiple frames of fingerprint images and may be configured to reconstruct the fingerprint based on the finger movement, a distortion factor of the fingerprint, and the multiple captured frames to create a blur-free fingerprint image. Such an image may be effectively used by a fingerprint recognition system to authenticate the fingerprint.

[0004] In a first aspect, a device is provided. The device includes a display component. The display component includes a fingerprint sensor configured to scan a fingerprint of a finger. The device further includes one or more processors operable to perform operations. The operations include detecting movement of the finger during a fingerprint authentication phase. The operations further include capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smearing of the fingerprint.The operations further include reconstructing, based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstruction reduces the smearing of the fingerprint to make it recognizable by a fingerprint matching component. The operations additionally include recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint to a stored fingerprint template.

[0005] In a second aspect, a computer-implemented method is provided. The method includes detecting, by a display component and during a fingerprint authentication phase, a movement of a finger. The method further includes capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smearing of the fingerprint, wherein the fingerprint sensor is configured to scan a fingerprint of the finger. The method further includes reconstructing, based on the movement of the finger and an estimated fingerprint distortion, an unsmeared fingerprint from the fingerprint data, wherein the reconstructing reduces the smearing of the fingerprint to make it detectable by a fingerprint matching component.The method additionally includes recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises comparing the reconstructed fingerprint with a stored fingerprint template.

[0006] In a third aspect, an article of manufacture is provided. The article of manufacture may include a non-transitory, computer-readable medium having stored thereon program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations. The operations include detecting, by a display component and during a fingerprint authentication phase, movement of a finger. The operations further include capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smear of the fingerprint, the fingerprint sensor being configured to scan a fingerprint of the finger.The operations further include reconstructing, based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstruction reduces the smearing of the fingerprint to make it recognizable by a fingerprint matching component. The operations additionally include recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint to a stored fingerprint template.

[0007] In a fourth aspect, a system is provided. The system includes means for detecting, by a display component and during a fingerprint authentication phase, a movement of a finger; means for detecting, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smearing of the fingerprint, the fingerprint sensor being configured to sample a fingerprint of the finger; means for reconstructing, based on the movement of the finger and an estimated fingerprint distortion, an unsmeared fingerprint from the fingerprint data, wherein the reconstruction reduces the smearing of the fingerprint to make it detectable by a fingerprint matching component;and means for recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises comparing the reconstructed fingerprint with a stored fingerprint template;

[0008] Other aspects, embodiments and implementations will become apparent to those skilled in the art upon reading the following detailed description, with reference, where appropriate, to the accompanying drawings. SHORT DESCRIPTION OF THE CHARACTERS Fig. 1 illustrates a computing device for blur detection according to example embodiments. Fig. 2A is an example block diagram illustrating fingerprint recognition by removing blur from an image, according to example embodiments. Fig. 2B illustrates example pixel configurations for motion detection according to example embodiments. Fig. 2C is an example block diagram illustrating fingerprint recognition by removing blur from an image using a machine learning model, according to example embodiments. Fig. 2D is an example block diagram illustrating fingerprint recognition using a matching model based on machine learning, according to example embodiments. Fig. 2E is an example block diagram illustrating fingerprint recognition using a matching and deception detection model based on machine learning, according to example embodiments. Fig. 2F is an example block diagram illustrating fingerprint recognition by removing blurring of an image based on force detection, according to example embodiments. Fig. 3 is a diagram illustrating training and inference phases of a machine learning model, according to example embodiments. Fig. 4 illustrates a distributed computing architecture according to example embodiments. Fig. 5 illustrates a network of computing clusters arranged as a cloud-based server system, according to example embodiments. Fig. 6 illustrates a method according to exemplary embodiments. DETAILED DESCRIPTION

[0009] Example methods, devices, and systems are described herein. It should be understood that the terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." An embodiment or feature described as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments. Other embodiments may be utilized and other changes may be made without departing from the scope of the subject matter presented herein.

[0010] Therefore, the exemplary embodiments described herein are not to be considered limiting. Aspects of the present disclosure, as generally described herein and illustrated in the figures, may be implemented, substituted, combined, separated, and configured in a wide variety of different configurations, all of which are contemplated herein.

[0011] Furthermore, unless the context suggests otherwise, the features illustrated in the individual figures may be used in combination with one another. Therefore, the figures should generally be considered component aspects of one or more overall embodiments, with the understanding that not all illustrated features are required for every implementation. Overview

[0012] Human fingerprints are detailed, virtually unique, difficult to alter, and last a lifetime. Fingerprint authentication can be used to enable individuals to gain access to secure devices, locations, and software features, such as a user device, a door entry, a safe, application software, and so on. During a fingerprint authentication phase, a fingerprint scanner scans a fingerprint and processes the scanned image for validation purposes. A user can place their finger on the fingerprint scanner. However, various factors can cause the scanned image to become blurred. For example, motion blur can be caused by finger movement, device movement, or both. Furthermore, poor lighting conditions, for example, can negatively impact the quality of the scanned image.In some situations, the pressure exerted by the finger on the scanner can cause distortion of the scanned image. In some other situations, image compression can cause errors in the scanned image. Furthermore, the finger may only be partially scanned due to, for example, the finger's placement relative to the fingerprint scanner.

[0013] The scanned image of a fingerprint can be compared to an existing fingerprint template. For example, during a fingerprint enrollment phase, the device can scan multiple images of the finger. For example, the user can be instructed to place the finger at specific locations on the display component, rotate the finger in different directions, at different speeds, with different pressure, etc. Such scanned images can then be stored in a database for the fingerprint authentication phase. During the fingerprint authentication phase, a scanned fingerprint can be compared to a stored fingerprint to determine if there is a match.

[0014] Conventional matching methods rely on the fine-grained features of a fingerprint. Such approaches are difficult to implement in situations where only small and / or incomplete fingerprint images are available. Some conventional techniques rely on pattern matching using Scale Invariant Feature Transform (SIFT). This has limited features, and the latency associated with this approach cannot be optimized through hardware acceleration.

[0015] A fingerprint scanner on a device can be configured to detect finger movements. Various sensors can be used, such as an optical sensor, an ultrasonic sensor, a direct pressure sensor, a capacitive sensor, a thermal sensor, etc. In general, a fingerprint on the device may not be stable, and the device may fail to identify the fingerprint, delaying or aborting the fingerprint authentication process. For example, finger movement during scanning can blur the captured image. Blurred fingerprints can cause the fingerprint recognition system to reject the verification attempt.

[0016] Conventional fingerprint scanners aren't equipped to detect smearing. Theoretically, smearing can be corrected with high frame rates and image stacking. Pixelflow techniques can also be used to estimate finger movement and correct the image. However, these can be resource-intensive.

[0017] Correcting a smudged fingerprint may involve considering various factors, such as ambient light intensity, variations in individual fingerprints, variations due to display components, variations due to different types of sensors, the amount of motion, the direction of motion, the amount of pressure applied to a display component, temperature changes, etc. Accordingly, performing such operations on a mobile device in near real-time can be a challenging task. Such a task can be achieved by employing a combination of hardware accelerators, on-device machine learning models, and / or enhanced image processing techniques.

[0018] Some techniques described in this paper address these problems by providing efficient methods for removing smearing from a scanned fingerprint, enabling a faster, more efficient, and more accurate fingerprint authentication process. In addition, anti-spoofing techniques, for example, can be implemented. Such operations can be performed in near real time on a mobile device, resulting in a significant improvement in the security of the device, data, and applications. Other advantages are also considered and will be apparent from the discussion provided in this paper. Example devices

[0019] Fig. 1 illustrates a computing device 100 according to example embodiments. The computing device 100 includes a display component 110, a fingerprint reconstruction module 120, one or more ambient light sensors 130, one or more fingerprint sensors 140, one or more other sensors 150, a network interface 160, a controller 170, a fingerprint matching component 180, and a motion / force detection component 190. In some examples, the computing device 100 may take the form of a desktop device, a server device, or a mobile device. The computing device 100 may be configured to interact with an environment. For example, the computing device 100 may obtain fingerprint information from an environment of the computing device 100. In addition, the computing device 100 may, for example, obtain environmental state measurements associated with an environment around the computing device 100 (e.g.,ambient light measurements, etc.).

[0020] The display component 110 may be configured to provide output signals to a user via one or more screens (including touchscreens), cathode ray tubes (CRTs), liquid crystal displays (LCDs), light-emitting diodes (LEDs), displays using digital light processing (DLP) technology, and / or other similar technologies. The display component 110 may also be configured to generate audible output, such as via a speaker, a speaker jack, an audio output port, an audio output device, headphones, and / or other similar devices.The display component 110 may be further configured with one or more haptic components that can generate haptic outputs, such as vibrations and / or other outputs detectable by touch and / or physical contact with the computing device 100.

[0021] In example embodiments, the display component 110 is configured to operate at a given brightness level. The brightness level may correspond to an operation performed by the display component. For example, if an under display fingerprint sensor (UDFPS) is enabled, the display component 110 may operate at a brightness of 800 or 900 nits. In example embodiments, the display component 110 may operate at a low brightness level of 2 nits to account for low ambient light intensity. In some other examples, the display component 110 may operate at a normal brightness level of 500 nits.

[0022] In certain embodiments, display component 110 may be a color display that utilizes a variety of color channels to generate images. For example, display component 110 may utilize, among others, the red, green, and blue (RGB) color channels or the cyan, magenta, yellow, and black (CMYK) color channels.

[0023] In some embodiments, display component 110 may include a plurality of pixels arranged in a pixel array defining a plurality of rows and columns. For example, if display component 110 had a resolution of 1024×500, each column of the array could include 500 pixels, and each row of the array could include 1024 groups of pixels, with each group including one red, one blue, and one green pixel, for a total of 3072 pixels per row. In example embodiments, the color of a particular pixel may depend on a color filter disposed over the pixel.

[0024] In example embodiments, display component 110 may receive signals from its pixel array. The signals may indicate movement. For example, a digital image of a fingerprint may include various image pixels corresponding to respective pixels of display component 110. Each pixel of the digital image may have a numerical value representing the luminance (e.g., lightness or darkness) of the digital image at a particular location. These numerical values ​​may be referred to as "gray levels." The number of gray levels may depend on the number of bits used to represent the numerical values. For example, if 8 bits were used to represent a numerical value, display component 110 could provide 256 gray levels, where a numerical value of 0 corresponds to pure black and a numerical value of 255 corresponds to pure white.

[0025] The fingerprint reconstruction module 120 may be configured with logic that compensates for inaccuracies that occur due to an error in fingerprint scanning. For example, movement of a fingerprint may result in motion blur in the fingerprint image. Also, for example, a pressure on the display component 110 may result in smearing of the fingerprint image. The fingerprint reconstruction module 120 may be configured with logic to reconstruct an unsmeared fingerprint image that may be used by a fingerprint matching component. In some embodiments, the fingerprint reconstruction module 120 may include one or more machine learning algorithms that perform smear removal. The fingerprint reconstruction module 120 may share one or more aspects with the smear removal components described herein (e.g.,with reference to . Fig. 2A-2F).

[0026] The ambient light sensor(s) 130 may be configured to receive light from the environment of the computing device 100 (e.g., within a distance of 1 meter (m), 5 m, or 10 m). The ambient light sensor(s) 130 may include one or more single-photon avalanche detectors (SPADs), avalanche photodiodes (APDs), complementary metal oxide semiconductor (CMOS) detectors, and / or charge-coupled devices (CCDs). For example, ambient light sensors 130 may include indium gallium arsenide (InGaAs) APDs configured to detect light with wavelengths around 1550 nanometers (nm). Other types of ambient light sensor(s) 130 are possible and are contemplated herein.

[0027] In some embodiments, the ambient light sensors 130 may include a plurality of photodetector elements arranged in a one-dimensional or two-dimensional array. For example, the ambient light sensor(s) 130 may include sixteen detector elements arranged in a single column (e.g., a linear array). The detector elements may be arranged along, or at least parallel to, a primary axis.

[0028] In some embodiments, computing device 100 may include one or more fingerprint sensors 140. In some embodiments, fingerprint sensor(s) 140 may include one or more image capture devices capable of capturing an image of a finger. Fingerprint sensor(s) 140 is used to authenticate a fingerprint. The image of the finger captured by one or more image capture devices is compared to a stored image for authentication purposes. Light from display component 110 is reflected back from the finger to fingerprint sensor(s) 140. There may be some loss of light originating from the display and some loss due to a small reflection.Generally, a high brightness level is required to sufficiently illuminate the finger to meet SNR requirements and avoid loss from the display and / or reflection. In some embodiments, the fingerprint sensor(s) 140 is / are configured with a time threshold within which the authentication process must be completed. If the authentication process is not completed within the time threshold, the authentication process fails. In some embodiments, authentication may fail due to defects in the scanned fingerprint. For example, a smudged fingerprint may be unidentifiable. In some embodiments, the display component 110 may attempt to reauthenticate the fingerprint. Such repetitive authentication processes may cause high power consumption.

[0029] The fingerprint sensor(s) 140 may include optical sensors, ultrasonic sensors, and / or capacitive sensors. For example, an under-display fingerprint sensor (UDFPS) is an optical sensor laminated beneath a display component 110 of the computing device 100. For the USFPS to function for fingerprint authentication, the light emitted by the display component 110 is reflected back to the sensor by a finger to be authenticated. Generally, the display component 110 may operate in a normal mode, corresponding to a low brightness level. In some embodiments, the display component 110 may switch to a high brightness mode to enable fingerprint scanning and recognition.

[0030] In some embodiments, the computing device 100 may include one or more other sensors 150. The other sensor(s) 150 may be configured to measure conditions within the computing device 100 and / or conditions in an environment (e.g., within 1 m, 5 m, or 10 m) of the computing device 100 and to provide data about those conditions. For example, the other sensor(s) 150 may include one or more of the following: (i) sensors for obtaining data about the computing device 100, such as, but not limited to, a thermal sensor for measuring thermal activity at or near the computing device 100, a thermometer for measuring a temperature of the computing device 100, a battery sensor for measuring the performance of one or more batteries of the computing device 100, and / or other sensors for measuring the condition of the computing device 100;(ii) an identification sensor for identifying other objects and / or devices, such as, but not limited to, a radio frequency identification (RFID) reader, a proximity sensor, a one-dimensional barcode reader, a two-dimensional barcode reader (e.g., quick response (QR) code), and / or a laser tracker, wherein the identification sensor may be configured to read identifiers, such as RFID tags, barcodes, QR codes, and / or other devices and / or objects configured to read them, and to provide at least identifying information; (iii) sensors for measuring locations and / or movements of the computing device 100, such as, but not limited to, a tilt sensor, a gyroscope, an accelerometer, a Doppler sensor, a global positioning system (GPS) device, a sonar sensor, a radar device, a laser displacement sensor, and / or a compass;(iv) an environmental sensor for obtaining data indicative of an environment of the computing device 100, such as, but not limited to, an infrared sensor, an optical sensor, a biosensor, a capacitive sensor, a touch sensor, a temperature sensor, a wireless sensor, a radio sensor, a motion sensor, a proximity sensor, a radar receiver, a microphone, a sound sensor, an ultrasonic sensor, and / or a smoke sensor; (v) a pressure sensor for measuring an amount of pressure exerted by a finger on the display component 110 during fingerprint enrollment and / or authentication;and / or (vi) a force sensor for measuring one or more forces (e.g., inertial forces and / or G-forces) acting on the computing device 100, such as, but not limited to, one or more sensors that measure forces in one or more dimensions, torque, ground force, friction, and / or a zero moment point (ZMP) sensor that identifies ZMPs and / or locations of the ZMPs. Many other examples of other sensors 150 are also possible.;

[0031] Data collected from ambient light sensors 130, fingerprint sensors 140, and other sensors 150 may be communicated to controller 170, which may use the data to perform one or more actions.

[0032] Network interface 160 may include one or more wireless interfaces and / or wired interfaces configurable for communication over a network. Wireless interfaces may include one or more wireless transmitters, receivers, and / or transceivers, such as a Bluetooth™ transceiver, a Zigbee® transceiver, a Wi-Fi™ transceiver, a WiMAX™ transceiver, an LTE™ transceiver, and / or other similar types of wireless transceivers configurable for communication over a wireless network.Wired interfaces may include one or more wired transmitters, receivers, and / or transceivers, such as an Ethernet transceiver, a Universal Serial Bus (USB) transceiver, or a similar transceiver that is configurable to communicate over a twisted pair cable, a coaxial cable, a fiber optic connection, or a similar physical connection to a wired network.

[0033] In some embodiments, the network interface 160 may be configured to enable reliable, secure, and / or authenticated communication. For each communication described herein, information to facilitate reliable communication (e.g., guaranteed message delivery) may be provided, possibly as part of a message header and / or footer (e.g., packet or message sequencing information, encapsulation headers and / or footers, size or timing information, and transmission verification information such as cyclic redundancy check (CRC) and / or parity check values). Communications may be secured using one or more cryptographic protocols and / or algorithms, including, but not limited to, the following (e.g.,encoded or encrypted) and / or decrypted / decoded using: a Data Encryption Standard (DES), Advanced Encryption Standard (AES), Rivest-Shamir-Adelman (RSA) algorithm, Diffie-Hellman algorithm, a secure sockets protocol such as Secure Sockets Layer (SSL) or Transport Layer Security (TLS), and / or a Digital Signature Algorithm (DSA). To secure (and subsequently decrypt or decode) the communication, another cryptographic protocol and / or algorithm may be used in addition to, or instead of, the cryptographic protocols and / or algorithms listed herein.

[0034] Computing device 100 may include a user interface module (not shown) operable to send and / or receive data to and / or from external user input / output devices. For example, the user interface module may be configured to send and / or receive data to and / or from user input devices such as a touch screen, a computer mouse, a keyboard, a keypad, a touchpad, a trackball, a joystick, a speech recognition module, and / or other similar devices.The user interface module may also be configured to provide output to user display devices, such as one or more cathode ray tubes (CRTs), liquid crystal displays, light-emitting diodes (LEDs), displays using digital light processing (DLP) technology, printers, light bulbs, and / or other similar devices, either now known or later developed. The user interface module may be configured to generate audible output, for example, via devices such as a speaker, a speaker jack, an audio output port, an audio output device, headphones, and / or other similar devices.The user interface module may be further configured with one or more haptic devices that can generate haptic outputs, such as vibrations and / or other signals detectable by touch and / or physical contact with the computing device 100. In some examples, the user interface module may be used to provide a graphical user interface (GUI) for using the computing device 100. For example, the user interface module may be used to provide instructions to a user during the fingerprint enrollment phase to guide the user in successfully enrolling their fingerprint.

[0035] Computing device 100 may also include a power system (not shown). The power system may include one or more batteries and / or one or more external power interfaces for providing electrical power to computing device 100. Each of the one or more batteries, when electrically coupled to computing device 100, may function as a source of stored electrical power for computing device 100. The one or more batteries of the power system may be configured to be portable. Some or all of the one or more batteries may be readily removable from computing device 100. In other examples, some or all of the one or more batteries may be located within computing device 100 and thus may not be readily removable from computing device 100.Some or all of the one or more batteries may be rechargeable. For example, a rechargeable battery may be recharged via a wired connection between the battery and another power supply, such as one or more power supplies located external to the computing device 100 and connected to the computing device 100 via the one or more external power interfaces. In other examples, some or all of the one or more batteries may be non-rechargeable batteries.

[0036] One or more external power interfaces of the power system may include one or more wired power interfaces, such as a USB cable and / or a power cable, enabling wired power connections to one or more power supplies located external to the computing device 100. The one or more external power interfaces may include one or more wireless power interfaces, such as a Qi wireless charger, enabling wireless electrical power connections, such as via a Qi wireless charger, to one or more external power supplies.Once an electrical power connection to an external power source is established using the one or more external power interfaces, computing device 100 may draw electrical power from the external power source using the established electrical power connection. In some examples, the power system may include related sensors, such as battery sensors associated with one or more batteries or other types of electrical power sensors.

[0037] The controller 170 may include one or more processors 172 and a data memory 174. Processor(s) 172 may include one or more general-purpose processors and / or one or more special-purpose processors (e.g., display driver integrated circuits (DDICs), digital signal processors (DSPs), tensor processing units (TPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), etc.). The processor(s) 172 may be configured to execute computer-readable instructions contained in the data memory 174 and / or other instructions as described herein.

[0038] The data storage 174 may include one or more non-transitory, computer-readable storage media that can be read and / or accessed by the processor(s) 172. The one or more non-transitory, computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic, or other data storage or disk storage, which may be integrated in whole or in part with at least one of the processors 172. In some examples, the data storage 174 may be implemented using a single physical device (e.g., an optical, magnetic, organic, or other data storage or disk storage device), while in other examples, the data storage 174 may be implemented using two or more physical devices.

[0039] The data store 174 may include computer-readable instructions and possibly additional data. In some examples, the data store 174 may include storage required to perform at least some of the methods, scenarios, and techniques described in this document and / or at least some of the functionality of the devices and networks described herein. In some examples, the data store 174 may include storage for a trained neural network model (e.g., a trained neural network model such as the networks described in this document). The data store 174 may also be configured to store a fingerprint template after enrollment.

[0040] In example embodiments, the processor(s) 172 is / are configured to execute instructions stored in the data memory 174 to perform operations.

[0041] The operations may include detecting a movement of the finger by the fingerprint sensor 140 during a fingerprint authentication phase.

[0042] The operations may further include capturing, by a fingerprint sensor of the display component 110, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smearing of the fingerprint.

[0043] The operations may further include reconstructing, by the fingerprint reconstruction module 120 and based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstructing reduces the blurring of the fingerprint to make it recognizable by a fingerprint matching component.

[0044] The operations may further include recognizing the fingerprint by the fingerprint matching component 180, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint with a stored fingerprint template.

[0045] The fingerprint matching component 180 may be configured with logic that compares a scanned fingerprint to a stored fingerprint template. For example, the fingerprint matching component 180 may be configured with logic to identify one or more features of a fingerprint, such as bumps, lines, patterns, depressions, scars, etc., and store these features as a fingerprint template. For this purpose, the fingerprint matching component 180 may include a data store for storing the one or more features.

[0046] The motion / force detection component 190 may include circuitry and / or logic that could detect motion and / or force applied to the display component 110. For example, the motion / force detection component 190 may receive pixel values ​​from an array of pixels and detect motion. Furthermore, the motion / force detection component 190 may, for example, calculate optical flow based on multiple consecutive frames and detect motion based on the optical flow. Furthermore, the motion / force detection component 190 may, for example, receive data from a thermal sensor and detect motion based on thermal activity. As another example, the motion / force detection component 190 may receive data from a pressure sensor and / or a fingerprint sensor to determine an amount of pressure a finger has applied to the display component 110.

[0047] Fig. 2A is an example block diagram illustrating fingerprint recognition by removing blurring from an image, according to example embodiments. Some embodiments include capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, where movement of the finger caused the fingerprint to blur. For example, during the fingerprint authentication phase, the fingerprint scanner 205 may capture one or more images of a fingerprint associated with a finger and generate one or more frames 215, such as frame 1, frame 2, ..., frame N.

[0048] The fingerprint scanner 205 may include a sensor, such as an optical sensor, an ultrasonic sensor, a direct pressure sensor, a capacitive sensor, a thermal sensor, etc. The fingerprint sensor may capture multiple frames and send them to an internal buffer. With ultrasonic sensors, multiple frames may be captured at specific intervals, and the final image may be reconstructed using a diffraction model.

[0049] Although the example techniques are described with reference to optical images, this is for illustrative purposes only. For example, image data based on one or more individual images 215 may be replaced with data from a capacitive sensor, a thermal sensor, etc.

[0050] A finger movement detector 210 can detect movement of the finger. In general, the movement of the finger can result in smearing of the fingerprint. For example, the one or more individual images 215 can exhibit motion blur. This can cause the fingerprint data to represent a smeared fingerprint, which can be unreliable for fingerprint authentication. The term "smear" / "smearing" as used herein can generally refer to image distortion or degradation that causes an error in the captured fingerprint image, making it difficult to identify by a fingerprint recognition system. For example, a smeared fingerprint can exhibit motion blur caused by the movement of the finger, the device, or both.In addition, even under ideal shooting conditions, inherent blur can occur due to sensor resolution, light diffraction, display errors, pixel saturation, and anti-aliasing filters. Similarly, image noise is also inherent in the capture of an image.

[0051] The term "image noise" used in this paper generally refers to blurring that causes an image to appear to have artifacts (e.g., specks, colored dots, etc.) resulting from a lower signal-to-noise ratio (SNR). For example, an SNR below a certain desired threshold can cause image noise. In some examples, image noise may occur due to an image sensor. Generally, when images are compressed before storage or transmission, e.g., using JPEG compression, such image compression artifacts can also degrade image quality. The term "image compression artifact" used in this paper generally refers to a degradation factor resulting from lossy image compression.For example, image data may be lost during compression, resulting in visible artifacts in a decompressed version of the image.

[0052] The term "pixel saturation" as used in this document generally refers to a state in which pixels are saturated with photons, and the photons then spill over to neighboring pixels. For example, a saturated pixel may be associated with an image intensity that is above a threshold intensity (e.g., above 245 or at 255, etc.). The image intensity may correspond to the intensity of a gray level or the intensity of a color component in red, blue, or green (RGB). For example, highly saturated pixels may appear brightly colored. Accordingly, spillover of photons from saturated pixels to neighboring pixels may cause perceptual errors in an image (e.g., causing one or more neighboring pixels to become saturated, distorting the intensity of one or more neighboring pixels, etc.).

[0053] Some embodiments may include detecting a movement of the finger by a display component and during the fingerprint authentication phase. For example, the finger movement detector 210 may detect the movement of the finger. In some embodiments, a motion vector representing the movement of the finger may be generated. For example, the displacement between respective pixels in two consecutive frames in the one or more frames 215 may be represented by a motion vector. Additionally, for example, one or more feature vectors may be generated for input to the various machine learning models described herein.

[0054] In some embodiments, detecting finger movement may be based on the respective pixel values ​​of one or more pixels in a pixel array of an image of the fingerprint. In some embodiments, the one or more pixels in the pixel array may be dedicated to motion detection. For example, when a finger moves, the fingerprint scanner 205 may capture images with different pixel values. The change in pixel values ​​in the one or more individual images 215 reflects the movement and may be processed to detect the movement.

[0055] Fig. Figure 2B illustrates example pixel configurations for motion detection according to example embodiments. Image 255 illustrates an example pixel arrangement where motion detection pixels 270 (represented by white squares) are located at the four corners and in the center of the pixel array, while the remaining portion of the pixel array is occupied by image capture pixels 265 (represented by white squares). Figure 260 illustrates an example pixel array where motion detection pixels 270 (represented by white squares) are located along the boundaries of the pixel array and surround image capture pixels 265 (represented by white squares) located in the central region of the pixel array. In some embodiments, the one or more pixels in the pixel array may be randomly distributed within the pixel array.

[0056] In some embodiments, the detection of finger movement is performed using an application-specific integrated circuit (ASIC) of the device. For example, a tensor processing unit (TPU) may include a single-chip ASIC with specialized algorithms for motion detection.

[0057] In some embodiments, the display component includes a touch-sensitive display panel, the fingerprint sensor may be an under-display fingerprint sensor (UDFPS), and includes the method of determining a thermal image indicative of movement on or near the touch-sensitive display panel. In some embodiments, the device includes a thermal sensor configured to detect thermal activity on or near the device and includes the method of determining a thermal image based on the detected thermal activity. In such embodiments, detecting movement of the finger may be based on a thermal image. A thermal image (or thermal imaging) identifies areas of interest by associating different thermal properties of the finger with different colors, such as colors ranging from red to blue.Generally, red, yellow, and orange represent regions with higher temperatures, while blue and green represent regions with lower temperatures. Temporary thermal imaging can enable detection of finger movement.

[0058] In some embodiments, the fingerprint sensor may be a capacitive fingerprint sensor. Detecting the movement of the finger may be based on a capacitive region of the display component. In general, capacitive fingerprint scanners may include an array of capacitor circuits to collect data about a fingerprint. The capacitors store electrical charge, which changes when a bump of a finger is placed on a conductive plate of the capacitive sensor. However, the electrical charge remains unchanged if an air gap is present. The changes in electrical charge may be tracked with an integrator circuit and recorded with an analog-to-digital converter (ADC). The captured fingerprint data may be processed to analyze features of the fingerprint.Motion detection can process changes in electrical charges over time to detect finger movement.

[0059] In some embodiments, the fingerprint sensor may be an ultrasonic fingerprint sensor. An ultrasonic pulse may be sent by the sensor against the finger placed on the fingerprint scanner 205. Generally, based on a location and / or configuration of the bumps, valleys, pores, scars, bumps, and other details unique to each fingerprint, a portion of the pulse may be absorbed and another portion may be reflected back to the sensor. In some embodiments, the fingerprint data generated by the sensor over time may be processed to detect finger movement.

[0060] Some embodiments include determining optical flow based on one or more images of the fingerprint. Detection of finger movement may be based on optical flow. For example, the finger may be tracked within one or more frames 215 to determine movement. Some optical flow techniques may be gradient-based. In general, displacement and / or velocity of finger movement may be determined using optical flow. Various methods may be used, such as phase correlation, differential methods (e.g., Lucas-Kanade, Hon-Schunk, Buxton-Buxton, etc.), and / or discrete optimization methods. In some embodiments, the device may include an optical sensor configured to measure optical flow or visual movement of the finger.For example, the optical flow sensor may be communicatively coupled to the ASIC, which includes one or more algorithms for detecting motion based on the optical flow measurements. In some embodiments, neuromorphic circuits may be implemented in an optical sensor to respond to optical flow.

[0061] Some embodiments include reconstructing, based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstruction reduces the smearing of the fingerprint to make it detectable by a fingerprint matching component. For example, a smear removal component 220 may be configured to reconstruct the non-blurred fingerprint from the fingerprint data (e.g., image data of images in the one or more frames 215) based on the motion data of the finger motion detector 210. The smear removal component 220 may reduce image distortions caused by finger movements, sensor resolution, light diffraction, display errors, pixel saturation, and / or anti-aliasing filters. Similarly, image noise may be reduced.Artifacts caused by pixel saturation and / or compression may be reduced and / or removed. Additional and / or alternative image enhancement techniques may be applied to reconstruct the unblurred image. For example, brightness correction may be applied to compensate for distortions due to low lighting conditions. In some embodiments, a geometric distortion map may be applied to correct the geometric properties of bumps and valleys in the fingerprint data.

[0062] In some embodiments, frame stacking techniques can be used to interpolate the reconstructed fingerprint from one or more frames 215. Frame interpolation is the process of synthesizing intermediate images from a given set of images. The technique can be used for temporal upsampling. For example, the sensor can capture images at a high frame rate, and frame interpolation can be used to interpolate between these nearly identical images.

[0063] In some embodiments, the estimated fingerprint distortion may indicate a baseline amount of smearing. For example, the estimated fingerprint distortion may be based on a normalized amount of smearing based on a plurality of users, a plurality of devices, or both. Some fingerprint distortion may be expected based on an individual finger, a device, an amount of light incidence, sensor configurations, etc. Accordingly, a baseline amount of smearing may be determined. In some embodiments, the estimated fingerprint distortion may be determined during the fingerprint enrollment phase. For example, at the time a user goes through the enrollment process, the device may determine the geometry of the finger, including specific configurations unique to the individual (e.g.,geometric features, arrangement of bumps and valleys, scars, etc.), an amount of pressure applied by the user, the manner in which the finger is moved from left to right to generate an impression of the fingerprint, and so on. In addition, device- and / or sensor-specific properties, for example, may be retrieved and stored to generate the baseline. In some embodiments, a machine learning model (e.g., one or more of the models described herein or a standalone distortion model) may be trained to predict the estimated fingerprint distortion. The machine learning model may, for example, predict possible variations of input fingerprint data. In some embodiments, training data may include a plurality of pairs of fingerprints and associated smeared fingerprints.The smeared fingerprints may be real data corresponding to fingerprint smearing (e.g., due to movement, pressure, perspiration, etc.) and / or synthetic data simulating fingerprint smearing due to movement, pressure, perspiration, etc. Furthermore, one or more geometric transformations may be applied to the fingerprint data to determine the estimated fingerprint distortion. The one or more geometric transformations may include, for example, rotations, translations, skews, contractions, expansions, etc., which may be applied to transform relative configurations of fingerprint features, such as bumps, pores, depressions, scars, etc.

[0064] In some embodiments, the estimated fingerprint distortion may be predicted by a machine learning model. For example, a machine learning model may be trained based on training data related to the finger, device, sensor configurations, varying light intensities, image degradations, and so on, to determine the estimated fingerprint distortion. In some embodiments, the estimated fingerprint distortion may be determined during the fingerprint authentication phase. For example, statistical properties based on the fingerprint data, the motion data, and so on may be determined to determine the estimated fingerprint distortion. In some embodiments, a trained machine learning model may be used to derive the estimated fingerprint distortion during the fingerprint enrollment phase.

[0065] Some embodiments include recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint with a stored fingerprint template. For example, a fingerprint matching component may include a fingerprint matching component 225 that may be configured to receive the reconstructed fingerprint (e.g., modified fingerprint data corresponding to the reconstructed fingerprint) and compare the respective features of the reconstructed fingerprint with the stored fingerprint template. In some embodiments, a similarity threshold may be determined at which the reconstructed fingerprint and the stored fingerprint template are determined to be a match if the respective feature sets are determined to be similar within the similarity threshold.In some embodiments, the fingerprint matching component 225 may determine a match score indicating the degree of match. For example, a higher match score may indicate a higher degree of match between the reconstructed fingerprint and the stored fingerprint template. Furthermore, for example, a lower match score may indicate a lower degree of match between the reconstructed fingerprint and the stored fingerprint template. In some embodiments, a match threshold may be used to determine whether there is a match. For example, the match threshold may be 70%, and a match score exceeding 70% may be determined to indicate a match.

[0066] Fig. 2C is an example block diagram illustrating fingerprint recognition by removing blur from an image using a machine learning model, according to example embodiments. Some embodiments include applying a machine learning model to perform reconstruction of the unblurred fingerprint. For example, image data of a single frame 230 and motion data from the finger motion detector 210 may be received by a machine learning-based blur removal model 235. The machine learning-based blur removal model 235 may be trained on various types of training data to reconstruct an unblurred image.For example, the training data may include a plurality of pairs of first data relating to blurred fingerprints, along with second data relating to respective non-blurred versions of the fingerprints, as well as corresponding motion data that caused the blurring. The machine learning-based blur removal model 235 may be trained on such training data to obtain fingerprint data for a blurred fingerprint and corresponding motion data to predict the non-blurred image. In some embodiments, conventional architectures may be used for the machine learning-based blur removal model 235.In some embodiments, the machine learning-based blur removal model 235 may include an image enhancement neural network trained to enhance optical images by removing image distortions due to motion blur, pixel saturation, image compression artifacts, and so on.

[0067] In some embodiments, the ASIC may be configured to accelerate inference for the machine learning-based blur removal model 235, such as one or more deep learning models. This is particularly useful for fingerprint recognition, where accurate recognition must be performed in real time on the device. Performing inference on the device also enables improved security for the device, as the data can remain on the device rather than being sent to a cloud server hosting the machine learning model. In some embodiments, the TPU may be an entire system, including custom ASIC chips, boards, and interconnects, configured to accelerate both training and inference for the machine learning-based blur removal model 235.As previously described, the fingerprint matching component 225 may perform a comparison between the reconstructed fingerprint and a stored fingerprint template.

[0068] Fig. 2D is an example block diagram illustrating fingerprint recognition using a matching model based on machine learning, according to example embodiments. Some embodiments include applying a machine learning model to perform the reconstruction of the unsmudged fingerprint and the recognition of the fingerprint. For example, a machine learning model for smear removal and matching 240 may be used to perform the operations of the smear removal component 220 and the fingerprint matching component 225. In some embodiments, the machine learning model for smear removal and matching 240 may consist of two separate models.For example, a first model may share one or more features with the machine learning-based blur removal model 235, and a second model may be a machine learning-based classifier that performs the operation of the fingerprint matching component 225. For example, a classifier may be trained to compare the reconstructed fingerprint to a plurality of fingerprint templates to determine if a match exists. In some embodiments, a binary classifier may perform such a matching task.

[0069] In some embodiments, the smear removal and matching machine learning model 240 may be a standalone model trained to predict a match between the reconstructed fingerprint and a stored fingerprint template based on the fingerprint data and the motion data, along with the estimated fingerprint distortion. In embodiments where the estimated fingerprint distortion is calculated by a machine learning model, such a model may operate independently or as part of the smear removal and matching machine learning model 240.

[0070] Fig. Figure 2E is an example block diagram illustrating fingerprint recognition using a matching and deception detection model based on machine learning, according to example embodiments. The term "deception detection" as used herein generally refers to techniques for detecting whether a forged fingerprint is being presented for verification. For example, a replica (e.g., engraved into a mold to create an impression of a fingerprint) or a digital impression of a finger or a portion thereof may be created and presented for verification, and the deception protection technique would be configured to detect that the fingerprint data does not correspond to an actual fingerprint. For some forms of deception, a sensor may be configured to modify the sampled data.In some aspects, machine learning-based techniques can be used to synthesize human fingerprints for deception attacks.

[0071] Some approaches to anti-spoofing may include instructing the user to perform one or more of dragging their finger toward the sensor, applying additional pressure, rotating their finger in a specified manner, and so on, to cause an intentionally smeared fingerprint. Such a smeared fingerprint may be compared to existing user data to detect spoofing. In some embodiments, a smeared fingerprint may have the smear removed and be compared to additional existing user data to detect spoofing.

[0072] Some anti-spoof approaches may also include hardware-based deception detection based on fingerprint characteristics, such as thermal properties, electrical charge levels, skin resistance, pulse oximetry, and so on. Additionally, anti-spoof approaches may include, for example, software-based deception detection. For example, the one or more individual images 215 may be processed to detect real-time distortions, perspiration changes, thermal image changes, capacitive changes, and so on.

[0073] Accordingly, some embodiments include applying a machine learning model to perform reconstruction of the unsmudged fingerprint, recognition of the fingerprint, and performance of deception detection. For example, the de-smear, matching, and deception detection machine learning model 245 may be configured to perform the operations of the de-smear component 220 and the fingerprint matching component 225 with de-smear protection algorithms for de-smear detection. The de-smear, matching, and de-smear detection machine learning model 245 may be three separate models or a combination of two or more models, each performing operations that combine at least two of de-smear, matching, and de-smear protection.In some embodiments, the machine learning model for deblurring, matching, and deception detection 245 may be a standalone model trained to perform deblurring, matching, and deception protection.

[0074] A deception detection model may be trained based on a variety of fingerprint data, thermal properties, electrical charge values, skin resistance, pulse oximetry, and so on, for real fingerprints and for synthetically generated data, including data corresponding to physical shapes that fingerprints capture. The deception detection model may be trained to detect a fake fingerprint based on the training data. In some embodiments, the deception detection model may be trained to perform deception detection by comparing an intentionally blurred fingerprint (e.g., when the user is instructed to perform one or more actions such as dragging the finger on the sensor, applying additional pressure, or rotating the finger in a specific manner) with the expected distortions of the user's fingerprint.

[0075] Fig. Figure 2F is an example block diagram illustrating fingerprint recognition by removing blurring from an image based on force detection, according to example embodiments. Some embodiments include detecting pressure applied by a finger through a pressure sensor. Further, these embodiments include measuring an amount of applied pressure. Reconstruction of the unsmudged fingerprint is based on the measured amount of applied pressure. For example, different people may apply different amounts of pressure to a fingerprint scanner. In some situations, the applied pressure may distort the fingerprint data (e.g., an image of a smudged fingerprint may appear smeared).The pressure data received from the fingerprint detector 250 may be combined with the motion data from the fingerprint motion detector 210, the fingerprint data from one or more individual images 215, and the estimated fingerprint distortion to determine the reconstructed image by the blur removal component 220.

[0076] In some embodiments, a machine learning model may be trained to reconstruct an image based on detecting an amount of pressure that may have been applied. For example, the training data may include a plurality of pairs of smeared fingerprints with associated pressure values ​​and non-smeared versions of the fingerprints. A machine learning model may then be trained on such training data to obtain a smeared image and pressure data and predict a non-smeared version of the fingerprint. Such a pressure-based smear removal machine learning model may be combined with the one or more machine learning models described herein (e.g., machine learning-based smear removal model 235, machine learning smear removal and matching model 240, and / or machine learning smear removal, matching, and deception detection model 245).

[0077] Generally speaking, a user may be provided with controls to determine whether and when the systems, programs, or features described herein enable the collection of user information (e.g., information about a user's fingerprint data, race, gender, social network, social contacts or activities, a user's preferences, or a user's current location, and so on) and whether the user is sent content or communications from a server. In addition, certain data may be handled in one or more ways before it is stored or used so that personally identifiable information is removed, secured, and encrypted.For example, a user's identity may be treated in such a way that no user data can be determined for the user, or a user's geographic location may be generalized when location information is obtained (such as at the city, zip code, or state level) so that a specific user location cannot be determined. Thus, the user can have control over what data concerning them is collected, how that information is used, what information is stored (e.g., on the user device, server, etc.), and what information is provided to the user. In addition to user controls in embodiments where user information is used for various aspects of fingerprint recognition, deception detection, etc.used, such user information is restricted to the user's device and is not shared with any server and / or other devices. Furthermore, the user may, for example, have the ability to delete or modify user information. Training machine learning models to generate conclusions / predictions

[0078] Fig. Figure 3 shows a diagram 300 illustrating a training phase 302 and a reasoning phase 304 of a trained machine learning model(s) 332 according to exemplary embodiments. Some machine learning techniques include training one or more machine learning algorithms on an input set of training data to identify patterns in the training data and provide output inferences and / or predictions about (patterns in the) training data. The resulting trained machine learning algorithm may be referred to as a trained machine learning model. For example, Fig. 3 illustrates the training phase 302, in which one or more machine learning algorithms 320 are trained on training data 310 to become a trained machine learning model 332. Then, during the inference phase 304, the trained machine learning model 332 may receive input data 330 (e.g., input fingerprint data, motion data, print data, estimated fingerprint distortion, and so on) and one or more inference / prediction requests 340 (perhaps as part of the input data 330), and in response, provide one or more inferences and / or predictions 350 as output (e.g., predicting an unsmudged version of a fingerprint, predicting whether the input fingerprint data matches a stored fingerprint template, etc.).

[0079] As such, a trained machine learning model(s) 332 may include one or more models of one or more machine learning algorithms 320. The machine learning algorithm(s) 320 may include, but are not limited to, an artificial neural network (e.g., a convolutional neural network, a recurrent neural network, a Bayesian network, a hidden Markov model, a Markov decision process, a logistic regression function, a support vector machine, a suitable statistical machine learning algorithm, and / or a heuristic machine learning system). The machine learning algorithm(s) 320 may be supervised or unsupervised and implement any suitable combination of online and offline learning.Supervised algorithms may include linear regression, decision trees, support vector machines, and / or a naive Bayesian classifier. Unsupervised algorithms may include hierarchical clustering, K-means clustering, self-organizing maps, and / or hidden Markov models.

[0080] Various types of architectures can be used to perform one or more of the fingerprint recognition and / or authentication processes described in this paper. For example, a ResNet architecture, a Generative Adversarial Network (GAN), auto-encoders, recurrent neural networks (RNN), etc.

[0081] In some examples, the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332 may be accelerated using on-device coprocessors such as graphics processing units (GPUs), tensor processing units (TPUs), digital signal processors (DSPs), and / or application-specific integrated circuits (ASICs). Such on-device coprocessors may be used to accelerate the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332. In some examples, the trained machine learning model(s) 332 may bethe trained machine learning models 332 are trained, stored, and executed to provide inferences for a particular computing device and / or otherwise produce inferences for the particular computing device.

[0082] During the training phase 302, the machine learning algorithm(s) 320 may be trained by providing at least training data 310 as training input using unsupervised, supervised, semi-supervised, and / or weakly supervised learning techniques. Unsupervised learning includes providing a portion (or all) of the training data 310 to the machine learning algorithm(s) 320, and the machine learning algorithm(s) 320 determining one or more conclusions based on the provided portion (or all) of the training data 310. Supervised learning includes providing a portion of training data 310 to the machine learning algorithm(s) 320, wherein the machine learning algorithm(s) 320the machine learning algorithms 320 determine one or more conclusions based on the provided portion of training data 310, and the conclusion output(s) is / are either accepted or corrected based on correct results associated with the training data 310. In some examples, the supervised learning of the machine learning algorithm(s) 320 may be guided by a set of rules and / or a set of labels for the training input, and the set of rules and / or the set of labels may be used to correct the conclusions of the machine learning algorithm(s) 320.

[0083] Semi-supervised learning includes having correct labels for a portion, but not all, of the trained data 310. During semi-supervised learning, supervised learning is used for a portion of the training data 310 that has correct results, and unsupervised learning is used for a portion of the training data 310 that does not have correct results. In some examples, the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332 may be trained using other machine learning techniques, including, without limitation, incremental learning and curriculum learning.

[0084] In some examples, the machine learning algorithm(s) 320 and / or the trained machine learning model(s) 332 may employ the use of transfer learning techniques. For example, transfer learning techniques may include pre-training a trained machine learning model(s) 332 on a set of data and additionally training using training data 310. In particular, the machine learning algorithm(s) 320 may be pre-trained on data from one or more computing devices, and a resulting trained machine learning model may be provided to a particular computing device, where the particular computing device is designated to execute the trained machine learning model during the reasoning phase 304.Subsequently, during the training phase 302, the pre-trained machine learning model can be further trained using training data 310, wherein the training data 310 can be derived from kernel and non-kernel data of the particular computing device. This further training of the machine learning algorithm(s) 320 and / or the pre-trained machine learning model using training data 310 of the particular computing device can be performed using either supervised or unsupervised learning. Once the machine learning algorithm(s) 320 and / or the pre-trained machine learning model have been trained on at least training data 310, the training phase 302 can be concluded. The trained resulting machine learning model can be used as at least one of the trained machine learning model(s) 332.

[0085] In particular, the trained machine learning model(s) 332 may be provided to a computing device, if not already present on the computing device, once the training phase 302 is complete. The reasoning phase 304 may begin after the trained machine learning model(s) 332 have been provided to the particular computing device.

[0086] During the inference phase 304, the trained machine learning model(s) 332 may receive input data 330 and generate and output one or more corresponding conclusions and / or predictions 350 about the input data 330. Thus, the input data 330 may be used as input to a trained machine learning model(s) 332 to provide corresponding conclusion(s) and / or prediction(s) 350 for kernel components and non-kernel components. For example, the trained machine learning model(s) 332 may generate the conclusion(s) and / or prediction(s) 350 in response to one or more inference / prediction requests 340. In some examples, the trained machine learning model(s) 332 maythe trained machine learning models 332 are executed by a piece of other software. For example, the trained machine learning model(s) 332 may be executed by a reasoning or prediction background program to provide inferences and / or predictions upon request. Input data 330 may include data from the particular computing device executing the trained machine learning model(s) 332 and / or input data from one or more computing devices other than the particular computing device.

[0087] The input data 330 may include fingerprint data, movement data and / or print data.

[0088] Conclusion(s) and / or prediction(s) 350 may include predicted non-blurred versions, results of anti-deception algorithms, a predicted output of a fitting model, a predicted estimated fingerprint bias, and / or other output data generated by trained machine learning models 332 operated on input data 330 (and training data 310). In some examples, the trained machine learning model(s) 332 may use output conclusion(s) and / or prediction(s) 350 as input feedback 360. The trained machine learning model(s) 332 may also rely on past conclusions as inputs to generate new conclusions.

[0089] A machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and so on, may be examples of machine learning algorithm(s) 320. After training, the trained version of such a neural network may be an example of a trained machine learning model(s) 332. In this approach, an example of an inference / prediction requirement orInference / prediction requirements 340 may be a requirement to predict a non-blurred version of a smeared fingerprint, to predict results of anti-spoofing algorithms, to predict an output of a matching model, to predict an estimated fingerprint distortion, and a corresponding example of inferences and / or predictions 350 may be an output indicating the respective outputs. In some examples, a given computing device may include a trained neural network (e.g., as illustrated in diagram 300), possibly after training the neural network.Then, the given computing device can receive queries to predict whether a non-blurred version of a smeared fingerprint exists, to predict results of anti-deception algorithms, to predict an output of a fitting model, to predict an estimated fingerprint distortion, and so on, and use the trained neural network to generate the prediction.

[0090] In some examples, two or more computing devices may be used to provide the prediction; e.g., a first computing device may generate and send requests to predict an unsmudged version of a smudged fingerprint, to predict results of anti-spoofing algorithms, to predict an output of a fitting model, and to predict an estimated fingerprint distortion. Then, the second computing device may use the trained versions of the neural networks, perhaps after training, to generate the prediction and respond to the requests from the first computing device. Upon receiving the responses to the requests, the first computing device may provide the requested output (e.g., using a user interface and / or a display, a printed copy, electronic communication, etc.). Example data network

[0091] Fig. 4 illustrates a distributed computing architecture 400 according to example embodiments. The distributed computing architecture 400 includes server devices 408, 410 configured to communicate with programmable devices 404a, 404b, 404c, 404d, 404e over a network 406. The network 406 may correspond to a local area network (LAN), a wide area network (WAN), a WLAN, a WWAN, a corporate intranet, the public internet, or any other type of network configured to provide a communication path between networked computing devices. The network 406 may also correspond to a combination of one or more LANs, WANs, corporate intranets, and / or the public internet.

[0092] Although Fig. 4 shows only five programmable devices, distributed application architectures may serve dozens, hundreds, or thousands of programmable devices. Furthermore, the programmable devices 404a, 404b, 404c, 404d, 404e (or any additional programmable devices) may be any type of computing device, such as a mobile computing device, a desktop computer, a wearable computing device, a head-mountable device (HMD), a network terminal, a mobile computing device, and so on. In some examples, such as illustrated by the programmable devices 404a, 404b, 404c, 404e, programmable devices may be directly connected to the network 406.In other examples, such as illustrated by programmable device 404d, programmable devices may be indirectly connected to network 406 via an associated computing device, such as programmable device 404c. In this example, programmable device 404c may act as an associated computing device to relay electronic communications between programmable device 404d and network 406. In other examples, such as illustrated by programmable device 404e, a computing device may be part of and / or located within a vehicle, such as a car, truck, bus, boat or ship, aircraft, etc. In other examples, such as illustrated by programmable device 404e, a computing device may be part of and / or located within a vehicle, such as a car, truck, bus, boat or ship, aircraft, etc. Fig. 4, a programmable device may be connected to the network 406 both directly and indirectly.

[0093] Server devices 408, 410 may be configured to perform one or more services as requested by programmable devices 404a-404e. For example, server device 408 and / or 410 may provide content to programmable devices 404a-404e. The content may include, but is not limited to, web pages, hypertext, scripts, binary data such as compiled software, images, audio data, and / or videos. The content may include compressed and / or uncompressed content. The content may be encrypted and / or unencrypted. Other types of content are also possible.

[0094] As another example, server devices 408 and / or 410 may provide programmable devices 404a-404e with access to software for database, search, computing, graphics, audio, video, World Wide Web / Internet usage, and / or other functions. Many other examples of server devices are also possible. Cloud-based servers

[0095] Fig. 5 illustrates a network 406 of computing clusters 509a, 509b, 509c arranged as a cloud-based server system, according to an exemplary embodiment. The computing clusters 509a, 509b, 509c may be cloud-based devices that store program logic and / or data from cloud-based applications and / or services; e.g., perform at least one function of and / or related to the neural networks and / or method 600.

[0096] In some embodiments, the computing clusters 509a, 509b, 509c may be a single computing device located in a single data center. In other embodiments, the computing clusters 509a, 509b, 509c may include multiple computing devices in a single data center or even multiple computing devices located in multiple data centers located in different geographical locations. For example, Fig. 5 illustrates each of the computing clusters 509a, 509b, and 509c at different physical locations.

[0097] In some embodiments, data and services in the compute clusters 509a, 509b, 509c may be encoded as computer-readable information stored on non-transitory, physical computer-readable media (or computer-readable storage media) and accessible by other computing devices. In some embodiments, the compute clusters 509a, 509b, 509c may be stored on a single disk drive or other physical storage medium, or may be implemented on multiple disk drives or other physical storage media located in one or more different geographic locations.

[0098] Fig. 5 illustrates a cloud-based server system according to an exemplary embodiment. In Fig. 5, the functionality of convolutional neural networks and / or a computing device may be distributed across the computing clusters 509a, 509b, 509c. The computing cluster 509a may include one or more computing devices 500a, cluster storage arrays 510a, and cluster routers 511a connected by a cluster local network 512a. Similarly, the computing cluster 509b may include one or more computing devices 500b, cluster storage arrays 510b, and cluster routers 511b connected by a cluster local network 512b. Likewise, the computing cluster 509c may include one or more computing devices 500c, cluster storage arrays 510c, and cluster routers 511c connected by a cluster local network 512c.

[0099] In some embodiments, each of the compute clusters 509a, 509b, and 509c may include an equal number of compute devices, an equal number of cluster storage arrays, and an equal number of cluster routers. However, in other embodiments, each compute cluster may include a different number of compute devices, a different number of cluster storage arrays, and a different number of cluster routers. The number of compute devices, cluster storage arrays, and cluster routers in each compute cluster may depend on the compute task(s) assigned to each compute cluster.

[0100] In the computing cluster 509a, the computing devices 500a may, for example, be configured to perform various computational tasks of a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device. In one embodiment, the various functions of a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device may be distributed among one or more of the computing devices 500a, 500b, 500c.Computing devices 500b and 500c in respective computing clusters 509b and 509c may be configured similarly to computing devices 500a in computing cluster 509a. On the other hand, in some embodiments, computing devices 500a, 500b, and 500c may be configured to perform different functions.

[0101] In some embodiments, computational tasks and stored data associated with a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device may be based at least in part on the processing requirements of a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device, the processing capabilities of the computing devices 500a, 500b, and 500c,the latency of the network connections between the computing devices within a cluster as well as between the clusters themselves and / or other factors that may contribute to cost, speed, fault tolerance, reliability, efficiency and / or other design goals of the overall system architecture, may be distributed among the computing devices 500a, 500b and 500c.

[0102] The cluster storage arrays 510a, 510b, 510c of the compute clusters 509a, 509b, 509c may be data storage arrays that include disk array controllers configured to manage read and write access to groups of disk drives. The disk array controllers, alone or in conjunction with their respective computing devices, may also be configured to manage backup or redundant copies of the data stored in the cluster storage arrays to protect against disk drive or other cluster storage array failures and / or network failures that prevent one or more computing devices from accessing one or more cluster storage arrays.

[0103] Similar to the way in which the functions of a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device may be distributed among the computing devices 500a, 500b, 500c of the computing clusters 509a, 509b, 509c, various active portions and / or backup portions of these components may also be distributed among the cluster storage arrays 510a, 510b, 510c.For example, some clustered storage arrays may be configured to store a portion of the data of a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device, while other clustered storage arrays may store another portion(s) of data of a neural network, a machine learning-based blur removal model 235, a machine learning model for blur removal and matching 240, and / or a machine learning model for blur removal, matching, and deception detection 245, and / or a computing device.Additionally, for example, some cluster storage arrays may be configured to store the data of a first neural network, while other cluster storage arrays may store the data of a second and / or third neural network. Furthermore, some cluster storage arrays may be configured to store backup versions of data stored in other cluster storage arrays.

[0104] The cluster routers 511a, 511b, 511c in the compute clusters 509a, 509b, 509c may include network equipment configured to provide internal and external communications for the compute clusters. For example, the cluster routers 511a in the compute cluster 509a may include one or more internet switching and routing devices configured to provide (i) local area network communication between the compute devices 500a and the cluster storage arrays 510a via the local cluster network 512a, and (ii) wide area network communication between the compute cluster 509a and the compute clusters 509b and 509c via the wide area network connection 513a to the network 406.Cluster routers 511b and 511c may include network equipment similar to cluster routers 511a, and cluster routers 511b and 511c may perform similar network functions for compute clusters 509b and 509b that cluster routers 511a perform for compute cluster 509a.

[0105] In some embodiments, the configuration of the cluster routers 511a, 511b, 511c may be based at least in part on the data communication requirements of the computing devices and the cluster storage arrays, the data communication capabilities of the network equipment in the cluster routers 511a, 511b, 511c, the latency and throughput of the cluster local area networks 512a, 512b, 512c, the latency, throughput, and cost of the wide area network connections 513a, 513b, 513c, and / or other factors that may contribute to the cost, speed, fault tolerance, resiliency, efficiency, and / or other design criteria of the moderation system architecture. Exemplary procedures

[0106] Fig. 6 illustrates a method 600 according to exemplary embodiments. The method 600 may include various blocks or steps. The blocks or steps may be performed individually or in combination. The blocks or steps may be performed in any order and / or serially or in parallel. Furthermore, the blocks or steps may be omitted or added to the method 600.

[0107] The blocks of method 600 may be performed by various elements of computing device 100, as described with respect to Fig. 1 illustrated and described.

[0108] Block 610 includes detecting a movement of a finger by a display component and during a fingerprint authentication phase.

[0109] Block 620 includes capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smear of the fingerprint, wherein the fingerprint sensor is configured to scan a fingerprint of the finger.

[0110] Block 630 includes reconstructing, based on the movement of the finger and an estimated fingerprint distortion, an unsmudged fingerprint from the fingerprint data, wherein the reconstructing reduces the smearing of the fingerprint to make it recognizable by a fingerprint matching component.

[0111] Block 640 includes recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint to a stored fingerprint template.

[0112] Some embodiments include applying a machine learning model to perform reconstruction of the unsmudged fingerprint. Some embodiments include training the machine learning model to predict a plurality of unsmudged variations of a given scanned fingerprint.

[0113] Some embodiments include applying a machine learning model to perform reconstruction of the non-blurred fingerprint and recognition of the fingerprint.

[0114] Some embodiments include applying a machine learning model to perform reconstruction of the unsmudged fingerprint, recognition of the fingerprint, and performance of deception detection.

[0115] Some embodiments include detecting pressure exerted by the finger using a pressure sensor. These embodiments further include measuring the amount of pressure applied. Reconstructing the unsmudged fingerprint is based on the measured amount of pressure applied.

[0116] In some embodiments, detecting the movement of the finger is based on the respective pixel values ​​of one or more pixels in a pixel array of an image of the fingerprint. In some embodiments, the one or more pixels in the pixel array are dedicated to motion detection.

[0117] In some embodiments, detecting the movement of the finger is performed using an application specific integrated circuit (ASIC) of the device.

[0118] In some embodiments, the display component includes a touch-sensitive display panel, the fingerprint sensor may be an under-display fingerprint sensor (UDFPS), and includes the method of determining a thermal image indicative of movement at or near the touch-sensitive display panel. Detecting the movement of the finger may be based on the thermal image.

[0119] In some embodiments, the device includes a thermal sensor configured to detect thermal activity on or near the device, and includes the method of determining a thermal image based on the detected thermal activity. Detecting the movement of the finger may be based on the thermal image.

[0120] In some embodiments, the fingerprint sensor may be a capacitive fingerprint sensor. Detecting the movement of the finger may be based on a capacitive region of the display component.

[0121] Some embodiments include determining an optical flow based on one or more images of the fingerprint. Detection of finger movement may be based on the optical flow.

[0122] In some embodiments, the stored fingerprint template may be predetermined during a fingerprint enrollment phase of the fingerprint, wherein the fingerprint enrollment phase occurs before the fingerprint authentication phase.

[0123] In some embodiments, the estimated fingerprint distortion may be determined during the fingerprint enrollment phase.

[0124] In some embodiments, the estimated fingerprint distortion may be determined during the fingerprint authentication phase.

[0125] In some embodiments, the estimated fingerprint distortion may be based on a normalized amount of blurring based on a plurality of users, a plurality of devices, or both.

[0126] In some embodiments, the estimated fingerprint distortion may be predicted by a machine learning model.

[0127] In some embodiments, the device may be a mobile computing device.

[0128] In some embodiments, reconstruction of the unsmudged fingerprint and recognition of the fingerprint may be performed on the device.

[0129] In some embodiments, the estimated fingerprint distortion may indicate a base amount of blurring.

[0130] The specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of the elements shown in a given figure. Furthermore, some of the illustrated elements may be combined or omitted. Furthermore, an exemplary embodiment may include elements not illustrated in the figures.

[0131] A step or block representing information processing may correspond to circuitry that can be configured to perform the specific logical functions of a method or technique described herein. Alternatively or additionally, a step or block representing information processing may correspond to a module, segment, or portion of program code (including associated data). The program code may include one or more processor-executable instructions for implementing specific logical functions or actions in the method or technique. The program code and / or associated data may be stored on any type of computer-readable medium, for example, on a storage device such as a floppy disk, hard disk, or other storage medium.

[0132] The computer-readable medium may also include non-transitory computer-readable media, such as computer-readable media that stores data for short periods of time, such as register memory, processor cache, and random access memory (RAM). The computer-readable media may also include non-transitory computer-readable media that stores program code and / or data for longer periods of time. The computer-readable media may therefore include secondary or persistent long-term storage, such as read-only memory (ROM), optical or magnetic disks, or compact disc read-only memory (CD-ROM). The computer-readable media may also be any other volatile or non-transitory storage systems. A computer-readable medium may, for example, be considered a computer-readable storage medium or a tangible storage device.

[0133] Although various examples and embodiments have been disclosed, other examples and embodiments will be apparent to those skilled in the art. The various disclosed examples and embodiments are illustrative and not limiting, with the true scope being indicated by the following claims. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 476 982

[0001]

Claims

[1] Device comprising: a display component comprising a fingerprint sensor configured to scan a fingerprint of a finger; and one or more processors operable to perform operations, the operations comprising: Detecting a movement of a finger during a fingerprint authentication phase; capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smearing of the fingerprint; Reconstructing, based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstructing reduces the blurring of the fingerprint to make it recognizable by a fingerprint matching component; and Recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint with a stored fingerprint template. [2] The apparatus of claim 1, wherein the operations further comprise: Applying a machine learning model to perform reconstruction of the unsmudged fingerprint. [3] The apparatus of any one of claims 1 or 2, wherein the operations further comprise: Training the machine learning model to predict a variety of non-blurred variations of a given scanned fingerprint. [4] The apparatus of any of claims 1-3, wherein the operations further comprise: Applying a machine learning model to perform the reconstruction of the unsmudged fingerprint and the recognition of the fingerprint. [5] The apparatus of any of claims 1-4, wherein the operations further comprise: Applying a machine learning model to perform non-blurred fingerprint reconstruction, fingerprint recognition, and deception detection. [6] The apparatus of any of claims 1-5, wherein the operations further comprise: Detecting pressure exerted by a finger using a pressure sensor; Measure an amount of pressure applied and where the reconstruction of the unsmudged fingerprint is based on the measured amount of pressure applied. [7] The device of any of claims 1-6, wherein detecting the movement of the finger is based on the respective pixel values ​​of one or more pixels in a pixel array of an image of the fingerprint. [8] The apparatus of claim 7, wherein the one or more pixels in the pixel array are for motion detection. [9] The device of any of claims 1-8, wherein the detection of the movement of the finger is performed using an application specific integrated circuit (ASIC) of the device. [10] The device of any of claims 1-9, wherein the display component comprises a touch-sensitive display panel, wherein the fingerprint sensor is an under-display fingerprint sensor (UDFPS), and the operations further comprise: Determining a thermal image indicating movement on or near the touch-sensitive display panel, and where the detection of the finger movement is based on the thermal image. [11] The device of any of claims 1-10, further comprising a thermal sensor configured to detect thermal activity on or near the device, and wherein the operations further comprise: Determining a thermal image based on the detected thermal activity and where the detection of the finger movement is based on the thermal image. [12] The device of any of claims 1-11, wherein the fingerprint sensor is a capacitive fingerprint sensor and wherein the detection of the movement of the finger is based on a capacitive area of ​​the display component. [13] The apparatus of any of claims 1-12, wherein the operations further comprise: Determining an optical flow based on one or more images of the fingerprint and where the detection of the finger movement is based on optical flow. [14] The apparatus of any of claims 1-13, wherein the stored fingerprint template was predetermined during a fingerprint registration phase of the fingerprint, the fingerprint registration phase occurring before the fingerprint authentication phase. [15] The apparatus of claim 14, wherein the estimated fingerprint distortion is determined during the fingerprint enrollment phase. [16] The apparatus of any of claims 1-15, wherein the estimated fingerprint distortion is determined during the fingerprint authentication phase. [17] The apparatus of any of claims 1-16, wherein the estimated fingerprint distortion is based on a normalized amount of blurring based on a plurality of users, a plurality of devices, or both. [18] The apparatus of any of claims 1-17, wherein the estimated fingerprint distortion is predicted by a machine learning model. [19] The device of any of claims 1-18, wherein the device is a mobile computing device. [20] The device according to any one of claims 1-19, wherein the reconstruction of the non-blurred fingerprint and the recognition of the fingerprint are performed on the device. [21] The apparatus of any of claims 1-20, wherein the estimated fingerprint distortion indicates a base amount of blurring. [22] Computer-implemented method comprising: Detecting a movement of a finger by a display component and during a fingerprint authentication phase; capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smear of the fingerprint, the fingerprint sensor being configured to scan a fingerprint of the finger; Reconstructing, based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstructing reduces the blurring of the fingerprint to make it recognizable by a fingerprint matching component; and Recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint with a stored fingerprint template. [23] An article of manufacture including a non-transitory, computer-readable medium having stored thereon program instructions that, when executed by one or more processors of a computing device, cause the computing device to perform operations comprising: Detecting a movement of a finger by a display component and during a fingerprint authentication phase; capturing, by a fingerprint sensor of the display component, fingerprint data associated with the fingerprint, wherein the movement of the finger caused a smear of the fingerprint, the fingerprint sensor being configured to scan a fingerprint of the finger; Reconstructing, based on the movement of the finger and an estimated fingerprint distortion, a non-blurred fingerprint from the fingerprint data, wherein the reconstructing reduces the blurring of the fingerprint to make it recognizable by a fingerprint matching component; and Recognizing the fingerprint by the fingerprint matching component, wherein recognizing the fingerprint comprises matching the reconstructed fingerprint with a stored fingerprint template.

Images