Method and system for performing fingerprint recognition

Abstract

The present disclosure provides a method and system for performing fingerprint recognition. The method includes: capturing an image of a finger depicting an object; using an object detection algorithm to detect a portion of a hand including at least one finger depicted in one of the images and the location of the detected portion in the image, wherein the object detection algorithm is a classifier trained to detect any area depicting at least one finger; identifying a fingertip segment of at least one finger from one of the images according to a segmentation algorithm, applying the segmentation algorithm based on the detection of the portion of the hand including at least one finger and according to the location, wherein the segmentation algorithm is a classifier trained to detect any fingertip segment within the area of ​​at least one finger, wherein the object detection algorithm and the segmentation algorithm are one or more of HOG, LBP, and Haar feature classifiers; generating a biometric identifier including distinguishing features extracted from the identified fingertip segment; and storing the biometric identifier in a storage medium.

Description

[0001] This application is a divisional application of the invention patent application with the application date of January 29, 2016, application number 201680020217.6 (international application number PCT / IB2016 / 000569), and invention name “System and method for performing fingerprint-based user authentication using images captured by mobile devices”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application is a continuation-in-part of U.S. non-provisional patent application serial number: 14 / 819,639, filed on August 6, 2015, entitled “SYSTEMS AND METHODS FOR PERFORMING FINGERPRINT BASED USER AUTHENTICATION USING IMAGERY CAPTURED USING MOBILE DEVICES,” which is based on and claims priority to U.S. provisional patent application serial number: 62 / 112,961, filed on February 6, 2015, entitled “SYSTEMS AND METHODS FOR PERFORMING FINGERPRINT BASED USER AUTHENTICATION USING IMAGERY CAPTURED USING MOBILE DEVICES,” the contents of which are hereby incorporated by reference as if expressly set forth in their entirety herein. Technical Field

[0004] The present invention relates to systems and methods for capturing and characterizing biometrics, and in particular, to systems and methods for capturing and characterizing biometrics using images of a finger captured by an embedded camera of a mobile device such as a smartphone. Background Art

[0005] Because biometrics are biological characteristics of an individual (e.g., fingerprints, hand geometry, retinal patterns, iris texture, etc.), biometric technology can be used as an additional verification factor because biometrics are generally more difficult to obtain than other non-biometric credentials. Biometrics can be used for identification and / or authentication (also known as identity assertion and / or verification).

[0006] Biometric assertions may require a certain level of security as dictated by the application. For example, authentication related to financial transactions or gaining access to secure locations requires a higher level of security. Therefore, it is preferred that the accuracy of the user's biometric representation is sufficient to ensure that the user is accurately authenticated and security is maintained.

[0007] Furthermore, lost, exchanged, mixed up and illegal adoption of newborns is a global challenge and the use of automated biometric recognition systems to identify newborns based on their face, iris, fingerprint, foot print and / or palm print has been proposed.

[0008] However, to the extent that iris, face, finger, and voice identity assertion systems exist and provide the necessary accuracy, such systems require specialized equipment and applications and are not easily implemented on traditional smartphones with limited camera resolution and lighting capabilities.

[0009] Electronic fingerprint sensors have been incorporated into smartphone devices, such as the iPhone 6 smartphone from Apple Inc. of Cupertino, California, and the Samsung S5 smartphone from Samsung Electronics Co. of South Korea. In these devices, users must register their fingerprint data by placing their finger on the sensor. At a later date, the user can verify their identity by repositioning their finger on the sensor. The fingerprint data is compared with the registered data, and if a match occurs, the user's identity is confirmed. If the fingerprints do not match, the user may be identified as an imposter. A disadvantage of these systems is that the fingerprint sensor increases the size, weight, and cost of the device. Furthermore, minimizing the size of the fingerprint sensor is advantageous, and as such, these sensors typically only capture a portion of the fingerprint, which reduces the effectiveness of identification. The smaller the area captured by the fingerprint sensor, the greater the chance that another finger will accidentally match, and any errors in the fingerprint data can lead to false rejection of the authentic user.

[0010] Furthermore, capturing a newborn's fingerprint by using a conventional fingerprint sensor is challenging due to the size of the fingers and the difficulty of holding a newborn's hand and placing it on the sensor.

[0011] In practice, this means that users (i.e., adults and newborns) experience a higher degree of inconvenience from false rejections, and the application of the sensor is limited to non-critical uses such as low-value payments. Fingerprint sensors can also be the target of spoofing attacks, where, for example, a mold of a real user's fingerprint is placed in the fingerprint sensor to enable an imposter to authenticate. This provides another reason to limit use to non-critical applications.

[0012] Another challenge is that only a small number of mobile devices are equipped with fingerprint sensors, which limits the number of people who can access the fingerprint authorization system and leads to inconsistent authentication methods between devices.

[0013] Systems have been proposed that use a mobile device's camera to analyze images of individual fingers, and while these systems may be more convenient, the minimum false acceptance and false rejection rates of such systems for imaging and analyzing individual fingers are still not reliable enough for applications requiring higher security, such as mid- to high-value procurement and enterprise systems (i.e., large-scale systems).

[0014] Therefore, there is a need for a more reliable and universal finger recognition system. Summary of the Invention

[0015] Technologies are provided herein that support systems and methods for performing fingerprint recognition.

[0016] According to a first aspect, a method for performing fingerprint recognition is provided. The method includes capturing one or more images of a plurality of fingers depicting an object using a mobile device having a camera, a storage medium, instructions stored on the storage medium, and a processor configured to execute the instructions. The method also includes detecting the plurality of fingers depicted in the one or more images using a finger detection algorithm. The method also includes identifying individual fingertip segments of each of the plurality of fingers from the one or more images based on a segmentation algorithm. Furthermore, the method includes extracting distinguishing features of each identified finger, generating a biometric identifier based on the extracted distinguishing features, and storing the biometric identifier in a memory.

[0017] According to another aspect, a method for performing fingerprint recognition is provided, the method comprising: capturing an image by a mobile device having a camera, a storage medium, instructions stored on the storage medium, and a processor configured by executing the instructions, the image depicting a finger of an object; using the processor, using an object detection algorithm configured to locate at least a portion of the hand, to detect a portion of the hand including at least one finger depicted in one of the images and a position of the detected portion in the image, wherein the object detection algorithm is a classifier trained to examine regions of an image to detect any region depicting at least one finger, and wherein the object detection algorithm is a classifier selected from the group consisting of HOG, LBP, and Haar feature classifiers. one or more of; identifying, with the processor, a fingertip segment of the at least one finger from one of the images based on a segmentation algorithm, wherein the segmentation algorithm is applied based on the detection of the portion of the hand including the at least one finger and according to the location, wherein the segmentation algorithm is a classifier trained to detect any fingertip segment within the area determined to depict the at least one finger, and wherein the segmentation algorithm is one or more of a HOG, LBP, and Haar feature classifier; generating a biometric identifier including the extracted distinguishing features, wherein generating the biometric identifier includes extracting the distinguishing features from the identified fingertip segment; and storing, with the processor, the generated biometric identifier in the storage medium.

[0018] According to yet another aspect, a system for performing fingerprint recognition is provided, comprising: a mobile device having a camera, a storage medium, a display, and a processor in operable communication with the camera, the display, and the storage medium; a software application comprising instructions stored in code on the storage medium, wherein the instructions are executable in the processor and configure the processor to: capture an image with the camera, the image depicting a finger of an object; detect a portion of a hand including at least one finger depicted in one of the images and a location of the detected portion in the image using an object detection algorithm configured to locate at least a portion of a hand, and wherein the object detection algorithm is a classifier trained to examine regions of an image to detect any region depicting at least one finger, and wherein wherein the object detection algorithm is one or more of a HOG, LBP, and Haar feature classifier; identifying a fingertip segment of the at least one finger from one of the images according to a segmentation algorithm, wherein the segmentation algorithm is applied based on the detection of the portion of the hand including the at least one finger and according to the location, wherein the segmentation algorithm is a classifier trained to detect any fingertip segment within any area determined to depict the at least one finger, and wherein the segmentation algorithm is one or more of a HOG, LBP, and Haar feature classifier; generating a biometric identifier including the extracted distinguishing features, wherein generating the biometric identifier includes extracting the distinguishing features from the identified fingertip segment; and storing the generated biometric identifier in the storage medium with the processor.

[0019] These and other aspects, features and advantages may be understood from a description of certain embodiments of the invention and from the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a high-level diagram of a computer system for authenticating a user based on a user's biometrics in accordance with at least one embodiment of the present disclosure;

[0021] Figure 2A is a block diagram of a computer system for authenticating a user based on a biometric characteristic of the user, in accordance with at least one embodiment of the present disclosure;

[0022] Figure 2B is a block diagram of a software module for authenticating a user based on a user's biometrics in accordance with at least one embodiment of the present disclosure;

[0023] Figure 2C is a block diagram of a computer system for authenticating a user based on a biometric characteristic of the user, in accordance with at least one embodiment of the present disclosure;

[0024] Figure 3is a flow chart illustrating a routine for generating a biometric identifier based on a user's biometrics and enrolling or authenticating the user in accordance with at least one embodiment disclosed herein;

[0025] Figure 4A is a flow chart illustrating a routine for detecting a finger from a visual image and a corresponding image in accordance with at least one embodiment of the present disclosure;

[0026] Figure 4B is a flow chart illustrating a routine for filtering fingertip regions detected from a visual image, in accordance with at least one embodiment of the present disclosure;

[0027] Figure 4C Description based on Figure 4B The routine for filtering the fingertip region to capture and generate images;

[0028] Figure 5A is a flow chart illustrating a routine for detecting liveness from a visual image of a finger in accordance with at least one embodiment of the present disclosure;

[0029] Figure 5B is a series of images captured according to a routine for detecting activity from a visual image of a finger;

[0030] Figure 5C is a series of images captured according to a routine for detecting activity from a visual image of a finger;

[0031] Figure 6A depicts a captured image of a finger and a corresponding ridge reflectivity image generated in accordance with at least one embodiment disclosed herein;

[0032] Figure 6B depicts a captured image of a finger and a corresponding ridge reflectivity image generated according to at least one embodiment disclosed herein;

[0033] Figure 6C depicts a captured image of a finger and a corresponding ridge reflectivity image generated in accordance with at least one embodiment disclosed herein;

[0034] Figure 6D depicts a captured image of a finger and a corresponding ridge reflectivity image generated in accordance with at least one embodiment disclosed herein;

[0035] Figure 6E depicts a captured image of a finger and a corresponding ridge reflectivity image generated in accordance with at least one embodiment disclosed herein;

[0036] Figure 6F Depicts a captured image of a finger and a corresponding ridge reflectivity image generated in accordance with at least one embodiment disclosed herein. DETAILED DESCRIPTION

[0037] By way of example only, for purposes of overview and introduction, embodiments of the present invention described below relate to a system and method for capturing a user's biometrics and generating an identifier that characterizes the user's biometrics using a mobile device, such as a smartphone. Images captured of multiple fingers of the user are preferably used to generate a biometric identifier that is used to authenticate / identify the user based on the captured biometrics and to determine the user's activity. This disclosure also describes additional techniques for preventing false authentication due to spoofing. In some examples, anti-spoofing techniques may include capturing one or more images of the user's biometrics and analyzing the captured images for activity indicators.

[0038] In some implementations, the system includes a cloud-based system server platform that communicates with stationary PCs, servers, and devices such as laptops, tablets, and user-operated smartphones. When a user attempts to access an access-controlled network environment (e.g., a website requiring a secure login), the user is prompted to authenticate using the user's pre-registered mobile device. Authentication can include verifying the user's identity and / or verifying that the user is alive (e.g., determining activity) by capturing biometric information in the form of at least an image of the user's finger, extracting unique features, and encoding the features into a biometric identifier that indicates the biometric characteristics and / or activity of the user using the mobile device. Thus, the user's identity and / or activity can be verified by the mobile device and / or the system server, or a combination of the foregoing, by analyzing the image, the generated biometric identifier, and / or comparing the image and / or biometric identifier with a biometric identifier generated during the user's initial registration with the system.

[0039] According to one aspect of the present application, the disclosed embodiments provide a reliable means of user identification / authentication using finger-based biometric recognition on ubiquitous and convenient mobile devices. The disclosed embodiments utilize the camera typically present on mobile devices to perform four-finger-based recognition, resulting in a design that does not incur additional bulk, cost, or weight, and allows for ubiquitous use. Another objective of the system is to provide protection against spoofing attacks.

[0040] The present invention simultaneously captures biometric information from multiple fingers, capturing a large print area from each finger. Furthermore, the present invention can be used to capture print information from other areas of the hand, including palm prints and fingerprints, to further improve system reliability. Furthermore, in the case of newborn identification, the present invention can be used to capture toes. Furthermore, the proposed innovation can be integrated with existing mobile face recognition systems. As a non-limiting example, exemplary systems and methods for biometric-based user authentication from images of facial features are described herein and in co-pending and commonly assigned U.S. patent application serial number 14 / 668,352, filed on May 13, 2016, entitled “SYSTEM AND METHOD FOR AUTHORIZING ACCESS TO ACCESSCONTROLLED ENVIRONMENTS,” which is a continuation-in-part of U.S. Patent No. 9 / 003,196, filed on May 13, 2014, entitled “SYSTEM AND METHOD FOR AUTHORIZING ACCESS TO ACCESSCONTROLLED ENVIRONMENTS,” and U.S. Patent No. 9,208,492, filed on March 7, 2014, entitled “SYSTEMS AND METHODS FOR BIOMETRIC AUTHENTICATION OF TRANSACTIONS,” which are incorporated herein by reference as if each were set forth in its entirety herein. Furthermore, the present invention can be used to process a captured finger photograph using a camera provided on a mobile device to generate a fingerprint image that corresponds to the captured finger photograph and can be matched with rolled and plain fingerprint images used in the Integrated Automated Fingerprint Identification Systems (IAFIS). IAFIS is a national automated fingerprint identification and criminal history system maintained by the Federal Bureau of Investigation (FBI). IAFIS provides automated fingerprint search capabilities, latent search capabilities, electronic image storage, and electronic exchange of fingerprints and responses.

[0041] The disclosed embodiments may be referred to as a multimodal biometric authentication system. Thus, compared to a single finger movement recognition system using a finger image or fingerprint captured by an embedded sensor in a smartphone, the presence of multiple independent biometrics (i.e., 4-10 fingers) provides the following advantages:

[0042] 1. Performance: Combining unrelated patterns (e.g., four fingers from a person and ten fingers from a newborn) can yield performance improvements over single-finger recognition systems. This improvement in accuracy occurs for two reasons. First, the fusion of biometric evidence from different fingers effectively increases distinguishing features and reduces overlap between features from different users. In other words, a combination of multiple fingers is more discriminative to an individual than a single finger. Second, noise and inaccuracies during acquisition of a subset of fingers (caused by factors such as dirt or ink smudges) can be addressed by the information provided by the remaining fingers.

[0043] 2. Universality: Addressing non-universality issues and reducing enrollment errors. For example, if a person cannot enroll a specific finger due to finger amputation, finger cut, injury, or worn ridges (i.e., worn ridges can physically occur in one or more fingers of a subject), he can still be identified using his other fingers.

[0044] 3. Spoofing attacks: Using the disclosed embodiments in which multiple fingers of a user are enrolled will increase the resistance of the authentication system to spoofing attacks. This is because it is increasingly difficult to circumvent or spoof multiple fingers simultaneously.

[0045] An exemplary system for authenticating a user and / or determining the activity of a user based on an image of the user's biometric 100 is shown as Figure 1 In one arrangement, the system is composed of a system server 105 and user devices including a mobile device 101a and a user computing device 101b. The system 100 may also include one or more remote computing devices 102.

[0046] System server 105 can be any computing device and / or data processing apparatus capable of communicating with user devices and remote computing devices, and receiving, sending, and storing electronic information and processing requests as further described herein. Similarly, remote computing device 102 can be any computing device and / or data processing apparatus capable of communicating with system servers and / or user devices, and receiving, sending, and storing electronic information and processing requests as further described herein. It should also be understood that the system server and / or remote computing device can be multiple networked or cloud-based computing devices.

[0047] In some implementations, computing device 102 may be associated with a business organization, such as a bank or website, that maintains user accounts ("business accounts"), provides services to business account holders, and requires authentication of users before providing user access to these systems and services.

[0048] The user device, mobile device 101a, and user computing device 101b can be configured to communicate with each other, as further described herein, and to send and receive electronic information from system server 105 and / or remote computing device 102. The user device can also be configured to receive user input and capture and process biometric information, such as a digital image and voice recording of user 124.

[0049] Mobile device 101a may be any mobile computing device and / or data processing apparatus capable of embodying the systems and / or methods described herein, including but not limited to a personal computer, tablet computer, personal digital assistant, mobile electronic device, cellular or smart phone device, etc. Computing device 101b is intended to represent various forms of computing devices with which a user may interact, such as a workstation, personal computer, laptop computer, dedicated point-of-sale system, ATM terminal, access control device, or other suitable digital computer.

[0050] As further described herein, system 100 facilitates authentication of user 124 using mobile device 101a based on the user's biometrics. In some implementations, identification and / or authentication of the user's biometrics utilizes the user's biometric information in a two-stage process. The first stage is referred to as the enrollment stage. During the enrollment stage, samples (e.g., images) of one or more appropriate biometric characteristics are collected from the individual. These biometric samples are analyzed and processed to extract features (or characteristics) present in each sample. The collection of features present in the individual's imaged biometrics constitutes an identifier for the person and can be used to authenticate the user and, in some instances, determine whether the user is a live subject. These identifiers are then stored to complete the enrollment stage. In the second stage, the same biometric characteristics of the individual are measured. Features from this biometric characteristic are extracted as in the enrollment stage to obtain a current biometric identifier. If the goal is to determine liveness, the features or characteristics can be analyzed to determine whether they represent a live subject. As further described herein, other features and characteristics of the captured biometric image can be analyzed to determine liveness. If the goal is identification, the identifier is searched for in a database of identifiers generated in the first stage. If a match occurs, the identification of the individual is revealed, otherwise the identification fails. If the goal is authentication, the identifier generated in the second phase is compared with the identifier generated for the specific person in the first phase. If a match occurs, the authentication is successful, otherwise the authentication fails.

[0051] It should be noted that although Figure 1 A system for authenticating a user 100 against a mobile device 101a and a user computing device 101b and a remote computing device 102 is depicted, but it should be understood that any number of such devices may interact with the system in the manner described herein. Figure 1A system is depicted for authenticating user 100 against user 124, but it should be understood that any number of users may interact with the system in the manner described herein.

[0052] It should be further understood that although the various computing devices and machines referenced herein, including but not limited to the mobile device 101a, the system server 105, and the remote computing device 102, are referred to herein as individual / single devices and / or machines, as known to those skilled in the art, in certain implementations, the referenced devices and machines and their associated and / or accompanying operations, features and / or functions may be combined or arranged or otherwise used across multiple such devices and / or machines, such as via network connections or wired connections.

[0053] It should also be understood that the exemplary systems and methods described herein in the context of mobile device 101a (also referred to as a smartphone) are not specifically limited to mobile devices and may be implemented using other enabled computing devices (eg, user computing device 102b).

[0054] refer to Figure 2A , the mobile device 101a of the system 100 includes various hardware and software components for implementing the operation of the system, including one or more processors 110, memory 120, microphone 125, display 140, camera 145, audio output 155, storage 190, and communication interface 150. The processor 110 is used to execute client applications in the form of software instructions that can be loaded into the memory 120. Depending on the specific implementation, the processor 110 can be multiple processors, a central processing unit CPU, a graphics processing unit GPU, multiple processor cores, or any other type of processor.

[0055] Preferably, the memory 120 and / or storage device 190 are accessible to the processor 110, thereby enabling the processor to receive and execute instructions encoded in the memory and / or on the storage device to cause the mobile device and its various hardware components to operate in accordance with aspects of the systems and methods described in more detail below. The memory may be, for example, random access memory (RAM) or any other suitable volatile or non-volatile computer-readable storage medium. Furthermore, the memory may be fixed or removable. The storage device 190 may take different forms depending on the specific implementation. For example, the storage device may include one or more components or devices, such as a hard drive, a flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The storage device may be fixed or removable.

[0056] One or more software modules 130 are encoded in the storage device 190 and / or the memory 120. The software modules 130 may include one or more software programs or applications having a computer program code or a set of instructions (also referred to as a "mobile authentication client application") that are executed in the processor 110. Figure 2B As shown, preferably included in the software modules 130 are a user interface module 170, a biometric capture module 172, an analysis module 174, an enrollment module 176, a database module 178, an authentication module 180, and a communication module 182, which are executed by the processor 110. Such computer program code or instructions configure the processor 110 to perform the operations of the systems and methods disclosed herein, and may be written in any combination of one or more programming languages.

[0057] The program code may execute entirely on the mobile device 101, as a stand-alone software package, partially on the mobile device, partially on the system server 105, or entirely on the system server or another remote computer / device. In the latter scenario, the remote computer may be connected to the mobile device 101 through any type of network including a local area network (LAN), a wide area network (WAN), a mobile communication network, a cellular network, or any type of network that can enable connection to an external computer (e.g., through the Internet using an Internet service provider).

[0058] It can also be said that, as known to those skilled in the art, the software module 130 and the program code of one or more computer-readable storage devices (such as memory 120 and / or storage device 190) can form a computer program product that can be manufactured and / or distributed according to the present invention.

[0059] It should be understood that in some illustrative embodiments, one or more of the software modules 130 can be downloaded from another device or system via a network to the storage device 190 via the communication interface 150 for use within the system 100. Furthermore, it should be noted that other information and / or data related to the operation of the present systems and methods (such as the database 185) may also be stored in the storage device. Preferably, such information is stored on a specially designated encrypted data storage device to securely store information collected or generated by the processor executing the security authentication application. Preferably, encryption is used to store information locally on the mobile device storage device and to transmit the information to the system server 105. For example, such data can be encrypted using a 1024-bit polymorphic cipher or, in accordance with export controls, AES 256-bit encryption. Furthermore, encryption can be performed using a remote key (seed) or a local key (seed). Alternative encryption methods, such as SHA256, as understood by those skilled in the art, can be used.

[0060] Furthermore, the user's biometric information, activity information, or mobile device information can be used as an encryption key to encrypt data stored on the mobile device 101a and / or the system server 105. In some implementations, a combination of the foregoing can be used to create a complex unique key for the user that can be encrypted on the mobile device using elliptic curve cryptography, preferably with a length of at least 384 bits. Furthermore, this key can be used to protect user data stored on the mobile device and / or the system server.

[0061] Also preferably stored on the storage device 190 is a database 185. As will be described in more detail below, the database contains and / or maintains various data items and elements used in various operations of the system and method for authenticating the user 100. The information stored in the database may include, but is not limited to, user biometric templates and profile information, which will be described in more detail herein. It should be noted that while the database is depicted as being configured locally to the mobile device 101a, in some implementations, the database and / or the various data elements stored therein may additionally or alternatively be remotely located (such as on the remote device 102 or system server 105 - not shown) and connected to the mobile device via a network in a manner known to those of ordinary skill in the art.

[0062] A user interface 115 is also operatively connected to the processor. The interface can be one or more input or output devices, such as switches, buttons, keys, touch screens, microphones, etc., as would be understood in the art of electronic computing devices. The user interface is used to facilitate the capture of commands from the user, such as on-off commands or user information, as well as settings related to the operation of the system for authenticating user 100. For example, the interface is used to facilitate the capture of certain information from mobile device 101, such as personal user information for registration with the system to create a user profile.

[0063] The computing device 101a may also include a display 140, which is also operatively connected to the processor 110. The display includes a screen or any other such presentation device that enables the system to indicate or otherwise provide feedback to the user regarding the operation of the system for authenticating the user 100. As an example, the display may be a digital display such as a dot matrix display or other two-dimensional display.

[0064] As a further example, the interface and display can be integrated into a touch screen display. Thus, the display is also used to display a graphical user interface, which can display various data and provide a "form" including fields that allow the user to enter information. Touching the touch screen at a location corresponding to the display of the graphical user interface allows a person to interact with the device to enter data, change settings, control functions, etc. Thus, when the touch screen is touched, the user interface communicates the change to the processor, and the settings can be changed, or the information entered by the user can be captured and stored in memory.

[0065] Mobile device 101a also includes a camera 145 capable of capturing digital images. The camera can be one or more imaging devices configured to capture images of at least a portion of a user's body, including the user's eyes and / or face, while utilizing mobile device 101a. The camera is used to facilitate the capture of images of the user in order to perform image analysis performed by the mobile device processor 110 of the secure authentication client application, which includes identifying biometric features used to authenticate the user from the image (biometrically) and determining the user's activity. Mobile device 101a and / or camera 145 may also include one or more light or signal emitters (e.g., LEDs, not shown), such as visible light emitters and / or infrared light emitters. The camera can be integrated into the mobile device, such as a front-facing or rear-facing camera in conjunction with a sensor, such as, but not limited to, a CCD or CMOS sensor. As will be understood by those skilled in the art, camera 145 may also include additional hardware, such as a lens, a light meter (e.g., a lux meter), and other conventional hardware and software features that can be used to adjust image capture settings, such as zoom, focus, aperture, exposure, shutter speed, etc. Alternatively, the camera can be external to mobile device 101a. Those skilled in the art will appreciate possible variations of the camera and light emitter.In addition, as will be appreciated by those skilled in the art, the mobile device may also include one or more microphones 104 for capturing audio recordings.

[0066] Audio output 155 is also operatively connected to processor 110. As will be appreciated by those skilled in the art, the audio output can be any type of speaker system configured to play electronic audio files. The audio output can be integrated into mobile device 101 or external to mobile device 101.

[0067] Various hardware devices / sensors 160 are also operatively connected to the processor. Sensors 160 may include: an onboard clock to track the time of day, etc.; a GPS-enabled device to determine the location of the mobile device; an accelerometer to track the orientation and acceleration of the mobile device; a gravity magnetometer to detect the Earth's magnetic field to determine the three-dimensional orientation of the mobile device; a proximity sensor to detect the distance between the mobile device and other objects; an RF radiation sensor to detect RF radiation levels; and other such devices as will be appreciated by those skilled in the art.

[0068] The communication interface 150 is also operatively connected to the processor 110 and can be any interface that enables communication between the mobile device 101a and external devices, machines, and / or components, including the system server 105. Preferably, the communication interface includes, but is not limited to, a modem, a network interface card (NIC), an integrated network interface, a radio frequency transmitter / receiver (e.g., Bluetooth, cellular, NFC), a satellite communication transmitter / receiver, an infrared port, a USB connection, and / or any other such interface for connecting the mobile device to other computing devices and / or communication networks, such as private networks and the Internet. Such connections can include wired connections or wireless connections (e.g., using the 802.11 standard), although it should be understood that the communication interface can be virtually any interface that enables communication to / from the mobile device.

[0069] At various points during operation of the system for authenticating user 100, mobile device 101a may communicate with one or more computing devices, such as system server 105, user computing device 101b, and / or remote computing device 102. Such computing devices send and / or receive data to / from mobile device 101a, thereby preferably enabling maintenance and / or enhancement of the operation of system 100, as will be described in greater detail below.

[0070] Figure 2C is a block diagram illustrating an exemplary configuration of the system server 105. The system server 105 may include a processor 210 operatively connected to various hardware and software components for enabling the system's operations of authenticating users 100. The processor 210 is configured to execute instructions to perform various operations related to user authentication and transaction processing, as will be described in greater detail below. Depending on the specific implementation, the processor 210 may be a plurality of processors, a multi-processor core, or any other type of processor.

[0071] In some implementations, the memory 220 and / or storage medium 290 are accessible to the processor 210, thereby enabling the processor 210 to receive and execute instructions stored on the memory 220 and / or storage device 290. The memory 220 may be, for example, a random access memory (RAM) or any other suitable volatile or non-volatile computer-readable storage medium. In addition, the memory 220 may be fixed or removable. The storage device 290 may take different forms depending on the specific implementation. For example, the storage device 290 may include one or more components or devices, such as a hard drive, flash memory, a rewritable optical disk, a rewritable magnetic tape, or some combination thereof. The storage device 290 may be fixed or removable.

[0072] The one or more software modules 130 are encoded in the storage device 290 and / or the memory 220. The one or more software modules 130 may comprise one or more software programs or applications having a computer program code or a set of instructions that are executed in the processor 210 (collectively referred to as "security authentication server applications"). As will be understood by those skilled in the art, such computer program code or instructions for performing the operations of aspects of the systems and methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be executed in its entirety on the system server 105 as a stand-alone software package, in part on the system server 105 and in part on a remote computing device (such as the remote computing device 102, the mobile device 101a and / or the user computing device 101b), or in its entirety on such a remote computing device. Figure 2B As shown, preferably, the software module 130 includes an analysis module 274 , a registration module 276 , an authentication module 280 , a database module 278 and a communication module 282 , which are executed by the processor 210 of the system server.

[0073] Also preferably stored on the storage device 290 is a database 280. As will be described in more detail below, the database 280 contains and / or maintains various data items and elements used in various operations of the system 100, including, but not limited to, user profiles as will be described in more detail below. It should be noted that while the database 280 is depicted as being locally configured to the computing device 205, in some implementations, the database 280 and / or the various data elements stored therein may be stored on a computer-readable memory or storage medium that is remotely located and connected to the system server 105 via a network (not shown), in a manner known to those of ordinary skill in the art.

[0074] A communication interface 255 is also operatively connected to the processor. The communication interface 255 can be any interface that enables communication between the system server 105 and external devices, machines, and / or components. In some implementations, the communication interface 255 includes, but is not limited to, a modem, a network interface card (NIC), an integrated network interface, a radio frequency transmitter / receiver (e.g., Bluetooth, cellular, NFC), a satellite communication transmitter / receiver, an infrared port, a USB connection, and / or any other such interface for connecting the computing device 205 to other computing devices and / or communication networks such as private networks and the Internet. Such connections can include wired connections or wireless connections (e.g., using the 802.11 standard), although it should be understood that the communication interface 255 can be virtually any interface that enables communication to / from the processor 210.

[0075] The operation of the system for authenticating a user 100 and the various elements and components described above will be further understood with reference to methods for facilitating the capture of biometric information and authentication, as described below. The processing described herein is shown from the perspective of the mobile device 101a and / or the system server 105, however, it should be understood that the processing can be performed in whole or in part by the mobile device 101a, the system server 105 and / or other computing devices (e.g., the remote computing device 102 and / or the user computing device 101b), or any combination of the foregoing. It should be understood that more or fewer operations than shown in the figures and described herein may be performed. These operations may also be performed in a different order than described herein. It should also be understood that one or more steps may be performed by the mobile device 101a and / or other computing devices (e.g., the computing device 101b, the system server 105, and the remote computing device 102).

[0076] Now turn to the figure reference Figure 3 , a flow chart illustrates a routine 300 for detecting a user's biometric features from one or more images, in accordance with at least one embodiment disclosed herein. Generally, the routine includes capturing and analyzing one or more images of at least a plurality of fingers of the user. Preferably, four fingers are captured, however, as described herein, more or fewer fingers may be captured and analyzed. As described above, the capture process may be performed during enrollment of the user and during subsequent authentication sessions, which will also be referenced. Figure 3 Provide a description.

[0077] It should be understood that in accordance with the disclosed embodiments, images can be captured and a biometric identifier indicative of a unique biometric and / or liveness of a user can be generated using a widely available mobile device (e.g., mobile device 101a) having a digital camera 145 capable of capturing images of a user's finger in at least the visible spectrum band.

[0078] The process begins at step 305, where the mobile device processor 110, configured by executing one or more software modules 130 (including preferably the capture module 172), causes the camera 145 to capture one or more images of the user (124) capturing at least a portion of all four (4) fingers of a hand. Preferably, the camera captures high-resolution images, such as using a rear-facing camera of a conventional smartphone device. If available, flash lighting may be used to capture the image to increase detail.

[0079] In some implementations, during the capture process, the user is prompted to position their finger in front of the camera. During this process, the user may be given visual feedback from the camera on the device display so that the user can position their finger appropriately. In some implementations, markers displayed on the display may be used to guide the user to place their finger at a specific location in the camera's field of view, and at a specific distance. For example, the user may be asked to align their finger with four (4) finger outlines overlaid on the camera image preview screen, so when the user fills in the finger outlines on the camera preview, the user will know that their finger is at the appropriate distance from the camera. In some implementations, the user may hold their index finger, middle finger, ring finger, and pinky finger together, rather than spacing them apart. Then, in some implementations, the user may be asked to capture the thumb of each hand separately. In some implementations, the user will be a newborn and an adult will help her / him to capture the image of the finger. The focus of the camera may be set by a configured processor to focus on the finger, which can be considered the location of the finger position guide on the screen. In some embodiments, a classifier will be trained to detect fingers in an image, and the classifier can trigger the camera to capture an image when a finger is detected and in focus. In certain implementations, the classifier that can be used to detect fingers in an image can be a Haar Cascade classifier that has been trained using conventional Haar features or a predefined and pre-designed classifier suitable for detecting fingers in hand images (and foot images, for example, in the case of a newborn identification system). In some implementations, an image enhancement procedure can be applied to the image before using the classifier to detect fingers in the captured image. In some implementations, the image enhancement procedure that can be applied before applying the finger classifier can be designed as a bandpass filter that passes fingerprint ridge frequencies but minimizes out of focus background frequencies.

[0080] The captured image can then be inspected and the quality of the captured biometric sample determined by analyzing the finger's ridges. This quality measurement can be a fusion of the following ridge characteristics: frequency, orientation, sharpness, and connectivity. If the quality measurement falls below a predetermined threshold, the user can be advised and guided to repeat the capture process.

[0081] After the fingers are captured, at step 310 , the regions of each finger are identified. Then, at step 315 , the regions are enhanced, and then, at step 320 , the distinguishing feature space of each finger can be independently extracted and stored separately. More specifically, at step 310 , the configured processor 110 can execute an automatic finger detection algorithm to detect fingers in the image. For example, an exemplary finger detection algorithm can include the application of a segmentation algorithm to distinguish fingers from background. This can be performed, for example, by dividing the image into homogeneous regions, then examining each region and classifying it as a finger or non-finger region. Furthermore, this can be performed, for example, by using a classifier that detects and classifies fingers and fingertips. In some implementations, the classifiers for detecting different fingers, classifying them, and detecting fingertips can be trained using a Haar Cascade classifier, a HOG Cascade classifier, a LBP Cascade classifier, or a combination of these classifiers. Classifier training can be performed on known example images as known in the art. Note that the classifier trained to find hands can first be used to narrow the search area of ​​other finger-finger-finding classifiers to improve speed and accuracy. Note also that such a classifier can be used in conjunction with other fingertip location finding techniques to provide improved accuracy. In some implementations, the regions identified using the classifier can be highlighted with a boundary and displayed to the user on a mobile device display. For example, a region identified as a fingertip segment can be defined by a boundary in an image that highlights the identified region. The boundary can be a variety of shapes including rectangular or elliptical boundaries, and different parts can be highlighted including a fingertip, a finger, a hand, a group of fingers, other finger regions, etc. User enrollment data can be used to help train the classifier once the user has confirmed the fingertip location. For example, in some implementations, the process of capturing and detecting fingers and fingertips can be summarized in the following steps. (1) Capture a hand image, then (2) call a cascade classifier trained to find a first fingertip region, then another classifier trained to find a second fingertip region, and so on.

[0082] By way of example and not limitation, the use of a classifier (e.g., an LBP classifier) ​​can be implemented to find fingertips in an image, and it may also be advantageous to first use a classifier to find a major region of the hand, such as the entire hand or four fingers of the hand, and then use a secondary method to locate minor regions (e.g., fingertips or middle phalanges) within the major region. The secondary method may be another classifier trained to locate each minor region. The results of the secondary classifier may be further filtered using knowledge of expected relationships, such as prescribed relationships between each minor region (e.g., when the fingers of a hand are closed and held flat, the four (4) fingers have a known positional relationship that can be used to exclude false matches). Further filtering may be applied by finding the location of other prominent hand features (e.g., connections between fingers) and using this information to filter the results from the classifier. Additionally, the primary classifier may be used in real time to follow the finger as the user presents it to the camera and ensure that focus and exposure are optimal for the hand before automatically triggering image capture and / or biometric matching.

[0083] As previously described and further described herein, the processor can be configured to detect and track the finger in real time as the user is presenting the finger to the camera and capturing an image using the camera. The tracked position of the image can be used to detect when the finger is sufficiently stable in position and to improve the quality of the verification image and the reliability of finger identification.

[0084] In some implementations, the processor can be configured to speed up real-time finger detection by dynamically switching between object detection methods and faster tracking methods such as template matching or optical flow. For example, when a set of four fingertips has been detected and it has been determined that they represent a hand, the configured processor can use an optical flow algorithm to track the fingers. As a result, finger positions can be tracked with significantly lower latency and at a higher frame rate than, for example, if a cascade classifier were also applied to subsequent image frames. High speed can be achieved by limiting the search space in the image searched by the processor to a local image area, and the processor can be configured to match only pixels that represent distinguishing features (such as finger centers).

[0085] If a finger is placed too far away or out of view, object tracking may fail. When the processor detects a failure, the processor can revert to an initial object detection method, such as a cascade classifier. In the case of tracking four fingers, the processor can measure the relative positions of the fingers (e.g., the distance between the centers of the fingers), and if it has been determined that the distance has changed significantly (e.g., above a specified threshold), the system can revert to object detection.

[0086] Preferably, the capture system is able to detect when the user's finger remains stationary (within a certain tolerance) to prevent motion blur in the verification image. This can be achieved, for example, by tracking the object between frames (e.g., calculating the vector of travel) and capturing a high-resolution verification image when the object's speed is below a threshold speed.

[0087] Small errors in locating an object's position can propagate into the calculation of the velocity vector. Methods such as cascade classifiers often introduce artificial fluctuations in object position from frame to frame (object center "jumps"). This position noise interferes with determining when an object is stationary. However, as described above, tracking using optical flow has lower noise and provides faster updates of object position, and can make stationary object detection significantly more reliable.

[0088] Furthermore, in some implementations, finger length information can be stored and used as part of the fingertip recognition algorithm, placing some weight on the fact that the expected fingerprint is at a particular relative position in the image, and this can improve the reliability of the fingerprint finding algorithm and, for example, help reject erroneously made fingerprint matches. The same is true for information about the height and width of each finger. Additionally, the color of the user's skin can be stored at enrollment and used as a further biometric identification and / or liveness verification measure. This has the advantage that spoofed fingerprints with the correct fingerprint but incorrect skin color (e.g., a pink silicone mold, or a black and white laser print from a recovered latent fingerprint) are rejected as spoofs.

[0089] FIG4 depicts an exemplary routine 400 for performing fingertip detection from captured finger images and corresponding images. As shown, a captured image including multiple fingers is obtained at step 405. An exemplary high-resolution image is shown as image 405a. At step 410, the image is reduced / converted to a grayscale image, and a primary cascade is applied to detect hands within the image. An exemplary grayscale image and multiple boundaries drawn around detected hand regions are depicted in image 410a. At step 415, the largest detected hand region is selected and expanded to include a surrounding region for further fingertip detection (e.g., an expanded region). An exemplary grayscale image and boundaries drawn around the selected and expanded finger regions are depicted in image 415a. Then, at step 420, one or more higher-sensitivity cascade classifiers are applied to detect secondary regions, i.e., the fingertip regions of each finger. An exemplary grayscale image and boundaries drawn around multiple detected fingertip regions are depicted in image 420a. As shown, the number of detected fingertip regions may exceed the number of actual fingertips in the image. Then, in step 425, the fingertip region is filtered. Figure 4BFiltering is further described. An exemplary grayscale image and a border drawn around the filtered detected fingertip region are depicted in image 425a. Then, at step 430, the fingertip region of interest (ROI) is adjusted (e.g., resized or extended downward) to correct the aspect ratio. An exemplary grayscale image and a border drawn around the detected and resized ROI are depicted in image 430a.

[0090] Finger detection is preferably robust to both indoor and outdoor lighting, where images captured using lighting can differ significantly. For example, in low-light environments, the background is often underexposed and darkened, whereas in strong, diffuse sunlight, the background brightness may exceed the brightness of the finger, and shadows may differ significantly. Therefore, in some implementations, the finger detection method can be improved by having the mobile device processor determine the amount of ambient light and, based on the detected light level, switching to a better light path for a specific light level in real time. For example, the light level can be read from a hardware-based light meter, such as those found on mobile phones for adjusting screen brightness, or estimated from camera exposure settings.

[0091] In one such implementation, one or more classifiers specific to various light levels can be stored and made available to the processor to perform finger segmentation. For example, a first cascade of classifiers for detecting one or more regions of a finger can be trained on images captured in high ambient light, while a second cascade of classifiers can be trained on images captured in low ambient light. Based on the measured light levels, the configured mobile device processor can apply the appropriate classifier. More specifically, the first classifier can be used by the processor as a default for detection unless the light level exceeds a threshold, in which case the second classifier can be used. Switching between classifiers can occur in real time, for example, if a sequence of image frames with different ambient lighting levels is being captured and analyzed. It will also be appreciated that the above-described method of applying ambient light-specific classifiers can be applied to images initially captured during processing (e.g., a low-resolution image captured when a user positions their finger in an appropriate position in front of the camera) or to subsequent high-resolution image captures (e.g., a high-resolution image captured after a finger is detected and determined to be in focus in the initial image capture).

[0092] Furthermore, in some implementations, based on the measured light level, the configured mobile device processor can selectively implement artificial flash image preprocessing steps as further described below. For example, when the light level is high enough, artificial flash processing can be applied to avoid illuminating the object with a flash.

[0093] No single method of finger detection guarantees 100% success, however, the processor can be configured to calculate a metric of detection quality and, based on that metric, apply a series of detection methods until a sufficiently high quality result is achieved. For example, in the event that all four fingers are detected, as further described herein, the configured processor can calculate a score to determine which set of four detections is most likely to represent a hand. If the score is poor (e.g., does not meet a specified threshold) or a finger is missing, the configured processor can apply a further detection technique. The further detection technique can be in the form of a differently trained classifier, or some other unrelated method. In addition, in some implementations, the configured processor can estimate the position of the missing finger based on known hand measurements of the user, for example, the known hand measurements of the user are determined from a previous enrollment or authentication capture. It will be understood that the specific order in which these methods are applied is not necessarily fixed, and that the specific detection techniques implemented and the order in which they are applied can be selectively applied by the processor as a function of measured environmental conditions, the capabilities of the specific mobile device hardware, or customized to a specific user over time (e.g., based on training and / or machine learning algorithms). In view of the above, it will be appreciated that, in order to improve detection speed, the processor may be configured to apply a tiered segmentation approach, wherein a faster (and potentially less accurate) segmentation algorithm is applied first, and if the quality of the results is insufficient, a transition is made to a more robust (and sometimes more processing-intensive) segmentation algorithm to more accurately detect the fingertip segments.

[0094] As described above and further described herein, an exemplary finger-based recognition algorithm implemented by a mobile device processor may include one or more image enhancement steps to improve finger detection and feature extraction. Because detection methods such as cascade classifiers typically work on grayscale images, for example, if only luminance (luma) is used as input, color information is lost. Therefore, by enhancing regions representing expected colors before converting to grayscale, methods for detecting objects with known color characteristics (such as human hands) can be beneficially improved.

[0095] In one exemplary implementation, an image preprocessing method suitable for finger detection implemented by a processor includes an adaptive skin model. More specifically, the processor can be configured to analyze one or more captured images and locate regions of known skin color, for example by detecting hands within the image, and then calculate a color model. The image is then converted to HSV color space, and a probability density function (PDF) is fitted to the distribution of both hue and saturation values ​​of pixels within a predetermined skin region. The remaining pixels in the image are located within the PDF, and a probability (p-value) is extracted, indicating the likelihood that the pixel represents skin. Preferably, this process is iterative, where all pixels exceeding a threshold p-value are used to refine a previous model, and the updated model is then applied to all pixels using the processor. In some implementations, by assuming that the skin region is continuous, pixels with low p-values ​​that are surrounded by pixels with higher p-values ​​can also be included in the model. This process can be paused after a fixed number of iterations or when the number of skin pixels no longer increases significantly (i.e., no longer increases by a specified amount, converges). The converged p-value can then be used directly (converted to a grayscale image) as input to further detection algorithms, or additionally or alternatively, it can be used to brighten skin areas in the image relative to background, non-skin areas (e.g., as an "artificial flash").

[0096] When a finger is presented to the mobile device camera in a relatively predetermined position (e.g., guided by on-screen guidelines), the processor can be configured to assume that a particular area is likely to represent skin color (e.g., the central area of ​​the hand in the guidelines). Thus, this assumed area can be used as an initial area for constructing a skin model. Additionally or alternatively, skin color can be recorded when the user enrolls with the system (e.g., done without using a skin model).

[0097] Figure 4B An exemplary routine 450 for filtering detected fingertip regions / segments is depicted (ie, Figure 4A Filtering generally refers to selecting the best set of fingertip segments (ie, selecting the fingertip segment for each finger that most likely corresponds to an actual fingertip segment).

[0098] The process begins at step 455, where fingertip detections are sorted in the horizontal ("X") direction (e.g., arranged perpendicular to the fingers according to their order). Then, at step 460, a combination of four fingertip regions is generated using the plurality of detected fingertip regions. An exemplary grayscale image and a boundary drawn around the combination of four detected fingertip regions are depicted in image 460a.

[0099] Then, in steps 465-480, each of the combined sets of four fingertip regions is scored. Scoring includes analyzing the fingertip regions to determine physical characteristics of individual fingertip regions and / or multiple fingertip regions, and comparing the measured characteristics to expected characteristics. As further described herein, scoring can be based on a comparative analysis of the physical characteristics of one or more fingertip segments relative to other fingertip segments, and additionally or alternatively, can be based on a comparative analysis of the physical characteristics of one or more fingertip segments relative to multiple fingers, such as the total width of a previously detected hand region (e.g., the "hand width" detected in step 415).

[0100] More specifically, in some implementations, the combined widths of the detections in the set can be compared to the hand width and scored based on the comparison. Additionally or alternatively, the detected width distribution (e.g., the center-to-center distance between adjacent finger segments) can also be scored relative to the expected width distribution of the finger segments given the hand width. The expected width distribution can be determined as the average of a training set of previously identified fingers. The training set and the set can be normalized based on hand width for accurate comparison. For example, image 470a, an exemplary grayscale image of four fingers, a boundary drawn around a combination of four detected fingertip regions / segments, and the measured middle-to-middle distances between adjacent segments d1, d2, and d3.

[0101] In some implementations, a weight can be assigned to each specific comparison so that the calculated score is a function of the weight. For example, less decisive / important measures (e.g., measurements with lower precision or accuracy or with lower reliability) can be discounted by being assigned a lower weight so as not to distort the overall result of the score. By way of example and not limitation, because the relative length of the pinky finger has a high degree of variability between individuals, the impact of the measured distance in Y determined in relation to the pinky finger can be correspondingly "down-weighted." Figure 4C Table 470B in

[0065] depicts exemplary widths, weights, and expected widths for scoring relative distance features. As shown in Table 470B, an exemplary expected relative distance between adjacent fingers is 1 / 4 of the total width of the four fingers, and each is assigned a weight of 1.

[0102] At step 475, the width of the fingertip segments can also be scored relative to the other fingertip segments. The comparison of finger widths can be based on the expected relative widths of specific fingers. For example, the index finger is expected to be larger than the distal fingers, so the relative width of the fingertip regions / segments can be scored based on this individual finger segment comparison. An exemplary grayscale image of a finger and four possible detected fingertip regions corresponding to the two middle fingers (index and middle) are depicted in image 475a.

[0103] Similarly, at step 480, the relative position of the fingertip region in the Y direction can be scored based on the expected lengths of the various fingertip segments. For example, the middle two fingers are generally expected to be taller in the Y direction relative to the distal fingers, and the fingertip segments can be scored based on such expected relative position characteristics. Therefore, the distribution of the heights of the fingertip segments in the Y direction (i.e., a direction parallel to the orientation of the fingers) can be analyzed. More specifically, analyzing the distribution in Y includes analyzing the "length pattern" of the fingers as depicted in 480a. That is, the index finger is expected to be shorter than the middle finger, which is longer than the ring finger, which is longer than the pinky finger. Therefore, the region of interest for this subject should have a position from the index finger to the pinky finger that follows an "up, down, down" pattern in Y. The exact expected pattern can be determined as the average of a training set of previously identified fingers. It will be appreciated that the training set and the set of fingertip segments can be normalized based on the respective finger and / or hand size to accurately compare relative lengths and / or positions in Y. Thus, the processor can be configured to calculate the distance in Y between the top boundaries of each region / segment of interest, resulting in three distances: index finger to middle finger, middle finger to ring finger, and ring finger to pinky finger. The processor can then normalize the distances using the hand width to make them comparable across hands of different sizes. Thereafter, the distances can be compared to the expected pattern, and combinations of fingers can be scored based on the comparison. An exemplary grayscale image of a finger and four possible detected fingertip regions compared in the Y direction are depicted in image 480a. It will also be appreciated that the relative height, width, Y position, and X position of the fingertip regions can also be weighted according to importance and / or reliability.

[0104] In addition to the above measures, the configured processor may score combinations of fingertip segments based on the illumination characteristics of the depicted fingertip segments. More specifically, fingers may be expected to appear in an image with approximately equal illumination. Therefore, the configured processor may measure the illumination across the combination of fingertip segments for each combination of fingertip segments and score the differences in illumination. For speed and accuracy, only the pixel values ​​at the center of each fingertip segment (e.g., within a 10×16 rectangle) may be added together and the differences in the four sums determined. A high difference means that one or more fingertip segments are incorrectly located and may be assigned a worse score.

[0105] The cumulative scores of the combinations of fingertip regions are then weighted, summed, and the best combination of segments is identified based on the calculated scores at step 485. An exemplary grayscale image of a finger and a boundary drawn around the best scoring combination of four detected fingertip regions are depicted in image 485a.

[0106] Furthermore, in some implementations, the check can be performed by analyzing the frequency and direction of edges within each region.Additionally or alternatively, segments of the image containing the finger can be identified as the segment(s) that primarily fill the location of the finger positioning guide on the screen.

[0107] In some implementations, where, for example, only four (4) fingers are used for identification, a robust procedure for enrolling and verifying four (4) fingerprints may operate as follows: a) Instruct the user to place their four (4) fingers in front of the camera and capture a flash image. b) Optionally use image processing algorithms (as described above) to identify the locations of the four fingerprints (and other areas of interest). c) Highlight these areas to the user by, for example, superimposing ellipses over the fingerprint areas and requesting the user to accurately accept the fingerprint identification, or to adjust misplaced fingerprint ellipses by dragging the ellipses to the correct location. This ensures accurate enrolled fingerprints. d) Use the accurate enrolled fingerprints for future verification procedures. This may include using the enrolled fingerprints to find the verification fingerprint in the verification image.

[0108] Furthermore, in some implementations, where four (4) fingers are captured, the detected image of the four fingers can be divided into four separate fingers by defining seams between each adjacent finger, for example by locating points where there are perturbations in the ridge direction, which are referred to as singular points. The K-means clustering algorithm can then be used to cluster the identified points into four (4) clusters representing the four fingers. In some implementations, K-means can use a special distance function to calculate the distance matrix to be used in the clustering algorithm. This special function will make the distance measurements of points located on the same finger smaller even if they are far relative to the traditional Euclidean distance. The region growing segmentation algorithm can then be used to segment each finger individually.

[0109] Then, for each finger, at least one region of the distal phalanx of each finger can be identified within the image. Preferably, the finger region located between the tip of the finger segment and the thicker line between the middle and distal phalanx is identified as having the most distinctive features containing details.

[0110] Both fingers and hands have a relatively limited space of possible shape configurations, so in some implementations, active shape models and active appearance models can be used to implement contactless fingerprint recognition. For example, to locate and segment a hand from a target image, a point distribution model is first calculated by placing a set of points on hand features (such as fingertip boundaries) in the example image. The model is then initialized within the target image by forming an initial estimate of the hand position using user hand placement guidance or other image processing techniques. For example, a cascade classifier can be used to provide an initial estimate of the hand position. The best fit of the model is then found by iteratively comparing it to the image data and updating the point positions.

[0111] The points of the adapted model are used to extract the region of interest for recognition. For example, the points describing the boundaries of a fingertip are used to extract a fingerprint.

[0112] Similarly, active shape models describing finger shapes can be used to segment individual fingertips. For example, a cascade classifier is first used to find image regions containing fingertips, and then the model is used to segment them to remove background and adjacent fingers. Furthermore, active shape models can be tailored to individual users. For example, given a correct model fit, confirmed by the user during system enrollment, the model is adjusted to better describe the shape of the individual's hand and fingers. This increases the speed and reliability of recognition, and deviations from the model can be used to detect spoofing.

[0113] To achieve maximum fingerprint extraction quality, users can be prompted to position their hands and fingers in the optimal position relative to the illumination source and the camera, for example, by providing visual guides or outlines of optimal finger placement on the device's display. This could be positioning the fingerprint near the center of the camera's field of view, with the fingerprint subtending a maximum angle of approximately + / - 20 degrees toward the camera. For example, the finger can be placed far enough from the light source to minimize the angle of incidence with the illuminating light, preventing loss of detail on tilted surfaces, while being close enough to sufficiently intense illumination. At the same time, the finger is positioned toward the camera to maximize reflected illumination and close enough to the camera to ensure sufficient pixel density for recognition.

[0114] The quality of the captured fingerprint can be further improved by adding additional illumination sources or extending the spatial range of the illumination source to the smart camera system. For example, by adding four LEDs in the corners of a smartphone or tablet, the light will be favorably reflected by more areas of the fingerprint, resulting in higher fingerprint capture quality.

[0115] Once the relevant regions of the finger are identified, then at step 315, the relevant regions can be enhanced. More specifically, the mobile device processor 110, configured by executing software modules 130 including a preferred analysis module 172, can process portions of the image to enhance the details of the image, for example using a set of Gabor filters tuned to a smooth ridge orientation map. In some implementations, the primary purpose of this image enhancement is to generate a fingerprint image that is similar to images of fingerprint impressions captured using a live scan sensor and typically stored in a traditional database such as IAFIS. This similarity means that the image captured using the mobile device simulates the same qualities and properties as an image captured from a live scan sensor. This similarity is desirable to ensure the likelihood of matching an image captured by the mobile device with an image of a fingerprint impression stored in a traditional database such as IAFIS.

[0116] To improve the extraction of distinctive features from fingertips, it is advantageous to use an enhancement filter to enhance the contrast between ridges and furrows. In some implementations, the mobile device processor can apply histogram equalization to improve local image contrast by evenly distributing the intensity over a range of possible values ​​(typically [0, 255] in grayscale images). This can be achieved by computing a cumulative histogram of pixel intensities, normalizing to the maximum value within the allowed range, and remapping source pixels according to their position in this distribution.

[0117] Contrast enhancement has the disadvantage of not distinguishing between, and therefore enhancing, background noise and the signal of interest. Therefore, it may be beneficial to isolate only the signal of interest by filtering before contrast enhancement. For example, a processor can apply a bandpass filter to remove signals with frequencies that do not correspond to the expected frequencies of fingerprint ridges. One such implementation removes high frequencies by subtracting a Gaussian blur filtered source image from the original source image. The result can then be filtered again by applying another Gaussian blur filter with a suitably smaller radius to remove low frequencies. Histogram equalization can then be applied to the bandpassed result to obtain an optimal image for feature extraction.

[0118] In step 320, minutiae are extracted for each finger and a biometric identifier is generated. As will be understood by those skilled in the art, minutiae refer to the points where ridges of a fingerprint end, and texture refers to the pattern defined by the ridges. More specifically, the mobile device processor 110, configured by executing software modules 130, including preferably analysis module 172, analyzes the enhanced image to extract features from at least the distal region of each finger using an algorithm, such as a minutiae extraction algorithm.

[0119] Most automated systems for fingerprint comparison are based on minutiae matching; therefore, reliable minutiae extraction is a critical task. Many such methods require converting a grayscale fingerprint image into a skeleton image. A simple image scan then allows the detection of pixels corresponding to minutiae where fingerprint ridges end and bifurcate. The extracted minutiae can be stored as a collection of points in a two-dimensional plane.

[0120] Finally, the configured processor can execute minutiae-based matching algorithms to generate similarity scores between fingerprints. These matching algorithms calculate similarity scores between fingerprints by finding an alignment between a template and an input minutiae set that results in a maximum number of minutiae pairings.

[0121] The features extracted from the distal region may be stored with other features similarly extracted from the remaining identified regions of the finger and / or hand.These features may be characterized in one or more biometric identifiers comprising one or more feature vectors.

[0122] During enrollment, such a feature vector is stored in memory as a biometric identifier (e.g., template) for use in the user verification step at step 325. Alternatively, during user verification (step 330), the biometric identifier is compared to the version stored during enrollment.

[0123] More specifically, during the verification process, the user's finger is captured and a biometric identifier is generated as described with respect to steps 305-320. However, at step 330, the query feature vector is then compared with the registered and stored feature vectors. Based on the comparison, a match score is generated by the configured processor 110 that correlates to the similarity of the match. If the match score indicates a sufficiently close match, the user can be determined to have passed the verification process.

[0124] In one or more implementations, the match score can be a combined match score based on individually matching the query fingerprint (e.g., query feature vector) with the enrolled fingerprints and determining a combined match score. More specifically, from a database of hand images, the images can be paired, with two different types of pairs: pairs of images of the same hand and pairs of images of different hands. For each pair of fingers on these hands (e.g., index finger and index finger), a match score can be calculated that measures the proximity of these hand images, where a higher score indicates a closer match.

[0125] These scores can be plotted as a score distribution. For each type of finger (e.g., ring finger), there are two distributions: the score of matching images of the same finger from the same hand and the score of matching images of the same finger from a different hand (i.e., imposters).

[0126] These score distributions can be thought of as probability distributions that give the probability that a given match score belongs to one of the distributions.These empirically derived distributions can be smoothed for noise and tightly characterized by matching them to known distributions, such as the gamma distribution.

[0127] Given an uncharacterized pair of finger images, the exemplary recognition system can be configured to determine a matching score. These adapted probability distributions can then be used to determine a ratio (likelihood) of the probabilities that the pair of finger images belong to the same finger or different fingers.

[0128] When performing full four-finger matching, the configured system can test an unknown image (a "probe" image) against previously enrolled images of known objects ("stock" images). For each pair of probe fingers versus stock fingers, the system can determine a likelihood ratio. These ratios can then be multiplied together, with the final result providing an overall measure of the chance that the probe image belongs to the object for which the stock image was provided.

[0129] This method has the advantage of not being degraded by particular fingers with poor predictive power; in particular, small fingers are less likely to provide significantly more predictive matches than other fingers. It also allows some tolerance for bad images; if one finger gets a poor match, that one finger can be compensated if another gets a particularly good match.

[0130] While combining scores across multiple fingers provides some tolerance for poor images, it's theoretically possible to make a single probe / gallery match score large enough to result in an overall pass. This could make spoofing more likely, for example, if an attacker were able to produce a very high-quality facsimile of one of the authorized user's fingers. An exemplary approach to mitigating this issue could include requiring a minimum number of probed fingers to individually produce a match score that crosses a secondary threshold during the matching and scoring process, and requiring a combination of match scores to pass a primary match threshold to determine a positive match. Thus, such a measure would require that any successful spoof successfully replicate a minimum number of fingers, a more difficult task than successfully replicating a single finger. It will be appreciated that the minimum number of fingers required to pass the secondary threshold and the value of the secondary threshold can be adjusted to mitigate the risk of spoofing based on the restoration of degraded image quality, depending on the security requirements of the implementation.

[0131] Typically, when comparing query finger data to enrolled finger data, it is important to ensure that the scale of the respective images is similar. Therefore, during analysis of the fingerprint image at step 320, the configured processor 110 may determine the fundamental frequency of the fingerprint ridges. During enrollment (e.g., step 325), the configured processor may store the fundamental frequency. During verification (e.g., step 330), the configured processor may scale the fundamental frequency of the verification fingerprint to match the fundamental frequency of the enrolled fingerprint prior to comparison. Additionally or alternatively, the processor may normalize the frequencies of the fingerprint to a specified reference frequency, such as one (1), so that the actual frequencies do not have to be stored. Thus, during recognition, the query fingerprint may be normalized to a specified reference value.

[0132] It should be understood that one or more pre-processing operations may be performed on the image frame prior to generating the feature vector or prior to comparing the feature vectors during authentication. By way of example and not limitation, pre-processing of the image data prior to analysis may include orienting the image frame in a coordinate space, etc., as will be understood by those skilled in the art.

[0133] Existing techniques for image-based fingerprint recognition that implement existing resizing algorithms typically incorrectly resize up to approximately 2% of fingerprints, leading to false rejections during authentication. This is due in part to those algorithms using an insufficient number of reference points in the image (i.e., using only two points, the tip / starting point of the fingerprint and the base / end point of the fingerprint) to reduce the size and resize accordingly. To improve the resizing operation, in accordance with one or more of the disclosed embodiments, a processor can implement an algorithm that analyzes the average frequency of the fingerprint (e.g., the typical number of lines per inch) and thus normalizes the size of the fingerprint. Because the technique determines resizing based on a larger set of fingerprint pixels captured at many points across the fingerprint area, a relatively high degree of reliability can be achieved during the resizing process.

[0134] Before and / or after the registration and verification steps, the method may further comprise a step of detecting activity. Figure 3 Step 335 depicts liveness detection. A liveness detection method can be implemented to verify that the captured four (4) finger images are from real fingers and not, for example, spoof prints or molds of fingers. More specifically, in some implementations, the mobile device processor 110, configured by executing software modules 130 (preferably including analysis module 172), can analyze the quality of the finger images and determine whether they are consistent with images from live fingers and / or fake fingers, where images of live fingers and / or fake fingers typically have noticeable artifacts, such as reduced resolution and sharpness.

[0135] For example, as further described herein, one liveness detection technique can be to prompt the user to rotate their hand during imaging, and the configured processor can determine whether the imaged hand is appropriately three-dimensional using, for example, depth from motion techniques and depth from focus techniques. Alternatively, the system can implement techniques for passive liveness detection, such as analyzing image quality to check whether it is sufficiently sharp and not low resolution (e.g., from a spoofed print of the hand). The configured processor can also analyze the color of the fingers to determine whether the coloring is consistent with the live hand image and / or the known color of the user's hand. Therefore, in some implementations, hand color consistency, in other words, color uniformity, can be performed by simultaneously detecting the fingertips and the hand. Then, a region of the hand consisting of the palm and lower phalanges (i.e., proximal and middle) that does not contain the fingertips is separated, and the color histogram of this region and the color histogram of the four detected fingertip regions are determined. Finally, in some implementations, comparing these two histograms can be used as a test of the color uniformity of the hand and fingers to determine a liveness measurement, especially if an attacker uses a stencil (i.e., a fake finger) to deceive the system. Additionally, the processor may be configured to request that the user perform one or more gestures with their fingers, such as opening and closing the fingers or moving certain fingers in a particular manner.

[0136] Furthermore, in some implementations, a classifier can be trained to distinguish between liveness and spoofing. The classifier can be trained so that it learns the difference between a real finger image and various spoofing images. The processor implementing the classifier can then be configured to provide a pass / fail result based on its training.

[0137] Furthermore, in some implementations, as an additional factor in biometric liveness, the position of the fingerprints in the image can be considered, i.e., a genuine user will have first, second, third, and fourth fingers of a particular length. Thus, when the user has the fingers of their hand extended and closed together, the positions of the four (4) fingerprints should have relative positioning consistent with that particular user. This information can be used as an additional security check to prevent spoofing attacks. For example, a hacker who discovers a latent fingerprint on a phone screen is unlikely to infer the length of the user's fingers and therefore is unlikely to present them correctly.

[0138] about Figure 3 Further exemplary systems and methods are described, contemplating various alternatives and permutations. In some implementations, the enrollment image of the user's finger need not be captured by the mobile device camera. Instead, the finger feature vector can be obtained from an alternative source, such as a pre-recorded database of finger images.

[0139] In some implementations, during the enrollment process, in order to capture the fingers (e.g., but not limited to, four or ten fingers with improved resolution), images of each finger can be captured sequentially in respective images. In this case, during the enrollment process, a processor configured to display finger guidance on the screen can prompt the user to place one finger on the screen at a time, and a segmentation algorithm can be used to separately identify the distal phalanx region and the fingerprint region of the finger.

[0140] In some implementations, in addition to limiting the matching process (e.g., step 330) to comparisons of the fingertip (e.g., distal phalanges) region, the comparison can include other parts of the finger in addition to or in lieu of fingerprints. For example, the region of interest can include any part of the hand with a detectable pattern, or the distal and middle phalanges or metacarpals. Some of these regions have the advantage that they are more resistant to spoofing attacks, thereby providing a higher level of security. For example, a user's fingertip fingerprints can often be found on smartphone casings or other surfaces touched by the user. These latent fingerprints could be copied by an imposter, and a mold could be created that could potentially pass verification. However, palm prints on the metacarpals are more difficult to find because it is uncommon for these areas of the hand to contact a surface to leave latent patterns.

[0141] In some implementations, instead of using a singularity point to separate the four finger clusters into individual fingers, the user can be prompted to spread their fingers during capture. A segmentation algorithm is then used to isolate the fingers, and a contour deformation method can be used to identify the location of each fingertip.

[0142] In some implementations, skin color, frequency, and orientation can be used to perform segmentation of the relevant finger region. For example, a Sobel operator can be implemented by a configured processor to emphasize the region of focus (i.e., the finger rather than the background) to aid in the segmentation process. Additionally or alternatively, segmentation can also be performed by simply extracting a fixed region from the captured image that is relevant to the region where the user was directed to place their finger during the capture process.

[0143] In some implementations, segmentation can be performed during the authentication process using enrolled fingerprint information. Segmenting and identifying and / or matching fingerprint features based on a fingerprint template generated during enrollment can provide improvements over existing techniques. For example, existing image-based fingerprint recognition techniques isolate fingerprints in the same manner during enrollment and authentication, and therefore have unsatisfactory success in isolating individual fingerprints from an image for reliable use. In some examples, successful isolation using existing methods occurs only 96% of the time, resulting in a 4% false rejection rate during authentication. This problem is further compounded by using the technique separately on multiple fingers.

[0144] However, in accordance with one or more of the disclosed embodiments, a different algorithm for fingerprint isolation is performed by the configured processor, i.e., an algorithm that uses the enrolled fingerprints to find the finger and isolate / match the fingerprint during authentication. This provides significantly more robust performance. In some implementations, the configured processor can implement the segmentation process, for example, by extracting finger features (e.g., minutiae) from the entire four finger images, and locating the finger region by exhaustively comparing all locations in the image with the finger features from the enrolled fingerprints. The finger region will be known to be located at the location where the enrolled finger is found to match the finger features in the image. In addition, to minimize the possibility of false matches of random features in the image, the validity of the matching region can be checked, for example, in the case of a four (4) finger capture procedure, by ensuring that the first, second, third, and fourth fingers are found in a generally expected manner based on user guidance overlaying the image, the skin color is as expected, etc. (e.g., using template matching of an enrolled template to guide the comparison). Moreover, rather than using the process to search the entire image to find the finger location, the search range can be limited to the region of the expected finger from the user guidance overlaying the image.

[0145] In addition to or alternatively to basing the finger resizing on the fundamental ridge frequency, the processor 110 may be configured to resize the finger based on one or more of: the width of the segmented four finger clusters, the width or length of each finger region of interest, or a specific point on the finger, such as a single point and thicker line at a phalangeal joint.

[0146] In addition to (or instead of) matching fingers based on minutiae, the processor 110 may also be configured to match fingerprints based on texture.

[0147] Furthermore, in some implementations, instead of using one image for a finger, multiple images can be used to enroll or authenticate a user. Multiple images can be captured by the configured processor 110 using the camera 145 at various exposures and / or focal lengths to create images with enhanced depth of field and / or dynamic range. Capturing images with such varying exposures and / or focal lengths can help ensure that the focus of the fingerprint at various locations on the hand is optimal. Thus, the configured processor can select and analyze the image(s) or portion(s) of the image(s) that have the best focus on the portion of the finger of interest.

[0148] Additionally or alternatively, liveness detection can be accomplished by checking other indicators consistent with a real finger rather than a print or video or molded finger spoof. These indicators can include analyzing the specular reflection of the flash captured from the image, analyzing the specular reflection from the flash compared to images taken without the flash, color, color indicators (thereby rejecting black and white and monochrome spoofs).

[0149] In some embodiments, activity can be detected by analyzing specular reflection or depth-from-focus information obtained from an image of a finger. As non-limiting examples, exemplary systems and methods for determining activity based on specular reflection and depth-from-focus information are described herein and in co-pending and commonly assigned U.S. patent application Ser. No. 62 / 066,957, filed Oct. 15, 2014, entitled “SYSTEMS AND METHODS FOR PERFORMING IRIS IDENTIFICATION AND VERIFICATION USING MOBILED EVICES UTILIZING VISIBLE SPECTRUM LIGHTING,” which is incorporated herein by reference as if fully set forth herein. Activity can also be detected by analyzing dynamic movement of a finger (e.g., a finger gesture), such as tilting the finger or expanding / contracting the finger as depicted by a sequence of images captured by a camera. As non-limiting examples, this document and co-pending and commonly assigned U.S. patent application serial number 62 / 041,803, filed on August 26, 2014, entitled “SYSTEM AND METHOD FOR DETERMINING LIVENESS,” describe exemplary systems and methods for liveness determination based on dynamic movement of biometrics and gestures, the entire contents of which are incorporated herein by reference.

[0150] In some implementations, activity can be detected by analyzing the reflectivity of light emitted onto the ridges of a finger during the capture of the finger image. The ridges of a live finger reflect light unevenly, while the ridges of a printed finger reflect light evenly. Therefore, the ridge reflectivity characteristics captured in the image of the finger can be analyzed to determine activity. Figures 5B-5C The corresponding images shown further illustrate an exemplary process for determining activity based on reflectivity. At step 505, input to the activity detection algorithm is obtained. The input includes a high-resolution image(s) of one or more fingers captured with the flash on, and a high-resolution image of the finger captured without the flash on. Figure 5B Exemplary flash-on 505A and non-flash images 505B of a finger are shown. At step 510, the flash-on image is resized so that the fingerprint within the image is isolated. At step 515, the region in the non-flash image that includes the corresponding finger is segmented (e.g., according to the exemplary fingerprint segmentation algorithm described above). Thus, the fingerprint depicted in the flash-on image and the corresponding fingerprint in the non-flash image are isolated for further processing. Exemplary isolated flash-on and non-flash images of a fingertip are shown as Figure 5BThen, at step 520, a high pass filter is applied to retain a portion of the image depicting the ridges. Exemplary filtered flash-on and flash-off images of the fingertip are shown as Figure 5B Then, at step 525, the activity score is calculated. In one exemplary arrangement, the activity score is calculated based on the standard deviation (a) of the histogram generated from the filtered non-flash image and the corresponding standard deviation (b) of the histogram generated from the filtered flash-on image (i.e., activity score = a / b). As an example, in Figure 5C A similar image obtained during application of process 500 to a spoof image of a fingerprint is depicted in . In other implementations, other measures can be calculated from the histograms of the filtered flash and no-flash images to calculate the liveness score. The following are some examples of measures that can be used: (1) the difference between the means of the histograms, (2) the difference between the means of the histogram frequencies, (3) the ratio of the standard deviations of the histogram frequencies, (4) the difference between the kurtosis of the histograms, and / or (5) the number of corresponding keypoints in the filtered flash and no-flash images. In some implementations, the difference in pixel intensity of the background of the flash and no-flash images can be used as a liveness measure.

[0151] Figures 6A-6F Depicted are exemplary ridge images of a finger captured at various positions relative to the camera's field of view. Specifically, Figure 6A Depicted are the captured images and corresponding ridge images of a finger that is too far from the camera and has low fingerprint resolution. Figure 6B Depicted are the captured images and corresponding ridge images of a finger that is too far from the camera and has low fingerprint resolution. Figure 6C Depicted are a captured image and corresponding ridge image showing good resolution due to placement of the finger centered in the field of view and sufficiently close to the camera. Figure 6D Depicted are captured images showing reflection loss at the edges of the index and pinky fingers due to high-angle LED reflections and corresponding ridge images. Figure 6E Depicted are captured images showing reflection loss at the tip of the finger due to high-angle LED reflections and corresponding ridge images when the finger is placed near the edge of the camera's field of view. Figure 6F Depicted are captured images showing reflection loss at the tip of the finger due to high-angle LED reflections and corresponding ridge images when the finger is placed near the edge of the camera's field of view.

[0152] In some implementations, fingerprint-based authentication can be further combined with facial recognition to provide enhanced security / reliability of multimodal biometrics. For example, in the case of a smartphone, the user's four fingers can be captured using the smartphone's rear camera simultaneously or sequentially with face and / or iris capture using the front camera. As a non-limiting example, exemplary systems and methods for generating hybrid biometric identifiers and performing identification / authentication using hybrid biometric identifiers are described herein and in co-pending and commonly assigned U.S. patent application Ser. No. 62 / 156,645, filed May 4, 2015, entitled "SYSTEM AND METHOD FOR GENERATING HYBRID BIOMETRIC IDENTIFIERS," the entire contents of which are incorporated herein by reference.

[0153] As a further example, in addition to characterizing the user by generating a finger feature vector according to routine 300, as described above, additional biometric features can be extracted from the image captured in step 305 or from separately captured biometric information. Such additional biometric features may include, for example, but are not limited to, soft biometric features and hard biometric features. "Soft biometric" features are physical, behavioral, or congenital human characteristics, while hard biometric features such as fingerprints, irises, and periocular characteristics are generally invariant. As a further example, soft biometric features may include physical characteristics such as skin texture or skin color. Soft biometric features may also include motion detected by a smartphone's gyroscope / accelerometer, eye movement characteristics detected by eye-tracking algorithms, and head movement characteristics detected by tracking facial and / or head movements. Such biometric features can be extracted and characterized according to the aforementioned methods and existing biometric analysis algorithms. Furthermore, the additional characterization of the user's biometrics may be encoded as part of the biometric identifier generated in step 320 or otherwise included in a composite biometric identifier including a fingerprint biometric identifier, such as by fusing multiple biometric identifiers.

[0154] In one or more exemplary embodiments, image capture of a finger can be performed at a greater distance than is typically performed by a user using a handheld device such as a smartphone. Exemplary embodiments can be similarly implemented using a system configured to capture images using possible short-range to long-range image acquisition modes. Image acquisition at a distance can be performed using optical models such as various optical-based systems using telephoto lenses, as well as laser-focus-based systems and sonar-based systems. Applications of these types of longer-range image capture models are crucial in law enforcement, military, and intelligence, and may eventually be deployed in commercial environments.

[0155] Additionally, image capture can be performed where the subject is not stationary, such an implementation being referred to herein as a Fingerprint on the move (FOM) system. This type of opportunistic capture can occur over time as a person's fingerprint becomes visible to the special operators responsible for this work in covert operations and / or surveillance mode.

[0156] For long-range capture, super-resolution techniques may be implemented to improve fingerprint quality by using data from multiple frames and stitching partial fingerprint regions from different frames into a larger fingerprint image. As non-limiting examples, exemplary systems and methods for performing super-resolution techniques to generate identifiers based on multiple image captures and for using the same for identification / authentication are described herein and in co-pending and commonly assigned U.S. patent application Ser. No. 62 / 066,957, filed Oct. 15, 2014, and entitled “SYSTEMS AND METHODS FOR PERFORMING IRIS IDENTIFICATION AND VERIFICATION USING MOBILE DEVICE SUTILIZING VISIBLE SPECTRUM LIGHTING,” which is incorporated herein by reference.

[0157] Furthermore, it will be appreciated that the above-described processes for performing fingerprint capture and identification can be similarly performed using images captured in both NIR light and the IR spectrum and using devices equipped with NIR and / or IR illuminators. Such implementations may be particularly useful for incorporating vein pattern recognition as an additional biometric factor. As non-limiting examples, exemplary systems and methods for capturing biometric images in the NIR and IR spectral bands and performing identification / authentication using NIR and IR illuminators are described herein and in co-pending and commonly assigned U.S. patent application Ser. No. 62 / 129,277, filed on March 6, 2015, entitled “SYSTEMS AND METHODS FOR PERFORMING IRIS IDENTIFICATION AND VERIFICATION USING MOBILE DEVICES,” which are incorporated herein by reference as if fully set forth herein.

[0158] In this context, it should be noted that while much of the foregoing description is directed to systems and methods for authenticating a user based on user biometrics captured using a conventional smartphone device, the systems and methods disclosed herein may be similarly deployed and / or applied in scenarios, situations, and settings beyond the referenced scenarios.

[0159] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on any implementation or the scope of what may be claimed, but rather as descriptions of specific features of specific embodiments of specific implementations. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Moreover, although features may be described above as occurring in certain combinations and even initially requested to be protected as such, one or more features from a claimed combination can in some cases be deleted from that combination, and a claimed combination can point to a sub-combination or a variation of a sub-combination.

[0160] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve satisfactory results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments. Instead, it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0161] The terms used herein are only used to describe the purpose of specific embodiments and are not intended to limit the present invention. The singular forms "one" and "the" are used herein for the purpose of also including plural forms, unless the context clearly indicates otherwise. It will also be understood that when a term includes and / or is included in this specification, the existence of the specified features, integral bodies, steps, operations, elements, and / or components is specified, but the existence or addition of one or more other features, integral bodies, steps, operations, elements, components and / or their groups is not excluded. It should be noted that the use of sequential terms such as "first", "second", "third" etc. in the claims does not itself mean that the priority, priority or order of a claim element exceeds another, or the chronological order of the effect of the method of execution, but is only used as a label to distinguish a claim element with a specific name from another element with the same name (but in order to use ordinal items) to distinguish claim elements. In addition, the wording and terminology used here are for the purpose of description and should not be considered as restriction. The use of "including", "comprising" or "having", "containing", "involving" and its variants is intended to include the project and its equivalent and additional items listed thereafter. It should be understood that like reference numerals in the drawings represent like elements across several figures, and that not all embodiments or arrangements require all components and / or steps described and illustrated with reference to the figures.

[0162] Thus, the illustrative embodiments and arrangements of the present system and method provide a computer-implemented method, computer system, and computer program product for authenticating a user based on the user's biometric characteristics. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to various embodiments and arrangements. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions recorded in the blocks may not be performed in the order shown in the figure. For example, depending on the functions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order. It will also be noted that each block in the block diagram and / or flowchart illustration, and combinations of blocks in the block diagram and / or flowchart illustration, may be implemented by a dedicated hardware-based system or a combination of dedicated hardware and computer instructions that performs the specified function or action.

[0163] The above subject matter is provided by way of illustration only and should not be construed as limiting. Various modifications and changes may be made to the subject matter described herein without following the exemplary embodiments and applications shown and described, and without departing from the true spirit and scope of the present invention as set forth in the appended claims.

Claims

1. A method for performing fingerprint recognition, the method comprising: capturing an image by a mobile device having a camera, a storage medium, instructions stored on the storage medium, and a processor configured by executing the instructions, the image depicting a finger of a subject, wherein the camera of the mobile device captures the image; detecting, with the processor, a portion of a hand including at least one finger depicted in one of the images and a location of the detected portion in the image using an object detection algorithm configured to locate at least a portion of a hand, and wherein the object detection algorithm is a classifier trained to examine regions of an image to detect any region depicting at least one finger, and wherein the object detection algorithm is one or more of a HOG, LBP, and Haar feature classifier; identifying, with the processor, a fingertip segment of the at least one finger from one of the images based on a segmentation algorithm, wherein the segmentation algorithm is applied based on detecting the portion of the hand including the at least one finger and based on the location, wherein the segmentation algorithm is a classifier trained to detect any fingertip segment within a region determined to depict the at least one finger, and wherein the segmentation algorithm is one or more of a HOG, LBP, and Haar feature classifier; generating a biometric identifier comprising the extracted distinguishing features, wherein generating the biometric identifier comprises extracting the distinguishing features from the identified fingertip segment; as well as The processor stores the generated biometric identifier in the storage medium.

2. The method of claim 1 , wherein the object detection algorithm is trained to detect at least one of a hand, a finger, a fingertip, a middle phalanx, a palm, a finger joint, and a group of fingers, and wherein the segmentation algorithm is applied to the image based on the position of the part detected by the object detection algorithm.

3. The method of claim 1 , wherein the step of detecting the portion of the hand comprises: detecting, with the processor, a region of a hand depicted in the image and a location of the detected region in the image using a first object detection algorithm, wherein the region includes one or more of a hand, a finger, and a group of fingers; as well as Based on detecting the area of ​​the hand using the first object detection algorithm, the processor detects the fingertip using a second object detection algorithm. The method of claim 1 , wherein the segmentation algorithm is a classifier trained differently from the object detection algorithm.

5. The method of claim 1 , further comprising: Capture a sequence of images; as well as The portion of the hand and the corresponding position of the detected portion within each image in the sequence of images are detected.

6. The method of claim 5, further comprising: The processor determines, based on the corresponding position, that the detected portion of the hand is stable in position.

7. The method of claim 5, further comprising: dynamically adjusting focus and exposure settings of the camera based on corresponding positions of the portions detected in the image; as well as The camera is triggered using the focus setting and the exposure setting to capture a high-resolution image of the finger, and wherein the identifying step is performed using the high-resolution image.

8. The method of claim 1 , further comprising: comparing, with the processor, the biometric identifier with a previously stored biometric identifier associated with the user; as well as Based on the comparison, it is determined that the object is the user.

9. The method of claim 5, further comprising: displaying, with the processor, a near real-time image feed comprising the sequence of images on a display of the mobile device; displaying, with the processor, finger placement guidelines on the display, wherein the guidelines are overlaid on the feed of images to prompt a user to align a finger displayed in the feed with the guidelines; detecting that the finger is aligned with the guideline based on determining that respective positions of the portion of the hand detected in one or more of the images correspond to a particular position in the field of view of the camera; as well as In response to detecting that the finger is aligned with the guideline, triggering, with the processor, the camera to capture a high-resolution image, wherein the identifying step is performed using the high-resolution image.

10. The method of claim 9, further comprising: A camera setting including at least one of a focus setting and an exposure setting is defined according to the position of the finger placement guide.

11. The method of claim 5, further comprising: displaying, with the processor, a near real-time image feed comprising the sequence of images on a display of the mobile device; Using the processor, display a border on the display around at least a portion of a finger detected by the object detection algorithm, wherein the border is overlaid on the corresponding image based on a corresponding position of the finger detected within the corresponding image in the sequence of images, and wherein the position of the border is dynamically adjusted as the corresponding position of the finger changes throughout the sequence.

12. A system for performing fingerprint recognition, comprising: a mobile device having a camera, a storage medium, a display, and a processor in operable communication with the camera, the display, and the storage medium; a software application comprising instructions stored in code form on the storage medium, wherein the instructions are executable in the processor and configure the processor to: capturing an image with the camera, the image depicting a finger of the subject; detecting a portion of a hand including at least one finger depicted in one of the images and a location of the detected portion in the image using an object detection algorithm configured to locate at least a portion of a hand, and wherein the object detection algorithm is a classifier trained to examine regions of an image to detect any region depicting at least one finger, and wherein the object detection algorithm is one or more of a HOG, LBP, and Haar feature classifier; identifying a fingertip segment of the at least one finger from one of the images according to a segmentation algorithm, wherein the segmentation algorithm is applied based on detecting the portion of the hand including the at least one finger and according to the location, wherein the segmentation algorithm is a classifier trained to detect any fingertip segment within any region determined to depict the at least one finger, and wherein the segmentation algorithm is one or more of a HOG, LBP, and Haar feature classifier; generating a biometric identifier comprising the extracted distinguishing features, wherein generating the biometric identifier comprises extracting the distinguishing features from the identified fingertip segment; as well as The processor stores the generated biometric identifier in the storage medium.

Images