Slap Segmentation of Contactless Fingerprint Images

Abstract

To provide a method to segment slap images and to generate accurately labelled individual fingerprints.SOLUTION: An image processing workflow includes: receiving input images from a contactless fingerprint reader under controlled lighting conditions; computing a variance in the received images to estimate a slap area as a foreground slap mask in the input images; identifying individual fingers by finding boundary of each finger; verifying some fingers and geometric constraints; calculating pose and orientation based on shape and geometry information; identifying effective fingertip area on each detected finger according the pose, orientation, as well as geometric information; and outputting individual fingerprints.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for slap segmentation for contactless fingerprint images. [Background technology]

[0002] More specifically, the contactless fingerprint image is captured by a contactless fingerprint reader. Contactless fingerprint readers have emerged in recent years. Unlike traditional live-scan fingerprint readers that only capture fingerprints that touch the device, contactless fingerprint readers capture a picture of every finger / slap / palm that is presented to the device. In order to identify the location and valid area of ​​each individual fingerprint, the captured slap image needs to be automatically segmented.

[0003] Biometric data such as fingerprints play an increasingly important role in data security, border control, law enforcement, finance, health care, and other fields. In an Automatic Fingerprint Identification System (AFIS), identification relies on a fingerprint, a unique pattern of ridges and valleys on the surface of a fingertip. Traditional fingerprint images are obtained from ink marks on a ten-print card or are otherwise captured by touch-based fingerprint capture devices such as optical and capacitive readers. In recent years, the increasing demand for faster fingerprint acquisition, enrollment, and matching has led to the development of non-contact fingerprint readers, by which fingerprint images can be conveniently captured without any physical contact between the subject's finger and the device. Despite the modality change, both contact and non-contact fingerprint acquisition share one crucial step: the precise segmentation of the complete desired individual fingerprint from images captured typically by placing left / right slaps of four fingers, i.e., index, middle, ring, pinky, or two thumbs, on an inked card or fingerprint reader, and in particular from slap images, which refer to multiple flat fingerprints captured simultaneously.

[0004] Given the nature of inked cards and contact fingerprint readers, traditional contact slap segmentation approaches primarily use shape, texture, orientation, and geometric constraints to identify each fingerprint.

[0005] Such methods for performing segmentation on contact fingerprint images can be found in the following documents: U.S. Pat. No. 7,072,496 B2; Craig I. Watson, "Slap Fingerprint Segmentation Evaluation II - Procedures and Results"; NIST Interagency / Internal Report (NISTIR) - 7553, 2009; Brad Ulery et al., "Slap Fingerprint Segmentation Evaluation 2004 Analysis Report"; NIST Interagency / Internal Report (NISTIR) - 7209, 2005; Sklansky, J., "Finding the Convex Hull of a Simple Polygon"; Pattern Recognition Letters, Vol. 1, No. 2, pp. 79-83, 1982; Christian Wolf et al., "Text Localization, Enhancement and Binarization in Multimedia Documents"; International Conference on Pattern Recognition, Volume 4, pp. 1037-1040, 2002, Yong-Liang Zhang, Gang Xiao, Yan-Miao Li, Hong-Tao Wu, and Ya-Ping Huang. 2010. Slap Fingerprint Segmentation for Live-Scan Devices and Ten-Print Cards. Proceedings of the 2010 20th International Conference on Pattern Recognition (ICPR '10).

[0006] Contactless fingerprint readers capture slap images by high-resolution cameras in open space without any touch with the subject, which is much faster and easier, but makes traditional slap segmentation algorithms for contact fingerprint images fail in many cases due to subject modality changes.

[0007] In fact, it is very common that fingers in a slap image from a contactless reader are not clearly separated from each other, which is the most challenging scenario for traditional slap segmentation algorithms on touch-based devices, which are developed with the strong assumption that fingerprints from different fingers are well separated from each other.

[0008] Also, with or without guidance, the slap, i.e., the subject to be imaged, can be positioned with different poses relative to the device, which can dramatically change the appearance of the slap, such as its rotation, orientation, and location in the captured image.

[0009] Finally, compared to touch-based slap images, the non-touch one captures the entire slap, which is problematic for traditional methods that rely on texture information because the upper palm can exhibit patterns similar to fingerprints.

[0010] Therefore, it is desirable to design a new method and apparatus for accurate identification and segmentation of each individual fingerprint from a slap image captured by a contactless fingerprint reader, which is expected to be adaptive to various hand shapes, hand postures, and distances of the subject.

[0011] Further alternative and advantageous solutions would therefore be desirable in the art. [Prior art documents] [Patent documents]

[0012] [Patent Document 1] U.S. Patent No. 7,072,496 [Non-patent literature]

[0013] [Non-Patent Document 1] Craig I. Watson, "Slap Fingerprint Segmentation Evaluation II - Procedures and Results". NIST Interagency / Internal Report (NISTIR) - 7553, 2009 [Non-Patent Document 2] Brad Ulery et al., "Slap Fingerprint Segmentation Evaluation 2004 Analysis Report". NIST Interagency / Internal Report (NISTIR) - 7209, 2005 [Non-Patent Document 3] Sklansky, J., "Finding the Convex Hull of a Simple Polygon". Pattern Recognition Letters, Vol. 1, No. 2, pp. 79-83, 1982. [Non-Patent Document 4] Christian Wolf et al., "Text Localization, Enhancement and Binarization in Multimedia Documents," International Conference on Pattern Recognition, Vol. 4, pp. 1037-1040, 2002. [Non-Patent Document 5] Yong-Liang Zhang, Gang Xiao, Yan-Miao Li, Hong-Tao Wu, and Ya-Ping Huang.2010.Slap Fingerprint Segmentation for Live-Scan Devices and Ten-Print Cards.Proceedings of the 2010 20th International Conference on Pattern Recognition(ICPR '10) Summary of the Invention [Problem to be solved by the invention]

[0014] The present invention aims to propose a segmentation method applicable to non-contact acquired slap images. [Means for solving the problem]

[0015] The present invention, in its broadest sense, is a method for segmenting a slap image and for generating accurately labeled individual fingerprints, comprising the following steps: - receiving an input image from a contactless fingerprint reader under controlled lighting conditions; - calculation of a variance in the received image to estimate a slap area as a foreground slap mask in the input image; - Identification of individual fingers by finding the boundaries of each finger, - validation of some finger and geometric constraints, - Calculating pose and orientation based on shape and geometry information; - Identifying the effective fingertip area on each detected finger according to its pose, orientation and geometric information; - Individual fingerprint output The present invention is defined as a method including:

[0016] The present invention is a unique and efficient solution that combines the capabilities of both hardware and software in a contactless fingerprint identification system. The present invention only requires two images to calculate the variance and a minimal hardware setup to solve the problem.

[0017] Advantageously, from a hardware standpoint, the images captured under controlled lighting conditions are images captured with and without flash light.

[0018] The software portion of the method can therefore receive two slap images of the same subject, one with flash light on and one with flash light off, and the device can then transmit, process and store the captured images and the processed individual fingerprint images. From a software perspective, the proposed algorithm can adaptively estimate the position of each individual fingerprint and then generate location information for those fingerprints.

[0019] According to an advantageous embodiment, the individual finger identification step comprises an adaptive binarization sub-step applied to the original illuminated image with the foreground slap mask by calculating the dynamic range of both the global and local dynamics according to the formula Thres(x,y)=α(x,y)+pα(x,y)(β(x,y)-q), where α(x,y) is a local dynamic factor, β(x,y) is a global dynamic factor and p, q are predefined parameters, this adaptive binarization sub-step outputs the individual finger masks.

[0020] The use of such an adaptive threshold makes it possible to automate the detection of individual fingers.

[0021] According to an advantageous feature of the invention, the step of pose and orientation calculation, based on shape and geometric information, includes a convexity check to detect the fingertip points.

[0022] This is a simple method for determining the direction of the slap.

[0023] According to a preferred embodiment of the present invention, the step of calculating pose and orientation based on shape and geometry information includes the sub-steps of defining a centerline of each individual finger mask, calculating a gradient along the direction of the finger centerline on the finger image, and detecting the first knuckle by finding the vertical line with maximum gradient within a certain distance to the fingertip point, where the fingertip mask is defined by the area extending between the first knuckle and the fingertip point.

[0024] The use of gradients allows different geometric characteristics of a captured finger to be automatically localized.

[0025] The present invention also relates to a contactless captured fingerprint image processor connected to at least a contactless fingerprint reader, the contactless fingerprint reader having various lighting conditions and configured to capture slap / finger images in a non-contact position in the vicinity of the reader under different lighting conditions for capturing a fingerprint of a user, the fingerprint image processor configured to perform segmentation on the slap images received from the contactless fingerprint reader under controlled lighting conditions and generate accurately labeled individual fingerprints, configured to calculate variance in the received images to estimate the slap area as a foreground slap mask in the input image, identify individual fingers by finding the boundaries of each finger, verify several finger and geometric constraints, calculate pose and orientation based on the shape and geometric information, and configured for identification of a valid fingertip area on each detected finger according to the pose, orientation and geometric information and output of individual fingerprints.

[0026] Such a processor makes it possible to implement the method of the invention for performing segmentation on a slap image as received from a contactless fingerprint reader.

[0027] Advantageously, the processor is configured to perform an adaptive binarization applied to the original illuminated image with the foreground slap mask by calculating the dynamic range of both the global and local dynamics according to the formula Thres(x,y)=α(x,y)+pα(x,y)(β(x,y)-q), where α(x,y) is a local dynamic factor, β(x,y) is a global dynamic factor and p, q are predefined parameters, and this adaptive binarization sub-step outputs masks of the individual fingers.

[0028] Also advantageously, the processor is configured to calculate pose and orientation based on the shape and geometry information, using a convexity check to detect fingertip points.

[0029] According to a preferred embodiment, the processor is configured to calculate the pose and orientation based on shape and geometry information including defining a centerline of each individual finger mask, calculating a gradient along the direction of the finger centerline on the finger image, and detecting the first knuckle by finding the vertical line with the maximum gradient within a certain distance to the fingertip point, where the fingertip mask is defined by an area extending between the first knuckle and the fingertip point.

[0030] To the accomplishment of the foregoing and related ends, the one or more embodiments comprise the features hereinafter fully described and particularly pointed out in the claims.

[0031] The following description and the annexed drawings set forth in detail certain illustrative aspects and point out but a few of the various ways in which the principles of the embodiments may be employed. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the drawings, and the disclosed embodiments are intended to include all such aspects and equivalents thereof. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 depicts a schematic diagram of a contactless fingerprint segmentation system. [Diagram 2] FIG. 1 shows a flowchart illustrating an image processing workflow. [Diagram 3] FIG. 13 shows a flowchart illustrating foreground slap mask estimation. [Figure 4] FIG. 13 illustrates an example of foreground mask estimation. [Diagram 5] FIG. 13 shows a flowchart illustrating individual finger identification. [Figure 6] FIG. 1 illustrates finger gap identification. [Figure 7] FIG. 1 illustrates individual finger identification. [Figure 8] FIG. 1 shows a flowchart illustrating fingerprint pose estimation and fingertip identification. [Figure 9] FIG. 1 illustrates individual fingertip identification. [Figure 10] FIG. 13 shows a flowchart illustrating the final fingerprint generation process. [Figure 11] FIG. 13 shows an example of a final individual fingerprint segmentation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] For a more thorough understanding of the present invention, the present invention will now be described in detail with reference to the accompanying drawings. The detailed description will illustrate and explain what are believed to be preferred embodiments of the present invention. It should of course be understood that various modifications and changes in form or detail can be readily made without departing from the spirit of the present invention. It is therefore intended that the present invention not be limited to the exact form and details shown and described herein, nor to anything less than the whole of the present invention as disclosed herein and hereinafter claimed. The same elements are designated by the same reference numerals in different drawings. For clarity, only those elements and steps that are useful for understanding the present invention will be shown and described in the drawings.

[0034] The present invention can be realized using a software or hardware system. Figure 1 is a schematic diagram of the proposed contactless fingerprint segmentation system 1, which includes at least a controller 16, including but not limited to a CPU, FPGA, DSP, or GPU. The controller manages all other modules, namely image I / O 11 and 14, processing by image processor 12, and storage in image memory 13.

[0035] The set of I / O ports 11 and 14 include, but are not limited to, USB2 / 3, Firewire, Thunderbolt, SATA, DMA, Ethernet, or Internet. These ports 11 and 14 take the input slap image from the contactless fingerprint reader 10 and send the output fingerprint 15 to other devices / systems.

[0036] The image processor 12 may be implemented in any programming language, including but not limited to C / C++, JAVA, Python, Assembly, or JavaScript. The image processor 12 or processing module identifies and extracts individual fingerprint ROIs 15 from the slap image in accordance with the present invention.

[0037] The image memory 13 includes, but is not limited to, a RAM, a ROM, an SSD, a hard drive, or a NAS. The image memory 13 or storage module stores the intermediate results and the final output fingerprint.

[0038] FIG. 2 is a flow chart illustrating the image processing workflow proposed in this invention, i.e., a method for segmenting a slap image and generating accurately labeled individual fingerprint ROIs. In the first step S0, a slap image is received from a contactless fingerprint reader 10 under controlled lighting conditions. The slap area in the input image is initially estimated as a foreground slap mask by calculating the variance in the received image in step S1. Then, in step S2, individual fingers are identified by finding the boundaries of each finger, while some finger and geometric constraints are verified. In the next step S3, for each finger, the pose and orientation are calculated based on the shape and geometric information. In step S4, the effective fingertip area is identified on each detected finger according to the pose, orientation and geometric information. The individual fingerprint ROI 15 is then output. The fingertip area is described as a rotated rectangular ROI, and both the segmented fingerprint image and the coordinates of the rectangular ROI are then saved to storage or sent to other systems.

[0039] Figure 3 is a flow chart illustrating the process for slap / foreground area identification as described in step S1 in Figure 2. A contactless fingerprint reader captures a slap image under controlled lighting conditions, for example using a flash light, in step I0. Under certain lighting conditions, the foreground slap area will show a larger lighting variance in the different images compared to the background area. Based on these findings, the variance is calculated in the images in step I1 to set up adaptive thresholds for different image areas.

[0040] Therefore, pixels with variance greater than the threshold are considered as foreground slap areas, and other pixels as background areas. The pixel values ​​M(x,y) of the slap mask are calculated according to the following formula:

number

[0041] FIG. 5 is a flow chart illustrating a process for single finger identification. One of the advantages of the contactless fingerprint reader 10 is that a slap can be placed against the reader without any physical touch to the device, which helps improve slap image acquisition speed and convenience. However, as a side effect, the contactless fingerprint reader 10 places much fewer constraints on finger position for image acquisition. Thereby, fingers may rotate, tilt, or bend during capture, and also fingers may be very close together in a four-finger slap image.

[0042] As a result, traditional slap segmentation methods that assume that the fingers are well separated will fail in the majority of cases in contactless fingerprint systems. Therefore, the most challenging part of identifying a valid fingerprint ROI is identifying each individual fingerprint.

[0043] The single finger identification method of the present invention includes a first step in which two images are received, one being the original image IF and the other being the foreground mask SM obtained as described on FIG.

[0044] FIG. 6 shows, on the lower image FS, the gap between the fingers that needs to be determined from the two images, especially when the fingers are close to each other.

[0045] In the first step F1, an adaptive binarization method is applied to the original image IF with the foreground mask as a constraint. In this method, the threshold for binarization is determined by calculating the dynamic range of both global and local dynamics, as shown in the following formula: Thres(x,y)=α(x,y)+pα(x,y)(β(x,y)-q), where α(x,y) is the local dynamic factor for each pixel (x,y), i.e., the average value inside the sliding window in which the pixel (x,y) is centered, and x and y are the coordinates of the pixel. β(x,y) is the global dynamic factor calculated based on the fluctuation of the whole image, and p and q are constant positive values, where p is the maximum standard deviation of the image, i.e., equal to 128 for 8-bit grayscale images, and q is equal to a preset bias in the range of [0.1,0.6] according to different applications. This binarization makes it possible to define the lines separating the fingers, as shown on the lower image FS of FIG.

[0046] Then, as illustrated in FIG. 7, in order to identify each individual finger, a convex hull that encloses the entire slap is first found in step F2, which helps to detect the finger tips. Then, in step F3, convexity is checked inside the convex hull to find the finger bases / joints. Usually, convexity defects, i.e. areas that do not belong to the slap but are located inside the convex hull, are calculated first. In this case, the areas refer to the areas between the two fingers and the areas on the two sides of the slap. So, the points in the convex defects with the maximum distance to the corresponding edge of the convex hull, i.e. the finger bases / joints, are detected as illustrated by the image TD. Finally, each individual finger is finally identified by combining the finger tip and finger base positions as illustrated by the image FD, and a corresponding mask FM is generated for each of those fingers in step F4.

[0047] After each individual finger has been identified in step F4, accurate detection of the fingertip position is required.

[0048] Figure 8 is a flow chart illustrating a process according to the present invention for finger pose estimation and fingertip segmentation. Figure 9 illustrates pose estimation and fingertip identification.

[0049] Finger pose is estimated by finding the centerline of the finger mask in step T1 on each individual finger mask, as illustrated diagrammatically on Fig. 9. In some special cases where the fingers are not straight, the average angle of the centerline is estimated.

[0050] Based on the result of step T1, the gradient along the direction of the finger centerline is calculated on the finger image FI, and then the first finger joint can be detected in step T2 of finger pose estimation by finding the vertical line with the maximum gradient within a certain distance to the fingertip point, which allows to determine the fingertip mask TM.

[0051] FIG. 10 is a flow chart illustrating the process for generating the final individual fingerprint image.

[0052] An input image is received in a first step P1, on the one hand a slap fingerprint image IF and on the other hand a set of individual fingertip masks TM, and an individual fingerprint image is generated by combining the slap image IF and the fingertip masks TM.

[0053] In step P2, post-processing is performed, including at least an image smoothing and denoising operation applied to the individual fingerprint image to remove random noise. Advantageously, a further morphological transformation is applied to the acquired fingerprint image to remove structural noise and smooth the fingerprint edges. The final individual fingerprint FIF is then generated as also shown on FIG.

[0054] In the above detailed description, reference has been made to the accompanying drawings which show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. The above detailed description is therefore not to be taken in a limiting sense, and the scope of the present invention is defined solely by the appended claims, appropriately interpreted in conjunction with the full scope of equivalents to which such claims are entitled.

Claims

1. A method for segmenting a slap image and generating accurately labeled individual fingerprints, comprising the steps of: receiving an input image from a contactless fingerprint reader under controlled lighting conditions; calculating a variance in the received image to estimate a slap area as a foreground slap mask in the input image; Identifying individual fingers by finding the boundaries of each finger; Verification of some finger and geometric constraints, Calculating pose and orientation based on shape and geometry information; Identifying a useful fingertip area on each detected finger according to pose, orientation, and geometric information; Individual fingerprint output Including, A method in which the step of calculating pose and orientation based on shape and geometry information includes a convexity check to detect fingertip points.

2. 2. The method for performing segmentation on a slap image of claim 1, wherein the images acquired under controlled lighting conditions are images acquired with and without a flash light.

3. 2. The method for performing segmentation on a slap image according to claim 1, wherein the individual finger identification step comprises an adaptive binarization sub-step applied to the original illuminated image with the foreground slap mask by calculating the dynamic range of both global and local dynamics according to the formula: Thres(x,y)=α(x,y)+pα(x,y)(β(x,y)-q), where α(x,y) is a local dynamic factor, β(x,y) is a global dynamic factor, and p, q are predefined parameters, and this adaptive binarization sub-step outputs the individual finger masks.

4. 2. The method for segmenting a slap image of claim 1, wherein the step of calculating pose and orientation based on shape and geometric information includes the sub-steps of defining a centerline of each individual finger mask, calculating a gradient along the direction of the finger centerline on the finger image, and detecting a first knuckle by finding a vertical line with a maximum gradient within a certain distance to a fingertip point, wherein the fingertip mask is defined by an area extending between the first knuckle and the fingertip point.

5. A contactless acquisition fingerprint image processor connected to at least a contactless fingerprint reader, the contactless fingerprint reader having various lighting conditions and configured to acquire slap / finger images in a non-contact position in the vicinity of the reader under different lighting conditions for acquiring a fingerprint of a user, the fingerprint image processor configured to perform segmentation on the slap images received from the contactless fingerprint reader under controlled lighting conditions and generate accurately labeled individual fingerprints, the fingerprint image processor configured to calculate variance in the received image to estimate the slap area as a foreground slap mask in the input image, identify individual fingers by finding the boundaries of each finger, verify several finger and geometric constraints, calculate pose and orientation based on the shape and geometric information, and configured for identification of a valid fingertip area on each detected finger according to the pose, orientation and geometric information and output of an individual fingerprint, A contactless acquired fingerprint image processor configured to calculate pose and orientation based on shape and geometry information using a convexity check to detect finger tip points.

6. 6. The contactless acquired fingerprint image processor of claim 5, configured to perform an adaptive binarization applied to the original illuminated image with foreground slap mask by calculating the dynamic range of both global and local dynamics according to the formula: Thres(x,y)=α(x,y)+pα(x,y)(β(x,y)-q), where α(x,y) is a local dynamic factor, β(x,y) is a global dynamic factor, and p, q are predefined parameters, and this adaptive binarization sub-step outputs masks of the individual fingers.

7. 6. The non-contact acquired fingerprint image processor of claim 5, wherein the contactless acquired fingerprint image processor is configured to calculate pose and orientation based on shape and geometry information including defining a centerline of each individual finger mask, calculating a gradient along the direction of the finger centerline on the finger image, and detecting a first knuckle by finding a vertical line with a maximum gradient within a specified distance to a fingertip point, the fingertip mask being defined by an area extending between the first knuckle and the fingertip point.

Images