Method of generating palm / finger foreground mask

Abstract

The invention relates to a method of generating a palm / finger foreground mask for subsequent image processing of a fingerprint on an image acquired using a non-contact fingerprint reader having at least a flash, the method comprising the steps of: - acquiring two images of a palm / finger in a non-contact position in the vicinity of the reader, one image being taken with the flash on and one image being taken without the flash, - computing a difference map between the image acquired with the flash and the image acquired without the flash, - computing an adaptive binarization threshold for each pixel of the image, the threshold for each pixel being the corresponding value in the difference map, to which is subtracted the corresponding flash compensation factor value determined in a flash compensation factor map using an image of a non-reflective blank target acquired with the flash, and to which is added the corresponding background enhancement factor value determined in a background enhancement factor map using the image acquired without the flash, - binarizing the difference map by attributing a first value to the pixels for which the adaptive binarization threshold is higher than the corresponding value in the difference map, and a second value to the pixels for which the adaptive binarization threshold is lower than the corresponding value in the difference map, the binarized image being the palm / finger foreground mask.

Description

Technical Field

[0001] The present invention relates to a method for generating a palm / finger foreground mask for subsequent image processing of fingerprints on images acquired using a contactless fingerprint reader having at least a flash.

[0002] The present invention also relates to a contactless fingerprint image processor, which is at least connected to a contactless fingerprint reader having at least a flash and adapted to acquire images of a palm / finger in a contactless location near the reader with or without the flash, for use in acquiring the fingerprint of a user implementing the method. Background Technology

[0003] Traditional contact-based methods capture fingerprints by directly imprinting a finger onto a recording medium or device, such as an ink, optical sensor, or electronic sensor. With the growing demand for faster capture speeds and better user experiences, contactless fingerprint devices have been introduced into the biometrics market. This invention relates to a novel method and apparatus for adaptive background subtraction of contactless fingerprint images.

[0004] More specifically, contactless fingerprint readers capture contactless fingerprint images. Unlike traditional contact fingerprint readers that only capture the palm / finger that touches the reader, contactless fingerprint readers take a picture of the entire palm / finger presented above the device. Therefore, due to the nature of the photographic representation from a contactless fingerprint reader, the captured picture contains both the palm / finger as the foreground and the scene behind the palm / finger, as well as noise as the background. In order to identify the position of the palm / finger (i.e., the foreground) in the captured image, the background scene and noise need to be removed.

[0005] Background subtraction becomes more challenging in contactless fingerprint images because:

[0006] 1. Typically, only a single image is captured for the same palm / finger, making traditional statistical background modeling, which requires a sequence of images, ineffective;

[0007] 2. For example, the dynamics of strong background light and the presence of strong background scene / noise can be significant in an image, and may even suppress foreground fingers;

[0008] 3. The uneven distribution of illumination / brightness from the captured image causes the foreground palm / fingers to have varying gray levels, and their positions vary across different subjects.

[0009] Background subtraction is a fundamental step in many image processing and computer vision applications or systems. Over the past few decades, various methods have been proposed for background subtraction to segment foreground objects from a background scene. In one typical application category, the foreground is identified by calculating the difference between frames with objects and frames without background. In other representative applications, it is assumed that the background can be statistically modeled and updated based on the image sequence.

[0010] Those general technologies are described, for example, in the following documents:

[0011] • US6411744B1 “Method and apparatus for performing a clean backgroundsubtraction”.

[0012] • Ahmed M. Elgammal, David Harwood, and Larry S. Davis. “Non-parametricModel for Background Subtraction.” In Proceedings of the 6th European Conference on Computer Vision, 2000.

[0013] • US5748775A “Method and apparatus for moving object extraction based background subtraction”

[0014] • M. Piccardi, “Background subtraction techniques: a review” 2004 IEEEInternational Conference on Systems, Man and Cybernetics

[0015] • Z. Zivkovic, “Improved adaptive Gaussian mixture model for background subtraction,” Proceedings of the 17th International Conference onPattern Recognition, 2004

[0016] • Sobral, Andrews & Vacavant, Antoine. “A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos”. Computer Vision and Image Understanding, 2014

[0017] • S. Liao, G. Zhao, V. Kellokumpu, M. Pietikäinen and S. Z. Li, “Modeling pixel process with scale invariant local patterns for background subtraction in complex scenes,” 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, 2010

[0018] • W. Kim and C. Kim, “Background Subtraction for Dynamic Texture Scenes Using Fuzzy Color Histograms,” in IEEE Signal Processing Letters, vol. 19, no. 3, pp. 127-130, March 2012

[0019] • Dar-Shyang Lee, “Effective Gaussian mixture learning for video background subtraction,” in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 5, pp. 827-832, May 2005

[0020] • J. Yao and J. Odobez, “Multi-Layer Background Subtraction Based on Color and Texture,” 2007 IEEE Conference on Computer Vision and PatternRecognition, Minneapolis, MN, 2007

[0021] • K. Shafique, O. Javed, and M. Shah, “A Hierarchical Approach to RobustBackground Subtraction using Color and Gradient Information,” Motion and VideoComputing, IEEE Workshop on (MOTION), Orlando, Florida, 2002.

[0022] However, in contactless fingerprint identification systems, both methods become challenging in capturing image sequences for statistical modeling, and the background scene may also change over time. Therefore, there is a need for a new approach for contactless fingerprint identification systems.

[0023] Therefore, alternative and advantageous solutions would be desirable in this field. Summary of the Invention

[0024] The present invention aims to provide a novel method for background subtraction in a specific field of contactless fingerprint identification systems.

[0025] This invention is defined in its broadest sense as a method for generating a palm / finger foreground mask for subsequent image processing of fingerprints on an image acquired using a contactless fingerprint reader with at least a flash, the method comprising the following steps:

[0026] - Capture two images of a hand / finger in a non-contact location near the reader: one image acquired with the flash on, and one image acquired without the flash.

[0027] - Calculate the difference map between images acquired with and without flash.

[0028] - Calculate the adaptive binarization threshold for each pixel of the image. The threshold for each pixel is the corresponding value in the difference map. Subtract this corresponding value multiplied by the corresponding flash compensation factor value determined in the flash compensation factor map of the image of the non-reflective blank target acquired with flash. Then add this corresponding value multiplied by the corresponding background enhancement factor value determined in the background enhancement factor map of the image acquired without flash.

[0029] The difference map is binarized by attributing a first value to pixels where the adaptive binarization threshold is higher than the corresponding value in the difference map, and a second value to pixels where the adaptive binarization threshold is lower than the corresponding value in the difference map. The binarized image is a palm / finger foreground mask.

[0030] With the aid of this invention, valid foreground regions, i.e., palm / finger regions, can be identified and extracted by adaptively calculating the differences between different areas. The extracted clean foreground is then represented by a foreground mask, which can be used for advanced fingerprint processing tasks in contactless fingerprint identification systems.

[0031] This invention is a unique and efficient solution combining the hardware and software capabilities of a contactless fingerprint identification system. It requires only two images and minimal hardware setup to solve the problem. On the hardware side, it receives two images of the same subject, one with the flash on and the other with the flash off; the device can then transmit, process, and save the captured images. On the software side, the proposed algorithm adaptively calculates the difference between the foreground and background regions based on the two captured images, and then generates an accurate foreground mask for the palm / finger.

[0032] In terms of functionality, this invention handles various scenarios, enabling the contactless fingerprint identification system to operate more robustly in different locations. In terms of performance, this invention not only accurately identifies foreground palm / finger areas but also improves the speed of traditional background modeling techniques, significantly enhancing the performance of the contactless fingerprint identification system. In terms of cost, this invention reduces the burden on product designs that require more complex viewports and overlays to remove cluttered background scenes and noise. In terms of business needs, this invention is one of the core features for customers.

[0033] Advantageously, the method further includes a noise removal step in the binarized image.

[0034] According to a specific feature of the invention, for each pixel, the flash compensation factor is defined by dividing the pixel’s standard illumination value by a reference illumination value of the pixel, such as that obtained in an image of a non-reflective blank target, which is a reference image.

[0035] According to another specific feature of the invention, for each pixel, the background enhancement factor is defined by dividing the background brightness in the image with the flash off by the average brightness of the palm / finger for different subjects with the flash on.

[0036] The present invention also relates to a contactless fingerprint acquisition image processor, which is at least connected to a contactless fingerprint reader, the contactless fingerprint reader having at least a flash and adapted to acquire images of a palm / finger at a contactless location near the reader, with or without the flash, for acquiring a user's fingerprint. The processor is adapted to generate a palm / finger foreground mask for subsequent image processing of the fingerprint. Upon receiving two images of the palm / finger at a contactless location near the reader—one image acquired with the flash on and one image acquired without the flash—the processor is adapted to calculate a difference map between the image acquired with the flash and the image acquired without the flash, to calculate an adaptive value for each pixel of the image. The binarization threshold for each pixel is the corresponding value in the difference map, which is subtracted from the corresponding value multiplied by the corresponding flash compensation factor value determined in the flash compensation factor map of the image of the non-reflective blank target acquired using a flash, and then added to the corresponding value multiplied by the corresponding background enhancement factor value determined in the background enhancement factor map of the image acquired without a flash. The processor is further adapted to binarize the difference map by attributing a first value to pixels where the adaptive binarization threshold is higher than the corresponding value in the difference map, and a second value to pixels where the adaptive binarization threshold is lower than the corresponding value in the difference map. The resulting binarized image is a palm / finger foreground mask, which will be used for subsequent image processing of fingerprints on images acquired using a contactless fingerprint reader.

[0037] Advantageously, the processor is further adapted to remove noise from the binarized image.

[0038] It is also advantageous to define the flash compensation factor for each pixel as the standard illumination value of the pixel divided by the reference illumination value of the pixel, such as in an image of a non-reflective blank target, which is a reference image.

[0039] The processor is further advantageously adapted to define the background enhancement factor for each pixel as the background brightness in the image with the flash off divided by the average brightness of the palm / finger for different subjects with the flash on.

[0040] To achieve the foregoing and related objectives, one or more embodiments include features that are fully described below and specifically pointed out in the claims. Attached Figure Description

[0041] The following description and accompanying drawings illustrate certain illustrative aspects in detail and indicate only a few of the various ways in which the principles of the embodiments can be employed. Other advantages and novel features will become apparent from the following detailed description when considered in conjunction with the accompanying drawings, and the disclosed embodiments are intended to include all such aspects and their equivalents.

[0042] · Figure 1 This is a schematic diagram of the environment in which the present invention is implemented;

[0043] · Figure 2 A flowchart is shown for the background subtraction method of non-contact fingerprint images used in this invention;

[0044] · Figure 3 The calculation of the image difference map with flash on / off is shown;

[0045] · Figure 4 The flowchart for calculating the flash enhancement factor map is shown;

[0046] · Figure 5 The flowchart for calculating the backlight enhancement factor map is shown;

[0047] · Figure 6 The illustration shows the adaptive thresholding of the difference map according to the present invention; and

[0048] · Figure 7 The noise removal and final palm / finger mask generation are shown. Detailed Implementation

[0049] For a more complete understanding of the invention, it will now be described in detail with reference to the accompanying drawings. The detailed description will illustrate and describe what is considered to be the preferred embodiments of the invention. It should be understood, of course, that various modifications and changes in form or detail can be readily made without departing from the spirit of the invention. Therefore, it is intended that the invention may not be limited to the exact forms and details shown and described herein, nor to anything less than the entirety of the invention disclosed herein and claimed hereinafter. The same elements are designated by the same reference numerals in different drawings. For clarity, only those elements and steps useful for understanding the invention are shown in the drawings and will be described therein.

[0050] Figure 1 This is a schematic diagram of the invented device 1. It includes three main components: two image I / O functions 11 and 14, an image processor 12, and an image memory 13. The invented device 1 is connected to at least one contactless fingerprint reader 10 to receive fingerprint capture.

[0051] Therefore, according to the present invention, the image I / O function 11 receives a raw contactless palm / finger image from the contactless fingerprint reader 10, which includes both flashing and non-flashing palm / finger images. Those images in Figure 3 As shown above.

[0052] The image I / O function 11 then transmits the image to the image processor 12 and the image memory 13. When applicable, it can also output the processed image to other components of the contactless fingerprint identification system.

[0053] The image processor 12 implements the present invention while processing the received image. At the end of the method of the present invention, an accurate foreground palm / finger mask is generated.

[0054] Image storage device 13, which includes ROM and RAM, is capable of storing captured or processed images.

[0055] Figure 2 A flowchart of the background subtraction method for contactless fingerprint images used in this invention is shown. First, in step S0, two palm / finger images of the same subject are received from the contactless fingerprint scanner 10, one captured with the flash on and the other captured with the flash off.

[0056] Second, in step S1, the difference between the two images is calculated and a difference map is generated. Third, due to uneven flash and background noise, adaptive thresholding is used to adaptively binarize the difference map in step S2. This generates an initial foreground mask. Fourth, in step S3, morphological operations are applied to the initial foreground map to remove random and structural noise. Finally, in step S4, a final accurate hand / finger mask is generated.

[0057] Figure 3 This illustration shows an example of how to generate an initial foreground image using flash-on / flash-off image difference map calculation. Two images received from a contactless fingerprint scanner 11 are continuously captured at intervals of less than 10 ms: one is the IF image with flash on and the other is the INF image with flash off. This therefore assumes no hand / finger movement between the two image IF and INF values. The difference between the two image IF and INF values ​​will be noise and the portion illuminated by the flash.

[0058] Because the background scene typically extends beyond the flash's range, the illuminated subject includes the palm / finger and parts of the scanner itself. Based on the specific pattern of the illuminated palm / finger, an initial foreground image, i.e., the palm / finger, can be generated by calculating the difference between the IF and INF values ​​of two images. Figure 3 As shown above, the difference map DM is obtained.

[0059] According to the present invention, a foreground image is generated by calculating the grayscale difference of each pixel between the flash image and the non-flash image according to the following equation: IF(x,y) is the pixel value of the flash image at position (x,y), and INF(x,y) is the pixel value of the non-flash image at the same position, and ID(x,y) is the absolute difference between IF and INF of the two images.

[0060] Figure 4 This is a flowchart illustrating the calculation of the flash enhancement factor map, showing how to compensate for the uneven flash distribution in the IF (Image Interpretation Function) of a hand / finger image. Due to hardware limitations, the flash in a hand / finger image is unevenly distributed, with the strongest flash received at the image center, and the flash intensity decreasing along the direction from the image center to the image boundaries. However, the physical shape and position of the hand / finger require that all areas in the image be uniformly illuminated, especially along the boundaries of the image IF where the fingers are more likely to be located.

[0061] This invention proposes a method to compensate for uneven flash by calculating a compensation factor. Special non-reflective targets are used to cover the field of view of the non-contact acquisition device 10.

[0062] Then, in step C1, an image of the non-reflective target is acquired with the flash on, serving as a reference image IR for the flash distribution. Then, in step C2, the acquired image is smoothed to obtain a pixel value map in step C3. Then, the flash compensation or enhancement factor map FCM—which is also essentially a pixel value map—is defined in step C4 as follows: ,in IS(x,y) is the flash compensation factor for the image pixel at position (x,y), IS(x,y) is the standard illumination value for pixel (x,y), which is equal to the average brightness level in the central region of the image without vignetting, and IR(x,y) is the reference pixel value at the same position. The flash compensation factor map FCM is then stored in image memory 13 and retrieved when a new difference map ID needs to be corrected.

[0063] Figure 5 This is a flowchart illustrating the calculation of the backlight enhancement factor map (BCM), showing how to obtain the BCM. As mentioned above, if conventional binarization methods are used, the presence of dynamic and strong background scenes / noise (e.g., strong background light) can be significant in the image and detract from the foreground hand / finger. According to the present invention, binarization is performed to adaptively increase / decrease the threshold used for binarization to adapt to the brightness in the captured image.

[0064] To determine the Background Enhancement Factor (BCM), multiple sample images were acquired across different subjects with the flash on before fabrication. The luminance of the palm / finger was then evaluated based on the collected sample images. The average intensity of the luminance of the palm / finger from multiple subjects was denoted as BF, which was generated and stored in the Software Development Kit (SDK) for all devices. During normal operation of the device—i.e., the device is installed in the field to capture fingerprints—an image was acquired for each capture at step BC1 with the flash off. This image was thresholded in step BC2. The average intensity of the luminance was denoted as BNF(x,y). The Background Enhancement Factor (BCM) was formulated in step BC3 as follows: Therefore, a background enhancement factor map (BCM) is obtained for the installation device that will capture fingerprints.

[0065] Figure 6 The adaptive thresholding used for the difference map and how the previously obtained difference map DM is transformed into an initial foreground mask IFM are illustrated. Three inputs are used for this transformation. The first is the initial difference map DM, which shows the essential difference between the flash-on and flash-off images. The second is the flash compensation map FCM, which compensates for the flash and eliminates inaccuracies introduced by flash intensity degradation along the image boundaries. The last is the background enhancement factor map BCM, which corrects for strong backgrounds present in the image.

[0066] Then, according to the present invention, the adaptive binarization threshold T is formulated as follows: .

[0067] The initial difference map ID is then binarized into an initial foreground mask IFM, as follows:

[0068]

[0069] Where M(x,y) is... Figure 6 The image shows the pixel values ​​at position (x,y) of the initial foreground mask IFM, which includes both the palm / finger and noise. Once the initial foreground map is generated, there is a need to identify and remove noise present in the image—including random noise and structural noise. Therefore, there is a need to remove noise to generate the final foreground mask FFM. In this invention, morphological operations are applied to remove structural and random noise.

[0070] In their experiments, the inventors discovered that structural noise is present in at least a portion of the captured image of the device casing. Background noise may also cause some additional random noise. This can be seen in the initial foreground mask IFM, as in... Figure 6 As shown in the image on the right.

[0071] Therefore, morphological operations are used to remove all types of noise. First, an opening operation is applied to the image, which removes small objects and random noise from the image, such as device objects in the image; then a closing operation is applied to the image, which removes small holes in the foreground and smooths the edges of the palm / fingers.

[0072] Figure 7 The noise removal and final palm / finger mask generation are shown as a comparison before and after noise removal, resulting in the final foreground mask FFM for the palm / finger. Once the final palm / finger foreground mask FFM is obtained, it is stored in image memory 13. The thus obtained final foreground mask FFM is then used as input to other modules in a contactless fingerprint identification system for advanced contactless fingerprint processing tasks.

[0073] In the above detailed description, reference has been made to the accompanying drawings, which illustrate specific embodiments by which the invention can be practiced. These embodiments have been described in sufficient detail to enable those skilled in the art to practice the invention. Therefore, the above detailed description should not be considered in a limiting sense, and the scope of the invention is defined only by the appended claims as properly interpreted.

Claims

1. A method for generating a palm / finger foreground mask for subsequent image processing of fingerprints on an image acquired using a non-contact fingerprint reader having at least a flash, the method comprising the steps of: - Acquire two images of the palm / finger in a non-contact position near the reader, one image taken with the flash on and one image taken without the flash; - Calculate the difference between the image acquired with the flash and the image acquired without the flash using the following equation: ID(x,y) = |IF(x,y) - INF(x,y)|, where IF(x,y) is the pixel value of the flash image at position (x,y), INF(x,y) is the pixel value of the non-flash image at the same position, and ID(x,y) is the absolute difference between the two images IF and INF; - Generate a difference map based on the calculation of the difference, wherein the difference map shows the part of the image including the palm / finger that can be illuminated by the flash; - Calculate a flash compensation factor based on a reference image (IR) of a non-reflective target with the flash on using the following equation: α(x,y) = IS(x,y) / IR(x,y), where α(x,y) is the flash compensation factor of the image pixel at position (x,y), IS(x,y) is the standard illumination value of the pixel (x,y), which is equal to the average luminance level in the central region of the image without vignetting effect, and IR(x,y) is the reference pixel value at the same position; - Generate a flash compensation factor map based on the calculation of the flash compensation factor; - Calculate a backlight enhancement factor using the following equation: β(x,y) = BNF(x,y) / BF, where BF is the average intensity of the luminance of the palm / finger when acquiring multiple sample images across different subjects with the flash on, and BNF(x,y) is the average intensity of the luminance with the flash off; - Generate a backlight enhancement factor map based on the calculation of the backlight enhancement factor; - Calculate an adaptive binary threshold (T) using the following equation: T(x,y) = ID(x,y)*(1 - α(x,y)+β(x,y)); and - Generate a foreground mask using the following equation: If T(x,y) ≥ ID(x,y), then M(x,y) = 255, and if T(x,y) < ID(x,y), then M(x,y) is 0, where M(x,y) is the pixel value of the initial foreground mask IFM at position (x,y).

2. The method according to claim 1, further comprising the step of noise removal in the binary image.

3. The method according to claim 2, wherein the noise is removed by morphological operations.

4. The method according to any one of claims 1 to 3, further comprising using the foreground mask as an input to other modules in a non-contact fingerprint identification system for advanced non-contact fingerprint processing tasks.

5. A contactless fingerprint acquisition system comprising at least one contactless fingerprint reader having at least one flash, and further comprising a contactless fingerprint acquisition image processor connected to the at least one contactless fingerprint reader, the contactless fingerprint reader being adapted to acquire an image of a palm / finger at a contactless location near the reader, with or without the flash, for acquiring a user's fingerprint, the processor being adapted to generate a palm / finger foreground mask from the image according to the method of claim 1.

6. The contactless fingerprint acquisition system according to claim 5, wherein, The processor is further adapted to remove noise from the binarized image.

7. The contactless fingerprint acquisition system of claim 6, wherein the noise is removed by morphological operations.

Images