Fingerprint verification device

Abstract

The fingerprint verification device includes a sensor unit configured to output a sensing signal by sensing a fingerprint, an image processing unit configured to generate a fingerprint image based on the sensing signal, a storage unit configured to store a template including an enrollment image, and a learning unit configured to generate a first pseudo image and add the first pseudo image to the template.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority and benefit to Korean Patent Application No. 10-2020-0076005, filed on June 22, 2020, which is incorporated herein by reference for all purposes, as if fully set forth herein. Technical Field

[0003] Exemplary implementations of the present invention generally relate to a fingerprint verification device, a display device including the fingerprint verification device, and a method for verifying fingerprints, and more specifically, to a fingerprint verification device using learning technology, a display device including the fingerprint verification device, and a method for verifying fingerprints. Background Technology

[0004] Recently, biometric authentication methods, such as using user fingerprints, have become widely used due to the widespread application of display devices such as smartphones and tablet PCs in many fields. To provide fingerprint sensing functionality, fingerprint sensors can be built into the display device or attached to the top and / or bottom of the display device. Display devices with integrated fingerprint sensors are called in-display fingerprint (FoD) devices.

[0005] FoD can include, for example, a photosensitive fingerprint sensor. A photosensitive FoD can use a light-emitting device included in the pixels as a light source and includes an optical sensor array as the fingerprint sensor. The optical sensor array can be implemented as, for example, a CMOS image sensor (CIS).

[0006] FoD can acquire fingerprint images through a fingerprint sensor and compare the fingerprint images with pre-registered fingerprint images (e.g., a template including the fingerprint image) to verify the fingerprint (i.e., verify that the sensed fingerprint is the user's registered fingerprint).

[0007] The accuracy of fingerprint recognition can be reduced depending on skin condition (e.g., when the skin is dry or when the skin is contracted) or the verification environment (e.g., when the surrounding environment is very bright or dark), and the fingerprint verification result of the same fingerprint can be altered.

[0008] The information disclosed in this background section is only for understanding the background technology of the inventive concept, and therefore may contain information that does not constitute prior art. Summary of the Invention

[0009] The applicant discovered that the accuracy of fingerprint recognition on the display device decreased when the user's skin condition or the verification environment was abnormal.

[0010] The fingerprint verification device of the display device constructed according to the principles and exemplary implementation of the present invention can improve the accuracy of fingerprint verification by using fingerprint image learning technology.

[0011] The fingerprint verification method according to the principles and exemplary implementation of the present invention can improve the accuracy of fingerprint verification by using fingerprint image learning techniques.

[0012] Further features of the inventive concept will be set forth in the following description and will be apparent in part from the description, or may be learned by practice of the inventive concept.

[0013] According to one or more exemplary embodiments of the present invention, a fingerprint verification device includes: a sensor unit configured to output a sensing signal by sensing a fingerprint; an image processing unit configured to generate a fingerprint image based on the sensing signal; a storage unit configured to store a template including a registered image; and a learning unit configured to generate a first pseudo image and add the first pseudo image to the template.

[0014] The signal-to-noise ratio of the first pseudo-image can be lower than that of the registered image.

[0015] The learning unit can be configured to convert the registered image into a first pseudo image by using a first image conversion model.

[0016] The image processing unit can be configured to generate a first fingerprint image during the fingerprint registration period and a second fingerprint image during the fingerprint verification period, wherein the registration image may include at least a portion of the first fingerprint image, and wherein the first pseudo-image may be different from the second fingerprint image.

[0017] The learning unit can be configured to set a first image transformation model by learning fingerprint images provided from outside the fingerprint verification device, wherein the fingerprint images for learning can include high-quality fingerprint images with a signal-to-noise ratio within a reference quality range and low-quality fingerprint images with a signal-to-noise ratio outside the reference quality range relative to multiple objects.

[0018] The fingerprint verification device may further include a verification unit configured to compare a fingerprint image generated by an image processing unit with a template and to output a comparison result. The image processing unit may be configured to generate a third fingerprint image corresponding to the second fingerprint image while changing the driving conditions of the sensor unit during the fingerprint verification period. The second and third fingerprint images are obtained from substantially the same fingerprint. The signal-to-noise ratio of the third fingerprint image may be less than that of the second fingerprint image. The storage unit may be configured to store the second and third fingerprint images based on the comparison result of the verification unit.

[0019] The sensor unit may include an optical sensor configured to sense light reflected by a fingerprint, wherein the image processing unit may be configured to generate a second fingerprint image by receiving the reflected light via the sensor unit for a first exposure time, and wherein the image processing unit may be configured to generate a third fingerprint image by receiving the reflected light via the sensor unit for a second exposure time.

[0020] The second exposure time can be shorter than the first exposure time.

[0021] The sensor unit may include an ultrasonic sensor configured to sense ultrasonic waves reflected by a fingerprint, wherein the image processing unit may be configured to generate a second fingerprint image by emitting ultrasonic waves having a first frequency via the sensor unit, and wherein the image processing unit may be configured to generate a third fingerprint image by emitting ultrasonic waves having a second frequency via the sensor unit.

[0022] The second frequency can be lower than the first frequency.

[0023] The learning unit can be configured to set a second image conversion model by learning a second fingerprint image and a third fingerprint image, can be configured to convert a first fingerprint image into a second pseudo image by using the second image conversion model, and can be configured to add the second pseudo image to the template.

[0024] The learning unit can be configured to update the first image conversion model by additionally learning the second and third fingerprint images, can be configured to convert the first fingerprint image into a third pseudo-image by using the updated first image conversion model, and can be configured to add the third pseudo-image to the template.

[0025] The image processing unit can be configured to generate a third fingerprint image corresponding to the second fingerprint image while changing the driving conditions of the sensor unit during the fingerprint verification period. The second and third fingerprint images are obtained from substantially the same fingerprint. The learning unit can be configured to set or update a first image conversion model by learning at least one of the first, second, and third fingerprint images. The signal-to-noise ratio of the third fingerprint image can be less than that of the second fingerprint image.

[0026] The sensor unit may include an optical sensor configured to sense light reflected by a fingerprint, wherein the image processing unit may be configured to generate a second fingerprint image by receiving the reflected light via the sensor unit for a first exposure time, and wherein the image processing unit may be configured to generate a third fingerprint image by receiving the reflected light via the sensor unit for a second exposure time.

[0027] The sensor unit may include an ultrasonic sensor configured to sense ultrasonic waves reflected by a fingerprint, wherein the image processing unit may be configured to generate a second fingerprint image by emitting ultrasonic waves having a first frequency via the sensor unit, and wherein the image processing unit may be configured to generate a third fingerprint image by emitting ultrasonic waves having a second frequency via the sensor unit.

[0028] The fingerprint verification device may further include a verification unit configured to compare a fingerprint image generated by an image processing unit with a template and to output a comparison result, wherein the image processing unit may be configured to generate a converted fingerprint image by increasing the signal-to-noise ratio of the fingerprint image using a first image conversion model, and wherein the verification unit may be configured to compare the converted fingerprint image with the template.

[0029] The verification unit can be configured to compare the transformed fingerprint image with the template based on a first comparison result obtained by comparing the fingerprint image with the template, and configured to output a second comparison result.

[0030] According to one or more other exemplary embodiments of the present invention, a display device includes a display panel and a fingerprint verification device. The display panel includes pixels that emit light, and the fingerprint verification device is disposed on the bottom of the display panel. The fingerprint verification device is configured to sense a fingerprint based on a signal reflected by the fingerprint. The fingerprint verification device includes: a sensor unit configured to output a sensing signal by sensing the fingerprint; an image processing unit configured to generate a fingerprint image based on the sensing signal; a storage unit configured to store a template including a registered image; and a learning unit configured to generate a first pseudo image and add the first pseudo image to the template.

[0031] The signal-to-noise ratio of the first pseudo-image can be less than that of the registered image, and the learning unit can be configured to convert the registered image into the first pseudo-image by using a first image conversion model.

[0032] The sensor unit may be an optical sensor configured to sense light reflected by a fingerprint, or an ultrasonic sensor configured to emit ultrasonic waves toward the fingerprint and receive at least a portion of the ultrasonic waves reflected by the fingerprint.

[0033] According to one or more exemplary implementations of the present invention, a method for verifying fingerprints includes the following steps: generating a first fingerprint image of a first object through a sensor unit; adding a registration image including at least a portion of the first fingerprint image to a template; generating a pseudo image based on the registration image; and adding the pseudo image to the template.

[0034] The signal-to-noise ratio (SNR) of a fake image can be lower than that of the registered image.

[0035] The steps of generating a pseudo image may include: setting a first image transformation model by learning a fingerprint; and converting a registered image into a pseudo image by using the first image transformation model, wherein the fingerprint image may include a high-quality fingerprint image with a signal-to-noise ratio within a reference quality range and a low-quality fingerprint image with a signal-to-noise ratio outside the reference quality range relative to a plurality of objects.

[0036] The method may further include the following steps: generating a second fingerprint image through a sensor unit; comparing the second fingerprint image with a template and outputting the comparison result.

[0037] The step of generating a second fingerprint image may include: generating a third fingerprint image corresponding to the second fingerprint image while changing the driving conditions of the sensor unit, wherein the second fingerprint image and the third fingerprint image are obtained from substantially the same object, and wherein the signal-to-noise ratio of the third fingerprint image is lower than that of the second fingerprint image.

[0038] The method may further include the following steps: storing a second fingerprint image and a third fingerprint image based on the comparison results; setting a second image conversion model by learning the second fingerprint image and the third fingerprint image; converting the registered image into a second pseudo image by using the second image conversion model; and adding the second pseudo image to the template.

[0039] The sensor unit may include an optical sensor configured to sense light reflected by a fingerprint, and the step of generating a third fingerprint image may include the following steps: generating a second fingerprint image by receiving the reflected light via the sensor unit for a first exposure time; and generating a third fingerprint image by receiving the reflected light via the sensor unit for a second exposure time.

[0040] The sensor unit may include an ultrasonic sensor configured to sense ultrasonic waves reflected by a fingerprint, and the step of generating a third fingerprint image may include the steps of: generating a second fingerprint image by emitting ultrasonic waves having a first frequency via the sensor unit; and generating a third fingerprint image by emitting ultrasonic waves having a second frequency via the sensor unit.

[0041] The method may further include the following steps: setting a first image conversion model by learning a fingerprint image; generating fingerprint images with different signal-to-noise ratios relative to a first object while changing the driving conditions of the sensor unit; and updating the first image conversion model by learning the fingerprint image.

[0042] The method may further include the following steps: generating a second fingerprint image through a sensor unit; comparing the second fingerprint image with a template and outputting a first comparison result; generating a converted fingerprint image based on the first comparison result by increasing the signal-to-noise ratio of the second fingerprint image using an image conversion model; and comparing the converted fingerprint image with a template and outputting a second comparison result.

[0043] It should be understood that both the foregoing general description and the following detailed description are exemplary and illustrative, and are intended to provide further explanation of the claimed invention. Attached Figure Description

[0044] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the invention and, together with the specification, serve to explain the inventive concept.

[0045] Figure 1A This is a block diagram schematically illustrating an exemplary embodiment of a display device constructed according to the principles of the present invention.

[0046] Figure 1B It is shown schematically. Figure 1A A block diagram of another exemplary embodiment of the display device.

[0047] Figure 2A It is shown Figure 1A A cross-sectional view of an example display device.

[0048] Figure 2B It is shown Figure 1A A cross-sectional view of another example of a display device.

[0049] Figure 2C It is shown Figure 1A A cross-sectional view of another example of a display device.

[0050] Figure 3 It is shown that it includes Figure 1A A block diagram of an example fingerprint verification device in a display device.

[0051] Figure 4 It is shown that it includes Figure 3 A circuit diagram of an example sensor pixel in a fingerprint verification device.

[0052] Figure 5 It is shown Figure 3 A block diagram of an example controller.

[0053] Figure 6 It is shown that it includes Figure 5 A diagram illustrating the operation of the image processing unit in the controller.

[0054] Figure 7 It is shown in Figure 5 A diagram illustrating the templates used in the controller.

[0055] Figure 8 and Figure 9 It is shown that it includes Figure 5 A diagram illustrating the operation of the learning unit in the controller.

[0056] Figure 10 It is shown that it is used for Figure 5 The controller learns fingerprint images in real time.

[0057] Figure 11 This is a flowchart illustrating an exemplary embodiment of a fingerprint verification method according to the principles of the present invention.

[0058] Figure 12 It is shown that it includes Figure 11 A diagram illustrating an example of the step in the method that requests fingerprint input.

[0059] Figure 13 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention.

[0060] Figure 14 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention.

[0061] Figure 15 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention.

[0062] Figure 16 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention. Detailed Implementation

[0063] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of various exemplary embodiments or implementations of the invention. As used herein, “implementation” and “method” are interchangeable terms and are non-limiting examples of apparatus or methods employing one or more of the inventive concepts disclosed herein. However, it will be apparent that various exemplary embodiments may be practiced without these specific details or using one or more equivalent arrangements. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the various exemplary embodiments. Furthermore, the various exemplary embodiments may be different, but are not necessarily exclusive. For example, a particular shape, configuration, and characteristic of one exemplary embodiment may be used or implemented in another exemplary embodiment without departing from the inventive concept.

[0064] Unless otherwise stated, the exemplary embodiments shown should be understood as exemplary features providing details of variations in some ways in which the inventive concept can be implemented in practice. Therefore, unless otherwise stated, features, components, modules, layers, films, panels, areas and / or aspects, etc. (hereinafter individually or collectively referred to as “elements”) of various embodiments may be combined, separated, interchanged and / or rearranged in other ways without departing from the inventive concept.

[0065] Crosshairs and / or shading are typically used in the accompanying drawings to clarify boundaries between adjacent elements. Therefore, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for particular materials, material properties, dimensions, proportions, commonalities between illustrated elements, and / or any other characteristics, properties, etc., of the elements. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of elements may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a particular sequence of processes may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously, or in the reverse order of their description. Similarly, the same reference numerals denote the same elements.

[0066] When an element (such as a layer) is referred to as being “on,” “connected to,” or “linked to” another element or layer, it may be directly on, directly connected to, or directly linked to that other element or layer, or there may be an intervening element or layer. However, when an element or layer is referred to as being “directly” on, directly connected to, or directly linked to another element or layer, there is no intervening element or layer. Therefore, the term “connection” can refer to a physical connection, electrical connection, and / or fluid connection, with or without an intervening element. For the purposes of this disclosure, “at least one of X, Y, and Z” and “at least one selected from the group consisting of X, Y, and Z” can be interpreted as only X, only Y, only Z, or any combination of two or more of X, Y, and Z, such as XYZ, XYY, YZ, and ZZ. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0067] Although the terms “first,” “second,” etc., may be used herein to describe various types of elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another. Therefore, without departing from the teachings of this disclosure, the first element discussed below may be referred to as the second element.

[0068] For descriptive purposes, spatial relative terms such as “below,” “under,” “below,” “down,” “above,” “above,” “higher,” and “side” (e.g., as in “sidewall”) may be used herein to describe the relationship between one element and another as shown in the accompanying drawings. In addition to the orientations depicted in the drawings, spatial relative terms are intended to encompass different orientations of the device during use, operation, and / or manufacture. For example, if the device in the drawings is flipped, an element described as “below” or “under” another element or feature would then be oriented “above” that other element or feature. Thus, the exemplary term “below” can encompass both above and below orientations. Furthermore, the device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and where so, the spatial relative descriptive terms used herein should be interpreted accordingly.

[0069] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when used in this specification, the terms “comprises,” “comprising,” “includes,” and / or “including” specify the presence of stated features, integrals, steps, operations, elements, components, and / or groups thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximations, not as terms of degree, and therefore to explain the inherent deviations in measurements, calculations, and / or provided values ​​that will be recognized by those skilled in the art.

[0070] As is customary in the art, the accompanying drawings describe and illustrate some exemplary embodiments in terms of functional blocks, units, and / or modules. Those skilled in the art will understand that these blocks, units, and / or modules are physically implemented by electronic (or optical) circuits (such as logic circuits, discrete components, microprocessors, hardwired circuits, memory elements, wiring connections, etc.) formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where blocks, units, and / or modules are implemented by microprocessors or other similar hardware, they can be programmed and controlled using software (e.g., microcode) to perform the various functions discussed herein, and may optionally be driven by firmware and / or software. It is also contemplated that each block, unit, and / or module may be implemented by dedicated hardware, or as a combination of dedicated hardware performing some functions and processors performing other functions (e.g., one or more programmed microprocessors and associated circuitry). Furthermore, without departing from the scope of the inventive concept, each block, unit, and / or module of some exemplary embodiments may be physically divided into two or more interactive and discrete blocks, units, and / or modules. Furthermore, without departing from the scope of the inventive concept, some exemplary embodiments of blocks, units and / or modules may be physically combined into more complex blocks, units and / or modules.

[0071] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms such as those defined in common dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant field and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0072] In the following detailed description of exemplary embodiments with reference to the accompanying drawings, this disclosure will be readily practiced by those skilled in the art. This disclosure may be implemented in various different forms and is not limited to the exemplary embodiments described herein.

[0073] Parts irrelevant to the description will be omitted to clearly describe this disclosure, and throughout the specification, the same or similar constituent elements will be denoted by the same reference numerals. Therefore, the same reference numerals may be used to identify the same or similar elements in different figures.

[0074] Furthermore, for better understanding and ease of description, the dimensions and thicknesses of each component shown in the accompanying drawings are arbitrarily illustrated, but the exemplary embodiments are not limited thereto. For clarity, the thicknesses of certain parts and regions have been exaggerated.

[0075] Figure 1AThis is a block diagram schematically illustrating an exemplary embodiment of a display device constructed according to the principles of the present invention. Figure 1B It is shown schematically. Figure 1A A block diagram of another exemplary embodiment of the display device shown. For ease of description, although... Figure 1A and Figure 1B The illustration shows a scenario where the display panel (DP) and driver (DRV) are separate from each other; however, the exemplary implementation is not limited to this. More specifically, the entire driver DRV or a portion of the driver DRV can be integrated with the display panel (DP).

[0076] refer to Figure 1A and Figure 1B The display device DD may include a display panel DP and a driver DRV. The driver DRV may include a panel driver DRV_DP and a fingerprint detector DRV_FP (e.g., a fingerprint verification unit).

[0077] The entire display device DD or at least a portion thereof may be flexible.

[0078] The display panel DP includes a display area AA and a non-display area NA. The display area AA is an area in which a plurality of pixels PXL (e.g., a plurality of subpixels) are disposed, and may be referred to as an active area. In various exemplary embodiments, each of the pixels PXL may include at least one light-emitting device. The display device DD drives the pixels PXL with image data provided from the outside, thereby displaying an image in the display area AA.

[0079] In an exemplary embodiment, the display area AA may include a fingerprint sensing area FSA. The fingerprint sensing area FSA may include at least some of the pixels PXL disposed in the display area AA.

[0080] In an exemplary implementation, such as Figure 1A As shown, at least a portion of the display area AA can be configured as the fingerprint sensing area FSA. In another exemplary embodiment, as... Figure 1B As shown, the entire display area AA can be set as the fingerprint sensing area FSA. When a fingerprint is detected, the fingerprint sensing operation can be performed only on the portion where the user's touch is actually made.

[0081] For example, although in Figure 1A The illustration shows an example in which only one fingerprint sensing area (FSA) is formed in the display area AA, but the exemplary implementation is not limited thereto. For example, multiple fingerprint sensing areas (FSAs) may be formed in the display area AA in a regular or irregular arrangement.

[0082] Furthermore, despite Figure 1AAn example is shown in which a fingerprint sensing region FSA is formed in at least a portion of the display area AA, but the exemplary implementation is not limited thereto. In various exemplary implementations, the display area AA and the fingerprint sensing region FSA may be configured to overlap each other only in at least a portion of the area.

[0083] The non-display area NA is the area located outside the display area AA, and can be referred to as a non-active area. For example, the non-display area NA can include line areas, pad areas, various dummy areas, etc.

[0084] In an exemplary embodiment, the display device DD may further include a plurality of sensor pixels SPXL disposed in the fingerprint sensing area FSA.

[0085] In an exemplary embodiment, each of the sensor pixels SPXL can be configured as an optical sensor for sensing light. When light emitted from a light source (e.g., pixel PXL) included in the display device DD is reflected by a user's finger, each of the sensor pixels SPXL can sense the reflected light and output a corresponding electrical signal (e.g., a voltage signal). The electrical signal of each of the sensor pixels SPXL can constitute a point in a fingerprint image (i.e., a pixel or a bright / dark point as the smallest unit constituting the fingerprint image). Depending on whether the reflected light is caused by the valleys or ridges of a fingerprint formed on the user's finger, the palm pattern, or the dermatoglyphics of the skin, the reflected light incident on each of the sensor pixels SPXL can have different optical characteristics (e.g., frequency, wavelength, size, etc.). Therefore, the sensor pixel SPXL can output a sensing signal SS with different electrical characteristics corresponding to the optical characteristics of the reflected light.

[0086] When sensor pixels SPXL are arranged in the fingerprint sensing area FSA, sensor pixels SPXL may overlap with pixels PXL or be located around the periphery of pixels PXL. For example, some or all of sensor pixels SPXL may overlap with pixels PXL or be located between pixels PXL. In various exemplary embodiments, sensor pixels SPXL and pixels PXL may have the same size or different sizes. The relative size and arrangement between sensor pixels SPXL and pixels PXL may vary.

[0087] When sensor pixel SPXL is positioned adjacent to or at least partially overlaps with pixel PXL, sensor pixel SPXL can use the light-emitting device included in each pixel PXL as a light source. Therefore, sensor pixel SPXL and the light-emitting device included in pixel PXL can constitute a photosensitive fingerprint sensor. As described above, when a display device DD with a built-in fingerprint sensor is configured by using pixel PXL as a light source without utilizing any external light source, the module thickness of the photosensitive fingerprint sensor or the display device DD with a photosensitive fingerprint sensor is reduced, and manufacturing costs are lowered.

[0088] In another exemplary embodiment, the sensor pixel SPXL can be configured as an ultrasonic sensor for sensing ultrasonic waves. The sensor pixel SPXL can emit ultrasonic signals and output corresponding electrical signals (e.g., sensing signals SS) by sensing ultrasonic waves reflected by the user's finger.

[0089] In various exemplary embodiments, sensor pixel SPXL may be arranged on another surface (e.g., the rear surface of display panel DP) opposite the surface on which the image is displayed (e.g., the front surface of display panel DP). However, exemplary embodiments are not limited thereto. For example, sensor pixel SPXL may be configured to be closer to the front surface of display panel DP than pixel PXL.

[0090] The driver DRV can drive the display panel DP. For example, the driver DRV can output a data signal DS corresponding to image data to the display panel DP. Furthermore, the driver DRV can output a drive signal for the sensor pixel SPXL and receive electrical signals (e.g., sensing signals SS) received from the sensor pixel SPXL. The driver DRV can detect the user's fingerprint shape using the received electrical signals.

[0091] In some exemplary embodiments, the driver DRV may include a panel driver DRV_DP and a fingerprint detector DRV_FP. Each of the panel driver DRV_DP and the fingerprint detector DRV_FP may be implemented as an integrated circuit and mounted on a flexible circuit board. The panel driver DRV_DP can be connected to the display panel DP via the flexible circuit board, and the fingerprint detector DRV_FP can be connected to the sensor pixel SPXL. Although in Figure 1A and Figure 1B The illustration shows a scenario where the panel driver DRV_DP and the fingerprint detector DRV_FP are separate from each other, but the exemplary implementation is not limited to this. For example, at least a portion of the fingerprint detector DRV_FP may be integrated with the panel driver DRV_DP, or may operate in conjunction with the panel driver DRV_DP.

[0092] The panel driver DRV_DP can provide the data signal DS corresponding to the image data to the pixels PXL while sequentially scanning the pixels PXL of the display area AA. Then, the display panel DP can display the image corresponding to the image data.

[0093] In an exemplary embodiment, the panel driver DRV_DP can provide a drive signal for fingerprint sensing operation to pixel PXL. The drive signal can be provided to pixel PXL such that pixel PXL emits light according to the drive signal and operates as a light source for sensor pixel SPXL. Therefore, the drive signal for fingerprint sensing operation can be provided to pixel PXL located in a specific area of ​​the display panel DP (e.g., pixel PXL located in the fingerprint sensing area FSA).

[0094] The fingerprint detector DRV_FP can detect or identify fingerprints based on the sensing signal SS transmitted from the sensor pixel SPXL. For example, the fingerprint detector DRV_FP can convert the sensing signal SS into a fingerprint image (e.g., image data) and perform fingerprint verification based on the fingerprint image. The sensor pixel SPXL and the fingerprint detector DRV_FP can constitute a fingerprint verification device FDD (e.g., a fingerprint sensing device).

[0095] In some exemplary embodiments, the fingerprint detector DRV_FP can generate or update a template based on a fingerprint image (i.e., a fingerprint image generated by the sensing operation of sensor pixels SPXL) during a fingerprint registration period (e.g., a first period). The template may include at least one fingerprint image used as a reference (e.g., a comparison reference) in fingerprint verification, or may include information about feature points extracted from at least one fingerprint image. Figure 7 Description template.

[0096] In some exemplary embodiments, the fingerprint detector DRV_FP can perform fingerprint verification by comparing a fingerprint image acquired during a fingerprint verification period (e.g., a second period) via sensing operations of the sensor pixel SPXL with a template. Hereinafter, the fingerprint image acquired during the fingerprint verification period is referred to as the "sensed fingerprint image". Reference will be made to... Figure 5 Describe fingerprint verification.

[0097] In some exemplary embodiments, the fingerprint detector DRV_FP can set up an image transformation model by learning (e.g., machine learning operations) various fingerprint images provided externally and / or fingerprint images acquired during the fingerprint verification period. Furthermore, the fingerprint detector DRV_FP can generate pseudo-images by using the image transformation model to transform or process registered or stored images (e.g., fingerprint images added to the template during the fingerprint registration period) and add the pseudo-images to the template. The various fingerprint images can include high-quality and low-quality fingerprint images of each of various objects (e.g., users), and the image transformation model can be a transformation algorithm (e.g., a function) between high-quality and low-quality fingerprint images. The quality of a fingerprint image can be determined by the brightness difference between ridges and valleys in the fingerprint image or by the signal-to-noise ratio (SNR) of the fingerprint image. High quality may mean a SNR greater than or equal to a reference SNR, and low quality may mean a brightness difference less than a reference brightness difference or a SNR less than a reference SNR. When a registered or stored image is transformed into a pseudo-image using the image transformation model, the quality of the pseudo-image may be lower than the quality of the registered image.

[0098] Please refer to later Figure 8 and Figure 9 Describe the scenario where the fingerprint detector DRV_FP sets up an image transformation model by learning (e.g., machine learning operations) and generates pseudo-images using the image transformation model.

[0099] The quality of sensed fingerprint images may decrease or degrade depending on skin condition (e.g., when skin is dry or wrinkled or contracted) or verification environment such as the fingerprint sensing environment (e.g., when the surrounding environment is very bright or dark). When fingerprint verification is performed using a template that only includes fingerprint images registered or stored during the fingerprint registration period, the similarity (e.g., consistency rate) between a relatively low-quality sensed fingerprint image and a relatively high-quality registered fingerprint image is low. Therefore, the accuracy of fingerprint recognition may be reduced, and fingerprint verification may fail. In particular, fingerprint verification may fail for extended periods until the skin condition returns to normal or the verification environment changes to a normal verification environment.

[0100] Therefore, the fingerprint detector DRV_FP can learn to set up an image transformation model. This model processes registered or stored fingerprint images to generate low-quality pseudo-images, and fingerprint verification is performed using a template incorporating these pseudo-images. Thus, although the quality of the sensed fingerprint image decreases or degrades when the user's skin condition or the verification environment is abnormal, the similarity between the low-quality sensed fingerprint image and the low-quality pseudo-image is high. This improves the accuracy of fingerprint recognition.

[0101] Furthermore, the fingerprint detector DRV_FP not only adds low-quality sensed fingerprint images to the template, but also converts registered fingerprint images into pseudo-images using an image transformation model. These converted pseudo-images (e.g., new fingerprint images for verification) reflect or incorporate various conditions (e.g., skin condition and verification environment). Therefore, fingerprints can be accurately identified in additional states of the user's skin and / or in unexpected verification environments (where additional states of the user's skin and unexpected verification environments were not considered during fingerprint registration).

[0102] Figure 2A It is shown Figure 1A A cross-sectional view of an example of the display device shown. Figure 2A It shows Figure 1A and Figure 1B The portion of the display device DD shown in the fingerprint sensing area FSA.

[0103] refer to Figure 1A , Figure 1B and Figure 2A The display device DD may include a display panel DP and a sensor PS (e.g., a fingerprint sensor) disposed on the lower surface of the display panel DP in the fingerprint sensing area FSA. Additionally, the display panel DP may include a substrate SUB and a circuit element layer BPL, a light-emitting device layer LDL, a first protective layer PTL1, a first adhesive layer ADL1, and a window WIN sequentially stacked or disposed on the upper surface of the substrate SUB. Furthermore, the display panel DP may include a second adhesive layer ADL2 and a second protective layer PTL2 sequentially stacked or disposed on the lower surface of the substrate SUB in the fingerprint sensing area FSA.

[0104] The substrate SUB is the base substrate of the display panel DP and can be substantially transparent. For example, the substrate SUB can be a transparent transmissive substrate. The substrate SUB can be a rigid substrate such as glass or tempered glass, or a flexible substrate formed of plastic. However, exemplary embodiments are not limited thereto, and the substrate SUB can be made of various materials.

[0105] The circuit element layer BPL may be disposed on the upper surface of the substrate SUB and includes at least one conductive layer. For example, the circuit element layer BPL may include multiple circuit elements constituting the pixel circuitry of the pixel PXL, as well as lines for providing various power supplies and signals for driving the pixel PXL. The circuit element layer BPL may include various types of circuit elements (such as at least one transistor and at least one capacitor) and multiple conductive layers for forming lines connected to the circuit elements. Furthermore, the circuit element layer BPL may include at least one insulating layer disposed between the multiple conductive layers.

[0106] The light-emitting device layer (LDL) can be disposed on the upper surface of the circuit element layer (BPL). The LDL may include multiple light-emitting devices (LDs) connected to the circuit elements and / or the circuit element layer (BPL) via contact holes or the like.

[0107] In an exemplary embodiment, at least one of a plurality of light-emitting devices (LDs) may be included in each pixel PXL. For example, the light-emitting device LD may be configured as an organic light-emitting diode or an inorganic light-emitting diode, such as a micro light-emitting diode or a quantum dot light-emitting diode. Furthermore, the light-emitting device LD may be a light-emitting device formed from a combination of organic and inorganic materials. Additionally, each of the pixels PXL includes a single light-emitting device LD. Alternatively, in another exemplary embodiment, each of the pixels PXL may include a plurality of light-emitting devices LDs, and the plurality of light-emitting devices LDs may be connected in parallel with each other, connected in series with each other, or connected in both series and parallel with each other.

[0108] The first protective layer PTL1 can be disposed on top of the light-emitting device layer LDL to cover the display area AA. The first protective layer PTL1 may include encapsulation components, such as a thin-film encapsulation (TFE) layer or an encapsulation substrate, and in addition to the encapsulation components, it may also include a protective film, etc.

[0109] The first adhesive layer ADL1 is disposed between the first protective layer PTL1 and the window WIN to connect the first protective layer PTL1 and the window WIN to each other. The first adhesive layer ADL1 may include a transparent adhesive such as optically clear adhesive (OCA), and in addition to transparent adhesives, it may include various adhesive materials.

[0110] A window (WIN) is a protective component located at the top of a display device (DD) including a display panel (DP), and can be substantially transparent. For example, a window (WIN) can be a transparent transmissive substrate. A window (WIN) can have a multi-layered structure, including a glass substrate, a plastic film, and / or a plastic substrate. A window (WIN) can include a rigid or flexible substrate, and the materials constituting the window (WIN) can vary or be modified.

[0111] In various exemplary embodiments, the display device DD may further include a polarizer, an anti-reflective layer, and / or a touch sensor layer (e.g., a touch electrode layer). For example, the display device DD may also include a polarizer and / or a touch sensor layer disposed between the first protective layer PTL1 and the window WIN.

[0112] The second protective layer PTL2 may be disposed on the lower surface of the substrate SUB. The second protective layer PTL2 may be bonded to the substrate SUB via a second adhesive layer ADL2. The second adhesive layer ADL2 may include a transparent adhesive such as OCA. The second adhesive layer ADL2 may include a pressure-sensitive adhesive (PSA) that serves as an adhesive material when pressure is applied to the adhesive surface of the second adhesive layer ADL2.

[0113] The second protective layer, PTL2, prevents oxygen and moisture from being introduced from the outside and can be in the form of a single layer or multiple layers. The second protective layer, PTL2, can be configured as a film to further provide flexibility to the display panel, DP. The second protective layer, PTL2, can be attached to the sensor, PS, via another adhesive layer comprising a transparent adhesive material such as OCA.

[0114] In various exemplary embodiments, when the sensor PS includes an optical sensor, a selective light-blocking film may also be provided on the bottom of the second protective layer PTL2. The selective light-blocking film blocks light (e.g., ultraviolet light) in a specific frequency range of external light introduced into the display device DD, so as to prevent the corresponding light from incident on the sensor pixels SPXL of the sensor PS.

[0115] The sensor PS is attached to the lower surface (e.g., the rear surface) of the display panel DP by an adhesive or the like to overlap with at least one area of ​​the display panel DP. For example, the sensor PS may be configured to overlap with the display panel DP in the fingerprint sensing area FSA. The sensor PS may include a plurality of sensor pixels SPXL arranged or dispersed at a predetermined resolution and / or predetermined distance.

[0116] In an exemplary embodiment, when the sensor PS includes an optical sensor, an optical system can be provided on the sensor PS to provide an optical path by focusing light transmitted toward the sensor PS. In the optical system, the width of the light transmission portion used to guide the light can be determined by considering sensing accuracy and light conversion efficiency. The focusing ratio of the light incident on the sensor PS can be improved by the optical system. In some exemplary embodiments, the optical system can be formed of optical fibers, silicon, etc.

[0117] The sensor pixels SPXL can have an appropriate number, size, and arrangement to generate a fingerprint image that can be identified from the electrical signals output by the sensor pixels SPXL. The distance between the sensor pixels SPXL can be set so that light reflected from the object to be observed (e.g., a fingerprint, etc.) (hereinafter also referred to as "reflected light") can be incident on at least two adjacent sensor pixels SPXL.

[0118] Figure 2B It is shown Figure 1AA cross-sectional view of another example of the display device shown.

[0119] refer to Figure 1A , Figure 1B , Figure 2A and Figure 2B The display device DD may further include a light-blocking layer PHL, which includes a pinhole PIH. The light-blocking layer PHL may be disposed within the display panel DP or between the display panel DP and the sensor pixel SPXL to block some of the light incident on the sensor pixel SPXL. For example, some of the light incident on the light-blocking layer PHL can be blocked, while other light can pass through the pinhole PIH to reach the sensor pixel SPXL below the light-blocking layer PHL.

[0120] Each of the pinhole PIHs may include an optical aperture and is a light-passing aperture. For example, a pinhole PIH may be a light-passing aperture with the smallest size (e.g., area) among light-passing apertures disposed on such a path when the layers of the display device DD overlap each other, wherein reflected light passes through the display panel DP in an oblique or vertical direction along the path and then incident on the sensor pixel SPXL.

[0121] The pinhole PIH can have a predetermined width (e.g., diameter), for example, a width (e.g., diameter) in the range of 5 μm to 20 μm. In this way, as each pinhole PIH becomes farther away from the light-blocking layer PHL (e.g., as the pinhole PIH moves away in the upward and downward directions), the width of the optical opening area to be required or ensured in each layer of the display device DD can gradually increase.

[0122] Only the reflected light passing through the pinhole PIH can reach the sensor pixel SPXL. The phase of the light reflected from the fingerprint and the phase of the image formed in the sensor PS have a phase difference of 180 degrees due to the very narrow width of the pinhole PIH.

[0123] The sensor pixel SPXL can output a sensing signal SS, such as a voltage signal corresponding to the reflected light received by the sensor pixel SPXL.

[0124] However, this is an illustrative example, and the configuration, arrangement, driving method, etc., of the optical sensor used to detect light reflected from a fingerprint are not limited to... Figure 2B Those are the sensors PS shown.

[0125] For example, although in Figure 2B A pinhole-type optical sensor is shown, but exemplary embodiments are not limited thereto. For example, a microlens-type optical sensor or a collimator-type optical sensor can be used.

[0126] Figure 2CIt is shown Figure 1A A cross-sectional view of yet another example of the display device shown.

[0127] refer to Figures 1A to 2A and Figure 2C The sensor PS can include an ultrasonic sensor. The sensor pixels SPXL (see...) Figure 2A This can be achieved using piezoelectric elements PZ. A filler PIL, used to isolate vibrations between the piezoelectric elements PZ, can be placed between them.

[0128] Each of the piezoelectric elements PZ may include a vibrator and electrodes disposed on the top and bottom surfaces of the vibrator. The vibrator may include a material that is easily vibrated, such as lead zirconate titanate (PZT).

[0129] When an AC voltage with a specific frequency is applied to the electrodes of the piezoelectric element PZ, ultrasonic waves with a specific frequency (e.g., 12 MHz) can be emitted while the vibrator vibrates vertically.

[0130] At the valleys of the fingerprint, most of the ultrasonic waves do not pass through the window WIN because they are mostly reflected due to the difference in medium between the window WIN and the air. On the other hand, at the ridges of the fingerprint that contact the window WIN, some of the ultrasonic waves can pass through the window WIN, and only the remaining ultrasonic waves can be reflected towards the sensor PS. For example, the intensity of the ultrasonic waves incident on each of the sensor pixels SPXL can vary depending on whether the ultrasonic waves are caused or reflected by the valleys or ridges of the fingerprint. Therefore, the sensor pixels SPXL can output sensing signals SS with different electrical characteristics corresponding to the intensity of the ultrasonic waves.

[0131] Figure 3 It is shown that it includes Figure 1A A block diagram illustrating an example of a fingerprint verification device in a display device. Hereinafter, the case where the sensor PS is an optical sensor is described exemplarily.

[0132] refer to Figure 1A , Figure 2A and Figure 3 A fingerprint verification device (FDD) may include a sensor (PS) and a fingerprint detector (DRV_FP).

[0133] The sensor PS may include an array of sensor pixels SPXL. In an exemplary embodiment, the sensor pixels SPXL may be arranged in a two-dimensional array, but the exemplary embodiment is not limited thereto. Each of the sensor pixels SPXL may include an optoelectronic device that converts incident light into electrical charge according to the amount of light.

[0134] The fingerprint detector DRV_FP may include a horizontal driver DRV_H, a vertical driver DRV_V, and a controller CON.

[0135] The horizontal driver DRV_H can be connected to the sensor pixel SPXL via drive lines H1 to Hn. The horizontal driver DRV_H can be configured as a shift register, address decoder, etc. In various exemplary embodiments, the horizontal driver DRV_H can apply drive signals to drive selected sensor pixels SPXL. For example, the horizontal driver DRV_H can apply drive signals on a row-by-row basis.

[0136] The sensor pixel SPXL, selected and driven by the horizontal driver DRV_H, senses light by using the optoelectronic device disposed therein and outputs an electrical signal (e.g., sensing signal SS) as a voltage signal in analog form corresponding to the sensed light.

[0137] The vertical driver DRV_V can be connected to the sensor pixel SPXL via signal lines V1 to Vm. The vertical driver DRV_V can process the electrical signals output from the sensor pixel SPXL.

[0138] For example, the vertical driver DRV_V can perform correlated double sampling (CDS) processing to remove noise from the electrical signal. Furthermore, the vertical driver DRV_V can convert the analog signal received from the sensor pixel SPXL into a digital signal. In an exemplary embodiment, an analog-to-digital converter can be provided for each column of sensor pixels to process the analog signals received from the column of sensor pixels in parallel.

[0139] The controller CON can control the horizontal driver DRV_H and the vertical driver DRV_V.

[0140] In an exemplary embodiment, the controller CON can generate image data (e.g., fingerprint data) corresponding to the sensing signal SS received from the vertical driver DRV_V, and perform processing on the image data. Furthermore, in an exemplary embodiment, the controller CON can detect fingerprints from the processed image data and verify the detected fingerprints or send the detected fingerprints externally. However, this is merely an example, and the generation of image data and fingerprint detection are not performed by the controller CON, but can be performed by an external host processor or the like.

[0141] Figure 4 It is shown that it includes Figure 3 A circuit diagram illustrating an example of sensor pixels in a fingerprint verification device. Figure 4 The image shows the sensor pixel SPXL positioned in the x-th ('x' is a natural number) sensor pixel row and the x-th sensor pixel column.

[0142] refer to Figure 3 and Figure 4 The sensor pixel SPXL includes an optical diode PD (as an example of an optoelectronic device), a transmission transistor TRTX, a reset transistor TRRX, and an amplification transistor TRAMP. Figure 4 The image shows an example in which an N-type transistor is used to implement a transistor. However, in various exemplary embodiments, at least some of the transistors can be implemented using P-type transistors, and correspondingly, various modifications can be made to the circuit structure of the sensor pixel SPXL.

[0143] The anode of the optical diode PD is grounded. The transfer transistor TRTX is connected between the cathode of the optical diode PD and the first node N1. The gate electrode of the transfer transistor TRTX is connected to the drive line Hx. When a drive signal is applied from the horizontal driver DRV_H through the drive line Hx, the transfer transistor TRTX is turned on. When the transfer transistor TRTX is turned on, the charge generated by photoelectric conversion in the optical diode PD can be transferred to the first node N1.

[0144] The reset transistor TRRX is connected between the reset power supply VRESET and the first node N1, and can receive a reset signal applied through the gate electrode connected to the reset line RST. When a reset signal is applied, the reset transistor TRRX turns on. When the reset transistor TRRX is on, the voltage of the first node N1 can be reset to the voltage of the reset power supply VRESET.

[0145] The TRAMP amplifier transistor is connected between the reset power supply VRESET and the signal line Vx. The gate electrode of the TRAMP amplifier transistor is connected to the first node N1. The TRAMP amplifier transistor operates as an amplifier, outputting a signal to the signal line Vx corresponding to the voltage of the first node N1.

[0146] In various exemplary embodiments, the structure of the sensor pixel SPXL is not limited to the structure described above, and the sensor pixel SPXL may include, for example, four or more transistors or two or fewer transistors.

[0147] Figure 5 It is shown Figure 3 A block diagram of an example controller. Figure 6 It is shown that it includes Figure 5 A diagram illustrating the operation of the image processing unit in the controller. Figure 7 It is shown in Figure 5 A diagram illustrating the templates used in the controller. Figure 8 and Figure 9 It is shown that it includes Figure 5 A diagram illustrating the operation of the learning unit in the controller. Figure 10 It is shown that it is used for Figure 5 The controller learns fingerprint images in real time.

[0148] refer to Figure 3 and Figure 5 The controller CON may include an image processing unit (IPU), a determination unit (DTU), a learning unit (MLU), and a storage unit (STU).

[0149] The image processing unit (IPU) can generate a fingerprint image (IMAGE) based on the sensing signal (SS) provided by the vertical driver (DRV_V) (or the sensor (PS)).

[0150] refer to Figure 6 The image processing unit (IPU) can convert the sensing signal SS into a raw image IMAGE_O. The raw image IMAGE_O can be transmitted via a pinhole camera (PIH) (see [link to IPU]). Figure 2B The light path caused by the light is represented as a grid pattern.

[0151] In an exemplary implementation, such as Figure 6 As shown, the image processing unit (IPU) can extract the effective region EA from the original image IMAGE_O. In the original image IMAGE_O, the effective region EA may refer to or correspond to the area where reflected light passes through a pinhole PIH (see Figure 1). Figure 2B The area to be received.

[0152] In an exemplary embodiment, the image processing unit (IPU) can perform image processing on the original image IMAGE_O or the valid region EA extracted from the original image IMAGE_O.

[0153] For example, the image processing unit (IPU) can perform a smoothing operation on the effective region EA. The IPU can increase the brightness difference within the effective region EA and remove noise from it. The smoothing operation can be performed by performing histogram analysis on each pixel in the original image IMAGE_O. For example, a median filter can be used for the smoothing operation. The smoothing operation can be performed using various algorithms known in the art, but exemplary embodiments are not limited thereto.

[0154] For example, the image processing unit (IPU) can perform binarization and thinning operations on the original image IMAGE_O or the effective region EA extracted from the original image IMAGE_O. The IPU can convert multiple gray levels (e.g., 256 gray levels) into values ​​corresponding to 0 (e.g., black) or 1 (e.g., white). Then, the ridges and valleys constituting the fingerprint can be clearly distinguished as black and white, respectively. Furthermore, the IPU can generate a line image from the binarized image where the width of the ridges is 1 pixel. Binarization and thinning operations are performed to improve the accuracy of fingerprint detection. However, when binarization and thinning operations are not required, they can be omitted.

[0155] In exemplary embodiments, at least one of the image processing operations described above may be omitted, or more image processing operations may be performed if necessary. For example, when the original image IMAGE_O is a sufficiently satisfactory image with low noise, a smoothing operation may not be performed.

[0156] When a valid region EA is extracted, the image processing unit (IPU) can generate a composite image IMAGE_C (e.g., composite data or composite image data) by synthesizing the image-processed valid region EA. For example, the image processing unit (IPU) can generate a composite image IMAGE_C by sequentially or continuously concatenating the extracted valid regions EA.

[0157] exist Figure 6 The process described herein involves a display device DD generating a fingerprint image IMAGE from a sensing signal SS corresponding to the reflected light transmitted through the pinhole PIH. However, exemplary embodiments are not limited thereto. When the display device DD senses a fingerprint using another method (e.g., when the display device DD acquires the sensing signal SS via an ultrasonic sensor), a fingerprint image IMAGE can be generated using various image processing techniques.

[0158] In the following text, for ease of description, the initial generated / stored fingerprint image (e.g., a synthetic image IMAGE_C) generated by the image processing unit IPU to perform fingerprint registration or template TEMP during the fingerprint registration period is defined as the first fingerprint image IMAGE1, and the fingerprint image generated by the image processing unit IPU to perform fingerprint verification during the fingerprint verification period is defined as the second fingerprint image IMAGE2. The second fingerprint image IMAGE2 can be used for fingerprint verification. A low-quality fingerprint image generated by the image processing unit IPU by changing the driving conditions of the sensor PS relative to a substantially identical object during the fingerprint verification period at substantially the same time as the second fingerprint image IMAGE2 is defined as the third fingerprint image IMAGE3. For example, by using driving conditions of the sensor PS that are different from those used by the sensor PS to obtain the first fingerprint image IMAGE1, the third fingerprint image IMAGE3 can be obtained from a substantially identical object (which was used to obtain the first fingerprint image IMAGE1) during the fingerprint verification period. The third fingerprint image IMAGE3 is not used for fingerprint verification and can be used for learning using artificial intelligence (AI) techniques described later.

[0159] Return to reference Figure 5 During the fingerprint registration period, the Determination Unit (DTU) can determine the quality of the first fingerprint image IMAGE1 provided by the Image Processing Unit (IPU). The DTU can then determine, based on the determined quality of the first fingerprint as the quality determination result, whether to add the first fingerprint image IMAGE1 to the template TEMP or include it in the template TEMP. The template TEMP can be stored in the storage unit (STU) and may include multiple fingerprint images or information on feature points extracted from each of the multiple fingerprint images. For ease of description, the following describes the case where the template TEMP includes multiple fingerprint images.

[0160] refer to Figure 7 The template TEMP can include multiple fingerprint images that can be acquired at several locations on a finger. For example, such as Figure 7 As shown, fingerprint images IMAGE_E1, IMAGE_E2, ..., IMAGE_Ek (e.g., "k" is 12) can be generated based on the position of the finger (e.g., based on the finger's posture, according to the finger's position relative to a reference point each time). Figure 3The locations of the parts of the window (WIN) that are in contact with the fingerprint image are categorized into several groups. For example, the fingerprint images IMAGE_E1, IMAGE_E2, ..., and IMAGE_Ek included in the template TEMP may overlap each other in at least a partial area. In another example, the fingerprint images IMAGE_E1, IMAGE_E2, ..., and IMAGE_Ek included in the template TEMP may not overlap each other. In the following text, for ease of description, the fingerprint images IMAGE_E1, IMAGE_E2, ..., and IMAGE_Ek acquired during the fingerprint registration period and to be added to the template TEMP (which includes fingerprint images registered by the user) will be referred to as the registration image IMAGE_E.

[0161] Return to reference Figure 5 When the quality of the first fingerprint image IMAGE1 is higher than or equal to the reference quality (or reference quality value), the determining unit DTU can generate a template TEMP based on the first fingerprint image IMAGE1. For example, when the quality of the first fingerprint image IMAGE1 is higher than or equal to the reference quality (or reference quality value), the determining unit DTU can generate or update a registration image IMAGE_E (e.g., the first registration image IMAGE_E1) that includes at least a portion of the first fingerprint image IMAGE1, and generate a template TEMP that includes the registration image IMAGE_E.

[0162] When the quality of the first fingerprint image IMAGE1 is lower than the reference quality, the determination unit DTU can reacquire the first fingerprint image IMAGE1 using the sensor PS and the image processing unit IPU, and determine the quality of the reacquired first fingerprint image IMAGE1 again. When the quality of the reacquired first fingerprint image IMAGE1 is higher than or equal to the reference quality (e.g., the reference quality value), the determination unit DTU can generate a template TEMP based on the reacquired first fingerprint image IMAGE1, for example, by adding the reacquired first fingerprint image IMAGE1 to the template TEMP.

[0163] The Determining Unit (DTU) can generate a fingerprint by determining the quality of multiple first fingerprint images (IMAGE1). Figure 7 The template TEMP for the multiple fingerprint images IMAGE_E1, IMAGE_E2, ... and IMAGE_Ek shown, wherein the multiple first fingerprint images IMAGE1 are acquired during the fingerprint registration period or for each location on the finger.

[0164] During the fingerprint verification period, the determining unit (DTU) can compare the second fingerprint image IMAGE2 provided by the image processing unit (IPU) with the template TEMP (e.g., the registered image IMAGE_E or the fingerprint image included in the template TEMP).

[0165] In an exemplary embodiment, the determining unit (DTU) can extract a first feature point from the second fingerprint image IMAGE2 and compare the first feature point with a second feature point in each of the fingerprint images (e.g., the registration image IMAGE_E) in the template TEMP. Each of the first and second feature points can be a point at the end of a ridge and / or a point at a ridge branch of the fingerprint. The determining unit (DTU) can calculate a consistency rate (e.g., a match score) between the first and second feature points. The determining unit (DTU) can verify the second fingerprint image IMAGE2 based on this consistency rate. For example, the determining unit (DTU) can verify that the user of the second fingerprint image IMAGE2 is a registered user. The determining unit (DTU) can provide the comparison result (e.g., a verification result) to an external source (e.g., a host processor or application processor).

[0166] In some exemplary embodiments, the determination unit (DTU) can generate a drive control signal by changing the drive conditions of the sensor (PS).

[0167] For example, when the sensor PS is implemented as an optical sensor and includes a reference Figure 4 When describing the sensor pixel SPXL, the determining unit DTU can generate a drive control signal for reducing the pulse width of the drive signal applied to the drive line Hx, or a drive control signal for reducing the exposure time of the sensor pixel SPXL. The drive control signal can be provided to the horizontal driver DRV_H. (Refer to...) Figure 10 Example description. When the sensor PS generates a sensing signal SS with an exposure time of approximately 100 ms as a general drive condition, the image processing unit IPU can generate a relatively high-quality second image IMAGE2_1. When the sensor PS generates a sensing signal SS with an exposure time of approximately 60 ms (which is a reduced exposure time) in response to a drive control signal as a low-quality drive condition, the image processing unit IPU can generate a relatively low-quality third image IMAGE3_1. Exposure times of 100 ms and 60 ms are exemplarily shown, and the exposure time with respect to the drive condition can be set differently depending on the product. In some exemplary embodiments, the image processing unit IPU can be configured to generate the second image IMAGE2_1 by receiving reflected light via the sensor PS for a first exposure time, and to generate the third image IMAGE3_1 by receiving reflected light via the sensor PS for a second exposure time, wherein the second exposure time can be shorter than the first exposure time.

[0168] In another example, when the sensor PS is implemented as an ultrasonic sensor, the determination unit DTU can generate a drive control signal to reduce the frequency of the ultrasonic waves emitted from the sensor PS. (See reference...) Figure 10Example description. When the sensor PS emits an ultrasonic signal with a frequency of approximately 12 MHz as a general drive condition, the image processing unit IPU can generate a relatively high-quality second image IMAGE2_2. When the sensor PS emits an ultrasonic signal with a frequency of approximately 10 MHz (which is a reduced frequency) as a low-quality drive condition in response to a drive control signal, the image processing unit IPU can generate a relatively low-quality third image IMAGE3_2. The frequencies of 12 MHz and 10 MHz are shown exemplary, and the frequencies for the drive conditions can be set differently depending on the product.

[0169] In some exemplary embodiments, the third fingerprint image IMAGE3 can be generated sequentially or sequentially together with the second fingerprint image IMAGE2. For example, when a user's finger touches the fingerprint sensing area FSA of the display device DD (see... Figure 1A When performing fingerprint verification, the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 can be generated sequentially. The third fingerprint image IMAGE3 can be generated regardless of the verification result of the second fingerprint image IMAGE2. Since the verification time of the second fingerprint image IMAGE2 is not included, the time required from user fingerprint input to generation of the third fingerprint image IMAGE3 can be shortened. As described above, the second fingerprint image IMAGE2 can be used for fingerprint verification in the determination unit DTU. When fingerprint verification is performed effectively or sufficiently using the relatively high-quality second fingerprint image IMAGE2, the relatively low-quality third fingerprint image IMAGE3 may not be used for fingerprint verification in the determination unit DTU. When fingerprint verification is successfully performed using the second fingerprint image IMAGE2, the third fingerprint image IMAGE3 can be stored together with the second fingerprint image IMAGE2 in the storage unit STU for use in the learning unit MLU, which will be described later.

[0170] For example, although it has been described that a third fingerprint image IMAGE3 is generated without considering the verification result of the second fingerprint image IMAGE2, the exemplary implementation is not limited thereto. In some exemplary implementations, the third fingerprint image IMAGE3 may be generated after fingerprint verification has been successfully performed using the second fingerprint image IMAGE2.

[0171] like Figure 8As shown, the learning unit MLU can transform the registered image IMAGE_E (e.g., the first fingerprint image IMAGE1) included in the template TEMP into a pseudo image IMAGE_P using an image transformation model. The image transformation model can be an algorithm used to transform an image with high quality (e.g., high signal-to-noise ratio) into an image with low quality (e.g., low signal-to-noise ratio). Therefore, the signal-to-noise ratio of the pseudo image IMAGE_P can be less than that of the registered image IMAGE_E (e.g., the first fingerprint image IMAGE1). For example, the learning unit MLU can transform a high-quality fingerprint image into a low-quality fingerprint image using an image transformation model.

[0172] A pseudo-image IMAGE_P can be added to the template TEMP. The pseudo-image IMAGE_P can be used when the determination unit (DTU) compares the pseudo-image IMAGE_P with the second fingerprint image IMAGE2 during the fingerprint verification period.

[0173] The pseudo image IMAGE_P can be included together with the registered image IMAGE_E in a template TEMP, but exemplary implementations are not limited to this. For example, the pseudo image IMAGE_P can be included in a separate template (e.g., an auxiliary template) that is different from the registered image IMAGE_E (e.g., template TEMP).

[0174] In an exemplary embodiment, the pseudo image IMAGE_P can be managed separately from the registration image IMAGE_E generated during the fingerprint registration period, or separately from the fingerprint image generated directly from the first fingerprint image IMAGE1. For example, the registration image IMAGE_E generated during the fingerprint registration period can be stored in the first sub-storage unit STU-1 of the storage unit STU, and the pseudo image IMAGE_P can be stored in one of the second sub-storage unit STU-2, the third sub-storage unit STU-3, and the fourth sub-storage unit STU-4.

[0175] In some exemplary implementations, the learning unit MLU can set or update the image transformation model by learning the fingerprint image IMAGE_L used for learning.

[0176] In an exemplary embodiment, the learning unit MLU can set up a first image transformation model by learning a fingerprint image IMAGE_L provided from an external source (e.g., a database) via the storage unit STU. The fingerprint image IMAGE_L for learning can include high-quality and low-quality fingerprint images of each of multiple objects (e.g., multiple fingerprints of a user).

[0177] refer to Figure 9The learning unit (MLU) can learn pairs of high-quality fingerprint images (IMAGE_HIGH) and low-quality fingerprint images (IMAGE_LOW) using artificial intelligence (AI) techniques based on deep learning methods, and set up a first image transformation model for converting between the high-quality and low-quality fingerprint images (IMAGE_HIGH and IMAGE_LOW). For example, AI techniques based on deep learning methods (e.g., learning algorithms) may include deep belief networks, autoencoders, convolutional neural networks (CNNs), recurrent neural networks (RNNs), deep Q-networks, etc. However, while AI techniques based on deep learning methods are illustrated exemplarily, exemplary implementations are not limited thereto.

[0178] For example, the learning unit MLU can set the coefficients and constants of the transformation function used to convert a high-quality fingerprint image IMAGE_HIGH to a low-quality fingerprint image IMAGE_LOW by learning the fingerprint image IMAGE_L used for learning. Additionally, the learning unit MLU can set the coefficients and constants of the transformation function used to convert a low-quality fingerprint image IMAGE_LOW to a high-quality fingerprint image IMAGE_HIGH by learning the fingerprint image IMAGE_L used for learning.

[0179] The process of setting up the first image transformation model by learning the fingerprint image IMAGE_L used for learning can be called pre-learning.

[0180] When the first image transformation model is set up through pre-learning, the learning unit MLU can transform the registered image IMAGE_E (e.g., the first fingerprint image IMAGE1) included in the template TEMP into a first pseudo image IMAGE_P1 using the first image transformation model. The first pseudo image IMAGE_P1 can be stored in the second sub-storage unit STU-2 of the storage unit STU. The first pseudo image IMAGE_P1 is not acquired through the sensing operation of the sensor PS, but is generated by transforming the registered image IMAGE_E included in the template TEMP. Therefore, the first pseudo image IMAGE_P1 can include fingerprint images corresponding to the position and number of registered images IMAGE_E (e.g., 12 first pseudo images IMAGE_P1). In addition, the learning unit MLU can update the first pseudo image IMAGE_P1 through further learning (e.g., learning fingerprint images IMAGE_L for learning).

[0181] In an exemplary embodiment, when verification of the second fingerprint image IMAGE2 is successfully performed, the learning unit MLU can set up a second image conversion model by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3. Similar to pre-learning, the learning unit MLU can learn the high-quality second fingerprint image IMAGE2 and the low-quality third fingerprint image IMAGE3, and can set up a second image conversion model for converting between the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3.

[0182] The process of setting up a second image transformation model by learning from the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 can be called real-time learning. The second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 are actually sensed by the sensor PS and are different from the fingerprint image IMAGE_L used for learning.

[0183] When a second image transformation model is set up through real-time learning, the learning unit MLU can transform the registered image IMAGE_E (e.g., the first fingerprint image IMAGE1) into a second pseudo-image IMAGE_P2 using the second image transformation model. The second pseudo-image IMAGE_P2 can be stored in the third sub-storage unit STU-3 of the storage unit STU. Similar to the first pseudo-image IMAGE_P1, since the second pseudo-image IMAGE_P2 is generated by transforming the registered image IMAGE_E included in the template TEMP, the second pseudo-image IMAGE_P2 can include fingerprint images corresponding to the position and number of registered images IMAGE_E (e.g., 12 second pseudo-images IMAGE_P2). The second pseudo-image IMAGE_P2 can be different from the third fingerprint image IMAGE3 generated by the sensing operation of the sensor PS. Furthermore, the learning unit MLU can update the second pseudo-image IMAGE_P2 through additional learning (e.g., by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 acquired each time fingerprint verification is successfully performed).

[0184] In another exemplary embodiment, when the verification of the second fingerprint image IMAGE2 is successfully performed, the learning unit MLU can update the first image conversion model or set the third image conversion model by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3.

[0185] For example, the learning unit MLU can update the first image transformation model based on the fingerprint image IMAGE_L used for learning by additionally learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 that were actually sensed. The process of updating the first image transformation model by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 that were actually sensed can be called complex learning.

[0186] When the first image transformation model is updated through complex learning (e.g., when a third image transformation model is generated), the learning unit MLU can transform the registered image IMAGE_E (e.g., the first fingerprint image IMAGE1) into a third pseudo-image IMAGE_P3 using the updated first image transformation model. The third pseudo-image IMAGE_P3 can be stored in the fourth sub-storage unit STU-4 of the storage unit STU. Similar to the second pseudo-image IMAGE_P2, since the third pseudo-image IMAGE_P3 is generated by transforming the registered image IMAGE_E included in the template TEMP, the third pseudo-image IMAGE_P3 can include fingerprint images corresponding to the position and number of registered images IMAGE_E (e.g., 12 third pseudo-images IMAGE_P3). The third pseudo-image IMAGE_P3 can be different from the third fingerprint image IMAGE3 generated by the sensing operation of the sensor PS. Furthermore, the learning unit MLU can update the third pseudo-image IMAGE_P3 through additional learning (e.g., by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 acquired each time fingerprint verification is successful).

[0187] The storage unit (STU) can store the template (TEMP). For example, the storage unit (STU) can be implemented as a memory device.

[0188] In an exemplary embodiment, during the fingerprint registration period, the storage unit STU can store the registration image IMAGE_E (e.g., fingerprint images IMAGE_E1, IMAGE_E2, ..., and IMAGE_Ek, which include at least a portion of the first fingerprint image IMAGE1) generated by the determination unit DTU (see [link to DTU]). Figure 7 Add to the TEMP template.

[0189] In an exemplary implementation, during the fingerprint verification period, the storage unit STU may store the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 provided by the determination unit DTU based on the comparison result (e.g., the verification result) of the determination unit DTU. The second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 may only be provided from the determination unit DTU to the storage unit STU if verification is successfully performed. The second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 may not be added to the template TEMP, but may be stored separately as learning data for the learning unit MLU. However, the exemplary implementation is not limited to this. For example, the registration image IMAGE_E may be replaced by the second fingerprint image IMAGE2. For example, the second fingerprint image IMAGE2 (or feature points included in the second fingerprint image IMAGE2 compared to the registration image IMAGE_E) may be used to update the registration image IMAGE_E (or the template TEMP). In another example, the third fingerprint image IMAGE3 may be added to the template TEMP and stored in the first sub-storage unit STU-1.

[0190] In an exemplary embodiment, the storage unit STU may include a first sub-storage unit STU-1 (or a first template), a second sub-storage unit STU-2 (or a second template), a third sub-storage unit STU-3 (or a third template), and a fourth sub-storage unit STU-4 (or a fourth template). The first sub-storage unit STU-1 may store the registration image IMAGE_E generated during the fingerprint registration period, the second sub-storage unit STU-2 may store the first pseudo-image IMAGE_P1 generated through pre-learning, the third sub-storage unit STU-3 may store the second pseudo-image IMAGE_P2 generated through real-time learning, and the fourth sub-storage unit STU-4 may store the third pseudo-image IMAGE_P3 generated through complex learning.

[0191] In some exemplary embodiments, the determining unit (DTU) may perform verification on the second fingerprint image IMAGE2 by using at least one of the registered image IMAGE_E, the first pseudo image IMAGE_P1, the second pseudo image IMAGE_P2, and the third pseudo image IMAGE_P3.

[0192] For example, the determining unit (DTU) may first perform verification on the second fingerprint image IMAGE2 using the registered image IMAGE_E, and then perform verification on the second fingerprint image IMAGE2 using at least one of the first pseudo image IMAGE_P1, the second pseudo image IMAGE_P2, and the third pseudo image IMAGE_P3.

[0193] In another example, when the quality of the second fingerprint image IMAGE2 is higher than or equal to the reference quality, the determining unit DTU can perform verification on the second fingerprint image IMAGE2 using the registered image IMAGE_E. When the quality of the second fingerprint image IMAGE2 is lower than the reference quality, the determining unit DTU can perform verification on the second fingerprint image IMAGE2 using at least one of the first pseudo-image IMAGE_P1, the second pseudo-image IMAGE_P2, and the third pseudo-image IMAGE_P3.

[0194] When verification is performed using a template TEMP that includes the registered image IMAGE_E, the accuracy of fingerprint recognition may decrease or degrade depending on skin condition (e.g., when skin is dry or wrinkled or contracted) or verification environment (e.g., when the surrounding environment is very bright or dark), and fingerprint verification may fail to be performed successfully. In particular, fingerprint verification may fail to be performed successfully for an extended period of time until the skin condition returns to normal or the verification environment changes to a normal verification environment.

[0195] Therefore, the controller CON (or fingerprint verification device FDD) can set up an image transformation model through various learning methods (i.e., pre-learning, real-time learning, and complex learning). This model generates a pseudo-image IMAGE_P corresponding to the registered image IMAGE_E, and then adds the pseudo-image IMAGE_P to the template TEMP. Fingerprint verification is then performed using the template TEMP, which includes the pseudo-image IMAGE_P. Thus, fingerprint verification can be performed normally even if the user's skin condition or verification environment is abnormal.

[0196] Specifically, the controller CON can convert the registered image IMAGE_E into a pseudo image IMAGE_P using an image transformation model and add the pseudo image IMAGE_P to the template TEMP. In other words, this differs from simply adding a low-quality fingerprint image (e.g., a third fingerprint image IMAGE3) sensed by the sensor PS to the template TEMP. When a low-quality fingerprint image sensed under a specific verification environment is added to the template TEMP, the low-quality fingerprint image is only used under that specific verification environment and may not be suitable for use under another verification environment. For example, in the description, the controller CON can generate a pseudo image IMAGE_P that reflects various conditions (e.g., skin condition and verification environment) (e.g., a new fingerprint image for verification). Therefore, fingerprint verification can be accurately performed under the user's skin condition or verification environment that was not considered in fingerprint registration.

[0197] In some exemplary embodiments, the image processing unit (IPU) can reverse-engineer the second fingerprint image (IMAGE2) into a fourth fingerprint image (e.g., a fingerprint image with improved quality) using an image transformation model, and the determining unit (DTU) can compare the fourth fingerprint image with the template (TEMP). Reference will be made later. Figure 16 Describe the generation and verification of the fourth fingerprint image.

[0198] Figure 11 This is a flowchart illustrating an exemplary embodiment of a fingerprint verification method according to the principles of the present invention. Figure 12 It is shown that it includes Figure 11 A diagram illustrating an example of the step in the method that requests fingerprint input.

[0199] refer to Figure 1A , Figure 3 , Figure 5 and Figure 11 , Figure 11 The method shown can be derived from Figure 1A The display device DD shown is Figure 3 The fingerprint verification device shown is FDD executed.

[0200] Figure 11 The method shown can request fingerprint input from the user (S1110).

[0201] For example, Figure 11 The method shown can display a message or image for requesting fingerprint input in the display area AA of the display device DD, or display an image (e.g., a fingerprint image) for requesting fingerprint input in the fingerprint sensing area FSA of the display device DD.

[0202] In some exemplary embodiments, Figure 11 The method shown can request the generation of a pseudo-image IMAGE_P or request permission to expand the template TEMP. For example, as... Figure 12 As shown, Figure 11 The method shown can display a message (or image) requesting permission to expand the template TEMP in the display area AA of the display device DD. When input allowing the expansion of the template TEMP is received, Figure 11 The method shown can perform operations to generate a pseudo-image IMAGE_P and add the pseudo-image IMAGE_P to the template TEMP or expand the template TEMP, wherein by using a reference Figure 5 The image transformation model is set up to generate pseudo-images IMAGE_P based on various learning methods (e.g., pre-learning, real-time learning, and complex learning).

[0203] When input (e.g., touch) from the user occurs to the fingerprint sensing area FSA of the display device DD, Figure 11 The method shown can generate a fingerprint image by sensing a fingerprint (S1120).

[0204] For reference Figure 5 As described, the sensor PS can generate a sensing signal SS corresponding to the fingerprint by using optical methods, ultrasonic methods, etc., and the image processing unit IPU can generate a fingerprint image (e.g., a first fingerprint image IMAGE1) based on the sensing signal SS.

[0205] Figure 11 The method shown can determine whether the quality of the fingerprint image meets a predetermined quality condition (S1130). When the quality of the fingerprint image meets the predetermined quality condition... Figure 11 The method shown can store a fingerprint image (e.g., a fingerprint image including at least a portion of a first fingerprint image IMAGE1) in a template TEMP (S1140).

[0206] For reference Figure 5 As described, when the quality of the first fingerprint image IMAGE1 is higher than or equal to the reference quality (e.g., the reference quality value), the determining unit DTU can generate a registered image IMAGE_E that includes at least a portion of the first fingerprint image IMAGE1, and add the registered image IMAGE_E to the template TEMP or store it in the template TEMP.

[0207] When the quality of the first fingerprint image IMAGE1 is lower than the reference quality, the determination unit DTU can reacquire the first fingerprint image IMAGE1 through the sensor PS and the image processing unit IPU, and determine again whether the quality of the reacquired first fingerprint image IMAGE1 meets the predetermined quality conditions.

[0208] Subsequently, Figure 11 The method shown can generate a pseudo image IMAGE_P from the registered image IMAGE_E using an image conversion model (S1150), and add the pseudo image IMAGE_P to the template TEMP or expand the template TEMP by using the pseudo image IMAGE_P (S1160).

[0209] For reference Figure 5 As described, Figure 11 The method shown can set up an image transformation model through various learning methods (e.g., pre-learning, real-time learning, and complex learning), and add a pseudo-image IMAGE_P corresponding to the registered image IMAGE_E to the template TEMP or expand the template TEMP by using the image transformation model. Subsequently, Figure 11The method shown performs fingerprint verification by using a template TEMP that includes a pseudo-image IMAGE_P, which enables fingerprint verification to be performed normally even when the user's skin condition or verification environment is abnormal.

[0210] Reference Figure 13 , Figure 14 and Figure 15 Various exemplary implementations in which pseudo-images are generated and then added to a template are described.

[0211] Figure 13 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention. Figure 13 The paper illustrates a method for fingerprint verification based on pre-learning.

[0212] refer to Figure 5 , Figure 11 and Figure 13 , Figure 13 The method shown can set or update the first image conversion model by learning a fingerprint image IMAGE_L provided from an external source (e.g., a database) for learning (S1310).

[0213] Figure 13 The method shown can convert the registered image IMAGE_E into a first pseudo image IMAGE_P1 using a first image conversion model (S1320) and add the first pseudo image IMAGE_P1 to the template TEMP, or expand the template TEMP by using the first pseudo image IMAGE_P1 (S1330).

[0214] Figure 14 This is a flowchart illustrating a method for verifying fingerprints according to another exemplary embodiment. Figure 14 The paper will demonstrate a method for verifying fingerprints based on real-time learning.

[0215] refer to Figure 1A , Figure 3 , Figure 5 , Figure 11 and Figure 14 , Figure 14 The method shown can request fingerprint input from the user (S1410) and sense the fingerprint corresponding to the fingerprint input.

[0216] Figure 14 The method shown can acquire a second fingerprint image IMAGE2 and a third fingerprint image IMAGE3 by sensing fingerprints while changing the driving conditions of the sensor PS (S1420).

[0217] In an exemplary implementation, as referenced Figure 5 and Figure 10 As described, when the sensor PS is implemented as an optical sensor, the sensor PS can generate a sensing signal SS with an exposure time of approximately 100 ms, and the image processing unit IPU can generate a relatively high-quality second image IMAGE2_1. Alternatively, the sensor PS can generate a sensing signal SS with an exposure time of approximately 60 ms in response to a drive control signal, and the image processing unit IPU can generate a relatively low-quality third image IMAGE3_1.

[0218] In another exemplary embodiment, as referenced Figure 5 and Figure 10 As described, when the sensor PS is implemented as an ultrasonic sensor, the sensor PS can emit an ultrasonic signal with a frequency of approximately 12 MHz, and the image processing unit IPU can generate a relatively high-quality second image IMAGE2_2. Alternatively, the sensor PS can emit an ultrasonic signal with a frequency of approximately 10 MHz in response to a drive control signal, and the image processing unit IPU can generate a relatively low-quality third image IMAGE3_2.

[0219] Subsequently, Figure 14 The method shown can perform fingerprint verification based on the second fingerprint image IMAGE2 (S1430).

[0220] When fingerprint verification is not successfully performed or when fingerprint verification fails, Figure 14 The method shown can repeatedly execute step S1410 of requesting fingerprint input to step S1430 of performing fingerprint verification.

[0221] When fingerprint verification is successfully performed, Figure 14 The method shown can store the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 (S1440), and set or update the second image conversion model by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 (S1450), and generate the second pseudo image IMAGE_P2 based on the second image conversion model (S1460).

[0222] Subsequently, Figure 14 The method shown can add a second pseudo image IMAGE_P2 to the template TEMP, or expand the template TEMP by using the second pseudo image IMAGE_P2 (S1470).

[0223] For reference Figure 14 As described, Figure 14The method shown can set or update the second image transformation model through real-time learning, generate a second pseudo image IMAGE_P2 corresponding to the registered image IMAGE_E by using the second image transformation model, and add the second pseudo image IMAGE_P2 to the template TEMP or expand the template TEMP.

[0224] Figure 15 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention. Figure 15 The paper illustrates a method for fingerprint verification based on complex learning.

[0225] refer to Figure 1A , Figure 3 , Figure 5 , Figure 11 , Figure 13 , Figure 14 and Figure 15 , Figure 15 The method shown can set up a first image conversion model by learning a fingerprint image IMAGE_L provided from an external source (e.g., a database) for learning (S1510).

[0226] Figure 15 The method shown can request fingerprint input from the user (S1520) and acquire a second fingerprint image IMAGE2 and a third fingerprint image IMAGE3 by sensing the fingerprint while changing the driving conditions of the sensor PS (S1530).

[0227] Subsequently, Figure 15 The method shown can perform fingerprint verification based on the second fingerprint image IMAGE2 (S1540).

[0228] When fingerprint verification is successfully performed, Figure 15 The method shown can store the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 (S1550), update the first image conversion model by learning the second fingerprint image IMAGE2 and the third fingerprint image IMAGE3 (S1560), and generate the third pseudo image IMAGE_P3 by using the updated first image conversion model (S1570).

[0229] Subsequently, Figure 15 The method shown can add a third pseudo-image IMAGE_P3 to the template TEMP, or expand the template TEMP by using the third pseudo-image IMAGE_P3 (S1580).

[0230] For reference Figure 15 As described, Figure 15The method shown can update the first image transformation model through complex learning, generate a third pseudo-image IMAGE_P3 corresponding to the registered image IMAGE_E by using the updated first image transformation model, and add the third pseudo-image IMAGE_P3 to the template TEMP or expand the template TEMP.

[0231] Figure 16 This is a flowchart illustrating another exemplary embodiment of a fingerprint verification method according to the principles of the present invention.

[0232] refer to Figure 1A , Figure 3 , Figure 5 , Figure 11 , Figure 13 , Figure 14 , Figure 15 and Figure 16 , Figure 16 The method shown can be derived from Figure 1A The display device DD shown is Figure 3 The fingerprint verification device shown is executed by FDD. Additionally, it can be executed... Figure 11 The method shown will then be executed. Figure 16 The method shown. For example, Figure 16 The method shown can be formed after various learning settings of the image transformation model.

[0233] Figure 16 The method shown can sense fingerprints (S1610). See reference... Figure 14 As described, Figure 16 The method shown can request fingerprint input from a user and sense the fingerprint corresponding to the fingerprint input.

[0234] Figure 16 The method shown can perform image processing on the fingerprint image generated by the fingerprint sensing operation (S1620).

[0235] For example, as referenced Figure 5 As described, Figure 16 The method shown can perform image processing on the fingerprint image through the image processing unit IPU, including smoothing, binarization, thinning and other operations.

[0236] Subsequently, Figure 16 The method shown can perform primary fingerprint verification (S1630) based on a fingerprint image (e.g., a second fingerprint image IMAGE2). For example, Figure 16The method shown can perform primary fingerprint verification by comparing a fingerprint image with a template TEMP (e.g., the registration image IMAGE_E, the first pseudo image IMAGE_P1, the second pseudo image IMAGE_P2, and the third pseudo image IMAGE_P3).

[0237] When the initial fingerprint verification is successfully performed, Figure 16 The method shown can output a successful verification result (S1640) to an external source (e.g., a host processor) indicating that fingerprint verification has been successfully performed.

[0238] When primary fingerprint verification is not successfully performed (or when primary fingerprint verification fails). Figure 16 The method shown can perform image processing on the fingerprint image (S1650).

[0239] In an exemplary embodiment, Figure 16 The method shown can be reversed by using a reference. Figure 9 The described image conversion model is used to convert a fingerprint image (e.g., a second fingerprint image IMAGE2) into a high-quality fingerprint image. For example, Figure 16 The method shown can take into account the failure of primary fingerprint verification due to low fingerprint image quality, and generate fingerprint images with improved quality by reverse-engineering an image transformation model.

[0240] Subsequently, Figure 16 The method shown can perform secondary fingerprint verification (S1660) based on a high-quality fingerprint image. Secondary fingerprint verification can be substantially the same as primary fingerprint verification. For example, Figure 16 The method shown can perform secondary fingerprint verification by comparing a high-quality fingerprint image with a template TEMP (e.g., the registration image IMAGE_E, the first pseudo image IMAGE_P1, the second pseudo image IMAGE_P2, and the third pseudo image IMAGE_P3).

[0241] For example, besides adding a low-quality pseudo-image IMAGE_P to the template TEMP or expanding the template TEMP, it is expected that a low-quality fingerprint image will be generated based on skin condition and verification environment. Figure 16 The method shown can improve the quality of fingerprint images by performing image processing on low-quality fingerprint images. Additionally, Figure 16 The method shown can also perform fingerprint verification based on an extended template TEMP and a fingerprint image with improved quality. Therefore, fingerprint verification can be performed more successfully.

[0242] Similar to when primary fingerprint verification is successfully performed, when secondary fingerprint verification is successfully performed, Figure 16The method shown can output a successful verification result (S1640) to an external source (e.g., a host processor) indicating that fingerprint verification has been successfully performed.

[0243] When secondary fingerprint verification is not successfully performed or when secondary fingerprint verification fails, Figure 16 The method shown can output a verification failure result indicating that fingerprint verification has failed to an external source (e.g., a host processor) (S1670).

[0244] For example, although in Figure 16 The example already illustrates the execution of image processing to improve the quality of the fingerprint image (S1650), but the exemplary implementation is not limited thereto. For example, the image processing to improve the quality of the fingerprint image may include step S1620. In step S1620, each of the fingerprint image and the high-quality fingerprint image may be generated. In step S1630, which performs primary fingerprint verification, fingerprint verification may be performed based on the fingerprint image. When primary fingerprint verification fails, in step S1660, which performs secondary fingerprint verification, fingerprint verification may be performed again based on the high-quality fingerprint image. For example, the image processing to improve the quality of the fingerprint image may be performed unrestricted between step S1610, which senses the fingerprint, and step S1660, which performs secondary fingerprint verification.

[0245] In the fingerprint verification device, display device including the fingerprint verification device, and fingerprint verification method according to this disclosure, a low-quality pseudo-image is generated from a registered fingerprint image by learning various fingerprint images. In addition to adding the registered fingerprint image to the template, the low-quality pseudo-image is also added to the template, thus expanding the template. By performing fingerprint verification using the expanded template, the accuracy of fingerprint verification can be improved even when the user's skin condition and verification environment change.

[0246] In particular, in the fingerprint verification device, the display device including the fingerprint verification device, and the method for verifying fingerprints according to the exemplary embodiments shown, a fake image can be generated by processing a registered fingerprint image instead of a low-quality fingerprint image that has been verified or acquired in the past. Therefore, the accuracy of fingerprint verification can be improved relative to a wider range of user skin conditions and verification environments.

[0247] While certain exemplary embodiments and implementations have been described herein, other embodiments and modifications will become apparent from the description. Therefore, the inventive concept is not limited to such embodiments, but rather to the broader scope of the appended claims and various obvious modifications and equivalent arrangements that will be apparent to those skilled in the art.

Claims

1. A fingerprint verification device, comprising: The sensor unit is configured to output a sensing signal by sensing a fingerprint; An image processing unit is configured to generate a fingerprint image based on the sensing signal; A storage unit configured to store a template including a registered image; The learning unit is configured to generate a first pseudo-image and add the first pseudo-image to the template. as well as The verification unit is configured to compare the fingerprint image generated by the image processing unit with the template, and is configured to output the comparison result. The learning unit is configured to convert the registered image into the first pseudo-image using a first image conversion model, wherein the signal-to-noise ratio of the first pseudo-image is lower than that of the registered image. The image processing unit is configured to generate a first fingerprint image during the fingerprint registration period and a second fingerprint image during the fingerprint verification period. The registration image includes at least a portion of the first fingerprint image, and the first pseudo-image differs from the second fingerprint image. The image processing unit is configured to generate a third fingerprint image corresponding to the second fingerprint image but not used for fingerprint verification during the fingerprint verification period, while changing the driving conditions of the sensor unit. The second fingerprint image and the third fingerprint image are obtained from the same fingerprint. The signal-to-noise ratio (SNR) of the third fingerprint image is lower than that of the second fingerprint image. The learning unit is configured to set the first image conversion model by learning a fingerprint image provided from outside the fingerprint verification device, or the learning unit is configured to set or update the first image conversion model by learning at least one of the first fingerprint image, the second fingerprint image, and the third fingerprint image. The fingerprint images used for learning include: high-quality fingerprint images with a signal-to-noise ratio (SNR) within a reference quality range relative to multiple objects, and low-quality fingerprint images with an SNR outside the reference quality range. The storage unit is configured to store the second fingerprint image and the third fingerprint image based on the comparison result of the verification unit. The learning unit is configured to set a second image conversion model by learning the second fingerprint image and the third fingerprint image, to convert the first fingerprint image into a second pseudo image by using the second image conversion model, and to add the second pseudo image to the template.

2. The fingerprint verification device according to claim 1, wherein, The sensor unit includes an optical sensor configured to sense light reflected by the fingerprint. The image processing unit is configured to generate the second fingerprint image by receiving reflected light via the sensor unit for a first exposure time, and The image processing unit is configured to generate the third fingerprint image by receiving the reflected light via the sensor unit for a second exposure time.

3. The fingerprint verification device according to claim 2, wherein, The second exposure time is shorter than the first exposure time.

4. The fingerprint verification device according to claim 1, wherein, The sensor unit includes an ultrasonic sensor configured to sense ultrasonic waves reflected by the fingerprint. The image processing unit is configured to generate the second fingerprint image by emitting ultrasonic waves of a first frequency via the sensor unit, and The image processing unit is configured to generate the third fingerprint image by emitting an ultrasonic wave with a second frequency via the sensor unit.

5. The fingerprint verification device according to claim 4, wherein, The second frequency is lower than the first frequency.

6. The fingerprint verification device according to claim 1, wherein, The learning unit is configured to update the first image conversion model by further learning the second fingerprint image and the third fingerprint image, to convert the first fingerprint image into a third pseudo image by using the updated first image conversion model, and to add the third pseudo image to the template.

Images