Display device with corrected sensor calibration image data and control method thereof

Abstract

Disclosed are a display device and a method for controlling the same. The display device includes a display portion including a plurality of pixels and displaying an image by light emitted from the plurality of pixels, a sensor including a plurality of light sensing pixels that receive light and acquire a first image when a test image including a marker is displayed on the display portion, a memory storing calibration image data for calibrating an image acquired by the sensor, and a sensor controller calculating an amount of movement of the marker in the first image and correcting or updating the calibration image data using at least one of the amount of movement of the marker.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to and the benefit of Korean Patent Application No. 10-2018-0133056, filed on November 1, 2018, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The disclosure relates to a display apparatus and a method for controlling the same, and more particularly, to a display apparatus capable of sensing a fingerprint. BACKGROUND

[0004] Recently, various terminals and wearable apparatuses have been provided with functions using personal information such as finance, security, etc. or for these functions, and thus, the importance of security authentication has increased. For example, at least one security mechanism can be installed in personal portable electronic apparatuses such as digital cameras, tablet PCs, notebook computers, etc. to protect personal information of a user.

[0005] User authentication with respect to an electronic apparatus or system can be performed through a single or multiple formats of biometric identifiers (IDs), and the authentication can be used alone or by adding an existing password authentication method. For example, an authentication method using biometrics can authenticate a user by using a fingerprint, an iris, a voice, a face, a blood vessel, etc. The biometric features used in the verification differ from person to person, and can be convenient, can have a limited risk of being stolen or imitated, and / or can not change frequently throughout a person's life.

[0006] In particular, due to various reasons such as convenience, security, and economy, biometric authentication is becoming more commercialized. A biometric sensor can be embedded in an electronic apparatus or system to acquire biometric data (e.g., a fingerprint image of a finger) of a user. Specifically, the biometric sensor can acquire a fingerprint image by directly or indirectly contacting a finger, and is capable of acquiring a fingerprint pattern from the fingerprint image.

[0007] The above information disclosed in this Background section is only for enhancing the understanding of the disclosure, and therefore it can include information that is not prior art to the disclosure. SUMMARY

[0008] Aspects of embodiments of the disclosure relate to a display apparatus performing biometric authentication by using light from a display portion and a control method thereof.

[0009] Aspects of embodiments of the disclosure relate to a display apparatus capable of acquiring a still fingerprint image even when a fingerprint moves with respect to a sensor and a control method thereof.

[0010] According to embodiments of the disclosure, a display device is provided, the display device including a display portion, a sensor, a memory, and a sensor controller, the display portion including a plurality of pixels configured to display an image by emitting light from the plurality of pixels; the sensor including a plurality of light sensing pixels configured to receive light and acquire a first image when a test image including a marker is displayed by the display portion; the memory configured to store calibration image data for a calibration image; and the sensor controller configured to calculate an amount of movement of the marker in the first image and update the calibration image data using the amount of movement of the marker.

[0011] The calibration image data can include a plurality of image pixels each having a gray value, and the sensor controller can generate a weight value map by calculating the amount of movement of the marker, and can apply the weight value map to the plurality of image pixels in the calibration image data to correct the calibration image data.

[0012] The memory can be configured to store a first image profile acquired when the calibration image data is generated by displaying the test image by the display portion, and the sensor controller can be configured to generate a second image profile from the first image, and calculate the amount of movement of at least one marker by comparing the first image profile with the second image profile.

[0013] The marker can have an X shape including a center point having a black gray value and an X-shaped portion having a white gray value.

[0014] The sensor controller can be configured to calculate the amount of movement of the marker by comparing a position corresponding to a minimum gray value of the marker in the first image profile and a position corresponding to a minimum gray value of the marker in the second image profile.

[0015] The sensor controller can be configured to calculate the amount of movement of the marker by comparing a minimum gray value of the marker in the first image profile and a minimum gray value of the marker in the second image profile.

[0016] The sensor controller can be configured to apply the updated calibration image data to a fingerprint image acquired by the sensor.

[0017] The sensor can be located at a rear side of the display portion.

[0018] The display portion can include a display area displaying an image and a non-display area located at at least one side of the display area, and the sensor can correspond to a fingerprint sensing area in the display area.

[0019] The display device can further include a touch sensor located at a front side of the display portion and a window located at a front side of the touch sensor.

[0020] When the touch sensor senses a touch at a touch area in the fingerprint sensing area, the display portion can be configured to control the pixels at the touch area to emit light, and the sensor can be configured to drive the light sensing pixels at the touch area.

[0021] According to another embodiment of the disclosure, a method of controlling a display device is provided, the method including: acquiring a first image through a sensor including a plurality of light sensing pixels when a display portion of the display device displays a test image including a marker; calculating a movement amount of the marker from the first image; and updating calibration image data based on the movement amount of the marker, wherein the calibration image data is to be applied to a fingerprint image acquired by the sensor.

[0022] The calibration image data includes a plurality of image pixels each having a gray value, and updating the calibration image data can include generating a weight value map by calculating the movement amount of the marker, and applying the weight value map to the plurality of image pixels in the calibration image data.

[0023] Generating the weight value map by calculating the movement amount of the marker can include generating a second image contour from the first image, and calculating the movement amount of the marker by comparing the second image contour with a first image contour acquired when the test image is displayed by the display portion to generate the calibration image data.

[0024] The marker can have an X shape including a center point having a black gray value and an X-shaped portion having a white gray value.

[0025] Calculating the movement amount of the marker can include comparing a position corresponding to a minimum gray value of the marker in the first image contour and a position corresponding to the minimum gray value of the marker in the second image contour.

[0026] Calculating the movement amount of the marker can include comparing a minimum gray value of the marker in the first image contour and a minimum gray value of the marker in the second image contour.

[0027] The method of controlling the display device can further include applying the updated calibration image data to the fingerprint image acquired by the sensor.

[0028] The display portion can include a display area displaying an image and a non-display area located at at least one side of the display area, and the sensor can correspond to a fingerprint sensing area in the display area.

[0029] Acquiring the first image can include sensing a touch at a touch area in the fingerprint sensing area by a touch sensor; emitting light from the pixels positioned in the touch area by the display portion; and driving the light sensing pixels positioned in the touch area by the sensor.

[0030] According to an exemplary embodiment, a fingerprint image from which noise is removed can be acquired.

[0031] According to an exemplary embodiment, a fingerprint recognition rate can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a plan view of a portion of a display device according to an embodiment of the present disclosure.

[0033] Figure 2 is a cross-sectional view of Figure 1 taken along line I-I'.

[0034] Figure 3 is a block diagram of a display device according to an embodiment of the present disclosure.

[0035] Figure 4 shows an example of a fingerprint image acquired by a fingerprint sensor when the fingerprint sensor moves.

[0036] Figure 5 is a flowchart of a method for storing calibration image data of a display device according to an embodiment of the present disclosure.

[0037] Figure 6 shows an outline of a test image displayed by a display portion and an image acquired by a fingerprint sensor according to an embodiment of the present disclosure.

[0038] Figure 7 is a flowchart of a method of acquiring a fingerprint using a display device according to an embodiment of the present disclosure.

[0039] Figures 8 to 12 shows an outline of an image acquired by a fingerprint sensor when the fingerprint sensor has moved in an x-axis direction according to an embodiment of the present disclosure.

[0040] Figure 13 is a graph showing a change in a maximum gray value, a minimum gray value, and an intermediate gray value of an outline of an image (e.g., Figures 9 to 12 ) acquired by a fingerprint sensor when the fingerprint sensor has moved in an x-axis direction.

[0041] Figure 14 shows an image acquired by a fingerprint sensor ("first image"), an x-axis weight value map, and a y-axis weight value map according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0042] In the following description, reference is made to the accompanying drawings which form a part hereof, and in which are shown, by way of illustration, example implementations. The drawings are not intended to be an exhaustive description of implementations of the disclosure as described herein. Implementations of the disclosure can be implemented in various forms and are not limited to the implementations set forth in the description below. Rather, these example implementations are provided as illustrative so that a thorough and complete comprehension of the disclosure will be achieved and will be conveyed by persons skilled in the art. Accordingly, processes, elements, and techniques that are not necessary for an understanding of the aspects and features of the disclosure are not described. Unless otherwise indicated, like reference numerals in the drawings and the written description denote like elements throughout the several views. In the drawings, the relative sizes of elements, layers, and regions can be exaggerated for clarity.

[0043] It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element or intervening elements can also be present. In contrast, when an element is referred to as being "directly on" another element, there are no intervening elements present. In addition, it will be understood that when an element or layer is referred to as being "between" two elements or layers, it can be the only element or layer between the two elements or layers or one or more intervening elements or layers can also be present. Spatially relative terms such as "beneath", "below", "lower", "under", "above", "upper", and the like can be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device is inverted in the figure, a spatially relative term such as "below" or "beneath" can be interpreted to mean "above" or "over" in view of the inverting. The spatially relative terms can be interpreted differently depending on the particular orientation of the device in use or operation. Accordingly, the example spatially relative terms "below", "beneath", and "under" can encompass both an orientation of above and below. The device can be otherwise oriented (e.g., rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0044] In addition, the word "comprise" and variations such as "comprises" or "comprising" will be understood to imply the inclusion of a stated element, but not the exclusion of any other elements.

[0045] Also, in the present specification, the phrase "on a plane" means to observe the target portion from the top, and the phrase "on a cross-sectional view" means to observe a cross-sectional view formed by cutting the target portion vertically from the side.

[0046] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" "comprising," "includes" and / or "including" when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of," when preceding a list of two or more items, modify the list of items as a whole and do not modify the individual items of the list in the absence of such express language.

[0047] As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art. Moreover, the use of "may" when describing embodiments of the present application is intended to mean that one or more embodiments of the present application. As used herein, the terms "use," "using," and "used" can be considered synonymous with the terms "utilize," "utilizing," and "utilized," respectively. Also, the term "exemplary" is intended to mean example or illustration.

[0048] The electronic device or electrical device and / or any other related device or component (such as the touch controller 315, display controller 325, display driver 320, sensor controller 335, and / or sensor driver 330) according to the embodiments of the present application described herein can be implemented using any suitable hardware, firmware (e.g., application specific integrated circuits), software, or combinations thereof. For example, various components of these devices can be formed on one integrated circuit (IC) chip or on separate IC chips. In addition, various components of these devices can be implemented on a flexible printed circuit film, tape carrier package (TCP), printed circuit board (PCB), or formed on one substrate. Further, various components of these devices can be processes or threads running on one or more processors in one or more computing devices, interacting with other system components for the performance of various functionality described herein. The computer program instructions are stored in memory that can be implemented using standard memory devices such as, for example, random access memory (RAM). The computer program instructions can also be stored in other non-transitory computer readable media such as, for example, CD-ROMs, flash drives, etc. Also, those skilled in the art will recognize that the functionality of various computing devices can be combined or integrated into a single computing device, or the functionality of a particular computing device can be distributed across one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the present application.

[0049] Figure 1 is a top view of a portion of a display device according to an embodiment of the present disclosure, and Figure 2 is a cross-sectional view taken along line I-I' of Figure 1 .

[0050] Referring to Figure 1 and Figure 2 , a display device 100 according to an exemplary embodiment can include a display panel 200 and a window 210 disposed on the display panel 200.

[0051] The display panel 200 can display visual information such as text, video, a photograph, a 2-dimensional image, or a 3-dimensional image, etc. through its surface. The display panel 200 displays an image, and can have various types including, but not limited to, LCD, LED, and OLED. Hereinafter, the display panel 200 will be exemplarily described as a panel including an organic light emitting diode (OLED) as a light emitting element. However, the type of the display panel 200 is not limited thereto. Those skilled in the art will recognize that there are other suitable types of display devices, and such suitable types of display devices are considered to be within the scope of the present disclosure.

[0052] The display panel 200 can have various shapes. For example, the display panel 200 can be formed in a rectangular shape having two pairs of parallel sides. For better understanding and ease of description, the display panel 200 is illustrated herein as a rectangle having one pair of long sides and one pair of short sides.

[0053] However, the shape of the display panel 200 is not limited thereto, and it can have various shapes. For example, in some embodiments, the display panel 200 can have various shapes such as a closed polygon having straight sides, a circle having curved sides, an ellipse, a semi-circle having straight sides and curved sides, a semi-ellipse, etc. At least a portion of the corners of the display panel 200 can be formed in the shape of a curve.

[0054] The display panel 200 can be flexible in whole or at least in part.

[0055] The display panel 200 can display an image. The display panel 200 can include a display portion 202, and the display portion 202 can include a display area DA displaying an image and a non-display area NDA arranged at at least one side of the display area DA. For example, the non-display area NDA can be formed in a shape surrounding the display area DA.

[0056] The display area DA can have a shape corresponding to the shape of the display panel 200. For example, similar to the shape of the display panel 200, the display area DA can include a closed polygon having straight sides, a circle or an ellipse including curved sides, a semi-circle or a semi-ellipse including straight sides and curved sides, etc. In an exemplary embodiment of the present disclosure, the display area DA is formed in a rectangular shape.

[0057] A plurality of pixels PX and a plurality of touch sensing pixels TS can be arranged in the display area DA. A display driver 320 (refer to FIG. 2) driving the plurality of pixels PX can be arranged in the non-display area NDA. Figure 3 ) can be arranged in the non-display area NDA.

[0058] A fingerprint sensing area SA in which a fingerprint is sensed can be arranged in the display area DA. The fingerprint sensing area SA can be formed in a size and shape capable of sensing a fingerprint of a user. For better understanding and ease of description, the fingerprint sensing area SA is described herein as being formed in a rectangular shape, but the present disclosure is not limited thereto, and other suitable shapes are considered to be within the scope of the present disclosure. In some embodiments, the fingerprint sensing area SA can have a circular shape, an elliptical shape, a semi-circular shape, a polygonal shape, etc.

[0059] In some embodiments, the fingerprint sensing area SA can be disposed in the display area DA. For example, the fingerprint sensing area SA can be disposed in the entire display area DA (e.g., can be disposed in the entire display area DA or can be coextensive with the display area DA), or can be disposed in a portion of the display area DA. The fingerprint sensing area SA can be disposed in various positions not limited to the display area DA. For example, a portion of the fingerprint sensing area SA can be disposed in the non-display area NDA.

[0060] In some embodiments, the transmittance of the window 210 of the fingerprint sensing area SA and at least a portion of the display panel 200 (e.g., disposed in the fingerprint sensing area SA) can be higher than the transmittance of the window 210 of the area other than the fingerprint sensing area SA and the display panel 200, to provide sensing of a fingerprint using light transmitted through the fingerprint sensing area SA. In some embodiments, as the transmittance of the fingerprint sensing area SA increases, light incident on the fingerprint sensor 220 can increase.

[0061] The window 210 can be disposed above the display panel 200. The window 210 can have a shape corresponding to the shape of the display panel 200, and can cover at least a portion of the entire surface of the display panel 200. For example, when the display panel 200 has a rectangular shape, the window 210 can also have a rectangular shape. Alternatively, when the display panel 200 has a circular shape, the window 210 can also have a circular shape.

[0062] An image displayed in the display panel 200 is transmitted to the outside through the window 210. The window 210 can prevent the display panel 200 from being broken or malfunctioning due to an external impact by absorbing the external impact. The external impact can be a force applied from the outside of the display device 100, which can be expressed as pressure, stress, etc., and can cause a defect in the display panel 200.

[0063] The window 210 can be flexible in whole or at least in part.

[0064] Specifically, the display panel 200 can include a substrate 206, a display portion 202 disposed on the substrate 206, a fingerprint sensor 220 disposed on the rear side of the substrate 206, and a touch sensor 204 disposed on the display portion 202.

[0065] The substrate 206 can be formed of, for example, various materials such as glass, polymeric metal, or the like. The substrate 206 can be an insulating substrate made of a polymeric organic material. The insulating substrate including the polymeric organic material can include polystyrene, polyvinyl alcohol, polymethyl methacrylate, polyethersulfone, polyacrylate, polyetherimide, polyethylene naphthalate, polyethylene terephthalate, polyphenylene sulfide, polyarylate, polyimide, polycarbonate, cellulose triacetate, cellulose acetate propionate, or the like. In some embodiments, the material forming the substrate 206 can be made of a glass fiber reinforced plastic (FRP).

[0066] The display portion 202 can be arranged above the substrate 206. The display portion 202 can display information input by a user or information provided to the user as an image. The display portion 202 can include a plurality of pixels PX. In some embodiments, the plurality of pixels PX can be organic light emitting elements each including an organic layer, but are not limited thereto. In some embodiments, the pixels PX can be implemented in various forms such as liquid crystal elements, electrophoretic elements, electro wetting elements, or the like. The plurality of pixels PX can be arranged in a display area DA of the substrate 206. Each pixel PX can include an organic light emitting element that emits white and / or colored light as a unit (e.g., a minimum unit) for displaying an image. In some embodiments, each pixel PX can emit light of any one of red, green, blue, and white colors, but the present disclosure is not limited thereto, and in some embodiments, each pixel PX can emit light of cyan, magenta, yellow, or the like. Each pixel PX can include a transistor connected to a plurality of signal lines and an organic light emitting diode electrically connected to the transistor.

[0067] The touch sensor 204 can be mounted on the display portion 202 in the form of a separate panel or film, or can be integrally formed with the display portion 202.

[0068] The touch sensor 204 can include a plurality of touch sensing pixels TS to sense a position of a touch when the touch is made by a user. The touch sensing pixels TS can sense a touch by using a mutual capacitance method or a self-capacitance method. The touch sensor 204 can receive a driving signal from the touch controller 315 (refer to FIG. 1), and can transmit a touch sensing signal that varies according to a touch of a user to the touch controller 315. Figure 3 ) receive a driving signal from the touch controller 315 (refer to FIG. 1), and can transmit a touch sensing signal that varies according to a touch of a user to the touch controller 315.

[0069] The fingerprint sensor 220 includes a plurality of light sensing pixels. In Figure 2 , the fingerprint sensor 220 is arranged at a rear side of the substrate 206 for convenience of illustration, but the present disclosure is not limited thereto. For example, in some embodiments, the fingerprint sensor 220 can be arranged between the substrate 206 and the display portion 202.

[0070] In addition, the fingerprint sensor 220 can be disposed in a fingerprint sensing area SA of the substrate 206. InFigure 2 In the middle, the fingerprint sensing area SA is arranged to correspond to a part of the display area DA, but as described above, the fingerprint sensing area SA can be arranged to correspond to the entire area of the display area DA. In this case, the fingerprint sensor 220 can be arranged to correspond to the entire area of the display area DA. In some embodiments, the fingerprint sensor 220 can be connected to the sensor driver 330 by, for example, a wiring, a flexible printed circuit board, a tape carrier package, a connector, or a chip on film (refer to FIG. 2). Figure 3 ).

[0071] The fingerprint sensor 220 can sense a fingerprint of a user by using light generated by the display portion 202. Specifically, light output by the display portion 202 can be reflected by a finger of a user in direct contact with the display device 100 (e.g., a surface of the direct contact window 210) or in the vicinity of the display device 100, and thus be incident on the display portion 202. The light incident on the display portion 202 by reflection by the finger of the user can have information of at least a part of the finger (e.g., information about a part of the finger of the user can be identified from the incident light). The information can include, for example, a spatial pattern of ridge and valley portions of the light-receiving part of the finger. Accordingly, the fingerprint sensor 220 can be formed to capture at least a part of the reflected light to detect a spatial pattern and a position of the ridge and valley portions of the light-receiving part of the finger, for example, by an optical imaging and / or light detection operation. The detected spatial pattern and position of the ridge and valley portions of the light-receiving part of the finger can be processed to form a fingerprint pattern and perform fingerprint recognition. For example, the fingerprint sensor 220 can determine whether the detected fingerprint matches a pre-stored, verified user fingerprint pattern.

[0072] The fingerprint sensor 220 can be an optical sensor or an image sensor. When the fingerprint sensor 220 is provided as an optical sensor, the fingerprint sensor 220 can identify a fingerprint of a user by distinguishing a difference in reflected light depending on whether a ridge portion or a valley portion of the fingerprint of the user reflects light on the display panel 200 (e.g., in contact with the display device 100 or the window 210). When the fingerprint sensor 220 is provided as an image sensor, an image of the fingerprint of the user reflecting light on the display panel 200 (e.g., in contact with the display device 100 or the window 200) is captured on the fingerprint sensor 220, and the captured image is compared with a reference fingerprint pattern to identify the fingerprint of the user. The image sensor can include an image-capturing constituent element such as an active pixel sensor (e.g., a CMOS sensor), a random digital image-capturing device (e.g., a CCD device), a random image-capturing device (e.g., a light-sensitive film camera), or the like.

[0073] The fingerprint sensor 220 can be able to identify a fingerprint when a fingerprint of a user touches and when a finger of a user moves during a touch state.

[0074] Figure 3 is a block diagram of a display device according to an embodiment of the present disclosure. Referring to Figure 3 , the display portion 202, the touch sensor 204, and the fingerprint sensor 220 are connected with a display driver 320, a touch controller 315, and a sensor driver 330, respectively. In addition, the display driver 320 and the sensor driver 330 are connected with a display controller 325 and a sensor controller 335, respectively.

[0075] The display driver 320 can include a gate driver and a data driver that supply signals to the pixels PX included in the display portion 202. The display controller 325 can control an image display operation of the display portion 202 by supplying a driving signal to the display driver 320.

[0076] For example, the gate driver can generate a gate signal based on a driving signal received from the display controller 325, and output the gate signal to a gate line connected with the pixel PX. The gate driver can be formed at the same time as the pixel PX through a thin film process. For example, the gate driver can be implemented in the form of an amorphous silicon thin film transistor (TFT) gate driver circuit (ASG) or an oxide semiconductor TFT gate driver circuit (OSG) in the non-display area NDA. The data driver can generate a gray scale voltage according to image data supplied from the display controller 325 based on a driving signal received from the display controller 325. The data driver can output the gray scale voltage as a data voltage to a data line connected with the pixel PX.

[0077] The display controller 325 can generate a driving signal using an image signal, a data enable signal, and a clock signal provided by an external image source. For example, the display controller 325 can receive an image signal and a control signal from the external image source, and the control signal can include a vertical synchronization signal that distinguishes frame portions, a horizontal synchronization signal that distinguishes lines in one frame, a data enable signal that is at a high level only at a period during which data is output, and a clock signal. In addition, the driving signal can include a gate / data driver driving signal.

[0078] The touch controller 315 can generate a driving signal output to the touch sensor 204, and can receive a sensing signal input from the touch sensor 204. The touch controller 315 can determine whether a touch screen is touched, a number of touch inputs, a touch input position, etc., by using the driving signal and the sensing signal.

[0079] The sensor driver 330 can include a scan driver that applies a scan signal to a light sensing pixel included in the fingerprint sensor 220, and a lead-out driver that receives a signal output from the light sensing pixel.

[0080] The sensor controller 335 may include at least one processor that controls the operation of the sensor driver 330, processes a fingerprint pattern based on a signal (e.g., an electrical signal) transmitted from the sensor driver 330, and determines whether the input fingerprint pattern is a fingerprint pattern of an authorized user. The fingerprint sensor 220 is used to obtain a fingerprint and compare the obtained fingerprint with a stored fingerprint to enable or disable functions of the display device 100 or a system including the display device 100.

[0081] Additionally, the sensor controller 335 may output a control signal to the display controller 325, causing the display portion 202 to illuminate to capture a fingerprint pattern when the sensor controller 335 performs fingerprint sensing operations. Here, pixels PX corresponding to the area where the user's finger is placed on the display portion 202 may emit light. For example, pixels PX located in a position corresponding to the fingerprint sensing area SA may emit light. Alternatively, when a position touched by a user's finger is sensed, pixels PX corresponding to the sensed position may emit light to sense the fingerprint. When the fingerprint sensor 220 is activated by a touch relative to the touch sensor 204, pixels PX located in the touched position may emit light with high brightness, thereby improving fingerprint sensing contrast.

[0082] The fingerprint image acquired by the fingerprint sensor 220 may include noise generated by image interference from the internal structure of the display panel 200 (such as electrode patterns, wiring patterns, etc.). The sensor controller 335 can correct the acquired fingerprint image by using a pre-stored calibration image. The calibration image data can be stored in the memory 340 as data for removing noise caused by image interference, etc. The calibration image may include a plurality of image pixels, and each image pixel may include a grayscale value.

[0083] That is, when acquiring a fingerprint image, the sensor controller 335 can correct the acquired fingerprint image by using the calibration image data stored in the memory 340. However, when the position of the fingerprint sensor 220 is changed in the display device 100, when the calibration image data stored in the memory 340 is used, the noise of the fingerprint image cannot be easily removed. Figure 4 Describe this.

[0084] Figure 4 An example of a fingerprint image acquired by the fingerprint sensor when the fingerprint sensor is moved is shown.

[0085] like Figure 4 As shown in FIG. 4( a ), an image 401 is acquired at an initial position of the fingerprint sensor 220 between reference lines RL1 and RL2 by the fingerprint sensor 220. When a calibration image 403 is applied to the image 401, an image 405 from which noise is removed is acquired.

[0086] As Figure 4 As shown in (b) of FIG. 4, when the fingerprint sensor 220 is moved to the outside of the reference line RL2 in the x-axis direction, an image 402 is acquired by the fingerprint sensor 220. When the calibration image 403 is applied to the image 402, an image 406 including noise is acquired. Since the position of the fingerprint sensor 220 at the time of generating the calibration image 403 and the position of the fingerprint sensor 220 at the time of acquiring the image 402 are different from each other, the value of the calibration image 403 pre-stored in the memory 340 cannot remove the noise of the image 402 or can not be as effective in removing the noise. Similarly, when the fingerprint sensor 220 is moved at the time of acquiring a fingerprint, the noise in the acquired fingerprint image cannot be removed (or can be more difficult to remove using the calibration image 403), and accordingly, the fingerprint recognition rate can be reduced.

[0087] The sensor controller 335 can perform a process for correcting the calibration image data. The sensor controller 335 can change or update (e.g., correct) the calibration image data periodically or by selection of a user (e.g., the user can trigger the sensor controller 335 to correct the calibration data). Even when sensing is performed due to an impact applied to the display apparatus 100 by other sensors (e.g., a gyro sensor, a temperature sensor, a barometric pressure sensor, a humidity sensor, etc.) included in the display apparatus 100, the sensor controller 335 can correct the calibration image data. When the other sensors sense an impact applied to the display apparatus 100, the sensor controller 335 can correct the calibration image data.

[0088] Specifically, the sensor controller 335 can correct the calibration image data by controlling the display portion 202 to display a test image and controlling the fingerprint sensor 220 to acquire the displayed test image. The sensor controller 335 can correct the calibration image data by calculating the amount of movement of the fingerprint sensor 220 using the test image acquired by the fingerprint sensor 220.

[0089] The memory 340 can store image profile data according to the test image data, the calibration image data, and the test image displayed at the time of generating the calibration image data.

[0090] Two or more of the touch controller 315, the display controller 325, and the sensor controller 335 can be integrated into one configuration. For example, the touch controller 315, the display controller 325, and the sensor controller 335 can be implemented as a single integrated circuit (IC).

[0091] Next, a method for storing calibration image data will be described with reference to Figure 5

[0092] Figure 5 ​is a flowchart of a method for storing calibration image data of a display device according to an embodiment of the present disclosure.

[0093] After the fingerprint sensor 220 is combined with the display panel 200 (e.g., attached to or fixed relative to it), calibration image data is generated (S100). The sensor controller 335 can generate the calibration image data by using image data acquired by the fingerprint sensor 220 in a specific environment (e.g., in a dark room with a specific reflector constantly irradiating light while being close to the fingerprint sensor 220). The calibration image data may include (e.g., may represent or reflect) processing deviations of the light-sensing pixels included in the fingerprint sensor 220 and noise caused by light path obstructions caused by the display panel 200.

[0094] Next, the display unit 202 displays a test image (S110). For example, the sensor controller 335 may control the display controller 325 to output a control signal and test image data so that the display unit 202 may display the test image. The test image may include at least one mark.

[0095] Next, the fingerprint sensor 220 acquires an image while displaying the test image (S120). The image acquired by the fingerprint sensor during the display of the test image may be referred to as a "first image" hereinafter.

[0096] The sensor controller 335 calculates an image contour from the first image, and stores the initially generated calibration image data in the memory 340 in association with the calculated image contour ( S130 ).

[0097] Now refer to Figure 6 The above steps S110 , S120 and S130 are described.

[0098] Figure 6 The test image displayed by the display unit and the outline of the image acquired by the fingerprint sensor are shown.

[0099] like Figure 6 As shown in (a) of FIG. , a test image 601 may include at least one mark 603. In some embodiments, the mark 603 may have a center point CP and may be in the shape of a cross or an X that intersects at the center point CP. The image may be a grayscale image, the cross or X-shaped portion may have a white grayscale value, and the center point CP may have a black grayscale value. The white grayscale value may be the highest grayscale value (e.g., 65535), and the black grayscale value may be the lowest grayscale value (e.g., zero).

[0100] Other suitable shapes that can be used to calculate the amount of movement of the marks on the image acquired by the fingerprint sensor 220 will be apparent to those skilled in the art, can be utilized in some embodiments, and are considered to be within the scope of the present disclosure.

[0101] Figure 6 (b) shows an image profile (e.g., an x-direction image profile) acquired along a line V-V' of the mark 603 in the image acquired by the fingerprint sensor 220. In the image profile, it can be determined that a portion N corresponding to the center point CP of the mark 603 has a gray value of 0, and portions N-10 to N and N to N+10 corresponding to the horizontal bars of the cross-shaped portion have a gray value of 65535. The sensor controller 335 can store the image profile data (e.g., generated based on the display) according to the display of the test image 601 when generating the calibration data by calculating the x-axis coordinate, the y-axis coordinate, and the gray value of the center point CP of one or more marks (e.g., each mark 603).

[0102] Next, with reference to Figure 7 a method for acquiring a fingerprint that removes noise even after moving the fingerprint sensor 220 will be described.

[0103] Figure 7 is a flowchart of a method for acquiring a fingerprint using a display device according to an embodiment of the present disclosure.

[0104] As previously described, the sensor controller 335 can perform a series of processes to change or update (e.g., correct) the calibration image data. The sensor controller 335 can correct the calibration image data periodically or upon receiving a command (e.g., from a user).

[0105] The display portion 202 displays a test image (S200). For example, the sensor controller 335 can output a control signal and a test image signal to the display controller 325 so that the display portion 202 can display the test image.

[0106] The test image can be displayed so that it is not visible to a user, or is difficult to view or perceive for a user. For example, the sensor controller 335 can cause the test image to be displayed when the display device 100 is turned on, or can cause the test image to be displayed when the display device 100 is turned off. In some embodiments, the display device 100 is included in a mobile terminal (e.g., a smartphone), and calibration of the calibration image data can be performed by displaying the test image when entering a lock mode or a release lock mode.

[0107] Next, the fingerprint sensor 220 acquires an image during a period in which the test image is displayed (S210).

[0108] The sensor controller 335 calculates an image profile from the first image (S220).

[0109] The sensor controller 335 compares the image profile stored in the memory 340 with the image profile calculated in step S220, and calculates the movement amount of at least one marker (S230).

[0110] Reference will be made to Figures 8 to 12 Steps S220 and S230 will be described.

[0111] Figures 8 to 12 A profile of an image acquired by the fingerprint sensor 220 when the fingerprint sensor 220 has been moved in the x-axis direction (e.g., due to generation of calibration image data) is shown in accordance with an embodiment of the present disclosure.

[0112] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction. Figure 8 Figure 8 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction.

[0113] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction. Figure 8 Figure 8 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction.

[0114] When the image profile 811 is compared with the image profile 813, it can be determined that the x-axis coordinate having the lowest gray value (0) moves from N to N-1 when the movement amount of the corresponding marker is -1.

[0115] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction. Figure 9 Figure 9 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction.

[0116] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction. Figure 9 Figure 9 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction.

[0117] When the image profile 911 is compared with the image profile 913, it can be determined that the x-axis coordinate having the lowest gray value (0) moves from N to N-1 when the movement amount of the corresponding marker is -2.

[0118] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generation of calibration image data (801), an image profile 811 of (b) is calculated in the x-axis direction. Figure 8 and​​​​Figure 9 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0119] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 10 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 10 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0120] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 10 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 10 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0121] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0122] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 11 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 11 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0123] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 11 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 11 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0124] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0125] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 12 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 12 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0126] As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 12 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction. Figure 12 As shown in (a) of FIG. 8, when the center point CP of the marker is positioned at (N, N) at the time of generating the calibration image data (801), the image profile 811 of (b) is calculated along the x-axis direction.

[0127] When image profile 1211 and image profile 1213 are compared, it can be determined that the minimum gray value increases from 0 to 6554 when the corresponding marker's movement amount is -0.9.

[0128] As shown in Figure 10 , Figure 11 and Figure 12 , the marker's movement amount can be determined (e.g., calculated) by using the change in the minimum gray value of the image profiles 1013, 1113, and 1213. Although image profile 1013 and image profile 1213 have the same minimum gray value, the marker's movement amount of each of the two image profiles 1013 and 1213 can be distinguished based on the position (e.g., x-axis coordinate) where the minimum gray value is located.

[0129] The marker's movement amount and the maximum gray value, the minimum gray value, and the intermediate gray value of the profile of the test image (e.g., the profile of the test image shown in Figures 9 to 12 ) associated with a given movement amount can be as shown in Figure 13 .

[0130] Figures 9 to 12 is a graph showing the change in the maximum gray value, the minimum gray value, and the intermediate gray value of the profile of the image (e.g., the image of Figure 13 ) acquired by the fingerprint sensor 220 when the fingerprint sensor 220 has moved along the x-axis.

[0131] As shown in Figure 7 , the change in the minimum gray value and / or the intermediate gray value of the profile of the image corresponding to the marker can be used (e.g., partially used) to determine the marker's movement amount.

[0132] Referring back to Figure 14 , the sensor controller 335 can generate a weight value map using the marker's movement amount (S240). The weight value map can be generated using the position of at least one marker and the marker's movement amount. In particular, the weight value map can be generated when the fingerprint sensor 220 moves in the z-axis direction.

[0133] The weight value map will be described with reference to Figure 14 .

[0134] Figure 14 shows an image acquired by the fingerprint sensor 220, an x-axis weight value map, and a y-axis weight value map according to an embodiment of the disclosure.

[0135] As shown in (a) of Figure 14 , when the fingerprint sensor 220 (e.g., a portion of the fingerprint sensor 220) is spaced apart from the display panel 200 in the z-axis direction, the fingerprint sensor 220 acquires Figure 14the first image 1401 (e.g., an image of the test image displayed by the display portion 202) of (b). Although the fingerprint sensor 220 is attached to the display panel 200 near the reference line RL1, there is a space between the fingerprint sensor 220 and the display panel 200 at the reference line RL2. Thus, the amount of movement of the marks acquired by the fingerprint sensor 220 near the reference line RL1 is greater than the amount of movement of the marks acquired by the fingerprint sensor 220 near the reference line RL2. When the amount of movement of each mark on the first image 1401 is calculated at step S230, the value of the amount of movement at the position between the corresponding marks can be calculated by interpolation. As ​ The x-axis weight value map 1403 including the x-axis amount of movement at the position between the corresponding marks can be generated (e.g., by interpolation) using the x-axis amount of movement M x00 to M x44 of each mark, and the y-axis weight value map 1405 including the y-axis amount of movement at the position between the corresponding marks can be generated (e.g., by interpolation) using the y-axis amount of movement M y00 to M y44 of each mark.

[0136] The sensor controller 335 can correct the calibration image data by using the weight value maps (e.g., the new calibration image data can be changed, updated, or generated to reflect the movement of the fingerprint sensor 220) (S242). For example, the sensor controller 335 can correct the calibration image data by applying the weight value maps to each image pixel of the calibration image data and the gray scale value corresponding to each image pixel. In some embodiments, the gray scale value corresponding to the image pixel is moved to correspond to a different image pixel based on the weight in the weight value map at the corresponding position in the weight value map. When the fingerprint sensor 220 moves only in the xy plane, the sensor controller 335 can correct the calibration image data by using the amount of movement of the marks calculated at S230 without using the weight value maps. Accordingly, in some embodiments, when the corresponding marks have the same amount of movement, the sensor controller 335 can determine that the fingerprint sensor 220 moves only in the xy plane, can correct the calibration image data by using the amount of movement, and in some embodiments can forgo generating the weight value maps.

[0137] The sensor controller 335 can acquire a fingerprint image when the user’s finger approaches (S250).

[0138] The sensor controller 335 can correct the acquired fingerprint image by applying the calibration image data (e.g., the updated calibration image data or the latest calibration image data) to the acquired fingerprint image (S260).

[0139] Table 1:

[0140]

[0141] Table 1 is a table showing a standard deviation value of a fingerprint image acquired by using corrected calibration data (e.g., a standard deviation value between a test image and an image acquired by the fingerprint sensor 220) according to an embodiment of the present disclosure. The table includes five entries, and for each entry, a value of an amount of movement of the fingerprint sensor 220 (original (shifted)), a correction amount applied to the calibration data (CAL data (shifted)), a maximum gray value (MAX), a minimum gray value (MIN), a middle gray value (AVG), and a standard deviation (STDEV) are listed.

[0142] As shown in Table 1, when the fingerprint sensor 220 does not move and no correction is performed on the calibration image data, the standard deviation value is 0. However, when the fingerprint sensor 220 moves and thus the mark moves up to 0.3 pixels, the standard deviation value increases to 582.01622. In addition, when the fingerprint sensor 220 moves and thus the mark moves up to 1.5 pixels, the standard deviation value further increases to 1172.823. That is, although the fingerprint image is corrected by using the calibration data (e.g., uncorrected or un-updated calibration data), after the fingerprint sensor 220 moves, the noise of the fingerprint image is not removed or not completely removed.

[0143] According to an embodiment of the present disclosure, when the calibration image is updated or corrected corresponding to the amount of movement of the mark, it can be determined that, when the fingerprint sensor 220 moves, the standard deviation STDEV can be 0.2862287 and thus the mark moves up to 0.3 pixels, and, when the fingerprint sensor 220 moves, the standard deviation STDEV can be 0.2499993 and thus the mark moves up to 1.5 pixels. That is, even when the fingerprint sensor 220 moves, the fingerprint image can be corrected by using the corrected calibration image and thus the noise of the fingerprint image can be removed.

[0144] According to an embodiment of the present disclosure, even when the fingerprint sensor 220 moves, a fingerprint image in which the noise is reduced or removed can be acquired.

[0145] In addition, according to an embodiment of the present disclosure, even when the fingerprint sensor 220 moves after the product is shipped, calibration image data is not generated again, and a fingerprint image in which the noise is reduced or removed can be acquired.

[0146] Furthermore, according to an embodiment of the present disclosure, even when the fingerprint sensor 220 moves, compared to the related art, a fingerprint recognition rate can be improved.

[0147] While the application has been described in connection with specific embodiments thereof, it will be understood that the described embodiments are illustrative of the application and not intended to be exhaustive or limiting of the application. Embodiments of the application will provide advantageous results, improvements, and / or improvements over the state of the art, including over methods, systems, and / or compositions described in the background section. Accordingly, it is intended to embrace all alternatives, modifications and variations of the application that have been discussed hereinabove generally or specifically, and also those that can be apparent to those who have the benefit of this disclosure.

Claims

1. A display device comprising: a display portion including a plurality of pixels, and configured to display an image by light emitted from the plurality of pixels; a sensor including a plurality of light sensing pixels configured to receive light and acquire a first image when the display portion displays a test image including a mark; a memory configured to store calibration image data used to calibrate a fingerprint image acquired by the sensor; and a sensor controller configured to calculate an amount of movement of the mark in the first image, correct the calibration image data using the amount of movement of the mark, and apply the corrected calibration image data to the fingerprint image acquired by the sensor, wherein the mark movement is due to the sensor movement.

2. The display device of claim 1, wherein, the calibration image data includes a plurality of image pixels each having a grayscale value, and the sensor controller is configured to generate a weight value map by calculating the amount of movement of the mark, and apply the weight value map to the plurality of image pixels in the calibration image data to correct the calibration image data.

3. The display device of claim 1, wherein, the memory is configured to store a first image profile acquired when the calibration image data is generated by displaying the test image by the display portion; and the sensor controller is configured to generate a second image profile from the first image, and calculate the amount of movement of the mark by comparing the first image profile with the second image profile.

4. The display device of claim 3, wherein, the mark has an X shape including a center point having a black grayscale value and an X-shaped portion having a white grayscale value.

5. The display device of claim 4, wherein, the sensor controller is configured to calculate the amount of movement of the mark by comparing a position corresponding to a minimum grayscale value of the mark in the first image profile and a position corresponding to a minimum grayscale value of the mark in the second image profile.

6. The display device of claim 4, wherein, the sensor controller is configured to calculate the amount of movement of the mark by comparing a minimum grayscale value of the mark in the first image profile and a minimum grayscale value of the mark in the second image profile.

7. The display device of claim 1, wherein, the sensor is located at a rear side of the display portion, and wherein the display portion includes a display area in which an image is displayed and a non-display area located at at least one side of the display area, and the sensor corresponds to a fingerprint sensing area in the display area. 8.The display device of claim 7, further comprising: a touch sensor located at a front side of the display portion; and a window located at a front side of the touch sensor, wherein, when the touch sensor senses a touch at a touch area in the fingerprint sensing area, the display portion is configured to control pixels at the touch area to emit light, and the sensor is configured to drive light sensing pixels at the touch area. 9.A method of controlling a display device, comprising: acquiring a first image by a sensor including a plurality of light sensing pixels when a display portion of the display device displays a test image including a mark; computing a movement amount of the marker from the first image; and correcting calibration image data based on the movement amount of the marker, wherein the corrected calibration image data is to be applied to a fingerprint image acquired by the sensor, wherein the marker movement is due to the sensor movement.

Images