Display device comprising an optical sensor and method of position measurement of an optical sensor

Abstract

A display device including an optical sensor and a measurement method for a position of the optical sensor are provided. The display device having the optical sensor includes a display panel including display pixels, and an optical sensor including a sensor controller and sensor pixels, wherein the sensor pixels are located on a first area of the display panel, the display pixels display a pattern including a plurality of markers in the first area, the sensor pixels image the pattern to generate a first image, the sensor controller divides the first image into sub-images corresponding to the markers, the sensor controller calculates first representative coordinates obtained by adding an integer part and a decimal part with respect to the sub-images, a unit of the integer part is a number of the sensor pixels, and each of the integer part and the decimal part independently includes a first dimension value for a first direction and a second dimension value for a second direction orthogonal to the first direction.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to Korean Patent Application No. 10-2020-0083050, filed on July 6, 2020, which is incorporated herein by reference for all purposes as if fully set forth herein. Technical Field

[0003] Exemplary embodiments and implementations of the present invention generally relate to display devices including optical sensors and methods for measuring the position of optical sensors. Background Technology

[0004] With the development of information technology, the importance of display devices as the connection medium between users and information has become increasingly prominent. Therefore, the use of display devices such as liquid crystal displays, organic light-emitting diode displays, and plasma displays has increased.

[0005] Depending on consumer needs, display devices can include various sensors, such as touch sensors, fingerprint sensors, illuminance sensors, and cameras. In particular, fingerprint sensors are attached to the back surface of the display panel and can identify the ridges and valleys of a fingerprint based on information from a medium (e.g., light or ultrasound) passing through the display panel.

[0006] As discussed in this article, because the carrier includes different background information depending on the internal layout of the display panel through which the carrier passes, it is necessary to perform an initial calibration to remove the background information and leave only the fingerprint information.

[0007] However, after the initial calibration, if the fingerprint sensor's position is unintentionally changed, the sensor's position needs to be updated to correctly use the calibration data. Other sensors attached to the rear surface of the display panel, besides the fingerprint sensor, may also have the same issue.

[0008] The information disclosed in this background section is only intended to enhance the understanding of the background technology of the present invention, and therefore may contain information that does not constitute prior art known to those skilled in the art in this country. Summary of the Invention

[0009] The present invention aims to provide a display device including an optical sensor and a method for measuring the position of the optical sensor, wherein the optical sensor can measure the position displacement and displacement direction of the optical sensor in decimal units and in integer units based on sensor pixels.

[0010] Additional features of the inventive concept will be set forth in the following description and will be apparent in part from the description or may be learned by practice of the inventive concept.

[0011] A display device including an optical sensor according to an embodiment of the present invention includes: a display panel including display pixels; and an optical sensor including a sensor controller and sensor pixels, the sensor pixels being located on a first region of the display panel, wherein the display pixels display a pattern including a plurality of marks in the first region, the sensor pixels image the pattern to generate a first image, the sensor controller divides the first image into sub-images corresponding to the marks, and the sensor controller calculates a first representative coordinate obtained by adding an integer part and a fractional part about the sub-image, the unit of the integer part being the number of sensor pixels, and each of the integer part and the fractional part independently including a first dimension value for a first direction and a second dimension value for a second direction orthogonal to the first direction.

[0012] For each sub-image in the sub-image, the sensor controller can determine the first dimension value corresponding to the maximum value among the average brightness values ​​with respect to the first dimension value as the first dimension value of the integer part, and can determine the second dimension value corresponding to the maximum value among the average brightness values ​​with respect to the second dimension value as the second dimension value of the integer part.

[0013] For each sub-image within a sub-image, the sensor controller can compare the current brightness of the first observation area with its previous brightness to determine the first dimension value of the fractional part.

[0014] Each of the first observation regions can be longer in the second direction than in the first direction, and the first observation regions can be positioned adjacent to each other in the first direction.

[0015] Each of the sub-images may include a first boundary, the brightness of which changes rapidly relative to a first direction, and at least one of the sub-images in the first observation region may overlap with the first boundary.

[0016] When the current brightness of the first observation area is less than its previous brightness, the sensor controller can determine that the first dimension value of the fractional part has a first sign, and when the current brightness of the first observation area is greater than its previous brightness, the sensor controller can determine that the first dimension value of the fractional part has a second sign opposite to the first sign.

[0017] When the current brightness of the first observation area is less than its previous brightness and the current brightness of the first observation area has a difference consistent with the previous brightness, the sensor controller can determine that the first dimension of the fractional part is 0.

[0018] For each sub-image within a sub-image, the sensor controller can compare the current brightness of the second observation area with its previous brightness to determine the second dimension value of the fractional part.

[0019] Each of the second observation regions can be longer in the first direction than in the second direction, and the second observation regions can be positioned adjacent to each other in the second direction.

[0020] Each of the sub-images may include a second boundary whose brightness changes rapidly relative to a second direction, and at least one of the second observation regions may overlap with the second boundary.

[0021] When the current brightness of the second observation area is less than its previous brightness, the sensor controller can determine that the second dimension value of the fractional part has a first sign, and when the current brightness of the second observation area is greater than its previous brightness, the sensor controller can determine that the second dimension value of the fractional part has a second sign.

[0022] The sensor controller can calculate the slope of the optical sensor using a first representative coordinate and a second representative coordinate of a second image generated at a time point different from that of the first image.

[0023] The sensor controller can calculate at least one first slope relative to a first direction and at least one second slope relative to a second direction; when at least one first slope is the same as at least one second slope, the sensor controller can determine that the third slope of a third direction orthogonal to the first and second directions is 0; and when at least one of the at least one first slope and at least one second slope is different, the sensor controller can determine that the third slope is non-zero.

[0024] Another embodiment of the present invention provides a method for measuring the position of an optical sensor located on a first region of a display panel. The method includes: displaying a pattern comprising a plurality of marks in the first region of the display panel; generating a first image by imaging the pattern by sensor pixels of the optical sensor; dividing the first image into sub-images corresponding to the marks by a sensor controller of the optical sensor; and calculating a first representative coordinate by the sensor controller by summing an integer part and a fractional part of the sub-images, wherein the unit of the integer part is the number of sensor pixels, and each of the integer part and the fractional part independently includes a first dimension value with respect to a first direction and a second dimension value with respect to a second direction orthogonal to the first direction.

[0025] The method for measuring the position of an optical sensor may further include: for each sub-image in a sub-image, the sensor controller determines a first dimension value as an integer part corresponding to the maximum value among the average brightness values ​​with respect to a first dimension value, and a second dimension value as an integer part corresponding to the maximum value among the average brightness values ​​with respect to a second dimension value.

[0026] The method for measuring the position of an optical sensor may further include: for each sub-image in the sub-image, the sensor controller compares the current brightness of a first observation area with its previous brightness to determine the first dimension value of the fractional part.

[0027] Each of the first observation regions may be longer in the second direction than in the first direction; the first observation regions may be positioned adjacent to each other in the first direction; each of the sub-images may include a first boundary, the brightness of which changes rapidly relative to the first direction, and at least one of the first observation regions may overlap with the first boundary.

[0028] The method for measuring the position of an optical sensor may further include: when the current brightness of a first observation area is less than its previous brightness, the sensor controller determines that the first dimension value of the fractional part has a first sign; and when the current brightness of the first observation area is greater than its previous brightness, the sensor controller determines that the first dimension value of the fractional part has a second sign opposite to the first sign.

[0029] The method for measuring the position of an optical sensor may further include: when the current brightness of a first observation area is less than its previous brightness and the current brightness of the first observation area has a difference consistent with the previous brightness, the sensor controller determines the first dimension value of the fractional part to be 0.

[0030] The method for measuring the position of the optical sensor may further include: for each sub-image in the sub-image, the sensor controller compares the current brightness of the second observation area with its previous brightness to determine the second dimension value of the fractional part.

[0031] The display device including an optical sensor and the method for measuring the position of the optical sensor according to the present invention can measure the position displacement and displacement direction of the optical sensor in both decimal units and integer units based on sensor pixels.

[0032] It will be understood that both the foregoing general description and the following detailed description are exemplary and illustrative, and are intended to provide further explanation of the claimed invention. Attached Figure Description

[0033] The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification. The drawings illustrate exemplary embodiments of the invention and, together with the description, serve to explain the inventive concept.

[0034] Figure 1 , Figure 2 and Figure 3 A schematic diagram is shown for explaining an embodiment of the concept according to the present invention.

[0035] Figure 4 A schematic diagram of an image captured by an optical sensor is shown to illustrate an embodiment of the concept according to the present invention.

[0036] Figure 5 A schematic diagram is shown for explaining a method for calculating the integer portion of representative coordinates of a sub-image according to an embodiment of the present invention.

[0037] Figure 6 , Figure 7 , Figure 8 and Figure 9 A schematic diagram is shown for explaining a method for calculating the fractional part of the representative coordinates of a sub-image according to an embodiment of the present invention.

[0038] Figure 10 A schematic diagram is shown for explaining a method for calculating the slope of an optical sensor according to an embodiment of the present invention. Detailed Implementation

[0039] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of various exemplary embodiments or implementations of the invention. As used herein, “implementation” and “method” are interchangeable terms and are non-limiting examples of apparatuses or methods employing one or more of the inventive concepts disclosed herein. However, it will be apparent, however, that various exemplary embodiments may be practiced without these specific details or with one or more equivalent arrangements. In other instances, well-known structures and apparatuses are illustrated in block diagram form to avoid unnecessarily obscuring the various exemplary embodiments. Furthermore, the various exemplary embodiments may be different, but not necessarily exclusive. For example, a particular shape, configuration, and characteristic of an exemplary embodiment may be used or implemented in another exemplary embodiment without departing from the inventive concept.

[0040] Unless otherwise stated, the exemplary embodiments shown should be understood as providing exemplary features detailing variations in some ways in which the inventive concept can be implemented in practice. Therefore, unless otherwise stated, features, components, modules, layers, films, panels, regions, and / or aspects (hereinafter individually or collectively referred to as “elements”) of various embodiments may be combined, separated, interchanged, and / or rearranged in other ways without departing from the inventive concept.

[0041] Crosshairs and / or shading are typically used in the accompanying drawings to clarify the boundaries between adjacent elements. Therefore, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for particular materials, material properties, dimensions, scale, commonalities between illustrated elements, and / or any other characteristics, properties, etc., of the elements, unless otherwise stated. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of elements may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a particular process sequence may be performed differently than the described sequence. For example, two consecutively described processes may be performed substantially simultaneously, or in the reverse order of their description. Furthermore, the same reference numerals denote the same elements.

[0042] When a component or layer is referred to as being "on," "connected to," or "attached to" another component or layer, it can be directly on, connected to, or attached to the other component or layer, or there may be intermediate components or layers present. However, when a component or layer is referred to as being "directly on," "directly connected to," or "directly attached to" another component or layer, there are no intermediate components or layers present. Therefore, the term "connection" can refer to a physical connection, electrical connection, and / or fluid connection with or without intermediate components. Furthermore, the D1-axis, D2-axis, and D3-axis are not limited to the three axes of a Cartesian coordinate system, such as the x-axis, y-axis, and z-axis, and can be interpreted in a broader sense. For example, the D1-axis, D2-axis, and D3-axis can be perpendicular to each other, or they can represent different directions that are not perpendicular to each other. For the purposes of this disclosure, “at least one of X, Y, and Z” and “at least one selected from the group consisting of X, Y, and Z” can be interpreted as only X, only Y, only Z, or any combination of two or more of X, Y, and Z, such as, for example, XYZ, XYY, YZ, and ZZ. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0043] Although the terms “first,” “second,” etc., may be used herein to describe various types of elements, these elements should not be limited by these terms. These terms are used to distinguish one element from another. Therefore, without departing from the teachings of this disclosure, the first element discussed below may be referred to as the second element.

[0044] For descriptive purposes, spatial relative terms such as “below,” “under,” “below,” “lower,” “above,” “upper,” “above,” “higher,” “side” (e.g., as in “sidewall”), etc., may be used herein to describe the relationship between one element and another as shown in the accompanying drawings. In addition to the orientations described in the drawings, spatial relative terms are also intended to include different orientations of the device in use, operation, and / or manufacture. For example, if the device in the drawings is flipped, an element described as “below” or “under” other elements or features would be oriented “above” those other elements or features. Thus, the exemplary term “below” can include both above and below orientations. Furthermore, the device may be oriented in other ways (e.g., rotated 90 degrees or in other orientations), and therefore, the spatial relative descriptive terms used herein shall be interpreted accordingly.

[0045] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. Furthermore, when used in this specification, the terms “comprise,” “comprising,” “include,” and / or “including” specify the presence of stated features, integrals, steps, operations, elements, components, and / or groups thereof, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms, not as terms of degree, and therefore, to explain the inherent biases in the measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0046] Unless otherwise specified, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms (such as those corresponding to those in commonly used dictionaries) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and shall not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0047] The inventive concept will be described more fully below with reference to the accompanying drawings, in which embodiments of the invention are illustrated. As will be appreciated by those skilled in the art, the described embodiments can be modified in various ways without departing from the spirit and scope of the inventive concept.

[0048] Parts irrelevant to the description will be omitted to clearly illustrate the inventive concept, and the same reference numerals denote the same elements throughout the specification. Therefore, the above reference numerals may be used in other figures.

[0049] Furthermore, the dimensions and thicknesses of each element are arbitrarily shown in the accompanying drawings for ease of description, and the inventive concept is not limited to the dimensions and thicknesses of the elements shown in the drawings. For clarity, the thicknesses of layers, films, panels, regions, etc., may be exaggerated in the drawings.

[0050] Figures 1 to 3 A schematic diagram is shown for explaining an embodiment of the concept according to the present invention. Figure 2 It shows along Figure 1 An exemplary cross-sectional view taken by line A-A'.

[0051] The first direction X, the second direction Y, and the third direction Z can be orthogonal to each other.

[0052] Reference Figures 1 to 3 The display device 9 may include a display panel 10 and an optical sensor 20.

[0053] Display panel 10 may include display pixels (DPX) and display controller 101. The display pixels (DPX) may be arranged in a plane defined by a first direction X and a second direction Y. The display pixels (DPX) are not necessarily arranged in a planar shape, but may be arranged in a curved shape. Although not shown, for example, the side surface of display panel 10 may be curved, and the display pixels (DPX) disposed on the side surface of display panel 10 may be arranged in a curved shape. The display pixels (DPX) may emit light in a third direction Z to display an image. When the first surface 10a of display panel 10 is viewed by a user, the displayed image can be viewed. Display controller 101 may receive RGB data, control signals, etc., from an external processor to provide data voltages, control signals, etc., for the display pixels (DPX). Detailed configuration of display panel 10 may be based on existing technology.

[0054] The optical sensor 20 may include sensor pixels (SPX), a sensor controller 201, and a sensor memory 202. The optical sensor 20 may be located on a first region AR1 of the display panel 10. For example, the optical sensor 20 may be attached to a second surface 10b of the display panel 10. The optical sensor 20 is a sensor in which light is the carrier, and may be a fingerprint sensor, a camera, an illuminance sensor, or a proximity sensor. For ease of description, it will be assumed below that the optical sensor 20 is a fingerprint sensor.

[0055] The sensor pixels SPX can be arranged in a plane defined by a first direction X and a second direction Y. The sensor pixels SPX can be configured according to the techniques used in conventional optical sensors such as CMOS sensors. The sensor pixels SPX can generate an image by transmitting and capturing a first area AR1 of the display panel 10.

[0056] The sensor memory 202 may include position data of the sensor pixels SPX and calibration data based on the position data. Because the layout of the display panel 10 through which the carrier transmits varies depending on the position of each sensor pixel SPX, it is useful to generate calibration data at least once before transporting the product. Furthermore, the calibration data can reflect the characteristic differences (distribution) between the individual sensor pixel SPXs.

[0057] The sensor controller 201 can generate a corrected image by applying calibration data to an image generated by the sensor pixels SPX. For example, when the image generated by the sensor pixels SPX includes both background information (e.g., the layout of the display panel 10) and a fingerprint shape, the corrected image can include only the fingerprint shape in which the background information has been removed. Therefore, user authentication, etc., can be performed using the obtained fingerprint shape.

[0058] Figure 4 A schematic diagram of an image captured by an optical sensor is shown to illustrate an embodiment of the concept according to the present invention.

[0059] One way to check whether the position of the optical sensor 20 has changed is by having the display device 9 periodically measure the position of the optical sensor 20. Alternatively, the position measurement of the optical sensor 20 can be performed by the user periodically.

[0060] First, the display pixel DPX can display a pattern including multiple markers MK1, MK2, MK3, and MK4 in the first region AR1. The sensor pixel SPX can image the pattern to generate the image ARi1.

[0061] Each of the markers MK1, MK2, MK3, and MK4 can have a brightness higher than the background of image ARi1. For example, each of the markers MK1, MK2, MK3, and MK4 may include a cross-shaped graphic 410. For example, each of the markers MK1, MK2, MK3, and MK4 may include a box graphic 420 with different gray levels within the cross-shaped graphic 410. However, the shapes of the markers MK1, MK2, MK3, and MK4 can vary depending on the implementation, and this implementation is not limited to this one. Figure 4 The shapes of markers MK1, MK2, MK3, and MK4. Markers MK1, MK2, MK3, and MK4 can be set at points of interest where the coordinates are to be known.

[0062] Sensor controller 201 can divide image ARi1 into sub-images Q1, Q2, Q3, and Q4 corresponding to markers MK1, MK2, MK3, and MK4. For example, image ARi1 can be a quadrilateral, and markers MK1, MK2, MK3, and MK4 can be located near the vertices of the quadrilateral. For example, sub-images Q1, Q2, Q3, and Q4 can be quadrants of image ARi1. Sensor controller 201 can calculate representative coordinates by summing the integer and fractional parts of the values ​​used for sub-images Q1, Q2, Q3, and Q4. In this case, the unit of the integer part can be the number of sensor pixels (SPX). For example, when the difference between the previous representative coordinates and the current representative coordinates is 3 relative to the first direction X, the result can be interpreted as the optical sensor 20 shifting in the first direction X by a distance corresponding to the width of three sensor pixels (SPX).

[0063] In this configuration, each of the integer and fractional parts can independently include a first dimension value for the first direction X and a second dimension value for the second direction Y, which is orthogonal to the first direction X. That is, the sensor controller 201 can calculate the first dimension value representing the coordinate by summing the first dimension values ​​of the integer and fractional parts after calculating the integer and fractional parts respectively. The sensor controller 201 can calculate the second dimension value representing the coordinate by summing the second dimension values ​​of the integer and fractional parts. The first dimension value can be a coordinate value along the x-axis, and the second dimension value can be a coordinate value along the y-axis.

[0064] According to the embodiments described herein, the displacement direction of both the fractional and integer parts can be measured. Therefore, even when the optical sensor 20 moves precisely in fractional units rather than integer units, the displacement direction and displacement amount of the optical sensor 20 can be accurately measured.

[0065] The method for calculating the representative coordinates of sub-image Q1 will be described below. The representative coordinates of the other sub-images Q2, Q3, and Q4 can also be calculated in the same way, so repeated descriptions are omitted.

[0066] Figure 5 A schematic diagram is shown for explaining a method for calculating the integer portion of representative coordinates of a sub-image according to an embodiment of the present invention.

[0067] For each sub-image Q1, Q2, Q3, and Q4, the sensor controller 201 can determine a first dimension value xi1 as the integer part of the first dimension value corresponding to the maximum value among the average brightness AVGX used for the first dimension value. Furthermore, for each sub-image Q1, Q2, Q3, and Q4, the sensor controller 201 can determine a second dimension value yi1 as the integer part of the second dimension value corresponding to the maximum value among the average brightness AVGY used for the second dimension value.

[0068] For example, the average brightness AVGX of the sensor pixel SPX at the same position relative to the first direction X can correspond to the height of the image. For example, since the sensor pixel SPX located relative to the first direction X corresponds to the background of the image ARi1, the average brightness AVGX can be minimized. At the same time, since the sensor pixel SPX located at the xi1th position relative to the first direction X includes the portion of the intersecting pattern extending in the second direction Y, the average brightness AVGX can be maximized.

[0069] For example, the average brightness AVGY of the sensor pixel SPX at the same position relative to the second direction Y can correspond to the height of the image. For example, since the sensor pixel SPX initially located relative to the second direction Y corresponds to the background of image ARi1, the average brightness AVGY can be minimized. At the same time, since the sensor pixel SPX located at the first yi1 relative to the second direction Y includes the portion of the intersecting pattern extending in the first direction X, the average brightness AVGY can be maximized.

[0070] Figures 6 to 9 A schematic diagram is shown for explaining a method for calculating the fractional part of the representative coordinates of a sub-image according to an embodiment of the present invention.

[0071] Reference Figure 6The sensor controller 201 can compare the current brightness of the first observation regions PVB, PVR, and PVD in sub-images Q1, Q2, Q3, and Q4 with their previous brightness to determine the first dimension value of the fractional part. The current brightness refers to the brightness of the first observation regions PVB, PVR, and PVD in image ARi1 captured at the current time point. The previous brightness refers to the brightness of the first observation regions PVB, PVR, and PVD in images captured at past time points.

[0072] Each of the first observation regions PVB, PVR, and PVD can have a shape that is longer in the second direction Y than in the first direction X. For example, each of the first observation regions PVB, PVR, and PVD can be a quadrilateral that is longer in the second direction Y than in the first direction X. The first observation regions PVB, PVR, and PVD can have the same size and shape.

[0073] The first observation regions PVB, PVR, and PVD can be positioned adjacent to each other in the first direction X. For example, the first observation regions PVB, PVR, and PVD can be in contact with each other in the first direction X.

[0074] Each of the sub-images Q1, Q2, Q3, and Q4 may include a first boundary EDG1, the brightness of which changes rapidly relative to a first direction X. For example, sub-image Q1 may include a first boundary EDG1, the brightness of which changes rapidly relative to the first direction X. Figure 6 In one implementation, the first boundary EDG1 of the white-framed graphic located in the first direction X and the second direction Y from the cross graphic is used.

[0075] At least one of the first observation regions PVB, PVR, and PVD can overlap with the first boundary EDG1. For example, the first observation region PVR can overlap with the first boundary EDG1. The length of the first observation region PVR relative to the second direction Y can be the same as the length of the first boundary EDG1. The first observation region PVB can overlap with the white box. The first observation region PVD can overlap with the background.

[0076] Reference Figure 7 The diagram exemplarily illustrates a graph XGR for the previous brightness of the first observation region PVB, PVR, and PVD, and graphs XGP and XGM for different cases of the current brightness of the first observation region PVB, PVR, and PVD. Each of the graphs XGR, XGP, and XGM is a graph that connects the brightness of the first observation region PVB, PVR, and PVD with straight lines.

[0077] When the current brightness of the first observation area PVB, PVR, and PVD is less than its previous brightness, the sensor controller 201 can determine that the first dimension value of the fractional part has a first sign. For example, when the current brightness of the first observation area PVB, PVR, and PVD corresponds to the graphic XGP, the current brightness is less than the previous brightness of the graphic XGR. In this case, because the sensor pixel SPX is shifted relative to the display panel 10 in the first direction X, the first dimension value of the fractional part can have a (+) sign. The amount of shift in the first direction X can be calculated using various algorithms based on the position, size, and shape of the first observation area PVB, PVR, and PVD. For example, in Figure 7 In this implementation, the first dimension value of the fractional part can be (+)0.2. This is a shift amount corresponding to 1 / 5 of the width of the sensor pixel SPX. Therefore, the first dimension value of the representative coordinate of sub-image Q1 can be xi1+0.2.

[0078] When the current brightness of the first observation area PVB, PVR, and PVD is greater than its previous brightness, the sensor controller 201 can determine that the first dimension value of the fractional part has a second sign opposite to the first sign. For example, when the current brightness of the first observation area PVB, PVR, and PVD corresponds to the graphic XGM, the current brightness is greater than the previous brightness of the graphic XGR. In this case, because the sensor pixel SPX is shifted relative to the display panel 10 in the opposite direction to the first direction X, the first dimension value of the fractional part can have a (-) sign. The amount of shift in the first direction X can be calculated using various algorithms based on the position, size, and shape of the first observation area PVB, PVR, and PVD. For example, in Figure 7 In this implementation, the first dimension value of the fractional part can be (-)0.2. This is a shift amount corresponding to 1 / 5 of the width of the sensor pixel SPX. Therefore, the first dimension value of the representative coordinate of sub-image Q1 can be xi1-0.2.

[0079] According to this embodiment, the shift direction and shift amount of the fractional and integer parts can be known. Therefore, when the shift amount of the integer part of the optical sensor 20 is 0 and the shift amount of the fractional part is not 0, that is, even when the optical sensor 20 is finely shifted, the precise shift direction can be known.

[0080] The representative coordinates of sub-images Q1, Q2, Q3, and Q4 calculated according to this embodiment can be stored in the sensor memory 202. Therefore, the calibration data can be used for image correction based on the corrected representative coordinates.

[0081] Reference Figure 8The diagram exemplarily illustrates a graph XGR for the previous brightness of the first observation region PVB, PVR, and PVD, and a graph XGD for the current brightness of the first observation region PVB, PVR, and PVD. Each of the graphs XGR and XGD is a graph that connects the brightness of the first observation region PVB, PVR, and PVD with a straight line.

[0082] When the current brightness of the first observation area PVB, PVR, and PVD is less than its previous brightness, and when the current brightness of the first observation area PVB, PVR, and PVD has a consistent difference from its previous brightness, the sensor controller 201 can determine that the first dimension of the fractional part is 0. The range of the consistent difference can be determined differently depending on the position, size, and shape of the first observation area PVB, PVR, and PVD.

[0083] For example, when the current brightness of the first observation area PVB, PVR, and PVD is less than its previous brightness, and when the current brightness of the first observation area PVB, PVR, and PVD has a consistent difference from its previous brightness, the current brightness pattern XGD can have substantially the same shape as the pattern XGR, but can have a shape that is shifted downwards relative to the brightness axis. This situation can occur where the sensor pixel SPX is not shifted, but the display pixel DPX degrades and therefore its brightness decreases. That is, although in Figure 7 The graphics of XGP and Figure 8 In both XGD images, the current brightness of the first observation region PVR is lower than the previous brightness, but it is certain that in... Figure 7 In the case of the XGP graphics, a shift occurred, while Figure 8 No shift occurred in the case of XGD graphics.

[0084] Therefore, according to the exemplary embodiment, multiple first observation regions PVB, PVR, and PVD are observed to measure relative displacement. Thus, it is possible to prevent a situation where only one first observation region PVR is observed, thereby incorrectly determining that displacement has occurred.

[0085] Reference Figure 9 The sensor controller 201 can compare the current brightness of the second observation regions PHB, PHR, and PHD of sub-images Q1, Q2, Q3, and Q4 with their previous brightness to determine the second dimension value of the fractional part. The current brightness refers to the brightness of the second observation regions PHB, PHR, and PHD in image ARi1 captured at the current time point. The previous brightness refers to the brightness of the second observation regions PHB, PHR, and PHD in images captured at past time points.

[0086] Each of the second observation regions PHB, PHR, and PHD can have a shape that is longer in the first direction X than in the second direction Y. For example, each of the second observation regions PHB, PHR, and PHD can be a quadrilateral that is longer in the first direction X than in the second direction Y. The second observation regions PHB, PHR, and PHD can have the same size and shape.

[0087] The second observation regions PHB, PHR, and PHD can be positioned adjacent to each other in the second direction Y. For example, the second observation regions PHB, PHR, and PHD can be in contact with each other in the second direction Y.

[0088] Each of the sub-images Q1, Q2, Q3, and Q4 may include a second boundary EDG2, the brightness of which changes rapidly relative to the second direction Y. For example, sub-image Q1 may include a second boundary EDG2, the brightness of which changes rapidly relative to the second direction Y. Figure 9 In one implementation, the second boundary EDG2 of the white-framed graphic located in the first direction X and the second direction Y from the cross graphic is used.

[0089] At least one of the second observation regions PHB, PHR, and PHD can overlap with the second boundary EDG2. For example, the second observation region PHR can overlap with the second boundary EDG2. The length of the second observation region PHR relative to the first direction X can be the same as the length of the second boundary EDG2. The second observation region PHB can overlap with the white box. The second observation region PHD can overlap with the background.

[0090] When the current brightness of the second observation areas PHB, PHR, and PHD is less than their previous brightness, the sensor controller 201 can determine that the second dimension value of the fractional part has a first sign. Furthermore, when the current brightness of the second observation areas PHB, PHR, and PHD is greater than their previous brightness, the sensor controller 201 can determine that the second dimension value of the fractional part has a second sign. This is because... Figure 7 The descriptions are basically the same, so repeated descriptions have been omitted.

[0091] When the current brightness of the second observation area PHB, PHR, and PHD is less than its previous brightness, and when the current brightness of the second observation area PHB, PHR, and PHD has a consistent difference from its previous brightness, the sensor controller 201 can determine that the second dimension value of the fractional part is 0. This is because... Figure 8 The descriptions are basically the same, so repeated descriptions have been omitted.

[0092] Figure 10 A schematic diagram is shown for explaining a method for calculating the slope of an optical sensor according to an embodiment of the present invention.

[0093] Reference Figure 10 The diagram exemplifies a first image ARi1 generated at a first time point and a second image ARi1' generated at a second time point different from the first time point. For example, the second time point could be the measurement time point following the first time point. It is assumed that the first representative coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) are calculated based on sub-images Q1, Q2, Q3, and Q4 of the first image ARi1, and that the second representative coordinates (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4') are calculated based on sub-images of the second image ARi1'. For ease of description, it is assumed that the first representative coordinates (x1, y1) and the second representative coordinates (x1', y1') are the same. For ease of description, it is assumed that the first image ARi1 is a quadrilateral connecting the four first representative coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4). For ease of description, assume that the second image ARi1' is a quadrilateral connecting four second representative coordinates (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4').

[0094] The sensor controller 201 can use first representative coordinates (x1, y1), (x2, y2), (x3, y3), and (x4, y4) and second representative coordinates (x1', y1'), (x2', y2'), (x3', y3'), and (x4', y4') of the second image ARi1' generated at a different time point than the first image ARi1 to calculate the slopes ag12, ag34, ag13, and ag24 of the optical sensor 20. For example, the sensor controller 201 can calculate at least one first slope ag12 or ag34 relative to a first direction X, and at least one second slope ag13 or ag24 relative to a second direction Y.

[0095] For example, the slope ag12 can be calculated using the first representative coordinates (x1, y1) and the second representative coordinates (x2', y2') through Equation 1 below.

[0096] [Equation 1]

[0097]

[0098] Other slopes ag34, ag13, and ag24 can also be calculated in essentially the same way, so redundant descriptions are omitted. Therefore, according to this embodiment, it is possible to calculate the amount and direction of slope shift in the plane of the optical sensor 20.

[0099] When at least one first slope ag12 or ag34 and at least one second slope ag13 or ag24 are the same, the sensor controller 201 can determine that the third slope perpendicular to the first direction X and the second direction Y is 0. For example, when all slopes ag12, ag34, ag13, and ag24 are the same, the optical sensor 20 can be determined that the slope is tilted only in the plane (first direction X and second direction Y) and not in the vertical direction (third direction Z).

[0100] When at least one of the first slopes ag12 or ag34 and at least one of the second slopes ag13 or ag24 is different, the sensor controller 201 can determine that the third slope is not zero. For example, when at least two of the slopes ag12, ag34, ag13 and ag24 are different, the optical sensors 20 can be determined to be tilted in the vertical direction and in the planar direction.

[0101] Therefore, according to this embodiment, it is possible to determine whether the slope in the planar direction of the optical sensor 20 and the slope in its vertical direction are different from each other.

[0102] While the invention has been described in conjunction with embodiments now considered practical exemplary embodiments, it will be understood that the invention is not limited to the disclosed embodiments, but rather, it is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Therefore, those skilled in the art will understand that various modifications and other equivalent embodiments of the inventive concept are possible. Consequently, the true technical scope of the inventive concept must be determined based on the technical spirit of the appended claims.

Claims

1. A display device, including: Display panel, including display pixels; as well as An optical sensor, including a sensor controller and sensor pixels, wherein the sensor pixels are located on a first area of ​​the display panel; The display pixels in the first area display a pattern including multiple marks. The sensor pixels image the pattern to generate a first image. The sensor controller divides the first image into sub-images corresponding to the markers. The sensor controller calculates a first representative coordinate obtained by adding the integer and fractional parts of the sub-image. Wherein, the unit of the integer part is the number of sensor pixels. Each of the integer part and the fractional part independently includes a first dimension value for a first direction and a second dimension value for a second direction orthogonal to the first direction. Specifically, for each sub-image within the sub-images, the sensor controller determines the first dimension value corresponding to the maximum value among the average brightness values ​​with respect to the first dimension value as the first dimension value of the integer part, and determines the second dimension value corresponding to the maximum value among the average brightness values ​​with respect to the second dimension value as the second dimension value of the integer part. Specifically, for each sub-image in the sub-image, the sensor controller compares the current brightness of the first observation area with its previous brightness to determine the first dimension value of the fractional part, and compares the current brightness of the second observation area with its previous brightness to determine the second dimension value of the fractional part.

2. The display device according to claim 1, wherein, Each of the first observation regions is longer in the second direction than in the first direction, and The first observation areas are positioned adjacent to each other in the first direction.

3. The display device according to claim 2, wherein, Each of the sub-images includes a first boundary, the brightness of which changes rapidly relative to the first direction, and Wherein, at least one of the first observation regions overlaps with the first boundary.

4. The display device according to claim 3, wherein, When the current brightness of the first observation area is less than its previous brightness, the sensor controller determines that the first dimension value of the fractional part has a first sign, and Wherein, when the current brightness of the first observation area is greater than its previous brightness, the sensor controller determines that the first dimension value of the fractional part has a second sign opposite to the first sign.

5. The display device according to claim 4, wherein, When the current brightness of the first observation area is less than its previous brightness and the current brightness of the first observation area has a difference consistent with the previous brightness, the sensor controller determines that the first dimension value of the fractional part is 0.

6. A method for measuring the position of an optical sensor located on a first area of ​​a display panel, the measurement method comprising: The display panel displays a pattern including multiple marks in the first area; The first image is generated by the sensor pixels of the optical sensor imaging the pattern; The sensor controller of the optical sensor divides the first image into sub-images corresponding to the markers; as well as The sensor controller calculates the first representative coordinates by summing the integer and fractional parts of the sub-image. Wherein, the unit of the integer part is the number of sensor pixels. Each of the integer part and the fractional part independently includes a first dimension value with respect to a first direction and a second dimension value with respect to a second direction orthogonal to the first direction, and The measurement method further includes: For each sub-image in the sub-image, the sensor controller determines a first dimension value corresponding to the maximum value among the average brightness values ​​with respect to a first dimension value as the first dimension value of the integer part, and a second dimension value corresponding to the maximum value among the average brightness values ​​with respect to a second dimension value as the second dimension value of the integer part; and For each sub-image in the sub-image, the sensor controller compares the current brightness of the first observation area with its previous brightness to determine the first dimension value of the fractional part, and compares the current brightness of the second observation area with its previous brightness to determine the second dimension value of the fractional part.

Images