Image processing system

Abstract

To provide an image processing system capable of reducing the display unevenness of an image displayed on a display device.SOLUTION: The image processing system includes a display device, an imaging device, and a learning device. The learning device stores a table representing information relating to correspondence between first image data and second image data generated by displaying an image corresponding to the first image data on the display device and capturing the image by the imaging device. The learning device generates teacher data on the basis of the table, to generate a machine learning model by using the generated teacher data. Image processing using the machine learning model is performed to image data inputted to the display device, to reduce the display unevenness of the image displayed on the display device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] One aspect of the present invention relates to an image processing system. [Background technology]

[0002] Display devices such as liquid crystal displays and organic electroluminescence displays can be manufactured by applying a resist onto a substrate, and then exposing the substrate through a mask to perform patterning. In order to enlarge the display device, the substrate must also be enlarged, but there are cases where the mask cannot be enlarged to match the size of the substrate. As a method for manufacturing a display device in such a case, Patent Document 1 discloses a method in which the substrate surface is divided into a plurality of exposure regions corresponding to the size of the mask, and each exposure region is exposed.

[0003] Furthermore, defective pixels may occur due to poor characteristics or deterioration of display elements, transistors, etc., that the pixels of the display device have. Defective pixels may be, for example, bright spots or dark spots. When viewing an image displayed on a display device, bright spots are more noticeable than dark spots, and therefore have a large impact on visibility. Therefore, if there are many bright spots, the display device may not be able to display a high-quality image. Patent Document 2 discloses a method of turning bright spots into dark spots during the manufacturing process of a display device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-198990 A [Patent Document 2] Special Publication No. 2018-514801 Summary of the Invention [Problem to be solved by the invention]

[0005] When performing divided exposure as described above, the exposure amount at the boundary of the exposure area may differ from the exposure amount in other areas due to the position of the mask being shifted from the exposure area, etc. As a result, the characteristics of the elements of the pixels provided at the boundary of the exposure area may differ from the characteristics of the elements of the pixels provided in other areas. As a result, even if the gradation is the same, the luminance of the light emitted by the pixels provided at the boundary of the exposure area may differ from the luminance of the light emitted by the pixels provided in other areas. The difference in luminance may be visually recognized as display unevenness.

[0006] As a method for making the display unevenness less noticeable, a method of performing image processing on image data input to the display device is considered. For example, a method of performing image processing using machine learning is considered. Specifically, a method of generating a machine learning model by a generating device and performing image processing by the machine learning model on image data input to the display device is considered. When the display device performs image processing using the machine learning model generated by the generating device, it can be said that the display device and the generating device constitute an image processing system.

[0007] Furthermore, even if a pixel is not a bright spot when the display device is manufactured, it may become a bright spot when the display device is used for a long period of time due to deterioration of the display elements, transistors, etc. that constitute the pixel, causing fluctuations in electrical characteristics, etc. Such bright spots are difficult to remove during the manufacturing process of the display device.

[0008] An object of one embodiment of the present invention is to provide an image processing system that can make display unevenness of an image displayed on a display device less noticeable. Another object of one embodiment of the present invention is to provide an image processing system that can display a high-quality image on a display device. Another object of one embodiment of the present invention is to provide an image processing system having a large display device. Another object of one embodiment of the present invention is to provide an image processing system having a display device that can display a high-definition image. Another object of one embodiment of the present invention is to provide an image processing system that can perform image processing in a short time. Another object of one embodiment of the present invention is to provide an image processing system having a highly reliable display device.

[0009] Another object of one embodiment of the present invention is to provide a novel image processing system, a novel image processing method, a novel generation device, a novel machine learning model generation method, a novel image processing device, a novel display device, or the like.

[0010] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract problems other than these from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0011] One aspect of the present invention relates to an image processing system including a display device, an imaging device, and a learning device, and a method for generating a machine learning model using the image processing system. The display device has pixels arranged in a matrix of m rows and n columns (m and n are integers equal to or greater than 2). The learning device has a database. The database stores a table generated based on first image data and second image data acquired by displaying an image corresponding to the first image data on the display device and capturing the image with the imaging device. The first image data has first gradation values ​​of m rows and n columns, and the second image data has second gradation values ​​of m rows and n columns. The table specifically represents the first gradation values ​​and the second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values.

[0012] When generating a machine learning model, first, an image corresponding to the first training image data is displayed on a display device, and the image displayed on the display device is captured by an imaging device to obtain second training image data. Next, the learning device performs image processing on the first training image data based on the second training image data to generate third training image data having a third gradation value in m rows and n columns. Specifically, the learning device performs image processing on the first training image data so as to approximate the second training image data to generate third training image data having a third gradation value in m rows and n columns. For example, image processing is performed on the first training image data so that the sum of the third gradation values ​​in the first row, first column to the m rows and n columns is equal to the sum of the gradation values ​​in the first row, first column to the m rows and n columns of the second training image data.

[0013] Then, the learning device selects the second gradation values ​​in the first row, first column to the m-th row, n-th column based on the third gradation values ​​in the first row, first column to the m-th row, n-th column. For example, the second gradation value that matches or is closest to the third gradation value is selected for each of the first row, first column to the m-th row, n-th column. Next, fourth learning image data is generated, which is image data including a first gradation value corresponding to the selected second gradation value. Then, the learning device generates a machine learning model such that image data output when the first learning image data is input matches the fourth learning image data.

[0014] The machine learning model generated by the learning device is supplied to the display device. This enables the display device to perform image processing using the machine learning model on image data input to the display device. For example, image processing such as reducing display unevenness can be performed on image data input to the display device using the machine learning model.

[0015] One aspect of the present invention includes a display device, an imaging device, and a learning device. The display device includes an input unit, a machine learning processing unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The learning device includes a database, an image processing unit, an image generation unit, and a learning unit. The database stores a table generated based on first image data input to the input unit and second image data acquired by causing the display unit to display an image corresponding to the first image data and capturing an image by the imaging device so as to include the image displayed on the display unit. The first image data has first gradation values ​​of m rows and n columns, and the second image data has second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. The image processing unit generates a table based on the second learning image data for the first learning image data input to the input unit. The image processing system has a function of generating third training image data by performing image processing based on the training image data, the second training image data being image data acquired by displaying an image corresponding to the first training image data on a display unit and capturing an image using an imaging device so as to include the image displayed on the display unit, the third training image data having a third gradation value of m rows and n columns, the image generation unit having a function of generating fourth training image data which is image data including a first gradation value corresponding to a second gradation value selected based on the third gradation value, the training unit having a function of generating a machine learning model such that image data output when the first training image data is input matches the fourth training image data and outputting the machine learning model to the machine learning processing unit, and the machine learning processing unit having a function of performing processing using the machine learning model on the content image data input to the input unit.

[0016] Alternatively, in the above aspect, the first learning image data may have a fourth gradation value in m rows and n columns, and the second learning image data may have a fifth gradation value in m rows and n columns, and the image processing unit may have a function of performing image processing such that the difference between the sum of the third gradation values ​​and the sum of the fifth gradation values ​​is smaller than the difference between the sum of the fourth gradation values ​​and the sum of the fifth gradation values.

[0017] Alternatively, in the above aspect, the machine learning model may be a neural network model.

[0018] Alternatively, one aspect of the present invention is a method for generating a machine learning model by an image processing system having a display unit in which pixels of m rows and n columns (m and n are integers of 2 or more) are arranged in a matrix, the method comprising: displaying an image corresponding to first image data having a first gradation value of m rows and n columns on the display unit by emitting light of a luminance corresponding to the first gradation value from the pixels; acquiring second image data having a second gradation value of m rows and n columns by capturing an image including the image corresponding to the first image data displayed on the display unit; generating a table representing the first gradation value and the second gradation value of a coordinate corresponding to the coordinate of the first gradation value; A method for generating a machine learning model, comprising: acquiring second training image data by displaying the second training image data on a display unit and capturing an image including an image corresponding to the first training image data displayed on the display unit; generating third training image data having third gradation values ​​of m rows and n columns by performing image processing on the first training image data based on the second training image data; generating fourth training image data based on the third gradation values, which is image data including a first gradation value corresponding to the selected second gradation value; and generating a machine learning model such that image data output when the first training image data is input matches the fourth training image data.

[0019] Alternatively, in the above aspect, the first learning image data has a fourth gradation value of m rows and n columns, and the second learning image data has a fifth gradation value of m rows and n columns, and the image processing may be performed such that the difference between the sum of the third gradation values ​​and the sum of the fifth gradation values ​​is smaller than the difference between the sum of the fourth gradation values ​​and the sum of the fifth gradation values.

[0020] Alternatively, in the above aspect, the machine learning model may be a neural network model.

[0021] Alternatively, one embodiment of the present invention includes a display device, an imaging device, and a generation device. The display device includes an input unit, a bright spot correction unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The generation device includes a database and an image generation unit. The database stores a table generated based on first database image data input to the input unit and second database image data acquired by causing the display unit to display an image corresponding to the first database image data and capturing an image including the image displayed on the display unit, the first database image data having first gradation values ​​of m rows and n columns, the second database image data having second gradation values ​​of m rows and n columns, and the table represents the first gradation values ​​and the second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. The imaging device has a function of acquiring second image data for bright spot correction by imaging the image displayed on the display unit when an image corresponding to first image data for bright spot correction input to the input unit is displayed on the display unit, the second image data for bright spot correction having a third gradation value of m rows and n columns, the image generation unit has a function of generating third image data for bright spot correction which is image data including a first gradation value corresponding to a second gradation value selected based on the third gradation value, the bright spot correction unit has a function of detecting the coordinates of the first gradation value that is below a threshold value as bright spot coordinates among the first gradation values ​​of m rows and n columns possessed by the third image data for bright spot correction, and the bright spot correction unit has a function of reducing the fourth gradation value at the same coordinates as the bright spot coordinates when content image data having a fourth gradation value of m rows and n columns is input to the input unit.

[0022] Alternatively, one embodiment of the present invention includes a display device, an imaging device, and a generating device. The display device includes an input unit, a bright spot correction unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The generating device includes a database and an image generating unit. The database stores a table generated based on first database image data input to the input unit and second database image data acquired by causing the imaging device to display an image corresponding to the first database image data on the display unit and capturing an image including the image displayed on the display unit. The first database image data has first gradation values ​​of m rows and n columns, and the second database image data has second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. When the imaging device causes the display unit to display an image corresponding to the first bright spot correction image data input to the input unit, the image generating unit has a function of generating third image data for bright spot correction, the third image data for bright spot correction being image data including a first gradation value corresponding to the second gradation value selected based on the third gradation value, and the bright spot correction unit has a function of generating a first gradation value corresponding to the second gradation value selected based on the third gradation value, the first gradation value being equal to or smaller than a first threshold value among the first gradation values ​​in the m rows and n columns of the third image data for bright spot correction. the bright spot correction unit has a function of detecting the coordinates of a third gradation value equal to or greater than a second threshold value as a first bright spot coordinate, and the bright spot correction unit has a function of detecting the coordinates of a third gradation value of m rows and n columns that is contained in the second bright spot correction image data and that is equal to or greater than a second threshold value as a second bright spot coordinate, and the bright spot correction unit has a function of reducing the fourth gradation value at the same coordinate as the first or second bright spot coordinate when content image data having a fourth gradation value of m rows and n columns is input to the input unit.

[0023] Alternatively, one embodiment of the present invention includes a display device, an imaging device, and a generation device. The display device includes an input unit, a bright spot correction unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The generation device includes a database and an image generation unit. The database stores a table generated based on first database image data input to the input unit and second database image data acquired by causing an image corresponding to the first database image data to be displayed on the display unit and capturing an image by the imaging device so as to include the image displayed on the display unit. The first database image data has first gradation values ​​of m rows and n columns, and the second database image data has second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. The imaging device has a function of acquiring the second bright spot correction image data by capturing the image displayed on the display unit when an image corresponding to the first bright spot correction image data input to the input unit is displayed on the display unit. The second bright spot correction image data has third gradation values ​​in m rows and n columns, the image generation unit has a function of generating third bright spot correction image data which is image data including a first gradation value corresponding to the second gradation value selected based on the third gradation value, and the bright spot correction unit detects, among the first gradation values ​​in m rows and n columns of the third bright spot correction image data, coordinates of the first gradation value which is equal to or smaller than a first threshold value and equal to or larger than a second threshold value as first bright spot coordinates, and coordinates of the first gradation value which is smaller than the second threshold value as second bright spot coordinates. the bright spot correction unit has a function of detecting, as third bright spot coordinates, the coordinates of a third gradation value that is equal to or greater than a third threshold value among the third gradation values ​​of m rows and n columns contained in the second bright spot correction image data, and the bright spot correction unit has a function of reducing the fourth gradation value at the same coordinates as both the first bright spot coordinates and the third bright spot coordinates, and a function of reducing the fourth gradation value at the same coordinates as the second bright spot coordinates, when content image data having a fourth gradation value of m rows and n columns is input to the input unit.

[0024] Alternatively, in the above aspect, the display device has a machine learning processing unit, and the generation device has an image processing unit and a learning unit, the image processing unit has a function of generating third training image data by performing image processing on first training image data input to the input unit based on second training image data, the second training image data being image data acquired by displaying an image corresponding to the first training image data on the display unit and capturing an image by the imaging device so as to include the image displayed on the display unit, the third training image data having a fifth gradation value of m rows and n columns, the image generation unit has a function of generating fourth training image data being image data including a first gradation value corresponding to the second gradation value selected based on the fifth gradation value, the learning unit has a function of generating a machine learning model such that image data output when the first training image data is input matches the fourth training image data and outputs the machine learning model to the machine learning processing unit, and the machine learning processing unit may have a function of performing processing using the machine learning model on the content image data input to the input unit.

[0025] Alternatively, in the above aspect, the first learning image data may have a sixth gradation value in m rows and n columns, and the second learning image data may have a seventh gradation value in m rows and n columns, and the image processing unit may have a function of performing image processing such that the difference between the sum of the fifth gradation values ​​and the sum of the seventh gradation values ​​is smaller than the difference between the sum of the sixth gradation values ​​and the sum of the seventh gradation values.

[0026] Alternatively, in the above aspect, the machine learning model may be a neural network model. Effect of the Invention

[0027] According to one embodiment of the present invention, an image processing system capable of making display unevenness of an image displayed on a display device less noticeable can be provided. According to another embodiment of the present invention, an image processing system capable of displaying a high-quality image on a display device can be provided. According to another embodiment of the present invention, an image processing system having a large display device can be provided. According to another embodiment of the present invention, an image processing system having a display device capable of displaying a high-definition image can be provided. According to another embodiment of the present invention, an image processing system capable of performing image processing in a short time can be provided. According to another embodiment of the present invention, an image processing system having a highly reliable display device can be provided.

[0028] According to one embodiment of the present invention, a novel image processing system, a novel image processing method, a novel generation device, a novel machine learning model generation method, a novel image processing device, a novel display device, or the like can be provided.

[0029] The effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. The other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from descriptions in the specification, drawings, etc., and can be appropriately extracted from these descriptions. One embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief description of the drawings]

[0030] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system. [Diagram 2] Fig. 2A is a block diagram showing an example of the configuration of a display unit, Fig. 2B1 and Fig. 2B2 are circuit diagrams showing an example of the configuration of a pixel. [Diagram 3]3A and 3B are schematic diagrams showing an example of an image processing method. [Figure 4] FIG. 4 is a flowchart showing an example of a method for generating a table. [Diagram 5] 5A and 5B are schematic diagrams showing an example of a method for generating a table. [Figure 6] 6A and 6B are schematic diagrams showing an example of a method for generating a table. [Figure 7] FIG. 7 is a flowchart illustrating an example of a method for generating a machine learning model. [Figure 8] 8A and 8B are schematic diagrams showing an example of a method for generating a machine learning model. [Figure 9] 9A and 9B are schematic diagrams showing an example of a method for generating a machine learning model. [Figure 10] FIG. 10 is a schematic diagram showing an example of a method for generating a machine learning model. [Figure 11] 11A and 11B are schematic diagrams showing an example of an image processing method. [Figure 12] 12A and 12B are graphs showing an example of an image processing method. [Figure 13] Figures 13A1 and 13A2 are graphs showing an example of an image processing method, and Figure 13B is a schematic diagram showing an example of an image processing method. [Figure 14] FIG. 14 is a graph showing an example of an image processing method. [Figure 15] Fig. 15A is a diagram showing an example of the configuration of a machine learning model, and Fig. 15B is a schematic diagram showing an example of a learning method. [Figure 16] 16A and 16B are schematic diagrams showing an example of a calculation by a machine learning model. [Figure 17] FIG. 17 is a graph showing the learning results according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be easily understood by those skilled in the art that the modes and details of the present invention can be modified in various ways without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0032] In the configuration of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and the repeated explanations are omitted. In addition, when referring to similar functions, the same hatch pattern may be used and no particular reference numeral may be used.

[0033] In addition, for ease of understanding, the position, size, range, etc. of each component shown in the drawings may not represent the actual position, size, range, etc. Therefore, the disclosed invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings.

[0034] In addition, the ordinal numbers "first," "second," and "third" used in this specification are used to avoid confusion of components and do not limit the numbers.

[0035] (Embodiment) In this embodiment, an image processing system or the like according to one embodiment of the present invention will be described with reference to drawings.

[0036] <Image processing system> 1 is a block diagram showing an example of the configuration of an image processing system 10. The image processing system 10 includes a display device 20, an imaging device 30, and a generating device 40. Here, the generating device 40 is preferably provided in a device having high computing power, such as a server.

[0037] The display device 20 includes an input unit 21, a display unit 22, a machine learning processing unit 23, and a bright spot correction unit 50. The generation device 40 includes a database 42, an image extraction unit 43, an image processing unit 44, an image generation unit 45, and a learning unit 46.

[0038] Fig. 2A is a block diagram showing an example of the configuration of the display unit 22. As shown in Fig. 2A, the display unit 22 has pixels 24 arranged in a matrix of m rows and n columns (m and n are integers of 2 or more). The pixels 24 in the same row are electrically connected to each other via the same wiring 134, and the pixels 24 in the same column are electrically connected to each other via the same wiring 126. The pixels 24 have display elements and can display an image on the display unit 22 using the display elements.

[0039] In this specification, the pixels 24 in m rows and n columns are distinguished by being described as pixels 24(1,1) to 24(m,n), respectively. Other elements may be described in a similar manner. For example, (1,1) to (m,n) may be referred to as coordinates.

[0040] In this specification and the like, a display element can be rephrased as a display device. A light-emitting element can be rephrased as a light-emitting device, and a liquid crystal element can be rephrased as a liquid crystal device. For other elements, too, an "element" can sometimes be rephrased as a "device."

[0041] In Fig. 1, arrows indicate data exchange between components of the image processing system 10. Note that the data exchange shown in Fig. 1 is an example, and for example, data and the like may be exchanged between components that are not connected by arrows. Also, even if components are connected by arrows, data may not be exchanged between them.

[0042] Image data is input to the input unit 21. The image data input to the input unit 21 can be output to the display unit 22, the machine learning processing unit 23, the database 42, the image processing unit 44, or the learning unit 46.

[0043] The image data input to the input unit 21 includes database image data DG IN , Learning image data LG IN , Bright spot correction image data BCG IN , and content image data CGIN etc. Database image data DG IN can be supplied from the input unit 21 to the display unit 22 and the database 42. IN can be supplied from the input unit 21 to the display unit 22 and the image processing unit 44. IN can be supplied from the input unit 21 to the display unit 22 and the image processing unit 44. Content image data CG IN can be supplied from the input unit 21 to the machine learning processing unit 23.

[0044] The display unit 22 has a function of displaying an image corresponding to image data. Here, the image data can be a set of gradation values. For example, the image data supplied to the display unit 22 can be configured to have gradation values ​​of m rows and n columns. In this case, the pixels 24 emit light with a luminance corresponding to the gradation value, thereby displaying an image on the display unit 22. Here, the gradation value can be a digital value. For example, when the gradation value is an 8-bit digital value, the gradation value can be an integer between 0 and 255.

[0045] The machine learning processing unit 23 has a function of performing image processing on image data based on the machine learning model generated by the generation device 40. Specifically, the machine learning processing unit 23 performs image processing on the content image data CG input from the input unit 21 based on the machine learning model MLM generated by the learning unit 46. IN The image data that has been image-processed by the machine learning processing unit 23 is converted into content image data CG ML to the bright spot correction unit 50.

[0046] As the machine learning model MLM, for example, a multilayer perceptron, a neural network model, etc. can be applied. In particular, when a neural network model is applied, image processing can be efficiently performed, and a high-quality image can be displayed on the display unit 22, which is preferable. Here, as the neural network model, for example, a generative model such as an autoencoder, U-net, or pix2pix can be used. Note that even machine learning models other than those mentioned above can be used as the machine learning model MLM as long as they are capable of Bayesian estimation. In addition, it is preferable that the machine learning model MLM can perform learning and inference by independently handling input and output values.

[0047] The bright point correction unit 50 corrects the content image data CG ML Specifically, the bright point correction unit 50 has a function of correcting the content image data CG ML The bright spot correction unit 50 has a function of correcting the gradation value of the content image data CG based on the bright spot correction image data BCG_1 generated by the image generation unit 45 or the bright spot correction image data BCG_2 generated by the image extraction unit 43, as will be described in detail later. ML The corrected image data is converted into the content image data CG COR is supplied to the display unit 22.

[0048] In this specification, the bright spot correction image data BCG_1 and the bright spot correction image data BCG_2 may be collectively referred to as bright spot correction image data BCG. Specifically, when the bright spot correction image data BCG is described, it indicates, for example, either the bright spot correction image data BCG_1 or the bright spot correction image data BCG_2. Note that the same description may be used for other data.

[0049] Here, the bright spot correction unit 50 corrects the content image data CG ML The bright point correction unit 50 corrects the content image data CG MLAs described above, by providing the bright spot correction section 50 in the display device 20, the quality of the image displayed on the display section 22 can be improved.

[0050] In this specification, the terms "darkening" and "making a pixel a dark point" refer to reducing the luminance of light emitted from a bright pixel 24. Therefore, even if a pixel 24 is darkened, the luminance of light emitted from the pixel 24 does not have to be 0.

[0051] The display device 20 does not need to have the bright spot correction unit 50. In this case, the bright spot correction image data BCG IN is not input to the input unit 21. In addition, the content image data CG ML can be provided to the display unit 22.

[0052] The imaging device 30 has a function of acquiring imaging data by capturing an image. Specifically, the imaging device 30 can capture an image including an image displayed on the display unit 22. The acquired imaging data is supplied to the image extraction unit 43. Here, the database image data DG IN The image data acquired by the imaging device 30 by capturing an image including the image is displayed on the display unit 22, and the captured image data IMG DG In addition, the learning image data LG IN The image data acquired by the imaging device 30 by capturing an image including the image is displayed on the display unit 22, and the captured image data IMG LG Furthermore, the bright spot correction image data BCG IN The image data acquired by the imaging device 30 by capturing an image including the image is displayed on the display unit 22, and the captured image data IMG BCG The imaging data can be a set of gradation values.

[0053] The image extraction unit 43 extracts the image data IMG DG , Image data IMG LG , and image data IMG BCGThe image extracting unit 43 has a function of extracting data of a portion representing the image displayed on the display unit 22 from the captured image data including the image displayed on the display unit 22. When the imaging device 30 captures an image including the image displayed on the display unit 22, an area other than the display unit 22 may also be captured. For example, the housing of the display device 20 may be captured in addition to the display unit 22. The image extracting unit 43 has a function of extracting data of a portion representing the image displayed on the display unit 22 when a portion other than the image displayed on the display unit 22 is included in the captured image data. The data extraction unit 43 can be performed by pattern matching, template matching, or the like. For example, when extracting data of a portion representing the image displayed on the display unit 22 from captured image data including the image displayed on the display unit 22 and the housing of the display device 20, a pattern representing the housing of the display device 20 can be specified, and a portion not including the pattern can be regarded as data of a portion representing the image displayed on the display unit 22. In addition, when the captured image data IMG DG , Image data IMG LG , and image data IMG BCG etc., and data of the portion representing the image displayed on the display unit 22 can be extracted.

[0054] The image extraction unit 43 extracts the image data IMG DG The data extracted from the database image data DG DP In addition, the image extraction unit 43 extracts the captured image data IMG LG The data extracted from the learning image data LG DP Furthermore, the image extraction unit 43 extracts the captured image data IMG BCG The data extracted from the bright spot correction image data BCG DP Let us assume that the database image data DG DP is supplied to the database 42, and the learning image data LG DP , and bright spot correction image data BCG DP is supplied to the image processing unit 44. The bright spot correction image data BCG DP may be supplied to the image generating unit 45 instead of to the image processing unit 44.

[0055] The image extraction unit 43 also extracts the bright spot correction image data BCGDP The bright spot correction image data BCG to be supplied to the bright spot correction unit 50 DP is assumed to be bright spot correction image data BCG_2.

[0056] As will be described later in detail, the image processing system 10 stores database image data DG IN and database image data DG DP The image processing system 10 also has a function of acquiring a table showing information on the correspondence between the learning image data LG IN And learning image data LG DP Furthermore, the image processing system 10 has a function of comparing the bright spot correction image data BCG IN and bright spot correction image data BCG DP It has the function of comparing.

[0057] Therefore, the database image data DG IN The resolution of the image represented by and the database image data DG DP It is preferable that the resolution of the image represented by the database image data DG is equal to that of the image represented by the database image data DG. IN The number of rows and columns of the gradation values ​​contained in the database image data DG DP It is preferable that the number of rows and columns of the gradation values ​​included in the database image data DG are equal to each other. IN If m rows and n columns of gradation values ​​are included in the database image data DG DP It is preferable that the gradation values ​​included in the learning image data L IN The resolution of the image represented by and the learning image data LG DP It is preferable that the resolution of the image represented by is equal to that of the learning image data LG IN The number of rows and columns of the gradation values ​​contained in and the learning image data LG DP It is preferable that the number of rows and columns of the gradation values ​​included in the learning image data L IN contains m rows and n columns of gradation values, the learning image data LG DPIt is preferable that the gradation values ​​included in the bright spot correction image data BCG are arranged in m rows and n columns. IN The image resolution represented by and the bright point correction image data BCG DP It is preferable that the resolution of the image represented by the bright spot correction image data BCG is equal to the resolution of the image represented by the bright spot correction image data BCG. IN The number of rows and columns of the gradation values ​​contained in the bright point correction image data BCG DP It is preferable that the number of rows and columns of the gradation values ​​included in the bright spot correction image data BCG are equal to each other. IN If m rows and n columns of gradation values ​​are included in DP It is preferable that the gradation values ​​contained in are also arranged in m rows and n columns.

[0058] In this specification, database image data DG IN The gradation value of the database image data DG DP The gradation value that the color filter has may be referred to as the second gradation value.

[0059] The image extraction unit 43 can perform up-conversion or down-conversion on the data extracted from the imaging data. DG The number of rows or columns of gradation values ​​contained in the data extracted from the database image data DG IN If the number of rows or columns is smaller than the number of rows or columns of the image data IMG DG The image extraction unit 43 can perform up-conversion on the data extracted from the image data IMG DG The number of rows or columns of gradation values ​​contained in the data extracted from the database image data DG IN If the number of rows or columns is greater than the number of rows or columns of the image data IMG DG Down conversion can be performed on the data extracted from the database image data DG DP The number of rows and columns of the gradation values ​​included in the database image data DG IN The number of rows and columns of the gradation values ​​included in the image data IMG can be made equal to the number of rows and columns of the gradation values ​​included in the image data IMG.LG , and image data IMG BCG The same is true for . Note that up-conversion and down-conversion can be performed using a nearest neighbor method, a bilinear method, a bicubic method, or the like.

[0060] The database 42 contains database image data DG IN and database image data DG DP A table T representing information on the correspondence between and can be stored. Specifically, the table T is a database of image data DG IN and the database image data DG DP and a second gradation value that the first gradation value has. Table T represents, for example, a first gradation value and a second gradation value at coordinates corresponding to the coordinates of the first gradation value. Table T represents, for example, a first gradation value and a second gradation value at coordinates that are the same as the coordinates of the first gradation value.

[0061] The image processing unit 44 receives the learning image data LG IN On the other hand, the learning image data LG DP By performing image processing based on the IP In other words, the learning image data LG IN and learning image data LG DP Based on the comparison results, the learning image data LG IN By performing image processing on the learning image data LG IP For example, the learning image data LG IN On the other hand, the learning image data LG DP By performing image processing to approximate the learning image data LG IP The image processing unit 44 also has a function of generating the bright spot correction image data BCG IN The same image processing is performed on the bright spot correction image data BCG IP It has the function of generating

[0062] The image processing unit 44 receives, for example, learning image data LG IP The sum of the gradation values ​​of the learning image data LG DP The difference between the sum of the gradation values ​​of and is the learning image data LG IP The sum of the gradation values ​​of the learning image data LG IN The learning image data LG IN The image processing unit 44 has a function of converting the gradation value of the learning image data L IP The sum of the gradation values ​​of the learning image data LG DP The learning image data LG IN The function to convert the gradation value of "Learning Image Data LG IP The sum of the gradation values ​​of the learning image data LG IP It may be the sum of all the gradation values ​​of the "learning image data LG IN The sum of the gradation values ​​of the learning image data LG IN It may be the sum of all the gradation values ​​of "learning image data LG DP The sum of the gradation values ​​of the learning image data LG DP It may be the sum of all the gradation values ​​that the color filter has, or it may be the sum of some of the gradation values.

[0063] In addition, the image processing unit 44 may, for example, IP Learning image data LG DP The peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) for the training image data LG IN The training image data LG IN The image processing unit 44 has a function of converting the gradation value of the learning image data L DPThe learning image data LG is chosen so that the PSNR or SSIM for IN The learning image data LG IP It has the function of generating

[0064] The image processing performed by the image processing unit 44 may be, for example, gamma correction. In this case, the image processing can be performed by setting the gamma value to an appropriate value.

[0065] The image processing unit 44 receives, for example, the bright spot correction image data BCG IP and the bright point correction image data BCG DP The difference between the sum of the gradation values ​​of and is the bright point correction image data BCG IP The sum of the gradation values ​​of the bright spot correction image data BCG IN The brightness correction image data BCG is calculated so that the difference between the sum of the gradation values ​​of IN The function of converting the gradation values ​​of the learning image data LG IP Bright spot correction image data BCG IP and read it as learning image data LG DP Bright spot correction image data BCG DP and read it as learning image data LG IN Bright spot correction image data BCG IN The above explanation can be applied by reading it as follows.

[0066] The image generating unit 45 generates learning image data LG IP For example, the database image data DG IN has first gradation values ​​of m rows and n columns, and database image data DG DP has second gradation values ​​in m rows and n columns, and the learning image data L IP has m rows and n columns of gradation values. In this case, the image generating unit 45 generates the learning image data LG IPBased on the gradation values ​​in the first row, first column to the m-th row, n-th column of the learning image data LG IP For example, the table T may be a table of k (k is an integer of 2 or more) database image data DG. IN and k database image data DG DP In this case, for example, the learning image data LG IP The image generating unit 45 can select a second gradation value that is equal to or closest to the gradation value in the i-th row and j-th column (i is an integer between 1 and m, and j is an integer between 1 and n) of the bright spot correction image data BCG from among the k second gradation values ​​in the i-th row and j-th column. IP The image extraction unit 43 has a function of selecting the second gradation value included in the table T by a method similar to the above method based on the gradation value of the bright spot correction image data BCG generated by the image extraction unit 43. DP When the bright spot correction image data BCG is supplied to the image generating unit 45, DP Based on the gradation value of , the second gradation value included in table T can be selected by a method similar to the above method.

[0067] In this specification and the like, for example, the gradation value in the i-th row and j-th column may be referred to as the "gradation value at coordinates (i, j)".

[0068] In addition, the image generating unit 45 generates learning image data LG IP The learning image data LG is image data including a first gradation value corresponding to a second gradation value selected based on GEN Similarly, the image generating unit 45 has a function of generating the bright spot correction image data BCG IP The function of generating the image data for bright spot correction BCG_1 is image data including the first gradation value corresponding to the second gradation value selected based on the learning image data LG IPIn the case where the table T contains a plurality of second gradation values ​​in the i-th row and j-th column that match the gradation value in the i-th row and j-th column of the learning image data LG, one of the plurality of second gradation values ​​can be selected. Then, the first gradation value corresponding to the selected second gradation value is calculated as the learning image data LG. GEN Bright spot correction image data BCG IP The same applies to the following:

[0069] Also, for example, the learning image data LG IP If the table T does not include a second gradation value in the i-th row and j-th column that matches the gradation value in the i-th row and j-th column of the learning image data LG GEN The gradation value in the i-th row and j-th column of the learning image data LG IP The bright spot correction image data BCG can be set to the same value as the gradation value in the i-th row and j-th column of the bright spot correction image data BCG. IP The same applies to the following:

[0070] The learning unit 46 receives learning image data LG IN And learning image data LG GEN For example, the learning unit 46 generates a machine learning model MLM using the learning image data LG IN The image data output when the input is the learning image data LG GEN The learning unit 46 has a function of generating a machine learning model MLM that matches the learning image data LG IN And learning image data LG GEN and the function of generating the machine learning model MLM by supervised learning using the above. In this way, the machine learning model MLM can be generated by learning. Therefore, the machine learning model MLM generated by the learning unit 46 can be said to be a trained machine learning model.

[0071] The machine learning model MLM generated by the learning unit 46 is supplied to the machine learning processing unit 23. The machine learning processing unit 23 can perform image processing on the image data by performing inference based on the machine learning model MLM.

[0072] 2B1 and 2B2 are circuit diagrams showing a configuration example of the pixel 24 shown in Fig. 2A. Specifically, these are circuit diagrams showing configuration examples of sub-pixels included in the pixel 24. The pixel 24 shown in Fig. 2B1 has a transistor 161, a transistor 171, a capacitor 173, and a light-emitting element 170. In the pixel 24 shown in Fig. 2B1, the light-emitting element 170 can be used as a display element.

[0073] One of a source or a drain of the transistor 161 is electrically connected to a gate of a transistor 171. The gate of the transistor 171 is electrically connected to one electrode of a capacitor 173. One of a source or a drain of the transistor 171 is electrically connected to one electrode of a light-emitting element 170.

[0074] The other of the source and the drain of the transistor 161 is electrically connected to a wiring 126. The gate of the transistor 161 is electrically connected to a wiring 134. The other of the source and the drain of the transistor 171 and the other electrode of the capacitor 173 are electrically connected to a wiring 174. The other electrode of the light-emitting element 170 is electrically connected to a wiring 175.

[0075] A constant potential can be supplied to the wiring 174 and the wiring 175. For example, as shown in FIG. 2B1, when an anode of the light-emitting element 170 is electrically connected to one of the source and the drain of the transistor 171 and a cathode of the light-emitting element 170 is electrically connected to the wiring 175, a high potential can be supplied to the wiring 174 and a low potential can be supplied to the wiring 175.

[0076] The light emitting element 170 can be, for example, an organic EL element or an inorganic EL element.

[0077] When the pixel 24 provided in the display unit 22 has the configuration shown in Fig. 2B1, an image can be displayed on the display unit 22 by controlling the magnitude of the current flowing through the light-emitting element 170 to control the light emission luminance of the light-emitting element 170. The larger the current flowing through the light-emitting element 170, the greater the light emission luminance of the light-emitting element can be.

[0078] The pixel 24 shown in Fig. 2B2 includes a transistor 162, a capacitor 181, and a liquid crystal element 180. In the pixel 24 shown in Fig. 2B2, the liquid crystal element 180 can be a display element.

[0079] One of the source and the drain of the transistor 162 is electrically connected to one electrode of the liquid crystal element 180. One electrode of the liquid crystal element 180 is electrically connected to one electrode of the capacitor 181.

[0080] The other of the source and the drain of the transistor 162 is electrically connected to a wiring 126. The gate of the transistor 162 is electrically connected to a wiring 134. The other electrode of the capacitor 181 is electrically connected to a wiring 182. The other electrode of the liquid crystal element 180 is electrically connected to a wiring 183.

[0081] A constant potential can be supplied to the wiring 182 and the wiring 183. A low potential, for example, can be supplied to the wiring 182 and the wiring 183.

[0082] When the pixel 24 provided in the display unit 22 has the configuration shown in FIG. 2B2, the liquid crystal molecules included in the liquid crystal element 180 are aligned according to a voltage applied between both electrodes of the liquid crystal element 180. The liquid crystal molecules can transmit light from, for example, a backlight unit that can be included in the display device 20, depending on the degree of alignment. As described above, the other electrode of the liquid crystal element 180 is electrically connected to the wiring 183 and a constant potential is supplied to it. As described above, by controlling the potential of one electrode of the liquid crystal element 180, the pixel 24 can emit light with a luminance according to the potential, and an image can be displayed on the display unit 22.

[0083] FIG. 3A shows the content image data CG IN When the content image G_1 is directly input to the display unit 22, the content image G_1 is displayed on the display unit 22. DP FIG. 3B is a schematic diagram showing an example of content image data CG IN When the content image G_2 is input to the display unit 22 via the machine learning processing unit 23 and the bright spot correction unit 50, the content image G_2 is displayed on the display unit 22. DP FIG.

[0084] Content image data CG IN is input to the display unit 22 without image processing, etc., the display unevenness, bright spots, etc. may occur as described above. Fig. 3A shows a state in which the display unevenness 25 and bright spots 51 have occurred in the image displayed on the display unit 22.

[0085] Content image data CG IN By performing image processing using the machine learning model MLM, the machine learning processing unit 23 generates content image data CG that cancels out the display unevenness. ML In FIG. 3B, the machine learning processing unit 23 generates the content image data CG IN By adding data to the data corresponding to the area 26 to cancel the display unevenness 25, the content image data CG ML For example, when the luminance of a portion where display unevenness 25 occurs is higher than the luminance of the surrounding area of ​​the portion, the luminance of the region 26 can be made lower than the luminance of the surrounding area of ​​the region 26.

[0086] In addition, by performing image processing on the content image data based on the bright spot correction image data BCG (for example, one of the bright spot correction image data BCG_1 or the bright spot correction image data BCG_2), the bright spot correction unit 50 corrects the bright spots to make them less noticeable by generating content image data CG COR For example, content image data CG CORIn FIG. 3B, the content image data CG ML The bright spot correction unit 50 adds data for making the bright spot 51 less noticeable to the data corresponding to the area 52 where the bright spot 51 occurs, thereby correcting the content image data CG COR This shows how to generate the

[0087] As described above, as shown in FIG. 3B, the content image data CG IN In contrast, the machine learning processing unit 23 and the bright spot correction unit 50 perform image processing, so that the display unit 22 can display an image in which display unevenness and bright spots are not noticeable.

[0088] As described above, when the display device 20 is enlarged and the area of ​​the display unit 22 is increased, display unevenness is likely to occur. Furthermore, when the pixels 24 provided in the display unit 22 are miniaturized and the pixel density of the display unit 22 is increased, the variation in characteristics of the display elements and transistors, etc., of the pixels 24 among the pixels 24 increases, and display unevenness is likely to occur. In one embodiment of the present invention, display unevenness displayed in the display unit 22 can be made less noticeable. As described above, according to one embodiment of the present invention, the display device 20 can be enlarged while suppressing the display unevenness from being visually recognized in the image displayed on the display unit 22. Furthermore, while suppressing the display unevenness from being visually recognized in the image displayed on the display unit 22, the density of the pixels 24 provided in the display unit 22 can be increased, and a high-definition image can be displayed on the display unit 22.

[0089] As described above, the pixel 24 may become a bright spot or a dark spot due to a defective characteristic or deterioration of the display element, transistor, etc., that the pixel 24 has. When viewing an image displayed on the display unit 22, a bright spot is more noticeable than a dark spot, and therefore has a large impact on visibility. In one aspect of the present invention, a bright spot is corrected by the bright spot correction unit 50 or the like to become, for example, a dark spot, thereby making it possible to display a high-quality image on the display unit 22. The correction of the bright spot can also be performed by the machine learning processing unit 23.

[0090] In the image processing system 10 having the configuration shown in FIG. 1, a learning unit 46 having a function of generating a machine learning model MLM can be provided in the generation device 40, and a machine learning processing unit 23 having a function of processing using the machine learning model MLM can be provided in the display device 20. This allows the display device 20 to perform processing using the machine learning model MLM even if the display device 20 does not generate the machine learning model MLM. A large amount of learning image data LG IN , and learning image data LG GEN For this reason, by providing the learning unit 46 in the generating device 40, the computing power of the display device 20 can be made lower than the computing power of the generating device 40.

[0091] <How to generate a machine learning model> A method for generating the machine learning model MLM will be described below with reference to the drawings. As shown in FIG. 2A, the display unit 22 has pixels 24 arranged in a matrix of m rows and n columns. The gradation value of the image data is an 8-bit digital value, and the smaller the gradation value, the lower the luminance of the light emitted from the pixel 24. For example, when the gradation value can be an integer between 0 and 255, the luminance of the light emitted from the pixel 24 is the smallest when the gradation value is 0.

[0092] [Table generation method] Fig. 4 is a flowchart showing an example of a method for generating a table T stored in the database 42. As shown in Fig. 4, the table T is generated by a method shown in steps S01 to S04. Figs. 5A and 5B, and Figs. 6A and 6B are schematic diagrams showing the operations in steps S01 to S04.

[0093] To generate a table T, first, the database image data DG IN The database image data DG input to the input unit 21 is IN is input to the display unit 22, and the database image data DGIN Specifically, an image corresponding to the database image data DG IN An image is displayed on the display unit 22 by the pixels 24 emitting light of a luminance corresponding to the first gradation value in the m rows and n columns.

[0094] In FIG. 5A, database image data DG IN It is assumed that the entire surface of the display unit 22 displays an image with the same luminance. In other words, it is assumed that all pixels 24 emit light with the same luminance. On the other hand, in the image displayed on the display unit 22, the luminance of light emitted from some pixels 24 differs from the luminance of light emitted from other pixels 24. In other words, display unevenness occurs. In FIG. 5A, the generated display unevenness is shown as display unevenness 27.

[0095] Next, the image displayed on the display unit 22 is captured by the imaging device 30. As a result, the imaging device 30 captures the captured image data IMG DG (Step S02).

[0096] Here, when an image displayed on the display unit 22 of the display device 20 is captured by the imaging device 30, an object other than the display unit 22 may be captured. For example, the housing of the display device 20 may be captured. In FIG. 5A, the imaging data IMG DG It is assumed that the portion of display device 20 shown in FIG. 5A enclosed by a dashed line is included.

[0097] Thereafter, the image extraction unit 43 extracts the image data IMG DG From database image data DG DP Specifically, the imaging data IMG DG 5B, in addition to the image displayed on the display unit 22, the housing of the display device 20 extracts the image data IMG DG If included in the image data IMG DGand removes the data representing the housing of the display device 20. DP Acquire the image data IMG DG The extraction of data from can be performed by pattern matching, template matching, etc., as described above.

[0098] As mentioned above, the database image data DG IN The resolution of the image represented by and the database image data DG DP For example, the resolution of the database image data DG IN has m rows and n columns of first gradation values, the database image data DG DP However, for example, when the data representing the image displayed on the display unit 22 is taken as the image data IMG DG When extracted from, the gradation values ​​of the extracted data may not be m rows and n columns. For example, there may be cases where there are fewer than m rows of gradation values, or there may be cases where there are more than m rows of gradation values. Also, there may be cases where there are fewer than n columns of gradation values, or there may be cases where there are more than n columns of gradation values.

[0099] As shown above, the image data IMG DG If the gradation values ​​of the data extracted from are not m rows and n columns, the image extraction unit 43 performs up-conversion or down-conversion on the data, so that the database image data DG output from the image extraction unit 43 DP For example, it is preferable that the image extraction unit 43 extracts the image data IMG DG If the data extracted from the image data IMG has fewer gradation values ​​than m rows or fewer than n columns, the image extraction unit 43 extracts the image data IMG DG The image extraction unit 43 can perform up-conversion on the data extracted from the image data IMG DG If the data extracted from the image data IMG has more than m rows or more than n columns of gradation values, the image extraction unit 43 extracts the image data IMGDG Down conversion can be performed on the data extracted from the database image data DG DP The number of rows and columns of the second gradation values ​​included in the database image data DG IN The number of rows and columns of the first gradation values ​​included in the first gradation value can be m rows and n columns. As described above, the up-conversion and down-conversion can be performed by the nearest neighbor method, the bilinear method, the bicubic method, or the like.

[0100] And the database image data DG IN and database image data DG DP A table T representing information on the correspondence between and is stored in the database 42 (step S04). As described above, the table T is specifically stored in the database image data DG IN and the database image data DG DP and a second gradation value that the first gradation value has. Table T represents, for example, a first gradation value and a second gradation value at coordinates corresponding to the coordinates of the first gradation value. Table T represents, for example, a first gradation value and a second gradation value at coordinates that are the same as the coordinates of the first gradation value.

[0101] 6A is a diagram showing an example of a table T. In FIG. 6A, to the left of the arrow, database image data DG IN The first gradation value of the database image data DG DP 4 shows the second gradation value that the

[0102] Here, it is assumed that pixel 24 has a function of emitting red light, green light, and blue light. In this case, the image data has a gradation value (red gradation value) that represents the luminance of red light, a gradation value (green gradation value) that represents the luminance of green light, and a gradation value (blue gradation value) that represents the luminance of blue light. For example, the red gradation value of the i-th row and j-th column is R, the green gradation value is G, and the blue gradation value is B, which is expressed as [R,G,B]. (i,j)For example, if the red gradation value of the first row, first column to the mth row, nth column is R, the green gradation value is G, and the blue gradation value is B, that is, all the red gradation values ​​are R, all the green gradation values ​​are G, and all the blue gradation values ​​are B, then it is represented as [R,G,B]. (1,1)~(m,n) The light emitted by the pixel 24 is not limited to red light, green light, and blue light. For example, the pixel 24 may emit white light. The pixel 24 may emit cyan light, magenta light, and yellow light. The pixel 24 does not have to emit red light, green light, or blue light. Furthermore, the number of colors of light emitted by the pixel 24 is not limited to three colors, and the pixel 24 may emit one or two colors of light, or may emit four or more colors of light.

[0103] When the red, green, and blue gradation values ​​are each 8-bit digital data, the values ​​that R, G, and B can take can each be integers from 0 to 255. Here, if the smaller the gradation value, the lower the brightness of the light emitted from the pixel 24, then a gradation value of 0 can represent that no light is emitted. Thus, for example, [0,0,0] can represent that no red, green, or blue light is emitted from any of the pixels 24. (1,1)~(m,n) This can be shown by writing:

[0104] From the above, the database image data DG IN The first gradation value is [0,0,0] (1,1)~(m,n) In this case, it can be assumed that no light is emitted from the pixels 24(1,1) to 24(m,n). DP The second tone value of the (1,1)~(m,n) It can be said that:

[0105] In the method for generating a machine learning model using the image processing system 10, the first gradation value is [1,0,0] (1,1)~(m,n) 〜[255,0,0] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DGDP Also, if the first gradation value is [0,1,0] (1,1)~(m,n) 〜[0,255,0] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DG DP In addition, if the first gradation value is [0,0,1], (1,1)~(m,n) 〜[0,0,255] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DG DP That is, the second gradation value of the database image data DG IN The image represented by can be, for example, an image having a single color and uniform brightness all over.

[0106] In this specification, the term "monochrome" refers to a color represented by a pixel emitting light of one color. For example, if a pixel has a function of emitting red light, green light, and blue light, a red image, a green image, and a blue image are called monochrome images.

[0107] In this specification, for example, database image data DG IN In the case where the red gradation value in the i-th row and j-th column is 1, the corresponding database image data DG DP The red gradation value in R1 DP For example, the database image data DG IN In the case where the red gradation value in the i-th row and j-th column is 255, the corresponding database image data DG DP The red gradation value in R255 DP For example, the database image data DG IN In the case where the green gradation value in the i-th row and j-th column is 1, the corresponding database image data DG DP The green gradation value in G1 DP For example, the database image data DG INIn the case where the green gradation value in the i-th row and j-th column is 255, the corresponding database image data DG DP The green gradation value in G255 DP For example, the database image data DG IN In the case where the blue gradation value in the i-th row and j-th column is 1, the corresponding database image data DG DP The blue gradation value in B1 DP (i,j) is written as (i,j). Furthermore, for example, the database image data DG IN In the case where the blue gradation value in the i-th row and j-th column is 255, the corresponding database image data DG DP The blue gradation value in B255 DP This is indicated as (i,j).

[0108] Here, the first gradation value and the second gradation value corresponding to the first gradation value are not necessarily the same. For example, in the database image data DG IN The first gradation value is [128,0,0] (1,1)~(m,n) Even if it was, R128 DP (1,1) to R128 DP (m,n) is not always 128. It may be greater than 128 or smaller. For example, R128 DP (1,1) to R128 DP (m, n) are not necessarily all the same value. In other words, as mentioned above, DP In the image displayed by the image display device, display unevenness and the like may occur.

[0109] In addition, for all gradation values, the database image data DG IN is input to the display device 20, and the database image data DG DP For example, it is not necessary to obtain the second gradation value of [1,0,0]. (1,1)~(m,n) 〜[255,0,0] (1,1)~(m,n) Among them, some of the gradation values ​​are database image data DG IN is input to the display device 20, and the database image data DG DPIn addition, the second gradation value having the gradation value [0,1,0] may be obtained. (1,1)~(m,n) 〜[0,1,0] (1,1)~(m,n) Among them, some of the gradation values ​​are database image data DG IN is input to the display device 20, and the database image data DG DP In addition, a second gradation value having a gradation value [0,0,255] may be obtained. (1,1)~(m,n) 〜[0,0,255] (1,1)~(m,n) Among them, some of the gradation values ​​are database image data DG IN is input to the display device 20, and the database image data DG DP may also obtain a second gradation value that the image signal has.

[0110] As shown above, the database image data DG for some gradation values IN is input to the display device 20, and the database image data DG DP When acquiring the second gradation value that the display device 20 has, the database image data DG IN Database image data DG corresponding to DP The second gradation value of the database image data DG IN It can be calculated based on the first gradation value and the second gradation value corresponding to the first gradation value. For example, it can be calculated by proportional interpolation. It can also be calculated using a predetermined formula.

[0111] For example, the tone value [127,0,0] (1,1)~(m,n) Database image data DG DP and [129,0,0] (1,1)~(m,n) Database image data DG DP is input to the display device 20, but the grayscale value [128,0,0] (1,1)~(m,n) Database image data DG DP is not input to the display device 20. For example, R127 DP The value of (i,j) is 120, R129 DP Let the value of (i,j) be 124, and R128 DP (i,j) to R127 DP(i,j) and R129 DP The calculation is done by proportional interpolation of (i,j). In this case, R128 DP The value of (i, j) can be set to 122. The green gradation value and the blue gradation value can also be calculated in a similar manner.

[0112] Database image data DG for some gradation values IN By inputting these values ​​to the display device 20, it is possible to reduce the number of calculations required to generate the table T. This allows the table T to be generated in a short time.

[0113] In FIG. 6A, database image data DG IN The image represented by is a monochrome image. IN and database image data DG DP The red gradation value, the green gradation value, and the blue gradation value of the database image data DG are acquired separately. However, this is not a limited embodiment of the present invention. FIG. 6B is a modified example of FIG. 6A. IN and database image data DG DP This differs from the case shown in FIG. 6A in that the red gradation value, the green gradation value, and the blue gradation value are acquired at the same time.

[0114] In the case shown in FIG. 6B, the first gradation value is [0,0,0]. (1,1)~(m,n) ~[255,255,255] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DG DP That is, the second gradation value of the database image data DG IN is defined as an image in which the red, green, and blue gradation values ​​are all the same (white image), and the database image data DG DP The red, green, and blue gradation values ​​of the image are obtained. The obtained gradation values ​​are represented by a table T. Specifically, the table T is a database of image data DG INand the database image data DG of the red gradation value and the coordinates corresponding to the coordinates of the red gradation value. DP The table T represents the red gradation value of the database image data DG IN and the database image data DG DP Furthermore, the table T represents the green gradation value of the database image data DG IN and the database image data DG DP The table T represents, for example, the database image data DG IN and the database image data DG DP The table T represents, for example, the database image data DG IN and the database image data DG DP and the green gradation value of the image data in the database DG IN and the database image data DG DP and the blue gradation value that

[0115] Database image data DG IN By making the image represented by the database image data DG input to the display device 20 a white image, IN Therefore, the number of calculations required to generate table T can be reduced. As a result, table T can be generated in a short time. IN The red, green, and blue gradation values ​​of the database image data DG IN Among the red, green, and blue gradation values ​​of the database image data DG, the gradation value of one color may be different from the gradation values ​​of the other colors. INThe red, green and blue gradation values ​​may be different from each other.

[0116] [How to generate a machine learning model] Fig. 7 is a flowchart showing an example of a method for generating a machine learning model MLM using a table T stored in the database 42. As shown in Fig. 7, the machine learning model MLM is generated by a method shown in steps S11 to S16. Figs. 8A, 8B, 9A, 9B, and 10 are schematic diagrams showing the operations in steps S11 to S16.

[0117] To generate a machine learning model MLM, first, we need to create learning image data LG IN The learning image data L IN is input to the display unit 22, and the learning image data LG IN Specifically, an image corresponding to the learning image data LG IN The pixels 24 emit light with a luminance corresponding to the gradation values ​​of m rows and n columns, thereby displaying an image on the display unit 22. Fig. 8A shows a state in which display unevenness 28 occurs in the image displayed on the display unit 22.

[0118] Next, the image displayed on the display unit 22 is captured by the imaging device 30. As a result, the imaging device 30 captures the captured image data IMG LG (Step S12).

[0119] As described above, when an image displayed on the display unit 22 of the display device 20 is captured by the imaging device 30, an object other than the display unit 22 may be captured. For example, the housing of the display device 20 may be captured. In FIG. 8A, the imaging data IMG LG It is assumed that the portion of display device 20 shown in FIG. 8A enclosed by a dashed line is included.

[0120] Thereafter, the image extraction unit 43 extracts the image data IMG LG Learning image data LGDP (Step S13). The operation in step S13 is to obtain the image data IMG DG Image data IMG LG and database image data DG DP The learning image data LG DP 5B as FIG. 8B, the explanation of the operation in step S03 can be referred to.

[0121] Next, the learning image data LG IN On the other hand, the learning image data LG DP The image processing unit 44 performs image processing so that the learning image data LG IP (Step S14). FIG. 9A shows the learning image data LG IN and learning image data LG DP and the learning image data LG output from the image processing unit 44. IP An example is shown below.

[0122] Learning image data LG IP The generation of learning image data LG IP The sum of the gradation values ​​of the learning image data LG DP The difference between the sum of the gradation values ​​of and is the learning image data LG IP The sum of the gradation values ​​of the learning image data LG IN The learning image data LG IN For example, the learning image data LG IP The sum of the gradation values ​​of the learning image data LG DP The learning image data LG IN The learning image data LG IP can be generated.

[0123] In addition, the learning image data LG IP The generation of learning image data LGDP The PSNR or SSIM for the learning image data LG IN The training image data LG IN For example, the learning image data LG DP The learning image data LG is chosen so that the PSNR or SSIM for IN The learning image data LG IP can be generated.

[0124] As described above, the image processing performed by the image processing unit 44 may be, for example, gamma correction. In this case, the image processing can be performed by setting the gamma value to an appropriate value.

[0125] Here, the image processing unit 44 receives the learning image data LG IN It is preferable to perform image processing for each color. IN When performing gamma correction on learning image data L IP The sum of the red gradation values ​​of the learning image data LG DP The difference between the sum of the red gradation values ​​of and is the learning image data LG IP The sum of the red gradation values ​​of the learning image data LG IN The learning image data LG IN It is preferable to convert the red gradation value of the learning image data L IP The sum of the green gradation values ​​of the learning image data LG DP The difference between the sum of the green gradation values ​​of and is the learning image data LG IP The sum of the green gradation values ​​of the learning image data LG IN The learning image data LG INIt is preferable to convert the green gradation value of the learning image data L IP The sum of the blue gradation values ​​of the learning image data LG DP The difference between the sum of the blue gradation values ​​of and is the learning image data LG IP The sum of the blue gradation values ​​of the learning image data LG IN The learning image data LG IN It is preferable to convert the blue gradation value of the learning image data LG IP The sum of the red gradation values ​​of the learning image data LG DP The learning image data LG IN It is preferable to convert the red gradation value of the learning image data L IP The sum of the green gradation values ​​of the learning image data LG DP The learning image data LG IN It is preferable to convert the green gradation value of the learning image data L IP The sum of the blue gradation values ​​of the learning image data LG DP The learning image data LG IN It is preferable to convert the blue gradation value of the color component by image processing.

[0126] As mentioned above, "Learning Image Data LG IP The sum of the gradation values ​​of the learning image data LG IP It may be the sum of all the gradation values ​​of the m rows and n columns of the "learning image data LG IN The sum of the gradation values ​​of the learning image data LG IN It may be the sum of all the gradation values ​​of the m rows and n columns of the "learning image data LG DP The sum of the gradation values ​​of the learning image data LGDP It may be the sum of all the gradation values ​​of m rows and n columns of the gradation values ​​of the pixel 100, or the sum of some of the gradation values.

[0127] In this specification, learning image data LG IP The red gradation value in the i-th row and j-th column of IP In addition, the learning image data LG IP The green gradation value in the i-th row and j-th column of IP (i,j) are written as (i,j). Furthermore, the learning image data LG IP The blue gradation value in the i-th row and j-th column of IP This is indicated as (i,j).

[0128] As described above, the learning image data LG output from the image processing unit 44 IP The image represented by is the learning image data LG DP On the other hand, the learning image data LG IN Since the image represented by does not include the display unevenness 28, the learning image data LG IP The image represented by does not include the display irregularity 28.

[0129] Then, based on the table T, the image generating unit 45 generates the learning image data LG IP Learning image data LG GEN 9B shows that the image generating unit 45 generates the table T and the learning image data LG IP is input, and the learning image data LG GEN The figure shows how the following is output:

[0130] Specifically, first, the learning image data LG IP Based on the gradation value of the learning image data LG, the second gradation value included in the table T is selected. IP The second gradation value that is the same as or closest to the gradation value of R is selected for each of the 1st row, 1st column to the mth row, nth column. IP The red gradation value that matches or is closest to (i,j) is R0DP (i,j) to R255 DP (i,j). For example, G IP The green gradation value that matches or is closest to (i,j) is G0 DP (i,j) to G255 DP (i,j). Furthermore, for example, B IP The blue gradation value that matches or is closest to (i,j) is B0 DP (i,j) to B255 DP In FIG. 9B, the selected second gradation value in the first row and first column is selected as [Ra DP (1,1),Gb DP (1,1),Bc DP (1,1)] (1,1) (a, b, and c are integers between 0 and 255). The second gradation value selected in the mth row and nth column is expressed as [Rs DP (m,n),Gt DP (m,n),Bu DP (m,n)] (m,n) (s, t, and u are integers between 0 and 255 inclusive).

[0131] Next, the image generating unit 45 generates learning image data LG 1 , which is image data including a first gradation value corresponding to the selected second gradation value, based on the table T. GEN Specifically, the image generating unit 45 generates learning image data LG, which is image data including the first gradation values ​​in the first row, first column to the mth row, nth column corresponding to the selected second gradation values ​​in the first row, first column to the mth row, nth column. GEN For example, in the example shown in FIG. 9B, the learning image data L GEN In the first row and first column, the red gradation value is a, the green gradation value is b, and the blue gradation value is c. In addition, the mth row and nth column has a red gradation value s, a green gradation value t, and a blue gradation value u.

[0132] For example, the learning image data LG IP If the table T does not include a second gradation value in the i-th row and j-th column that matches the gradation value in the i-th row and j-th column of the learning image data LGGEN The gradation value in the i-th row and j-th column of the learning image data LG IP The value can be the same as the gradation value of the i-th row and j-th column of R IP The red gradation value that matches (i,j) is R0 DP (i,j) to R255 DP If it is not included in (i,j), the learning image data LG GEN The red gradation value in the i-th row and j-th column of IP (i,j). For example, G IP The green gradation value that matches (i,j) is G0 DP (i,j) to G255 DP If it is not included in (i,j), the learning image data LG GEN The green gradation value in the i-th row and j-th column of IP (i,j). Furthermore, for example, B IP The blue gradation value that matches (i,j) is B0 DP (i,j) to B255 DP If it is not included in (i,j), the learning image data LG GEN The blue gradation value in the i-th row and j-th column of IP It can be (i,j).

[0133] As mentioned above, the learning image data LG IP The image represented by the database image data DG having the second gradation value does not include display unevenness. DP includes display unevenness, etc. Also, database image data DG having a first gradation value IN does not include display irregularities. IP The learning image data LG is image data including a first gradation value corresponding to a second gradation value selected based on the gradation value of GEN is the learning image data LG DP In FIG. 9B and other figures, the learning image data LG GEN1, data that counteracts display unevenness 28 is added to the data corresponding to region 29. For example, if the luminance of the portion where display unevenness 28 occurs is higher than the luminance of the surrounding area of ​​said portion, the luminance of region 29 can be made lower than the luminance of the surrounding area of ​​region 29.

[0134] After step S15, the learning unit 46 receives the learning image data LG IN And learning image data LG GEN A machine learning model MLM is generated using the above (step 16). For example, the learning image data LG IN The image data output when the input is the learning image data LG GEN Such a machine learning model MLM is generated based on, for example, the learning image data LG IN And learning image data LG GEN The learning image data LG IN And learning image data LG GEN and are input to the learning unit 46, and the image data output from the learning unit 46 is the learning image data LG GEN In this example, the machine learning model MLM is generated so that the image data is the same as that in the example shown in FIG. As described above, a neural network model, for example, can be applied as the machine learning model MLM.

[0135] The above is an example of a method for generating the machine learning model MLM using the image processing system 10. As described above, by performing image processing on the image data input to the input unit 21 using the machine learning model MLM, the machine learning processing unit 23 can generate image data that cancels out the display unevenness. By inputting the image data output from the machine learning processing unit 23 to the display unit 22, the display unit 22 can display an image in which the display unevenness is made less noticeable. Therefore, as described above, the display device 20 can be enlarged while suppressing the display unevenness from being visually recognized in the image displayed on the display unit 22. In addition, while suppressing the display unevenness from being visually recognized in the image displayed on the display unit 22, the density of the pixels 24 provided in the display unit 22 can be increased, and a high-definition image can be displayed on the display unit 22.

[0136] Furthermore, by performing image processing using the machine learning model MLM, in addition to the display unevenness described above, it is possible to cancel out factors that degrade the image quality of the displayed image. For example, line defects, point defects, etc. can be canceled out. Therefore, the display unit 22 can display a high-quality image.

[0137] Here, even if image data is input to the input unit 21 and image data generated by the image generating unit 45 is input to the display unit 22 by performing the same processing as steps S11 to S15, an image in which display unevenness and the like are not noticeable can be displayed on the display unit 22. However, high computing power is required to perform steps S11 to S15. On the other hand, image processing using the generated machine learning model MLM can be performed with a lower computing power than steps S11 to S15. Therefore, image processing can be performed in a short time by performing image processing using the machine learning model MLM. In addition, image processing can be performed inside the display device 20 without using a device with high computing power such as a server.

[0138] If the image processing for making the display unevenness less noticeable on the image data input to the input unit 21 can be performed by, for example, a device having high computing power, the image processing may be performed by a method similar to steps S11 to S15 without using the machine learning model MLM. Also, if the computing power of the display device 20 is sufficiently high, for example, the image processing for making the display unevenness less noticeable on the image data input to the input unit 21 may be performed by a method similar to steps S11 to S15 without using the machine learning model MLM. When performing image processing without using the machine learning model MLM, the generating device 40 may be configured not to include the learning unit 46.

[0139] <Bright spot correction method> Hereinafter, a bright spot correction method, which is an image processing method according to one embodiment of the present invention, will be described with reference to the drawings.

[0140] In a bright spot correction method according to one aspect of the present invention, after detecting a pixel 24 that will become a bright spot, correction is performed on the detected pixel 24. Figures 11A and 11B are diagrams showing an example of a method for detecting a pixel 24 that will become a bright spot. Here, the method shown in Figure 11A is referred to as method M1, and the method shown in Figure 11B is referred to as method M2. It is noted that in Figures 11A and 11B, it is assumed that there is no display unevenness or the like.

[0141] In method M1, first, the bright spot correction image data BCG_1 IN is input to the input unit 21 of the display device 20. Then, the learning image data L IN Bright spot correction image data BCG_1 IN And the image data IMG LG Image data IMG BCG And learning image data LG DP Bright spot correction image data BCG DP And learning image data LG IP Bright spot correction image data BCG IP And learning image data LG GENand the bright spot correction image data BCG_1 are read as the bright spot correction image data BCG_2, and the same operations as steps S11 to S15 shown in FIG. 7 and the like are performed (steps S11' to S15'). For example, in steps S11' and S12', IN An image corresponding to the image is displayed on the display unit 22, and the image displayed on the display unit 22 is captured by the imaging device 30 to generate imaging data IMG BCG In addition, the bright spot correction image data BCG obtained in step S13' is obtained. DP can have m rows and n columns of gradation values. Furthermore, in step S15′, the database image data DG IN and the database image data DG DP Based on a table T representing information on the correspondence between the second gradation value of and the bright spot correction image data BCG IP The bright spot correction image data BCG_1 is generated from the bright spot correction image data BCG DP In this case, in step S15′, the image generating unit 45 does not receive the bright spot correction image data BCG DP The bright spot correction image data BCG_1 is generated from the

[0142] Here, when the pixel that becomes a bright spot is included in pixels 24(1,1) to 24(m,n), the bright spot correction image data BCG DP Among the m rows and n columns of gradation values ​​of the bright spot correction image data BCG, the gradation value of the coordinates corresponding to the coordinates of the pixel 24 that is a bright spot, for example, the gradation value of the same coordinates as the pixel 24 that is a bright spot, is high. DP can take any integer value from 0 to 255, the gradation value of the coordinates corresponding to the coordinates of the pixel 24 that is a bright spot will be 255 or a value close to that. DP is higher than the gradation values ​​of the coordinates surrounding the coordinates, the gradation value can be lowered in the bright spot correction image data BCG_1.

[0143] After step S15', the bright spot correction unit 50 of the display device 20 detects bright spot coordinates, which are the coordinates of the pixel 24 that becomes a bright spot, based on the bright spot correction image data BCG_1. Specifically, the coordinates of the gradation values ​​of m rows and n columns of the bright spot correction image data BCG_1 that are equal to or less than a threshold value can be set as the bright spot coordinates. The detection of the bright spot coordinates may be performed by the generation device 40. The detection of the bright spot coordinates may be performed, for example, by the image generation unit 45 of the generation device 40.

[0144] Here, the bright spot correction image data BCG DP In both the bright spot correction image data BCG_1 and the bright spot correction image data BCG_2, it is preferable that the difference between the gradation value of the bright spot coordinate and the gradation value of the coordinates surrounding the bright spot coordinate is large, since the bright spot coordinate can be detected with high accuracy. IN It is preferable that the gradation value of the bright spot correction image data BCG_1 is an intermediate gradation. IN It is preferable that all of the gradation values ​​of m rows and n columns of the bright spot correction image data BCG_1 be 127 or close to that. IN The gradation values ​​of all the bright point correction image data BCG are set to the same intermediate gradation value. DP 11A shows an example in which a gradation value 53 higher than the surrounding gradation values ​​is included as a bright spot. In the example shown in Fig. 11A, among the gradation values ​​of the bright spot correction image data BCG_1, gradation value 54 at the same coordinate as gradation value 53 can be made lower than the surrounding gradation values.

[0145] In method M2, first, the bright spot correction image data BCG_2 IN is input to the input unit 21 of the display device 20. Then, the learning image data L IN Bright spot correction image data BCG_2 IN And the image data IMG LG Image data IMG BCG And learning image data LG DPis read as the bright spot correction image data BCG_2, and the same operations as steps S11 to S13 shown in FIG. 7 and the like are performed (steps S11'' to S13''). For example, in steps S11'' and S12'', IN An image corresponding to the image is displayed on the display unit 22, and the image displayed on the display unit 22 is captured by the imaging device 30 to generate imaging data IMG BCG The bright spot correction image data BCG_2 acquired in step S13'' can have gradation values ​​of m rows and n columns.

[0146] Here, when a pixel that will become a bright spot is included among pixels 24(1,1) to 24(m,n), the gradation value of the coordinates corresponding to the coordinates of pixel 24 that will become a bright spot, for example the gradation value of the same coordinates as pixel 24 that will become a bright spot, among the gradation values ​​of m rows and n columns that image data for bright spot correction BCG_2 has will be high. For example, when the gradation values ​​that image data for bright spot correction BCG_2 has can take any integer value from 0 to 255, the gradation value of the coordinates corresponding to the coordinates of pixel 24 that will become a bright spot will be 255 or a value close to that.

[0147] After step S13'', the bright spot correction unit 50 of the display device 20 detects bright spot coordinates, which are the coordinates of the pixel 24 that becomes a bright spot, based on the bright spot correction image data BCG_2. Specifically, the coordinates of the gradation values ​​of m rows and n columns contained in the bright spot correction image data BCG_2 that are equal to or greater than a threshold value can be set as the bright spot coordinates. The detection of the bright spot coordinates may be performed by the generation device 40. The detection of the bright spot coordinates may be performed, for example, by the image extraction unit 43 of the generation device 40.

[0148] Here, in the bright spot correction image data BCG_2, if the difference between the gradation value of the bright spot coordinate and the gradation value around the bright spot coordinate is large, it is preferable because the bright spot coordinate can be detected with high accuracy. On the other hand, if the gradation value of the image data input to the display unit 22 is too small, even pixels 24 that could become bright spots depending on the gradation value will not become bright spots, and the bright spot coordinate cannot be detected with high accuracy. Bright spot correction image data BCG_2IN It is preferable to determine the gradation value of the bright spot correction image data BCG_2 based on the above. IN It is preferable that all of the gradation values ​​of m rows and n columns of the pixel 1111 are set to 0 or more and 127 or less, or 31 or more and 127 or less, or 63 or more and 127 or less. Fig. 11B shows an example in which the bright spot correction image data BCG_2 includes a bright spot with a gradation value of 55 that is higher than the surrounding gradation values.

[0149] By detecting the coordinates of the bright spots, the bright spot correction unit 50 can have a function of correcting the bright spots. For example, the content image data CG ML The gradation values ​​of the coordinates corresponding to the bright spot coordinates, for example, the same coordinates as the bright spot coordinates, among the gradation values ​​of m rows and n columns of the content image data CG COR By generating and supplying the content image data CG to the display unit 22, for example, a pixel 24 that would be a bright spot can be made a dark spot. As described above, when viewing an image displayed on the display unit 22, a bright spot stands out more than a dark spot, and therefore has a large effect on visibility. COR By displaying an image corresponding to the above on the display unit 22, the image displayed on the display unit 22 can be of high quality.

[0150] FIG. 12A shows the actual measured gradation values ​​of the bright spot correction image data BCG_1 and the IN This is a graph showing the relationship between the brightness correction image data BCG_1 and the gradation value, which can be created by the method M1. IN In FIG. 12A, the same gradation value of the bright spot correction image data BCG_1 IN There are multiple plots for the gradation value of BCG_1, but this is because the actual measured gradation values ​​of the bright spot correction image data BCG_1 are plotted for multiple coordinates.

[0151] The line 56 shown in FIG. 12A represents the bright spot correction image data BCG_1 IN12A shows the average of the gradation values ​​of the plotted image data for bright spot correction BCG_1 at each gradation value. As shown in FIG. 12A, the average of the gradation values ​​of the image data for bright spot correction BCG_1 and the average of the gradation values ​​of the image data for bright spot correction BCG_1 IN The relationship between the gradation value of and can be linearly approximated.

[0152] Here, as described above, the gradation value of the bright spot correction image data BCG_1 at the bright spot coordinates is lower than the gradation value at the non-bright spot coordinates. IN For example, among the gradation values ​​of the image data for bright spot correction BCG_1 of m rows and n columns, the coordinates of the gradation values ​​less than the threshold value can be set as bright spot coordinates. In Fig. 12A, the threshold value is indicated by a line 57. The line 57 can be expressed by a linear equation with a positive slope.

[0153] In the method M1, a plurality of image data for bright spot correction BCG_1 having different gradation values ​​are IN By preparing the above and generating the bright spot correction image data BCG_1 for each of them, it is possible to prevent the coordinates of the pixel 24 that becomes a bright spot from being determined to be not a bright spot coordinate, and the coordinates of the pixel 24 that does not become a bright spot from being determined to be a bright spot coordinate. Therefore, the bright spot correction unit 50 and the like can detect the bright spot coordinates with high accuracy.

[0154] FIG. 12B shows the bright spot correction image data BCG generated by the image extraction unit 43. DP The actual measured gradation value of the bright spot correction image data BCG input to the input unit 21 IN 12B is a graph showing the relationship between the gradation value of the bright spot correction image data BCG DP The gradation value of the bright spot correction image data BCG IN The relationship between the gradation values ​​cannot be approximated linearly, but is approximated by, for example, a sigmoid curve.

[0155] Next, an example of the pixel 24 that can be detected by the method M2 will be described.IN 1 is a graph showing the relationship between the gradation value and the

[0156] A graph 61 shown in FIG. 13A1 is a graph showing the average value of the gradation values ​​of m rows and n columns of the bright spot correction image data BCG_2 at the time of manufacturing the display device 20, for example, and the average value of the gradation values ​​of the bright spot correction image data BCG_2 IN The relationship between the gradation values ​​can be set as follows. IN The gradation values ​​of the bright spot correction image data BCG_2 can be the same value over the entire surface. Here, some of the gradation values ​​of the bright spot correction image data BCG_2 behave as shown in graph 63. In other words, when the gradation value of the image data input to the display unit 22 becomes high, the luminance of the light emitted from some of the pixels 24 decreases. The pixels 24 that behave as shown in graph 63 are prone to deterioration, and when the pixels 24 are used for a long period of time, that is, when a voltage is supplied to the display element of the pixels 24 for a long period of time, the behavior changes to that of graph 63A shown in FIG. 13A2. On the other hand, the pixels 24 that behave as shown in graph 61 are unlikely to deteriorate, and do not behave as shown in graph 63A even when used for a long period of time.

[0157] The pixel 24 that behaves as shown in the graph 63A can be said to be a bright spot. Therefore, the pixel 24 that behaves as shown in the graph 63 of FIG. 13A1 is not a bright spot at the time of manufacturing the display device 20, for example, but is highly likely to become a bright spot as the display device 20 is used. As described above, if the pixel 24 becomes a bright spot, it has a large impact on visibility. Therefore, it is preferable to make the pixel 24 that behaves as shown in the graph 63 of FIG. 13A1 a dark spot, for example, by reducing the voltage supplied to the display element. This can prevent the pixel 24 from becoming a bright spot, thereby improving the reliability of the display device 20.

[0158] 13B is a diagram showing an example of a method for detecting a pixel 24 that exhibits the behavior shown in the graph 63. As shown in FIG. 13B, IN The gradation value of the bright spot correction image data BCG_2 is high. INAll of the gradation values ​​of m rows and n columns of the bright spot correction image data BCG_2 are set to 255 or close to that. IN When steps S11'' to S13'' are performed based on this, the gradation values ​​of the bright spot correction image data BCG_2 corresponding to the pixels 24 that behave as shown in graph 63 become lower than the gradation values ​​of the surrounding pixels. FIG. 13B shows an example in which the bright spot correction image data BCG_2 includes a gradation value 65 that is lower than the gradation values ​​of the surrounding pixels. The pixel 24 corresponding to the gradation value 65 can be regarded as the pixel 24 that behaves as shown in graph 63, that is, the pixel 24 that becomes a bright spot after long-term use.

[0159] Therefore, by using method M2, it is possible to detect not only pixels 24 that are already bright spots, but also pixels 24 that are likely to become bright spots as the display device 20 is used. On the other hand, in method M1, among the gradation values ​​of the bright spot correction image data BCG_1, the gradation value corresponding to the pixel 24 that behaves as shown in graph 63 is higher than the gradation values ​​of the surrounding pixels. Therefore, the bright spot correction image data BCG_1 input to the input unit 21 IN When the gradation value of the bright spot correction image data BCG_1 is increased, it becomes difficult to detect the pixel 24 that behaves as shown in the graph 63. IN When the gradation value is lowered, the graph 63 becomes closer to the graph 61, and it becomes difficult to detect the pixel 24 that behaves as shown in the graph 63. For the above reasons, it is difficult to detect the pixel 24 that behaves as shown in the graph 63 with the method M1.

[0160] As described above, by performing both method M1 and method M2, it is possible to detect not only pixels 24 that are bright spots with high accuracy, but also pixels 24 that are likely to become bright spots as the display device 20 is used. In this way, by performing both method M1 and method M2, it is possible to comprehensively detect pixels 24 that should be dark spots, for example.

[0161] FIG. 14 shows the actual measured gradation values ​​of the bright spot correction image data BCG_1 and the IN12B is a graph showing the relationship between the gradation value and the line 57, and differs from the graph shown in FIG. 12A in that lines 57A and 57B are shown instead of line 57.

[0162] 14, among the gradation values ​​of the bright spot correction image data BCG_1, the gradation value indicated by line 57A is set as a first threshold value, and the gradation value indicated by line 57B is set as a second threshold value. The first threshold value is smaller than the value indicated by line 56, and the second threshold value is smaller than the first threshold value. Lines 57A and 57B can be expressed by linear equations with positive slopes, similar to line 57.

[0163] An example of an image processing method using the image processing system 10 will be described with reference to Fig. 14. First, the graph shown in Fig. 14 is created by method M1. In the graph shown in Fig. 14, for example, among the gradation values ​​of the bright spot correction image data BCG_1 of m rows and n columns, the coordinates of the gradation values ​​that are equal to or smaller than the first threshold value and equal to or larger than the second threshold value are set as first bright spot coordinates. Also, the coordinates of the gradation values ​​that are smaller than the second threshold value are set as second bright spot coordinates.

[0164] Furthermore, the bright spot coordinates are detected by the method M2, and the detected bright spot coordinates are set as third bright spot coordinates.

[0165] And content image data CG ML is input to the bright point correction unit 50, the content image data CG ML Among the gradation values ​​of m rows and n columns of the pixel 22, for example, the gradation value of the coordinates that are the same as the first luminance point coordinates and also the third luminance point coordinates is reduced. Also, for example, the gradation value of the coordinates that are the same as the second luminance point coordinates is reduced regardless of whether they are the same as the third luminance point coordinates. This makes it possible to correct the luminance point and display a high-quality image on the display unit 22.

[0166] By carrying out image processing in the above-described manner, it is possible to prevent pixels 24 that do not have a significant effect on visibility even if they are not made into dark spots from being made into dark spots. This makes it possible to prevent the quality of the image displayed on the display unit 22 from being reduced by making pixels 24 into dark spots. In the example shown in FIG. 14, among the pixels 24 in m rows and n columns, for example, pixel 24 with coordinates that are the same as the first bright spot coordinates but different from the third bright spot coordinates can be made into a pixel that does not have a significant effect on visibility even if it is not made into a dark spot. Thus, for example, content image data CG ML Among the gradation values ​​of m rows and n columns of the pixel 101, the gradation values ​​of the coordinates that are the same as the first bright-spot coordinates but different from the third bright-spot coordinates may not be corrected.

[0167] The above is an example of the bright spot correction method, which is an image processing method according to one embodiment of the present invention.

[0168] <Example of machine learning model configuration> FIG. 15A is a diagram showing a configuration example of a machine learning model MLM. As shown in FIG. 15A, the machine learning model MLM can be a neural network model having an input layer IL, an intermediate layer ML1, an intermediate layer ML2, an intermediate layer ML3, and an output layer OL. The input layer IL, the intermediate layer ML1, the intermediate layer ML3, and the output layer OL each have a plurality of layers composed of neurons, and the neurons provided in each layer are connected to each other. Image data can be input to the input layer IL.

[0169] The image data input to the input layer IL can have matrices with m rows and n columns, the number of which is the same as the number of types of sub-pixels that the display unit 22 has. For example, if the pixel 24 has a sub-pixel that emits red (R) light, a sub-pixel that emits green (G) light, and a sub-pixel that emits blue (B) light, the image data has a matrix with m rows and n columns whose components are red gradation values, a matrix with m rows and n columns whose components are green gradation values, and a matrix with m rows and n columns whose components are blue gradation values. In other words, the image data can be configured to have three matrices.

[0170] If the image data includes a matrix as described above, the number of neurons in the input layer IL can be the same as the number of components in the matrix. For example, if the image data includes three matrices with 1200 rows and 1920 columns, the number of neurons in the input layer IL can be 1920×1200×3. Also, if the image data includes a matrix, the number of neurons in the output layer OL can be the same as the number of components in the matrix. For example, if the image data includes three matrices with 1200 rows and 1920 columns as described above, the number of neurons in the output layer OL can be 1920×1200×3.

[0171] The hidden layer ML1 has a function of generating data D1 to be supplied to the hidden layer ML2. The data D1 can be a matrix having h elements x (h is an integer equal to or greater than 2).

[0172] In this specification, for example, h components x are referred to as components x1 to x h The other ingredients are described in the same way.

[0173] The number of neurons in the hidden layer ML1 is set to be greater than the number of neurons in the input layer IL. This allows the number of components in the data D1 to be greater than the number of components in the image data input to the input layer IL. Details of the arithmetic processing that can be performed by the hidden layer ML1 will be described later.

[0174] The hidden layer ML2 has a function of converting a component x into a component y. For example, the hidden layer ML2 converts the components x1 to xy into a component y by a nonlinear polynomial function of one variable. h are components y1 to y h An example of this function is shown below.

[0175]

number

[0176] Here, i can be an integer between 1 and h. The above formula is a function with component x as an independent variable, component y as a dependent variable, and a as a coefficient. The function has a term including x raised to the power d (d is an integer of 2 or more). In FIG. 15A, it is assumed that the hidden layer ML2 performs the arithmetic processing shown in the above formula.

[0177] Another example of a nonlinear polynomial function of one variable is shown below.

[0178]

number

[0179] The above formula is a function with x as an independent variable, y as a dependent variable, and a and b as coefficients. The function has a term including the cosine of component x and a term including the sine of component x. Note that the function does not necessarily have to have a term including the cosine of component x. Also, the function does not necessarily have to have a term including the sine of component x.

[0180] As a result, the hidden layer ML2 is composed of components y1 to y h This data is referred to as data D2. Data D2 can be a matrix, similar to data D1.

[0181] The hidden layer ML3 has a function of generating data to be supplied to the output layer OL. The number of neurons in the hidden layer ML3 is made greater than the number of neurons in the output layer OL. This allows the number of components in the image data output from the output layer OL to be less than the number of components in the data D2. Details of the arithmetic processing that can be performed by the hidden layer ML3 will be described later.

[0182] Note that two or more hidden layers may be provided between the input layer IL and the hidden layer ML2. Also, two or more hidden layers may be provided between the hidden layer ML2 and the output layer OL.

[0183] Fig. 15B is a diagram showing an example of a method for generating the machine learning model MLM when the machine learning model MLM has the configuration shown in Fig. 15A. As shown in Fig. 7, the machine learning model MLM is generated in step S16. Therefore, Fig. 15B can be said to be a diagram showing an example of the operation of step S16 when the machine learning model MLM has the configuration shown in Fig. 15A.

[0184] As described above, the machine learning model MLM can be generated by the learning unit 46. Specifically, for example, the learning image data LG IN And learning image data LG GEN For example, the learning image data LG IN The image data output when the input is the learning image data LG GEN The coefficient a is set to match 1,0 or coefficient a n,k By acquiring the values ​​of a and b by learning, the learning unit 46 can generate the machine learning model MLM. When the intermediate layer ML2 performs the calculation shown in Equation 2, in addition to the value of the coefficient a, the value of the coefficient b is also acquired by learning.

[0185] 16A and 16B are diagrams illustrating an example of a calculation by the machine learning processing unit 23 to which the machine learning model MLM is applied. As shown in FIG. 16A, the hidden layer ML1 calculates the content image data CG IN In the example shown in FIG. 16A, the content image data CG IN contains three matrices with 1200 rows and 1920 columns. In other words, the content image data CG IN The data has a width of 1920, a height of 1200, and a channel count of 3. The number of channels of the filter fa is 3, and the content image data CG IN By performing such a multiplication and addition operation, data D1 having a height of 1200, a width of 1920, and 9 channels can be output from the hidden layer ML1. The data D1 is made up of components x1 to x1920×1200×9 This includes:

[0186] The components x1 to x2 of the data D1 1920×1200×9 In the hidden layer ML2, the components y1 to y 1920×1200×9 The components y1 to y 1920×1200×9 The data including this is referred to as data D2.

[0187] As shown in FIG. 16B, the intermediate layer ML3 can perform a multiply-and-accumulate operation between data D2 and filter fb. Here, data D2 can be data with a height of 1200, a width of 1920, and a number of channels of 9, similar to data D1. Furthermore, filter fb has 9 channels, and intermediate layer ML3 performs a multiply-and-accumulate operation between data D2 and three filters fb (filters fb1 to fb3). By performing such a multiply-and-accumulate operation, data with a width of 1920, a height of 1200, and a number of channels of 3 can be output from intermediate layer ML3. This data is represented by content image data CG ML It can be said that:

[0188] From the above, using the machine learning model MLM, for example, content image data CG IN Content image data CG ML can be converted to

[0189] As described above, in the machine learning model MLM having the configuration shown in FIG. 15A, in the intermediate layer ML2, the components x1 to x h are expressed as components y1 to y2 using a nonlinear polynomial function of one variable. h This allows us to convert, for example, the components x1 to x using a linear function of one variable or a monomial function of one variable. h are components y1 to y hInference using the machine learning model MLM can be performed with higher accuracy than when converting to . In addition, since the number of filters fa shown in Fig. 16A and the number of channels of the filter fb shown in Fig. 16B can be reduced, the amount of calculation required for generating the machine learning model MLM by learning and for inference using the machine learning model MLM can be reduced. Therefore, learning and inference can be performed at high speed. EXAMPLES

[0190] In this embodiment, the results of learning performed to obtain the machine learning model MLM shown in FIG. 15A will be described.

[0191] In this embodiment, a machine learning model MLM was generated by supervised learning using image data with a width of 1920, height of 1200, and number of channels of 3 as training data and correct answer data. The component x included in the data D1 was converted to a component y by the intermediate layer ML2 using Formula 1, Formula 2, or the formula "y=ax+b". In Formula 1 and Formula 2, d=5.

[0192] Moreover, the hidden layer ML1 performs the calculation shown in Fig. 16A, and the hidden layer ML3 performs the calculation shown in Fig. 16B. When converting a component x included in data D1 to a component y using formula 1 or formula 2, the hidden layer ML1 performs a product-sum calculation between the image data and filters fa1 to fa9 with three channels, and the hidden layer ML3 performs a product-sum calculation between the data D1 and filters fa1 to fa3 with nine channels. When converting a component x included in data D1 to a component y using formula "y=ax+b", the hidden layer ML1 performs a product-sum calculation between the image data and filters fa1 to fa9 with three channels, and the hidden layer ML3 performs a product-sum calculation between the data D1 and filters fa1 to fa3 with nine channels. 162 The hidden layer ML3 performs a multiply-and-accumulate operation between the data D2 and the filters fb1 to fb3, each having 162 channels.

[0193] FIG. 17 is a graph showing the relationship between SSIM and the number of learning epochs. SSIM was calculated using test data and correct answer data. The larger the SSIM, the higher the similarity between the test data and correct answer data, which indicates that the machine learning model MLM can make inferences with high accuracy. The test data was image data with a width of 1920, a height of 1200, and three channels, just like the learning data and correct answer data.

[0194] As described above, when the component x included in the data D1 is converted to the component y using the formula 1 or 2, the number of filters fa and the number of channels of the filter fb are smaller than when the component x is converted to the component y using the formula "y=ax+b". Despite this, as shown in Fig. 17, when the number of learning times is 200 or more, when the component x is converted to the component y using the formula 1 or 2, the SSIM is larger than when the component x is converted to the component y using the formula "y=ax+b". [Explanation of symbols]

[0195] 10: image processing system, 20: display device, 21: input unit, 22: display unit, 23: machine learning processing unit, 24: pixel, 26: area, 29: area, 30: imaging device, 33: pixel, 40: generation device, 42: database, 43: image extraction unit, 44: image processing unit, 45: image generation unit, 46: learning unit, 50: bright spot correction unit, 51: bright spot, 52: area, 53: gradation value, 54: gradation value, 55: gradation value, 56: line, 57: line, 57A: line, 57B: line, 61: graph, 63: graph, 63A: graph, 65: gradation value, 126: wiring, 134: wiring, 161: transistor, 162: transistor, 170: light emitting element, 171: transistor, 173: capacitance, 174: wiring, 175: wiring, 180: liquid crystal element, 181: capacitance, 182: wiring, 183: wiring

Claims

[Claim 1] The present invention has an input unit, a machine learning processing unit, a display unit in which pixels are arranged in a matrix of m rows and n columns (m and n are integers of 2 or more), a database, an image processing unit, an image generation unit, and a learning unit, The database stores a table generated based on first image data input to the input unit and second image data acquired by capturing an image of the display unit displaying an image corresponding to the first image data; The first image data has first gradation values ​​in m rows and n columns, the second image data has second gradation values ​​in m rows and n columns; the table represents the first gradation value and the second gradation value at a coordinate corresponding to the coordinate of the first gradation value; The image processing unit has a function of generating third learning image data by performing image processing on the first learning image data input to the input unit based on the second learning image data, The second learning image data is acquired by capturing an image of the display unit that displays an image corresponding to the first learning image data, The third learning image data has third gradation values ​​in m rows and n columns, the image generation unit has a function of generating fourth learning image data including the first gradation value corresponding to the second gradation value selected based on the third gradation value; The learning unit has a function of generating a machine learning model such that image data output when the first learning image data is input matches the fourth learning image data, and outputting the machine learning model to the machine learning processing unit; The machine learning processing unit is an image processing system having a function of performing processing using the machine learning model on content image data input to the input unit.

Images