Image Processing System

Abstract

To provide an image processing system capable of reducing the display unevenness of an image displayed on a display device.SOLUTION: The image processing system includes a display device, an imaging device, and a learning device. The learning device stores a table representing information relating to correspondence between first image data and second image data generated by displaying an image corresponding to the first image data on the display device and capturing the image by the imaging device. The learning device generates teacher data on the basis of the table, to generate a machine learning model by using the generated teacher data. Image processing using the machine learning model is performed to image data inputted to the display device, to reduce the display unevenness of the image displayed on the display device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] One aspect of the present invention relates to an image processing system. [Background technology]

[0002] Display devices such as liquid crystal displays and organic electroluminescence displays can be manufactured by applying a resist to a substrate and then patterning the resist by exposing it through a mask. In order to increase the size of the display device, the substrate must also be made larger, but it may not be possible to enlarge the mask to match the size of the substrate. Patent Document 1 discloses a method for manufacturing a display device in such a case, in which the substrate surface is divided into multiple exposure regions corresponding to the size of the mask, and exposure is performed for each exposure region.

[0003] Furthermore, defective pixels may occur due to poor characteristics or deterioration of display elements, transistors, etc., contained in the pixels of a display device. Defective pixels may be, for example, bright spots or dark spots. When viewing an image displayed on a display device, bright spots are more noticeable than dark spots, and therefore have a significant impact on visibility. Therefore, if there are many bright spots, the display device may not be able to display a high-quality image. Patent Document 2 discloses a method for turning bright spots into dark spots during the manufacturing process of a display device. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-198990 [Patent Document 2] Special Publication No. 2018-514801 Summary of the Invention [Problem to be solved by the invention]

[0005] When performing divided exposure as described above, the exposure amount at the boundary of the exposure area may differ from the exposure amount in other areas due to factors such as the mask being misaligned from the exposure area. As a result, the characteristics of the elements of the pixels located at the boundary of the exposure area may differ from the characteristics of the elements of the pixels located in other areas. As a result, even if the gradation is the same, the brightness of the light emitted by the pixels located at the boundary of the exposure area may differ from the brightness of the light emitted by the pixels located in other areas. This difference in brightness may be visually recognized as display unevenness.

[0006] One possible method for making display unevenness less noticeable is to perform image processing on image data input to a display device. For example, one possible method is to perform image processing using machine learning. Specifically, one possible method is to generate a machine learning model using a generation device, and perform image processing using the machine learning model on image data input to the display device. When the display device performs image processing using the machine learning model generated by the generation device, the display device and the generation device can be said to constitute an image processing system.

[0007] Furthermore, even if a pixel is not a bright spot when the display device is manufactured, it may become a bright spot when the display device is used for a long period of time due to deterioration of the display elements, transistors, etc. that make up the pixel, causing fluctuations in electrical characteristics, etc. Such bright spots are difficult to remove during the manufacturing process of the display device.

[0008] An object of one embodiment of the present invention is to provide an image processing system that can make display unevenness of an image displayed on a display device less noticeable.Another object of one embodiment of the present invention is to provide an image processing system that can display a high-quality image on a display device.Another object of one embodiment of the present invention is to provide an image processing system including a large-sized display device.Another object of one embodiment of the present invention is to provide an image processing system including a display device that can display a high-resolution image.Another object of one embodiment of the present invention is to provide an image processing system that can perform image processing in a short time.Another object of one embodiment of the present invention is to provide an image processing system including a highly reliable display device.

[0009] Another object of one embodiment of the present invention is to provide a novel image processing system, a novel image processing method, a novel generation device, a novel machine learning model generation method, a novel image processing device, a novel display device, or the like.

[0010] Note that the description of these problems does not preclude the existence of other problems. Note that one embodiment of the present invention does not necessarily solve all of these problems. Note that problems other than these will become apparent from the description of the specification, drawings, claims, etc., and it is possible to extract other problems from the description of the specification, drawings, claims, etc. [Means for solving the problem]

[0011] One aspect of the present invention relates to an image processing system including a display device, an imaging device, and a learning device, and a method for generating a machine learning model using the image processing system. The display device has pixels arranged in a matrix of m rows and n columns (m and n are integers equal to or greater than 2). The learning device has a database. The database stores a table generated based on first image data and second image data acquired by displaying an image corresponding to the first image data on the display device and capturing the image with the imaging device. The first image data has first gradation values ​​of m rows and n columns, and the second image data has second gradation values ​​of m rows and n columns. Specifically, the table represents the first gradation values ​​and second gradation values ​​at coordinates corresponding to the coordinates of the first gradation values.

[0012] When generating a machine learning model, first, an image corresponding to the first training image data is displayed on a display device, and the image displayed on the display device is captured by an imaging device to obtain second training image data. Next, the training device performs image processing on the first training image data based on the second training image data to generate third training image data having third gradation values ​​in m rows and n columns. Specifically, the training device performs image processing on the first training image data to approximate the second training image data, thereby generating third training image data having third gradation values ​​in m rows and n columns. For example, image processing is performed on the first training image data so that the sum of the third gradation values ​​in the first row, first column through the m rows and n columns is equal to the sum of the gradation values ​​in the first row, first column through the m rows and n columns of the second training image data.

[0013] The learning device then selects second gradation values ​​from row 1, column 1 to column m, column n based on the third gradation values ​​from row 1, column 1 to column m, column n. For example, a second gradation value that matches or is closest to the third gradation value is selected for each of row 1, column 1 to column m, column n. Next, fourth training image data is generated, which is image data including first gradation values ​​that correspond to the selected second gradation values. The learning device then generates a machine learning model such that image data output when the first training image data is input matches the fourth training image data.

[0014] The machine learning model generated by the learning device is supplied to the display device. This enables the display device to perform image processing using the machine learning model on image data input to the display device. For example, the machine learning model can be used to perform image processing on image data input to the display device to reduce display unevenness.

[0015] One aspect of the present invention includes a display device, an imaging device, and a learning device. The display device includes an input unit, a machine learning processing unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The learning device includes a database, an image processing unit, an image generation unit, and a learning unit. The database stores a table generated based on first image data input to the input unit and second image data acquired by displaying an image corresponding to the first image data on the display unit and capturing an image by the imaging device so as to include the image displayed on the display unit. The first image data has first gradation values ​​of m rows and n columns, and the second image data has second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. The image processing unit generates a table based on the first learning image data input to the input unit and the second learning image data. The image processing system has a function of generating third training image data by performing image processing based on the second training image data, the second training image data being image data acquired by displaying an image corresponding to the first training image data on a display unit and capturing an image using an imaging device so as to include the image displayed on the display unit, the third training image data having third gradation values ​​of m rows and n columns, the image generation unit having a function of generating fourth training image data being image data including a first gradation value corresponding to a second gradation value selected based on the third gradation value, the learning unit having a function of generating a machine learning model such that image data output when the first training image data is input matches the fourth training image data and outputting the machine learning model to the machine learning processing unit, and the machine learning processing unit having a function of processing content image data input to the input unit using the machine learning model.

[0016] Alternatively, in the above aspect, the first learning image data may have fourth gradation values ​​in m rows and n columns, and the second learning image data may have fifth gradation values ​​in m rows and n columns, and the image processing unit may have a function of performing image processing so that the difference between the sum of the third gradation values ​​and the sum of the fifth gradation values ​​is smaller than the difference between the sum of the fourth gradation values ​​and the sum of the fifth gradation values.

[0017] Alternatively, in the above aspect, the machine learning model may be a neural network model.

[0018] Alternatively, one aspect of the present invention is a method for generating a machine learning model by an image processing system having a display unit in which pixels of m rows and n columns (m and n are integers of 2 or more) are arranged in a matrix, the method comprising: displaying an image corresponding to first image data having first gradation values ​​of m rows and n columns on the display unit by emitting light of a luminance corresponding to the first gradation values ​​from the pixels; capturing an image including the image corresponding to the first image data displayed on the display unit; acquiring second image data having second gradation values ​​of m rows and n columns; generating a table representing the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values; This is a method for generating a machine learning model, which involves acquiring second training image data by capturing an image that includes an image corresponding to the first training image data displayed on the display unit, performing image processing on the first training image data based on the second training image data to generate third training image data having third gradation values ​​of m rows and n columns, generating fourth training image data based on the third gradation values, which is image data including a first gradation value that corresponds to the selected second gradation value, and generating a machine learning model such that the image data output when the first training image data is input matches the fourth training image data.

[0019] Alternatively, in the above aspect, the first learning image data may have fourth gradation values ​​in m rows and n columns, and the second learning image data may have fifth gradation values ​​in m rows and n columns, and image processing may be performed so that the difference between the sum of the third gradation values ​​and the sum of the fifth gradation values ​​is smaller than the difference between the sum of the fourth gradation values ​​and the sum of the fifth gradation values.

[0020] Alternatively, in the above aspect, the machine learning model may be a neural network model.

[0021] Alternatively, one embodiment of the present invention includes a display device, an imaging device, and a generating device. The display device includes an input unit, a bright spot correction unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The generating device includes a database and an image generating unit. The database stores a table generated based on first database image data input to the input unit and second database image data acquired by displaying an image corresponding to the first database image data on the display unit and capturing an image to include the image displayed on the display unit, the first database image data having first gradation values ​​of m rows and n columns, and the second database image data having second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​at coordinates corresponding to the coordinates of the first gradation values. The imaging device has a function of acquiring second image data for bright spot correction by imaging the image displayed on the display unit when an image corresponding to first image data for bright spot correction input to the input unit is displayed on the display unit, the second image data for bright spot correction having a third gradation value of m rows and n columns, the image generation unit has a function of generating third image data for bright spot correction, which is image data including a first gradation value corresponding to a second gradation value selected based on the third gradation value, the bright spot correction unit has a function of detecting, as bright spot coordinates, the coordinates of the first gradation values ​​of m rows and n columns that are below a threshold value, and the bright spot correction unit has a function of reducing the fourth gradation value at the same coordinates as the bright spot coordinates when content image data having a fourth gradation value of m rows and n columns is input to the input unit.

[0022] Alternatively, one embodiment of the present invention includes a display device, an imaging device, and a generating device. The display device includes an input unit, a bright spot correction unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The generating device includes a database and an image generating unit. The database stores a table generated based on first database image data input to the input unit and second database image data acquired by the imaging device by displaying an image corresponding to the first database image data on the display unit and capturing an image to include the image displayed on the display unit. The first database image data has first gradation values ​​of m rows and n columns, and the second database image data has second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. When the imaging device displays an image corresponding to the first bright spot correction image data input to the input unit on the display unit, and a function to acquire second image data for bright spot correction by capturing an image displayed on the display unit, the second image data for bright spot correction having third gradation values ​​of m rows and n columns, the image generation unit having a function to generate third image data for bright spot correction, the third image data being image data including first gradation values ​​corresponding to the second gradation values ​​selected based on the third gradation values, and the bright spot correction unit having a function to select first gradation values ​​of m rows and n columns of the third image data for bright spot correction that are equal to or less than a first threshold value. The image processing system has a function of detecting the coordinates of the gradation value of the first or second bright spot coordinate as the first bright spot coordinate, and the bright spot correction unit has a function of detecting the coordinates of the third gradation value of m rows and n columns that is equal to or greater than a second threshold value as the second bright spot coordinate, and the bright spot correction unit has a function of reducing the fourth gradation value of the same coordinate as the first or second bright spot coordinate when content image data having a fourth gradation value of m rows and n columns is input to the input unit.

[0023] Alternatively, one aspect of the present invention includes a display device, an imaging device, and a generation device. The display device includes an input unit, a bright spot correction unit, and a display unit in which pixels of m rows and n columns (m and n are integers equal to or greater than 2) are arranged in a matrix. The generation device includes a database and an image generation unit. The database stores a table generated based on first database image data input to the input unit and second database image data acquired by displaying an image corresponding to the first database image data on the display unit and capturing an image by the imaging device so as to include the image displayed on the display unit. The first database image data has first gradation values ​​of m rows and n columns, and the second database image data has second gradation values ​​of m rows and n columns. The table represents the first gradation values ​​and second gradation values ​​of coordinates corresponding to the coordinates of the first gradation values. The imaging device has a function of acquiring the second bright spot correction image data by capturing the image displayed on the display unit when an image corresponding to the first bright spot correction image data input to the input unit is displayed on the display unit. The second bright spot correction image data has third gradation values ​​in m rows and n columns, the image generation unit has a function of generating third bright spot correction image data which is image data including first gradation values ​​corresponding to second gradation values ​​selected based on the third gradation values, and the bright spot correction unit detects, among the first gradation values ​​in m rows and n columns of the third bright spot correction image data, coordinates of first gradation values ​​that are equal to or less than a first threshold value and equal to or greater than a second threshold value as first bright spot coordinates, and coordinates of first gradation values ​​that are less than the second threshold value as second bright spot coordinates. The bright spot correction unit has a function of detecting, as third bright spot coordinates, coordinates of third gradation values ​​that are equal to or greater than a third threshold value among the third gradation values ​​of m rows and n columns contained in the second bright spot correction image data, and the bright spot correction unit has a function of reducing the fourth gradation values ​​of coordinates that are the same as both the first bright spot coordinates and the third bright spot coordinates, and a function of reducing the fourth gradation value of coordinates that are the same as the second bright spot coordinates, when content image data having fourth gradation values ​​of m rows and n columns is input to the input unit.

[0024] Alternatively, in the above aspect, the display device has a machine learning processing unit, and the generation device has an image processing unit and a learning unit, the image processing unit has a function of generating third training image data by performing image processing on first training image data input to the input unit based on second training image data, the second training image data being image data acquired by displaying an image corresponding to the first training image data on the display unit and using an imaging device to capture an image that includes the image displayed on the display unit, the third training image data having a fifth gradation value of m rows and n columns, the image generation unit has a function of generating fourth training image data being image data including a first gradation value corresponding to a second gradation value selected based on the fifth gradation value, the learning unit has a function of generating a machine learning model such that image data output when the first training image data is input matches the fourth training image data, and outputting the machine learning model to the machine learning processing unit, and the machine learning processing unit may have a function of processing content image data input to the input unit using the machine learning model.

[0025] Alternatively, in the above aspect, the first learning image data may have a sixth gradation value of m rows and n columns, and the second learning image data may have a seventh gradation value of m rows and n columns, and the image processing unit may have a function of performing image processing so that the difference between the sum of the fifth gradation values ​​and the sum of the seventh gradation values ​​is smaller than the difference between the sum of the sixth gradation values ​​and the sum of the seventh gradation values.

[0026] Alternatively, in the above aspect, the machine learning model may be a neural network model. [Effects of the Invention]

[0027] According to one embodiment of the present invention, an image processing system capable of making display unevenness of an image displayed on a display device less noticeable can be provided. According to another embodiment of the present invention, an image processing system capable of displaying a high-quality image on a display device can be provided. According to another embodiment of the present invention, an image processing system including a large-sized display device can be provided. According to another embodiment of the present invention, an image processing system including a display device that can display a high-resolution image can be provided. According to another embodiment of the present invention, an image processing system that can perform image processing in a short time can be provided. According to another embodiment of the present invention, an image processing system including a highly reliable display device can be provided.

[0028] According to one embodiment of the present invention, a novel image processing system, a novel image processing method, a novel generation device, a novel machine learning model generation method, a novel image processing device, a novel display device, or the like can be provided.

[0029] Note that the effects of one embodiment of the present invention are not limited to the effects listed above. The effects listed above do not preclude the existence of other effects. Note that the other effects are effects not mentioned in this section, which will be described below. Effects not mentioned in this section can be derived by a person skilled in the art from the description in the specification, drawings, etc., and can be extracted as appropriate from these descriptions. Note that one embodiment of the present invention has at least one of the effects listed above and / or other effects. Therefore, one embodiment of the present invention may not have the effects listed above in some cases. [Brief explanation of the drawings]

[0030] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system. [Figure 2] Fig. 2A is a block diagram showing an example of the configuration of a display unit, Fig. 2B1 and Fig. 2B2 are circuit diagrams showing an example of the configuration of a pixel. [Figure 3]3A and 3B are schematic diagrams showing an example of an image processing method. [Figure 4] FIG. 4 is a flowchart showing an example of a method for generating a table. [Figure 5] 5A and 5B are schematic diagrams showing an example of a method for generating a table. [Figure 6] 6A and 6B are schematic diagrams showing an example of a method for generating a table. [Figure 7] FIG. 7 is a flowchart illustrating an example of a method for generating a machine learning model. [Figure 8] 8A and 8B are schematic diagrams showing an example of a method for generating a machine learning model. [Figure 9] 9A and 9B are schematic diagrams showing an example of a method for generating a machine learning model. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of a method for generating a machine learning model. [Figure 11] 11A and 11B are schematic diagrams showing an example of an image processing method. [Figure 12] 12A and 12B are graphs showing an example of an image processing method. [Figure 13] Figures 13A1 and 13A2 are graphs showing an example of an image processing method, and Figure 13B is a schematic diagram showing an example of an image processing method. [Figure 14] FIG. 14 is a graph showing an example of an image processing method. [Figure 15] Fig. 15A is a diagram showing an example of the configuration of a machine learning model, and Fig. 15B is a schematic diagram showing an example of a learning method. [Figure 16] 16A and 16B are schematic diagrams showing an example of calculations by a machine learning model. [Figure 17] FIG. 17 is a graph showing the learning results according to the example. DETAILED DESCRIPTION OF THE INVENTION

[0031] The embodiments will be described in detail with reference to the drawings. However, the present invention is not limited to the following description, and it will be readily understood by those skilled in the art that various changes can be made in form and detail without departing from the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited to the description of the embodiments shown below.

[0032] In the configuration of the invention described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and repeated explanations thereof will be omitted. Furthermore, when referring to similar functions, the same hatch pattern may be used and no particular reference numeral may be assigned.

[0033] Furthermore, for ease of understanding, the position, size, range, etc. of each component shown in the drawings may not represent the actual position, size, range, etc. Therefore, the disclosed invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings.

[0034] Furthermore, the ordinal numbers "first," "second," and "third" used in this specification are used to avoid confusion of components and are not intended to limit the numbers.

[0035] (Embodiment) In this embodiment, an image processing system and the like according to one embodiment of the present invention will be described with reference to drawings.

[0036] <Image processing system> 1 is a block diagram showing an example configuration of an image processing system 10. The image processing system 10 includes a display device 20, an imaging device 30, and a generating device 40. Here, the generating device 40 is preferably provided in a device with high computing power, such as a server.

[0037] The display device 20 includes an input unit 21, a display unit 22, a machine learning processing unit 23, and a bright spot correction unit 50. The generation device 40 includes a database 42, an image extraction unit 43, an image processing unit 44, an image generation unit 45, and a learning unit 46.

[0038] 2A is a block diagram showing an example of the configuration of the display unit 22. As shown in Fig. 2A, the display unit 22 has pixels 24 arranged in a matrix of m rows and n columns (m and n are integers of 2 or greater). The pixels 24 in the same row are electrically connected to each other via the same wiring 134, and the pixels 24 in the same column are electrically connected to each other via the same wiring 126. The pixels 24 have display elements, and can display images on the display unit 22 using the display elements.

[0039] In this specification, the pixels 24 in m rows and n columns are distinguished by being referred to as pixel 24(1,1) through pixel 24(m,n), respectively. Similar notations may be used for other elements. For example, (1,1) through (m,n) may be referred to as coordinates.

[0040] In this specification and the like, a display element can be referred to as a display device. A light-emitting element can be referred to as a light-emitting device, and a liquid crystal element can be referred to as a liquid crystal device. Regarding other elements, the term "element" can also be referred to as a "device."

[0041] In Fig. 1, arrows indicate data exchange between components of the image processing system 10. Note that the data exchange shown in Fig. 1 is an example, and there are cases where data exchange can occur between components that are not connected by arrows. Also, there are cases where data exchange does not occur between components that are connected by arrows.

[0042] Image data is input to the input unit 21. The image data input to the input unit 21 can be output to the display unit 22, the machine learning processing unit 23, the database 42, the image processing unit 44, or the learning unit 46.

[0043] The image data input to the input unit 21 includes database image data DG IN , learning image data LG IN , bright spot correction image data BCG IN , and content image data CGIN Database image data DG IN can be supplied from the input unit 21 to the display unit 22 and the database 42. IN can be supplied from the input unit 21 to the display unit 22 and the image processing unit 44. Bright spot correction image data BCG IN can be supplied from the input unit 21 to the display unit 22 and the image processing unit 44. Content image data CG IN can be supplied from the input unit 21 to the machine learning processing unit 23.

[0044] The display unit 22 has a function of displaying an image corresponding to image data. Here, the image data can be a set of gradation values. For example, the image data supplied to the display unit 22 can be configured to have gradation values ​​in m rows and n columns. In this case, the pixels 24 emit light with a brightness corresponding to the gradation values, thereby displaying an image on the display unit 22. Here, the gradation values ​​can be digital values. For example, if the gradation values ​​are 8-bit digital values, the gradation values ​​can take integers from 0 to 255.

[0045] The machine learning processing unit 23 has a function of performing image processing on image data based on the machine learning model generated by the generation device 40. Specifically, the machine learning processing unit 23 performs image processing on the content image data CG input from the input unit 21 based on the machine learning model MLM generated by the learning unit 46. IN The image data that has undergone image processing by the machine learning processing unit 23 is converted into content image data CG ML to the bright spot correction unit 50.

[0046] As the machine learning model MLM, for example, a multilayer perceptron, a neural network model, etc. can be applied. In particular, applying a neural network model is preferable because it enables efficient image processing and enables high-quality images to be displayed on the display unit 22. Here, as the neural network model, for example, a generative model such as an autoencoder, U-net, or pix2pix can be used. Note that machine learning models other than those mentioned above can also be used as the machine learning model MLM as long as they are capable of Bayesian estimation. Furthermore, it is preferable that the machine learning model MLM can perform learning and inference by independently handling input and output values.

[0047] The bright spot correction unit 50 corrects the content image data CG ML Specifically, the bright spot correction unit 50 corrects the content image data CG ML The bright spot correction unit 50 has a function to correct the gradation values ​​of the content image data CG based on the bright spot correction image data BCG_1 generated by the image generation unit 45 or the bright spot correction image data BCG_2 generated by the image extraction unit 43, as will be described in detail later. ML The corrected image data is converted into content image data CG COR is supplied to the display unit 22 as

[0048] In this specification, the bright spot correction image data BCG_1 and the bright spot correction image data BCG_2 may be collectively referred to as the bright spot correction image data BCG. Specifically, the bright spot correction image data BCG refers to either the bright spot correction image data BCG_1 or the bright spot correction image data BCG_2. Note that similar notations may be used for other data.

[0049] Here, the bright spot correction unit 50 corrects the content image data CG ML The bright spot correction unit 50 corrects the content image data CG MLAs described above, by providing the bright spot correction section 50 in the display device 20, the quality of the image displayed on the display section 22 can be improved.

[0050] In this specification, the terms "darkening" and "making a pixel a dark point" refer to reducing the luminance of light emitted from a bright pixel 24. Therefore, even if a pixel 24 is darkened, the luminance of light emitted from the pixel 24 does not have to be 0.

[0051] The display device 20 does not need to have the bright spot correction unit 50. In this case, the bright spot correction image data BCG IN is not input to the input unit 21. In addition, the content image data CG ML can be supplied to the display unit 22.

[0052] The imaging device 30 has a function of capturing an image and acquiring imaging data. Specifically, the imaging device 30 can capture an image including an image displayed on the display unit 22. The acquired imaging data is supplied to the image extraction unit 43. Here, the database image data DG IN The image corresponding to the image is displayed on the display unit 22, and the imaging data acquired by the imaging device 30 by capturing an image including the image is converted into imaging data IMG DG In addition, the learning image data LG IN The image corresponding to the image is displayed on the display unit 22, and the imaging data acquired by the imaging device 30 by capturing an image including the image is converted into imaging data IMG LG Furthermore, the bright spot correction image data BCG IN The image corresponding to the image is displayed on the display unit 22, and the imaging data acquired by the imaging device 30 by capturing an image including the image is converted into imaging data IMG BCG The imaging data can be a set of gradation values.

[0053] The image extraction unit 43 extracts the image data IMG DG , image data IMG LG , and image data IMG BCGThe image extraction unit 43 has a function of extracting data of a portion representing the image displayed on the display unit 22 from the captured image data including the image displayed on the display unit 22. When the imaging device 30 captures an image including the image displayed on the display unit 22, areas other than the display unit 22 may also be captured. For example, the housing of the display device 20 may be captured in addition to the display unit 22. The image extraction unit 43 has a function of extracting data of a portion representing the image displayed on the display unit 22 when the captured image data includes areas other than the image displayed on the display unit 22. Data extraction can be performed by pattern matching, template matching, etc. For example, when extracting data of a portion representing the image displayed on the display unit 22 from captured image data including the image displayed on the display unit 22 and the housing of the display device 20, a pattern representing the housing of the display device 20 can be specified, and the portion not including the pattern can be used as data of the portion representing the image displayed on the display unit 22. In addition, when the captured image data IMG DG , image data IMG LG , and image data IMG BCG By performing edge detection on the above, data of the portion representing the image displayed on the display unit 22 can be extracted.

[0054] The image extraction unit 43 extracts the image data IMG DG The data extracted from the database image data DG DP In addition, the image extraction unit 43 extracts the captured image data IMG LG The data extracted from the training image data LG DP Furthermore, the image extraction unit 43 extracts the captured image data IMG BCG The data extracted from the image data for bright spot correction BCG DP Database image data DG DP is supplied to the database 42, and the learning image data LG DP , and bright spot correction image data BCG DP is supplied to the image processing unit 44. The bright spot correction image data BCG DP may be supplied to the image generating unit 45 instead of to the image processing unit 44.

[0055] The image extraction unit 43 also extracts the bright spot correction image data BCGDP can be supplied to the bright spot correction unit 50. The bright spot correction image data BCG supplied to the bright spot correction unit 50 DP is set as bright spot correction image data BCG_2.

[0056] As will be described in detail later, the image processing system 10 stores database image data DG IN and database image data DG DP The image processing system 10 also has a function of acquiring a table showing information on the correspondence between the learning image data LG IN and learning image data LG DP Furthermore, the image processing system 10 has a function of comparing the bright spot correction image data BCG IN and bright spot correction image data BCG DP It has the function to compare.

[0057] Therefore, the database image data DG IN The resolution of the image represented by and the database image data DG DP It is preferable that the resolution of the image represented by is equal to that of the database image data DG IN The number of rows and columns of the gradation values ​​included in the database image data DG DP It is preferable that the number of rows and columns of the gradation values ​​included in the database image data DG are equal. IN contains m rows and n columns of gradation values, the database image data DG DP It is also preferable that the gradation values ​​included in the learning image data LG IN The resolution of the image represented by and the learning image data LG DP It is preferable that the resolution of the image represented by is equal to that of the learning image data LG. IN and the number of rows and columns of the gradation values ​​contained in the learning image data LG DP It is preferable that the number of rows and columns of the gradation values ​​included in the learning image data LG IN contains m rows and n columns of gradation values, the learning image data LG DPIt is preferable that the gradation values ​​included in the bright spot correction image data BCG are arranged in m rows and n columns. IN The image resolution represented by and the bright spot correction image data BCG DP It is preferable that the resolution of the image represented by the bright spot correction image data BCG is equal to the resolution of the image represented by the bright spot correction image data BCG. IN and the number of rows and columns of the gradation values ​​included in the bright spot correction image data BCG DP For example, the number of rows and columns of the gradation values ​​included in the bright spot correction image data BCG are preferably equal to each other. IN If m rows and n columns of gradation values ​​are included in DP It is preferable that the gradation values ​​included in are also arranged in m rows and n columns.

[0058] In this specification, database image data DG IN The gradation value of the database image data DG DP The gradation value that the color image has is sometimes referred to as the second gradation value.

[0059] The image extraction unit 43 can perform up-conversion or down-conversion on the data extracted from the imaging data. DG The number of rows or columns of the gradation values ​​of the data extracted from the database image data DG IN If the number of rows or columns is less than the number of rows or columns of the image data IMG DG Furthermore, the image extraction unit 43 can perform up-conversion on the data extracted from the image data IMG DG The number of rows or columns of the gradation values ​​of the data extracted from the database image data DG IN If the number of rows or columns is greater than the number of rows or columns of the image data IMG DG Down-conversion can be performed on the data extracted from the database image data DG DP The number of rows and columns of the gradation values ​​included in the database image data DG IN The number of rows and columns of the gradation values ​​included in the image data IMG can be made equal to the number of rows and columns of the gradation values ​​included in the image data IMG.LG , and image data IMG BCG The same applies to

[0000] . Note that up-conversion and down-conversion can be performed using the nearest neighbor method, the bilinear method, the bicubic method, or the like.

[0060] The database 42 contains database image data DG IN and database image data DG DP The table T can store information about the correspondence between the database image data DG IN and the database image data DG DP and second gradation values ​​that correspond to the coordinates of the first gradation values. Table T represents, for example, first gradation values ​​and second gradation values ​​at coordinates that correspond to the coordinates of the first gradation values. Table T represents, for example, first gradation values ​​and second gradation values ​​at coordinates that are the same as the coordinates of the first gradation values.

[0061] The image processing unit 44 receives the learning image data LG IN On the other hand, the learning image data LG DP By performing image processing based on IP In other words, it has the function of generating the learning image data LG IN and learning image data LG DP Based on the comparison results, the learning image data LG IN By performing image processing on the learning image data LG IP For example, the learning image data LG IN On the other hand, the learning image data LG DP By performing image processing to make the learning image data LG IP The image processing unit 44 also has a function of generating the bright spot correction image data BCG IN The same image processing is performed on the bright spot correction image data BCG IP It has the function of generating

[0062] The image processing unit 44 receives, for example, learning image data LG IP and the sum of the gradation values ​​of the learning image data LG DP The difference between the sum of the gradation values ​​of and is the learning image data LG IP and the sum of the gradation values ​​of the learning image data LG IN The learning image data LG is calculated so that the difference between the sum of the gradation values ​​of IN The image processing unit 44 has a function of converting the gradation values ​​of the learning image data LG IP The sum of the gradation values ​​of the learning image data LG DP The learning image data LG IN It has the function of converting the gradation values ​​of the "learning image data LG" by image processing. IP The sum of the gradation values ​​of the learning image data LG IP It may be the sum of all the gradation values ​​that "learning image data LG" has, or it may be the sum of some of the gradation values. IN The sum of the gradation values ​​of the learning image data LG IN It may be the sum of all the gradation values ​​of the "learning image data LG" or the sum of some of the gradation values. DP The sum of the gradation values ​​of the learning image data LG DP It may be the sum of all the gradation values ​​that the color filter has, or the sum of some of the gradation values.

[0063] Furthermore, the image processing unit 44 may, for example, IP Learning image data LG DP The peak signal-to-noise ratio (PSNR) or structural similarity (SSIM) of the training image data LG IN The training image data LG is set so that the PSNR or SSIM is larger than the IN The image processing unit 44 has a function of converting the gradation values ​​of the learning image data LG DPThe training image data LG is used to maximize the PSNR or SSIM for IN The learning image data LG is obtained by converting the gradation values ​​of IP It has the function of generating

[0064] The image processing performed by the image processing unit 44 can be, for example, gamma correction. In this case, the image processing can be performed by setting the gamma value to an appropriate value.

[0065] The image processing unit 44 receives, for example, the bright spot correction image data BCG IP and the bright spot correction image data BCG DP The difference between the sum of the gradation values ​​of and is the bright spot correction image data BCG IP The sum of the gradation values ​​of the bright spot correction image data BCG IN The bright spot correction image data BCG is calculated so that the difference between the sum of the gradation values ​​of IN The image processing function converts the gradation values ​​of the learning image data LG IP Bright spot correction image data BCG IP and read as learning image data LG DP Bright spot correction image data BCG DP and read as learning image data LG IN Bright spot correction image data BCG IN The above explanation can be applied by reading it as follows:

[0066] The image generation unit 45 generates learning image data LG IP For example, the database image data DG IN has m rows and n columns of first gradation values, and the database image data DG DP has second gradation values ​​arranged in m rows and n columns, and the learning image data L IP has m rows and n columns of gradation values. In this case, the image generating unit 45 generates the learning image data LG IPBased on the gradation values ​​in the first row and the first column to the m-th row and the n-th column of the learning image data LG, the second gradation values ​​in the first row and the first column to the m-th row and the n-th column of the learning image data LG can be selected. IP For example, if the table T contains k (k is an integer equal to or greater than 2) database image data DG, the second gradation value that matches or is closest to the first gradation value can be selected for each of the first row and first column to the mth row and nth column. IN and k database image data DG DP In this case, for example, the learning image data LG IP The image generating unit 45 can select a second gradation value that matches or is closest to the gradation value in the ith row and jth column (i is an integer of 1 to m, and j is an integer of 1 to n) of the bright spot correction image data BCG from among the k second gradation values ​​in the ith row and jth column. IP The image extracting unit 43 has a function of selecting the second gradation value included in the table T by the same method as the above method based on the gradation value of the bright spot correction image data BCG DP to the image generating unit 45, the bright spot correction image data BCG DP Based on the gradation value of the first gradation value, the second gradation value included in the table T can be selected by a method similar to the above method.

[0067] In this specification and the like, for example, the gradation value at the ith row and jth column may be referred to as the "gradation value at coordinates (i, j)."

[0068] The image generating unit 45 also generates learning image data LG IP The learning image data LG is image data including a first gradation value corresponding to a second gradation value selected based on GEN Similarly, the image generating unit 45 has a function of generating the bright spot correction image data BCG IP The function is to generate the image data for bright spot correction BCG_1, which is image data including the first gradation value corresponding to the second gradation value selected based on the learning image data LG IPIf the table T contains multiple second gradation values ​​in the i-th row and j-th column that match the gradation value in the i-th row and j-th column of the learning image data LG, one second gradation value can be selected from the multiple second gradation values. Then, the first gradation value corresponding to the selected second gradation value is used as the learning image data LG. GEN Bright spot correction image data BCG IP The same applies to the following:

[0069] Also, for example, learning image data LG IP If the table T does not contain a second gradation value in the ith row and jth column that matches the gradation value in the ith row and jth column of the learning image data LG GEN The tone value in the i-th row and j-th column of the learning image data LG IP The bright spot correction image data BCG can be set to the same value as the gradation value in the i-th row and j-th column of the image data BCG. IP The same applies to the following:

[0070] The learning unit 46 receives the learning image data LG IN and learning image data LG GEN For example, the learning unit 46 generates a machine learning model MLM using the learning image data LG IN The image data output when input is the learning image data LG GEN The learning unit 46 has a function of generating a machine learning model MLM that matches the learning image data LG. IN and learning image data LG GEN and has the function of generating the machine learning model MLM through supervised learning using the above. In this way, the machine learning model MLM can be generated through learning. Therefore, the machine learning model MLM generated by the learning unit 46 can be said to be a trained machine learning model.

[0071] The machine learning model MLM generated by the learning unit 46 is supplied to the machine learning processing unit 23. The machine learning processing unit 23 performs inference based on the machine learning model MLM, thereby performing image processing on the image data.

[0072] 2B1 and 2B2 are circuit diagrams showing examples of the configuration of the pixel 24 shown in FIG. 2A. Specifically, these are circuit diagrams showing examples of the configuration of sub-pixels included in the pixel 24. The pixel 24 shown in FIG. 2B1 includes a transistor 161, a transistor 171, a capacitor 173, and a light-emitting element 170. In the pixel 24 shown in FIG. 2B1, the light-emitting element 170 can be used as a display element.

[0073] One of the source and the drain of the transistor 161 is electrically connected to the gate of the transistor 171. The gate of the transistor 171 is electrically connected to one electrode of the capacitor 173. One of the source and the drain of the transistor 171 is electrically connected to one electrode of the light-emitting element 170.

[0074] The other of the source and the drain of the transistor 161 is electrically connected to a wiring 126. The gate of the transistor 161 is electrically connected to a wiring 134. The other of the source and the drain of the transistor 171 and the other electrode of the capacitor 173 are electrically connected to a wiring 174. The other electrode of the light-emitting element 170 is electrically connected to a wiring 175.

[0075] A constant potential can be supplied to the wiring 174 and the wiring 175. For example, as shown in FIG. 2B1, when the anode of the light-emitting element 170 is electrically connected to one of the source and drain of the transistor 171 and the cathode of the light-emitting element 170 is electrically connected to the wiring 175, a high potential can be supplied to the wiring 174 and a low potential can be supplied to the wiring 175.

[0076] The light emitting element 170 can be, for example, an organic EL element or an inorganic EL element.

[0077] 2B1, an image can be displayed on the display unit 22 by controlling the magnitude of the current flowing through the light-emitting element 170 to control the light emission luminance of the light-emitting element 170. The larger the current flowing through the light-emitting element 170, the greater the light emission luminance of the light-emitting element can be.

[0078] The pixel 24 shown in Fig. 2B2 includes a transistor 162, a capacitor 181, and a liquid crystal element 180. In the pixel 24 shown in Fig. 2B2, the liquid crystal element 180 can be used as a display element.

[0079] One of the source and the drain of the transistor 162 is electrically connected to one electrode of the liquid crystal element 180. One electrode of the liquid crystal element 180 is electrically connected to one electrode of the capacitor 181.

[0080] The other of the source and the drain of the transistor 162 is electrically connected to a wiring 126. The gate of the transistor 162 is electrically connected to a wiring 134. The other electrode of the capacitor 181 is electrically connected to a wiring 182. The other electrode of the liquid crystal element 180 is electrically connected to a wiring 183.

[0081] A constant potential can be supplied to the wiring 182 and the wiring 183. A low potential, for example, can be supplied to the wiring 182 and the wiring 183.

[0082] When the pixel 24 provided in the display unit 22 has the configuration shown in FIG. 2B2, the liquid crystal molecules included in the liquid crystal element 180 are aligned in accordance with the voltage applied between both electrodes of the liquid crystal element 180. The liquid crystal molecules can transmit light from, for example, a backlight unit that can be included in the display device 20, depending on the degree of alignment. As described above, the other electrode of the liquid crystal element 180 is electrically connected to the wiring 183 and is supplied with a constant potential. As described above, by controlling the potential of one electrode of the liquid crystal element 180, the pixel 24 can emit light with a brightness corresponding to the potential, thereby displaying an image on the display unit 22.

[0083] FIG. 3A shows the content image data CG input to the input unit 21. IN When the content image G_1 is directly input to the display unit 22, the content image G_1 is displayed on the display unit 22. DP 3B is a schematic diagram showing an example of content image data CG IN is input to the display unit 22 via the machine learning processing unit 23 and the bright spot correction unit 50, the content image G_2 displayed on the display unit 22 is DP FIG.

[0084] Content image data CG IN is input to the display unit 22 without image processing, etc., the display unevenness, bright spots, etc. may occur as described above. Fig. 3A shows the state in which the display unevenness 25 and bright spots 51 occur in the image displayed on the display unit 22.

[0085] Content image data CG IN By performing image processing using the machine learning model MLM, the machine learning processing unit 23 can obtain content image data CG that cancels out the display unevenness. ML In FIG. 3B, the machine learning processing unit 23 generates the content image data CG IN By adding data that cancels out the display unevenness 25 to the data corresponding to the area 26, the content image data CG ML For example, if the brightness of a portion where display unevenness 25 occurs is higher than the brightness of the surrounding area of ​​the portion, the brightness of region 26 can be made lower than the brightness of the surrounding area of ​​region 26.

[0086] Furthermore, by performing image processing on the content image data based on the bright spot correction image data BCG (for example, one of the bright spot correction image data BCG_1 or the bright spot correction image data BCG_2), the bright spot correction unit 50 corrects the bright spots to make them less noticeable. COR For example, it is possible to generate content image data CG CORIn FIG. 3B, the content image data CG ML By adding data that makes the bright spot 51 less noticeable to the data corresponding to the area 52 where the bright spot 51 occurs, the bright spot correction unit 50 corrects the content image data CG COR This shows how to generate

[0087] As shown in FIG. 3B, the content image data CG IN In contrast, the machine learning processing unit 23 and the bright spot correction unit 50 perform image processing, so that the display unit 22 can display an image in which display unevenness and bright spots are not noticeable.

[0088] As described above, as the display device 20 becomes larger and the area of ​​the display unit 22 increases, display unevenness becomes more likely to occur. Furthermore, as the pixels 24 provided in the display unit 22 become smaller and the pixel density of the display unit 22 increases, the characteristics of the display elements, transistors, and the like included in the pixels 24 vary more widely among the pixels 24, making display unevenness more likely to occur. In one embodiment of the present invention, display unevenness displayed in the display unit 22 can be made less noticeable. As described above, according to one embodiment of the present invention, the display device 20 can be increased in size while suppressing visually noticeable display unevenness in an image displayed on the display unit 22. Furthermore, while suppressing visually noticeable display unevenness in an image displayed on the display unit 22, the density of the pixels 24 provided in the display unit 22 can be increased, making it possible to display a high-definition image on the display unit 22.

[0089] Furthermore, as described above, pixels 24 may become bright spots or dark spots due to poor characteristics or deterioration of the display elements, transistors, etc. contained in the pixels 24. When viewing an image displayed on the display unit 22, bright spots are more noticeable than dark spots, and therefore have a significant impact on visibility. In one aspect of the present invention, a bright spot correction unit 50 or the like corrects bright spots to, for example, dark spots, thereby enabling a high-quality image to be displayed on the display unit 22. Note that bright spot correction can also be performed by the machine learning processing unit 23.

[0090] In the image processing system 10 configured as shown in FIG. 1, a learning unit 46 having a function of generating a machine learning model MLM can be provided in the generation device 40, and a machine learning processing unit 23 having a function of performing processing using the machine learning model MLM can be provided in the display device 20. This allows the display device 20 to perform processing using the machine learning model MLM even if the display device 20 does not generate the machine learning model MLM. To generate the machine learning model MLM, a large amount of learning image data LG IN , and learning image data LG GEN Therefore, by providing the learning unit 46 in the generation device 40, the calculation capacity of the display device 20 can be made lower than the calculation capacity of the generation device 40.

[0091] <How to generate a machine learning model> The method for generating the machine learning model MLM will be described below with reference to the drawings. It is assumed that the display unit 22 has m rows and n columns of pixels 24 arranged in a matrix, as shown in FIG. 2A. The gradation values ​​of the image data are 8-bit digital values, and the smaller the gradation value, the lower the brightness of the light emitted from the pixel 24. For example, if the gradation values ​​are integers between 0 and 255, the brightness of the light emitted from the pixel 24 will be the lowest when the gradation value is 0.

[0092] [Table generation method] Fig. 4 is a flowchart showing an example of a method for generating a table T to be stored in the database 42. As shown in Fig. 4, the table T is generated by a method shown in steps S01 to S04. Figs. 5A and 5B, and 6A and 6B are schematic diagrams showing the operations in steps S01 to S04.

[0093] To generate a table T, first, the database image data DG IN is input to the input unit 21 of the display device 20. The database image data DG IN is input to the display unit 22, and the database image data DGIN Specifically, the image corresponding to the database image data DG IN The pixels 24 emit light of a luminance corresponding to the first gradation value of m rows and n columns, thereby displaying an image on the display unit 22.

[0094] In FIG. 5A, the database image data DG IN is assumed to display an image with the same brightness across the entire surface. In other words, it is assumed that all pixels 24 emit light with the same brightness. On the other hand, in the image displayed on the display unit 22, the brightness of light emitted from some pixels 24 differs from the brightness of light emitted from other pixels 24. In other words, display unevenness occurs. In FIG. 5A, the generated display unevenness is shown as display unevenness 27.

[0095] Next, the image displayed on the display unit 22 is captured by the imaging device 30. As a result, the imaging device 30 captures the captured image data IMG DG is acquired (step S02).

[0096] Here, when an image displayed on the display unit 22 of the display device 20 is captured by the imaging device 30, an object other than the display unit 22 may be captured. For example, the housing of the display device 20 may be captured. In FIG. 5A, the imaging data IMG DG It is assumed that the part of the display device 20 shown in FIG. 5A enclosed by the dashed line is included.

[0097] Thereafter, the image extracting unit 43 extracts the image data IMG DG From database image data DG DP Specifically, the image data IMG DG 5B, in addition to the image displayed on the display unit 22, the housing of the display device 20 extracts the image data IMG DG , the data representing the image displayed on the display unit 22 is taken as the image data IMG DGand removes the data representing the housing of the display device 20. DP Get the image data IMG DG As mentioned above, data extraction from can be performed by pattern matching, template matching, etc.

[0098] As mentioned above, the database image data DG IN The resolution of the image represented by and the database image data DG DP For example, the resolution of the database image data DG IN has m rows and n columns of first gradation values, the database image data DG DP However, for example, if the data representing the image displayed on the display unit 22 is taken as the image data IMG DG When extracted from, the gradation values ​​that the extracted data has may not be m rows and n columns. For example, there may be cases where there are fewer than m rows of gradation values, or there may be cases where there are more than m rows of gradation values. There may also be cases where there are fewer than n columns of gradation values, or there may be cases where there are more than n columns of gradation values.

[0099] As mentioned above, the image data IMG DG If the gradation values ​​of the data extracted from the image extraction unit 43 are not m rows and n columns, the image extraction unit 43 performs up-conversion or down-conversion on the data, thereby obtaining the database image data DG output from the image extraction unit 43. DP For example, it is preferable that the image data IMG DG If the data extracted from the image data IMG has fewer gradation values ​​than m rows or fewer than n columns, the image extraction unit 43 extracts the image data IMG DG Furthermore, the image extraction unit 43 can perform up-conversion on the data extracted from the image data IMG DG If the data extracted from the image data IMG has more than m rows or more than n columns of gradation values, the image extraction unit 43 extracts the image data IMGDG Down-conversion can be performed on the data extracted from the database image data DG DP The number of rows and columns of the second gradation values ​​included in the database image data DG IN The number of rows and columns can be m and n, which are equal to the number of rows and columns of the first gradation values ​​included in the image data. As mentioned above, up-conversion and down-conversion can be performed using the nearest neighbor method, bilinear method, bicubic method, etc.

[0100] And the database image data DG IN and database image data DG DP A table T representing information on the correspondence between and is stored in the database 42 (step S04). As described above, the table T specifically stores the database image data DG IN and the database image data DG DP and second gradation values ​​that correspond to the coordinates of the first gradation values. Table T represents, for example, first gradation values ​​and second gradation values ​​at coordinates that correspond to the coordinates of the first gradation values. Table T represents, for example, first gradation values ​​and second gradation values ​​at coordinates that are the same as the coordinates of the first gradation values.

[0101] 6A is a diagram showing an example of a table T. In FIG. 6A, the database image data DG IN The first gradation value of the database image data DG is shown on the right side of the arrow. DP 2 shows the second gradation value that the

[0102] Here, it is assumed that pixel 24 has the function of emitting red light, green light, and blue light. In this case, the image data has a gradation value (red gradation value) that represents the brightness of red light, a gradation value (green gradation value) that represents the brightness of green light, and a gradation value (blue gradation value) that represents the brightness of blue light. For example, the red gradation value of the i-th row and j-th column is R, the green gradation value is G, and the blue gradation value is B, which is expressed as [R,G,B]. (i,j)For example, if the red gradation value of the first row, first column to the mth row, nth column is R, the green gradation value is G, and the blue gradation value is B, that is, all red gradation values ​​are R, all green gradation values ​​are G, and all blue gradation values ​​are B, then it is expressed as [R,G,B] (1,1)~(m,n) The light emitted by the pixel 24 is not limited to red light, green light, and blue light. For example, the pixel 24 may emit white light. The pixel 24 may also emit cyan light, magenta light, and yellow light. The pixel 24 does not have to emit red light, green light, or blue light. Furthermore, the number of colors of light emitted by the pixel 24 is not limited to three, and the pixel 24 may emit light of one or two colors, or light of four or more colors, for example.

[0103] When the red gradation value, green gradation value, and blue gradation value are each 8-bit digital data, the values ​​that R, G, and B can take can each be integers between 0 and 255. Here, if the brightness of light emitted from a pixel 24 decreases as the gradation value decreases, the gradation value 0 can represent that no light is emitted. Therefore, for example, [0,0,0] can represent that none of the red light, green light, and blue light is emitted from all pixels 24. (1,1)~(m,n) It can be shown by writing:

[0104] From the above, the database image data DG IN The first gradation value is [0,0,0] (1,1)~(m,n) In this case, it can be assumed that no light is emitted from the pixels 24(1,1) to 24(m,n). DP The second tone value is also [0,0,0] (1,1)~(m,n) It can be said that:

[0105] In the method for generating a machine learning model using the image processing system 10, the first gradation value is [1,0,0] (1,1)~(m,n) 〜[255,0,0] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DGDP Also, if the first gradation value is [0,1,0] (1,1)~(m,n) 〜[0,255,0] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DG DP Furthermore, if the first gradation value is [0,0,1] (1,1)~(m,n) 〜[0,0,255] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DG DP That is, the second gradation value of the database image data DG IN The image represented by can be, for example, an image whose entire surface is monochromatic and of the same brightness.

[0106] In this specification, the term "monochromatic" refers to a color represented by a pixel emitting light of one color. For example, if a pixel has the function of emitting red light, green light, and blue light, a red image, a green image, and a blue image are called monochromatic images.

[0107] In this specification, for example, database image data DG IN In the case where the red gradation value in the i-th row and j-th column is 1, the corresponding database image data DG DP The red gradation value in R1 DP (i, j) is written as (i, j). For example, the database image data DG IN In the case where the red gradation value in the i-th row and j-th column is 255, the corresponding database image data DG DP The red gradation value in R255 DP (i, j) is written as (i, j). For example, the database image data DG IN In the case where the green gradation value in the i-th row and j-th column is 1, the corresponding database image data DG DP The green gradation value in G1 DP (i, j) is written as (i, j). For example, the database image data DG INIn the case where the green gradation value in the i-th row and j-th column is 255, the corresponding database image data DG DP The green gradation value in G255 DP (i, j) is written as (i, j). For example, the database image data DG IN In the case where the blue gradation value in the i-th row and j-th column is 1, the corresponding database image data DG DP The blue gradation value in B1 DP (i, j). Furthermore, for example, database image data DG IN In the case where the blue gradation value in the i-th row and j-th column is 255, the corresponding database image data DG DP The blue gradation value in B255 DP This is indicated as (i,j).

[0108] Here, the first gradation value and the second gradation value corresponding to the first gradation value are not necessarily the same. For example, in the database image data DG IN The first gradation value is [128,0,0] (1,1)~(m,n) Even if it was, R128 DP (1,1) to R128 DP (m,n) is not always 128. It may be larger or smaller than 128. For example, R128 DP (1,1) to R128 DP (m, n) are not necessarily all the same value. DP In some cases, display irregularities may occur in the image represented by the image.

[0109] In addition, for all gradation values, the database image data DG IN is input to the display device 20, and the database image data DG DP For example, the second gradation value [1,0,0] may not be acquired. (1,1)~(m,n) 〜[255,0,0] (1,1)~(m,n) Among them, some gradation values ​​of the database image data DG IN is input to the display device 20, and the database image data DG DPAlternatively, the second gradation value [0,1,0] may be obtained. (1,1)~(m,n) 〜[0,1,0] (1,1)~(m,n) Among them, some gradation values ​​of the database image data DG IN is input to the display device 20, and the database image data DG DP Furthermore, a second gradation value having a gradation value [0,0,255] may be obtained. (1,1)~(m,n) 〜[0,0,255] (1,1)~(m,n) Among them, some gradation values ​​of the database image data DG IN is input to the display device 20, and the database image data DG DP may acquire a second gradation value that the

[0110] As mentioned above, the database image data DG of some gradation values IN is input to the display device 20, and the database image data DG DP When acquiring the second gradation value that the display device 20 has, the database image data DG of the gradation value that is not input to the display device 20 is IN Database image data DG corresponding to DP The second gradation value of the database image data DG IN The gradation value can be calculated based on the first gradation value of the pixel and the second gradation value corresponding to the first gradation value. For example, the gradation value can be calculated by proportional interpolation. Alternatively, the gradation value can be calculated using a predetermined formula.

[0111] For example, the tone value [127,0,0] (1,1)~(m,n) Database image data DG DP and [129,0,0] (1,1)~(m,n) Database image data DG DP is input to the display device 20, but the gradation value [128,0,0] (1,1)~(m,n) Database image data DG DP is not input to the display device 20. For example, R127 DP The value of (i,j) is 120, R129 DP Let the value of (i,j) be 124, and R128 DP (i,j) to R127 DP(i,j) and R129 DP It is calculated by proportional interpolation of (i,j). In this case, R128 DP The value of (i, j) can be set to 122. The green gradation value and the blue gradation value can also be calculated in a similar manner.

[0112] Database image data DG for some gradation values IN By inputting these values ​​into the display device 20, it is possible to reduce the number of calculations required to generate the table T. This allows the table T to be generated in a short time.

[0113] In FIG. 6A, the database image data DG IN The image represented by is a monochrome image. IN and database image data DG DP The red gradation value, the green gradation value, and the blue gradation value of the database image data DG are acquired separately. However, one aspect of the present invention is not limited to this. Fig. 6B is a modified example of Fig. 6A. IN and database image data DG DP This differs from the case shown in FIG. 6A in that the red, green, and blue gradation values ​​are acquired at the same time.

[0114] In the case shown in FIG. 6B, the first gradation value is [0,0,0] (1,1)~(m,n) ~[255,255,255] (1,1)~(m,n) The database image data DG IN are input to the display device 20, and the database image data DG DP That is, the second gradation value of the database image data DG IN is an image in which all of the red, green, and blue gradation values ​​are the same (white image), and the database image data DG DP The red, green, and blue gradation values ​​of the image data DG are obtained. The obtained gradation values ​​are represented by a table T. Specifically, the table T is INand the database image data DG of the red gradation values ​​and the coordinates corresponding to the coordinates of the red gradation values. DP The table T represents the red gradation value of the database image data DG IN and the database image data DG of the green gradation values ​​and the coordinates corresponding to the coordinates of the green gradation values. DP Furthermore, the table T represents the green gradation value of the database image data DG IN and the database image data DG of the blue gradation values ​​and the coordinates corresponding to the coordinates of the blue gradation values. DP The table T represents, for example, the database image data DG IN and the database image data DG DP and the red gradation value of the image data DG. IN and the database image data DG DP and the green gradation value of the image data DG IN and the database image data DG DP and the blue gradation value of

[0115] Database image data DG IN By making the image represented by the database image data DG input to the display device 20 a white image, IN Therefore, the number of calculations required to generate the table T can be reduced. This allows the table T to be generated in a short time. IN The red gradation value, green gradation value, and blue gradation value of the database image data DG IN Among the red, green, and blue gradation values ​​of the database image data DG, the gradation value of one color may be different from the gradation values ​​of the other colors. INThe red gradation value, the green gradation value, and the blue gradation value may be different from each other.

[0116] [How to generate a machine learning model] Fig. 7 is a flowchart showing an example of a method for generating a machine learning model MLM using a table T stored in the database 42. As shown in Fig. 7, the machine learning model MLM is generated by a method shown in steps S11 to S16. Figs. 8A, 8B, 9A, 9B, and 10 are schematic diagrams showing the operations in steps S11 to S16.

[0117] To generate a machine learning model MLM, first, we need to create the training image data LG. IN is input to the input unit 21 of the display device 20. The learning image data LG IN is input to the display unit 22, and the display unit 22 displays the learning image data LG IN Specifically, the image corresponding to the learning image data LG IN The pixels 24 emit light of brightness corresponding to the gradation values ​​of m rows and n columns, thereby displaying an image on the display unit 22. Fig. 8A shows a state in which display unevenness 28 occurs in the image displayed on the display unit 22.

[0118] Next, the image displayed on the display unit 22 is captured by the imaging device 30. As a result, the imaging device 30 captures the captured image data IMG LG is acquired (step S12).

[0119] As described above, when an image displayed on the display unit 22 of the display device 20 is captured by the imaging device 30, an object other than the display unit 22 may be captured. For example, the housing of the display device 20 may be captured. In FIG. 8A, the imaging data IMG LG It is assumed that the part of the display device 20 shown in FIG. 8A that is surrounded by a dashed line is included.

[0120] Thereafter, the image extracting unit 43 extracts the image data IMG LG Learning image data LGDP (Step S13). The operation in step S13 is to acquire the image data IMG DG Image data IMG LG and database image data DG DP The learning image data LG DP and FIG. 5B is replaced with FIG. 8B, the explanation of the operation in step S03 can be referred to.

[0121] Next, the learning image data LG IN On the other hand, the learning image data LG DP The image processing unit 44 performs image processing so that the learning image data LG IP (Step S14). FIG. 9A shows the learning image data LG IN and learning image data LG DP and the learning image data LG output from the image processing unit 44. IP An example of this is shown below.

[0122] Learning image data LG IP The generation of learning image data LG IP The sum of the gradation values ​​of the learning image data LG DP The difference between the sum of the gradation values ​​of and is the learning image data LG IP The sum of the gradation values ​​of the learning image data LG IN The learning image data LG is calculated so that the difference between the sum of the gradation values ​​of IN For example, the learning image data LG IP The sum of the gradation values ​​of the learning image data LG DP The learning image data LG IN The learning image data LG IP can be generated.

[0123] In addition, the learning image data LG IP The generation of learning image data LGDP The PSNR or SSIM for the training image data LG IN The training image data LG is set so that the PSNR or SSIM is larger than the IN For example, the learning image data LG DP The training image data LG is used to maximize the PSNR or SSIM for IN The learning image data LG is obtained by converting the gradation values ​​of IP can be generated.

[0124] As described above, the image processing performed by the image processing unit 44 can be, for example, gamma correction. In this case, the image processing can be performed by setting the gamma value to an appropriate value.

[0125] Here, the image processing unit 44 receives the learning image data LG IN It is preferable to perform image processing for each color. IN When performing gamma correction on the learning image data LG, it is preferable to calculate the gamma value for each color. IP and the sum of the red gradation values ​​of the learning image data LG DP The difference between the sum of the red gradation values ​​of and is the learning image data LG IP and the sum of the red gradation values ​​of the learning image data LG IN The learning image data LG is calculated so that the sum of the red gradation values ​​of is smaller than the difference between IN It is preferable to convert the red gradation value of the learning image data LG by image processing. IP and the sum of the green gradation values ​​of the learning image data LG DP The difference between the sum of the green gradation values ​​of and is the learning image data LG IP and the sum of the green gradation values ​​of the learning image data LG IN The learning image data LG is calculated so that the difference between the sum of the green gradation values ​​of INIt is preferable to convert the green gradation value of the learning image data LG by image processing. IP and the sum of the blue gradation values ​​of the learning image data LG DP The difference between the sum of the blue gradation values ​​of and is the learning image data LG IP and the sum of the blue gradation values ​​of the learning image data LG IN The learning image data LG is calculated so that the sum of the blue gradation values ​​of is smaller than the difference between IN It is preferable to convert the blue gradation value of the learning image data LG by image processing. IP The sum of the red gradation values ​​of the learning image data LG DP The learning image data LG IN It is preferable to convert the red gradation value of the learning image data LG by image processing. IP The sum of the green gradation values ​​of the learning image data LG DP The learning image data LG IN It is preferable to convert the green gradation value of the learning image data LG by image processing. IP The sum of the blue gradation values ​​of the learning image data LG DP The learning image data LG IN It is preferable to convert the blue gradation value of the color by image processing.

[0126] As mentioned above, "Learning Image Data LG IP The "total of gradation values ​​of the learning image data LG IP It may be the sum of all the gradation values ​​of the m rows and n columns of the "learning image data LG IN The "total of gradation values ​​of the learning image data LG IN It may be the sum of all the gradation values ​​of the m rows and n columns of the "learning image data LG DP The "total of gradation values ​​of the learning image data LGDP It may be the sum of all the gradation values ​​of the m rows and n columns of the image data, or the sum of some of the gradation values.

[0127] In this specification, learning image data LG IP The red gradation value of the i-th row and j-th column of IP (i, j) are written as (i, j). IP The green gradation value in the ith row and jth column of IP (i, j) are written as (i, j). Furthermore, the learning image data LG IP The blue gradation value in the ith row and jth column of IP This is indicated as (i,j).

[0128] As described above, the learning image data LG output from the image processing unit 44 IP The image represented by is the learning image data LG DP On the other hand, the learning image data LG IN Since the image represented by does not include the display unevenness 28, the learning image data LG IP The image represented by does not include the display irregularity 28.

[0129] Thereafter, based on the table T, the image generating unit 45 generates the learning image data LG IP Learning image data LG GEN 9B shows how the image generating unit 45 generates the table T and the learning image data LG. IP is input, and the learning image data LG GEN The figure shows how the following is output.

[0130] Specifically, first, the learning image data LG IP For example, the second gradation value included in the table T is selected based on the gradation value of the learning image data LG IP The second gradation value that matches or is closest to the gradation value of R is selected for each of the first row and first column to the mth row and nth column. IP The red gradation value that matches or is closest to (i,j) is R0DP (i,j) to R255 DP (i,j). Also, for example, G IP The green gradation value that matches or is closest to (i,j) is G0 DP (i,j) to G255 DP (i,j). Furthermore, for example, B IP The blue gradation value that matches or is closest to (i,j) is B0 DP (i,j) to B255 DP In FIG. 9B, the selected second gradation value in the first row and first column is selected from [Ra DP (1,1),Gb DP (1,1),Bc DP (1,1)] (1,1) (a, b, c are integers between 0 and 255). The second gradation value selected at the mth row and nth column is expressed as [Rs DP (m,n),Gt DP (m,n),Bu DP (m,n)] (m,n) (where s, t, and u are integers between 0 and 255).

[0131] Next, the image generating unit 45 generates learning image data LG 1 , which is image data including the first gradation value corresponding to the selected second gradation value, based on the table T. GEN Specifically, the image generation unit 45 generates learning image data LG, which is image data including the first gradation values ​​in the first row, the first column to the m-th row, the n-th column corresponding to the selected second gradation values ​​in the first row, the first column to the m-th row, the n-th column. GEN For example, in the example shown in FIG. 9B, the learning image data LG GEN In the image, the red gradation value of the first row and first column is a, the green gradation value is b, and the blue gradation value is c. The red gradation value of the mth row and nth column is s, the green gradation value is t, and the blue gradation value is u.

[0132] For example, the learning image data LG IP If the table T does not contain a second gradation value in the ith row and jth column that matches the gradation value in the ith row and jth column of the learning image data LGGEN The tone value in the i-th row and j-th column of the learning image data LG IP Specifically, for example, R IP The red gradation value that matches (i,j) is R0 DP (i,j) to R255 DP If it is not included in (i,j), the learning image data LG GEN The red gradation value in the i-th row and j-th column of IP (i,j). Also, for example, G IP The green gradation value that matches (i,j) is G0 DP (i,j) to G255 DP If it is not included in (i,j), the learning image data LG GEN The green gradation value in the ith row and jth column of IP (i,j). Furthermore, for example, B IP The blue gradation value that matches (i,j) is B0 DP (i,j) to B255 DP If it is not included in (i,j), the learning image data LG GEN The blue gradation value in the i-th row and j-th column of IP It can be (i,j).

[0133] As mentioned above, the learning image data LG IP On the other hand, the image represented by the database image data DG having the second gradation value does not include display unevenness. DP includes display irregularities, etc. Also, database image data DG having a first gradation value IN does not include display irregularities. IP The learning image data LG is image data including a first gradation value corresponding to a second gradation value selected based on the gradation value of GEN is the learning image data LG DP In FIG. 9B and other figures, the learning image data LG GEN1, data that counteracts display unevenness 28 is added to the data corresponding to region 29. For example, if the brightness of the portion where display unevenness 28 occurs is higher than the brightness of the surrounding area of ​​that portion, the brightness of region 29 can be made lower than the brightness of the surrounding area of ​​region 29.

[0134] After step S15, the learning unit 46 uses the learning image data LG IN and learning image data LG GEN A machine learning model MLM is generated using the above (step 16). For example, the learning image data LG IN The image data output when input is the learning image data LG GEN Such a machine learning model MLM is generated based on, for example, the learning image data LG IN and learning image data LG GEN The learning image data LG IN and learning image data LG GEN and are input to the learning unit 46, and the image data output from the learning unit 46 is the learning image data LG GEN 1 shows how a machine learning model MLM is generated so that the image data is the same as that of the image data of the image sensor 1. As mentioned above, a neural network model, for example, can be applied as the machine learning model MLM.

[0135] The above is an example of a method for generating the machine learning model MLM using the image processing system 10. As described above, by performing image processing using the machine learning model MLM on image data input to the input unit 21, the machine learning processing unit 23 can generate image data that cancels out display unevenness. By inputting the image data output from the machine learning processing unit 23 to the display unit 22, the display unit 22 can display an image in which display unevenness is less noticeable. Therefore, as described above, the display device 20 can be enlarged while suppressing visual recognition of display unevenness in an image displayed on the display unit 22. Furthermore, while suppressing visual recognition of display unevenness in an image displayed on the display unit 22, the density of the pixels 24 provided in the display unit 22 can be increased, allowing a high-definition image to be displayed on the display unit 22.

[0136] Furthermore, by performing image processing using the machine learning model MLM, it is possible to cancel out factors that degrade the image quality of a displayed image, in addition to the display unevenness described above. For example, it is possible to cancel out line defects, point defects, etc. Therefore, the display unit 22 can display a high-quality image.

[0137] Here, even when image data is input to the input unit 21 and image data generated by the image generation unit 45 by performing the same processing as steps S11 to S15 is input to the display unit 22, an image in which display unevenness, etc., is made less noticeable can be displayed on the display unit 22. However, performing steps S11 to S15 requires high computing power. On the other hand, image processing using the generated machine learning model MLM can be performed with lower computing power than steps S11 to S15. Therefore, by performing image processing using the machine learning model MLM, image processing can be performed in a short time. Furthermore, image processing can be performed inside the display device 20 without using a device with high computing power such as a server.

[0138] Note that, if the image processing for making display unevenness less noticeable on the image data input to the input unit 21 can be performed by, for example, a device with high computing power, the image processing may be performed by a method similar to steps S11 to S15 without using the machine learning model MLM. Also, if the computing power of the display device 20 is sufficiently high, for example, the image processing for making display unevenness less noticeable on the image data input to the input unit 21 may be performed by a method similar to steps S11 to S15 without using the machine learning model MLM. When image processing is performed without using the machine learning model MLM, the generation device 40 may be configured without including the learning unit 46.

[0139] <Bright spot correction method> Hereinafter, a bright spot correction method, which is an image processing method according to one embodiment of the present invention, will be described with reference to the drawings.

[0140] In a bright spot correction method according to one embodiment of the present invention, a pixel 24 that will become a bright spot is detected, and then correction is performed on the detected pixel 24. FIGS. 11A and 11B are diagrams showing an example of a method for detecting a pixel 24 that will become a bright spot. Here, the method shown in FIG. 11A is referred to as Method M1, and the method shown in FIG. 11B is referred to as Method M2. Note that in FIGS. 11A and 11B, it is assumed that there is no display unevenness or the like.

[0141] In method M1, first, the bright spot correction image data BCG_1 IN is input to the input unit 21 of the display device 20. Then, the learning image data LG IN Bright spot correction image data BCG_1 IN And the image data IMG LG Image data IMG BCG and learning image data LG DP Bright spot correction image data BCG DP and learning image data LG IP Bright spot correction image data BCG IP and learning image data LG GENare read as the bright spot correction image data BCG_1, and the same operations as steps S11 to S15 shown in FIG. 7 and the like are performed (steps S11' to S15'). For example, in steps S11' and S12', IN An image corresponding to the image is displayed on the display unit 22, and the image displayed on the display unit 22 is captured by the imaging device 30, thereby generating image data IMG BCG In addition, the bright spot correction image data BCG obtained in step S13' is obtained. DP can have m rows and n columns of gradation values. Furthermore, in step S15′, the database image data DG IN and the database image data DG DP The image generating unit 45 generates the bright spot correction image data BCG based on the table T which represents information on the correspondence between the second gradation values ​​of IP The bright spot correction image data BCG_1 is generated from the bright spot correction image data BCG DP is not supplied to the image processing unit 44 but to the image generating unit 45, the operation shown in step S14' is not performed. In this case, in step S15', the image generating unit 45 DP The bright spot correction image data BCG_1 is generated from the

[0142] Here, if the pixel that becomes a bright spot is included in the pixels 24(1,1) to 24(m,n), the bright spot correction image data BCG DP Among the gradation values ​​of m rows and n columns of the image data for bright spot correction BCG, the gradation values ​​of the coordinates corresponding to the coordinates of the pixel 24 that will be the bright spot, for example, the gradation values ​​of the coordinates that are the same as the pixel 24 that will be the bright spot, will be high. DP can take any integer value between 0 and 255, the gradation value of the coordinates corresponding to the coordinates of the pixel 24 that is a bright spot will be 255 or a value close to that. DP is higher than the gradation values ​​of the coordinates surrounding the coordinates, the gradation value can be lowered in the bright spot correction image data BCG_1.

[0143] After step S15', the bright spot correction unit 50 of the display device 20 detects bright spot coordinates, which are the coordinates of the pixel 24 that becomes a bright spot, based on the bright spot correction image data BCG_1. Specifically, the coordinates of the gradation values ​​of m rows and n columns contained in the bright spot correction image data BCG_1 that are equal to or less than a threshold value can be set as bright spot coordinates. The detection of bright spot coordinates may be performed by the generation device 40. The detection of bright spot coordinates may be performed, for example, by the image generation unit 45 of the generation device 40.

[0144] Here, the bright spot correction image data BCG DP In both the image data for bright spot correction BCG_1 and the image data for bright spot correction BCG_2, it is preferable that the difference between the gradation value of the bright spot coordinate and the gradation value of the coordinates around the bright spot coordinate is large, because the bright spot coordinate can be detected with high accuracy. IN It is preferable that the gradation value of the bright spot correction image data BCG_1 is an intermediate gradation. IN It is preferable that all of the gradation values ​​of m rows and n columns of the image data for bright spot correction BCG_1 are set to 127 or close to that. IN The gradation values ​​of all the bright spot correction image data BCG are set to the same intermediate gradation value. DP 11A shows an example in which a bright spot includes a gradation value 53 that is higher than the surrounding gradation values. In the example shown in Fig. 11A, among the gradation values ​​of the bright spot correction image data BCG_1, gradation value 54 at the same coordinate as gradation value 53 can be made lower than the surrounding gradation values.

[0145] In method M2, first, the bright spot correction image data BCG_2 IN is input to the input unit 21 of the display device 20. Then, the learning image data LG IN Bright spot correction image data BCG_2 IN And the image data IMG LG Image data IMG BCG and learning image data LG DPis read as the bright spot correction image data BCG_2, and the same operations as steps S11 to S13 shown in FIG. 7 and the like are performed (steps S11'' to S13''). For example, in steps S11'' and S12'', IN An image corresponding to the image is displayed on the display unit 22, and the image displayed on the display unit 22 is captured by the imaging device 30, thereby generating image data IMG BCG The bright spot correction image data BCG_2 acquired in step S13'' can have gradation values ​​of m rows and n columns.

[0146] Here, when a pixel that will become a bright spot is included among pixels 24(1,1) to 24(m,n), the gradation values ​​of the m rows and n columns of the bright spot correction image data BCG_2 will be high for coordinates that correspond to the coordinates of the pixel 24 that will become a bright spot, for example, the gradation value of the same coordinates as the pixel 24 that will become a bright spot. For example, when the gradation values ​​of the bright spot correction image data BCG_2 can take any integer value between 0 and 255, the gradation value of the coordinates that correspond to the coordinates of the pixel 24 that will become a bright spot will be 255 or a value close to that.

[0147] After step S13'', the bright spot correction unit 50 of the display device 20 detects bright spot coordinates, which are the coordinates of the pixel 24 that becomes a bright spot, based on the bright spot correction image data BCG_2. Specifically, the coordinates of gradation values ​​equal to or greater than a threshold value among the m rows and n columns of gradation values ​​contained in the bright spot correction image data BCG_2 can be set as bright spot coordinates. The detection of bright spot coordinates may be performed by the generation device 40. The detection of bright spot coordinates may also be performed by, for example, the image extraction unit 43 of the generation device 40.

[0148] Here, in the bright spot correction image data BCG_2, it is preferable that the difference between the gradation value of the bright spot coordinate and the gradation value of the surrounding bright spot coordinate is large, because the bright spot coordinate can be detected with high accuracy. On the other hand, if the gradation value of the image data input to the display unit 22 is too small, even pixels 24 that could become bright spots depending on the gradation value will not become bright spots, and the bright spot coordinate cannot be detected with high accuracy. Bright spot correction image data BCG_2IN It is preferable to determine the gradation value of the bright spot correction image data BCG_2 based on the above. IN It is preferable that all of the gradation values ​​of m rows and n columns of BCG_2 be between 0 and 127, or between 31 and 127, or between 63 and 127. Fig. 11B shows an example in which the bright spot correction image data BCG_2 includes a bright spot with a gradation value of 55 that is higher than the surrounding gradation values.

[0149] By detecting the coordinates of the bright spots, the bright spot correction unit 50 can have the function of correcting the bright spots. For example, the content image data CG ML Among the gradation values ​​of m rows and n columns of the content image data CG, the gradation values ​​of the coordinates corresponding to the bright spot coordinates, for example, the gradation values ​​of the coordinates that are the same as the bright spot coordinates, can be reduced, for example, to 0. COR By generating and supplying the content image data CG to the display unit 22, it is possible to make a pixel 24 that would otherwise be a bright spot a dark spot. As described above, when viewing an image displayed on the display unit 22, a bright spot stands out more than a dark spot, and therefore has a large effect on visibility. COR By displaying an image corresponding to the above on the display unit 22, the image displayed on the display unit 22 can be made high quality.

[0150] FIG. 12A shows the actual measured gradation values ​​of the bright spot correction image data BCG_1 and the IN This is a graph showing the relationship between the brightness correction image data BCG_1 and the gradation value, which can be created by method M1. IN In FIG. 12A, the same gradation value of the bright spot correction image data BCG_1 IN There are multiple plots for the gradation value of BCG_1, but this is because the actual measured gradation values ​​of the bright spot correction image data BCG_1 are plotted for multiple coordinates.

[0151] The line 56 shown in FIG. 12A represents the bright spot correction image data BCG_1 IN12A shows the average of the gradation values ​​of the plotted image data for bright spot correction BCG_1 at each gradation value. IN The relationship between the gradation value of and can be linearly approximated.

[0152] As described above, the gradation values ​​of the bright spot correction image data BCG_1 at the bright spot coordinates are lower than the gradation values ​​at the non-bright spot coordinates. IN For example, among the gradation values ​​of the image data for bright spot correction BCG_1 with m rows and n columns, the coordinates of gradation values ​​less than the threshold value can be set as bright spot coordinates. In Fig. 12A, the threshold value is indicated by line 57. Line 57 can be expressed by a linear equation with a positive slope.

[0153] In the method M1, a plurality of image data for bright spot correction BCG_1 having different gradation values IN and generate the bright spot correction image data BCG_1 for each of them, it is possible to prevent the coordinates of the pixel 24 that becomes a bright spot from being determined to be not a bright spot coordinate, and the coordinates of the pixel 24 that does not become a bright spot from being determined to be a bright spot coordinate. Therefore, the bright spot correction unit 50 and the like can detect the bright spot coordinates with high accuracy.

[0154] FIG. 12B shows the bright spot correction image data BCG generated by the image extraction unit 43. DP and the bright spot correction image data BCG input to the input unit 21. IN 12B is a graph showing the relationship between the gradation value of the bright spot correction image data BCG DP The gradation value of the bright spot correction image data BCG IN The relationship between the gradation values ​​cannot be approximated linearly, but is approximated by a sigmoid curve, for example.

[0155] Next, an example of the pixel 24 that can be detected by the method M2 will be described.IN 10 is a graph showing the relationship between the gradation values.

[0156] Graph 61 shown in FIG. 13A1 is a graph showing the relationship between the average value of the m-row, n-column gradation values ​​of the bright spot correction image data BCG_2 at the time of manufacturing the display device 20, for example, and the average value of the m-row, n-column gradation values ​​of the bright spot correction image data BCG_2 IN The relationship of the gradation values ​​can be as follows. IN The gradation values ​​of the bright spot correction image data BCG_2 can be, for example, the same value across the entire surface. Here, it is assumed that some of the gradation values ​​of the bright spot correction image data BCG_2 behave as shown in graph 63. In other words, as the gradation value of the image data input to the display unit 22 increases, the brightness of light emitted from some of the pixels 24 decreases. Pixels 24 that behave as shown in graph 63 are prone to deterioration, and when the pixels 24 are used for a long period of time, that is, when a voltage is supplied to the display element of the pixels 24 for a long period of time, the behavior changes to that of graph 63A shown in FIG. 13A2. On the other hand, pixels 24 that behave as shown in graph 61 are unlikely to deteriorate, and do not behave as shown in graph 63A even when used for a long period of time.

[0157] A pixel 24 exhibiting the behavior shown in graph 63A can be said to be a bright spot. Therefore, a pixel 24 exhibiting the behavior shown in graph 63 of FIG. 13A1 is not a bright spot at the time of manufacturing the display device 20, for example, but is likely to become a bright spot as the display device 20 is used. As described above, if a pixel 24 becomes a bright spot, it will have a significant impact on visibility. Therefore, it is preferable to turn a pixel 24 exhibiting the behavior shown in graph 63 of FIG. 13A1 into a dark spot, for example, by reducing the voltage supplied to the display element. This can prevent the pixel 24 from becoming a bright spot, thereby improving the reliability of the display device 20.

[0158] 13B is a diagram showing an example of a method for detecting a pixel 24 that exhibits the behavior shown in the graph 63. As shown in FIG. 13B, IN The gradation value of the bright spot correction image data BCG_2 is high. INAll of the gradation values ​​of m rows and n columns of the image data for bright spot correction BCG_2 are set to 255 or close to that. IN When steps S11'' to S13'' are performed based on this, the gradation values ​​of the bright spot correction image data BCG_2 that correspond to pixels 24 that behave as shown in graph 63 become lower than the surrounding gradation values. FIG. 13B shows an example in which the bright spot correction image data BCG_2 includes a gradation value 65 that is lower than the surrounding gradation values. The pixel 24 that corresponds to the gradation value 65 can be considered to be a pixel 24 that behaves as shown in graph 63, that is, a pixel 24 that becomes a bright spot after long-term use.

[0159] Therefore, method M2 can detect not only pixels 24 that are already bright spots, but also pixels 24 that are likely to become bright spots as the display device 20 is used. On the other hand, method M1 detects the gradation values ​​of the bright spot correction image data BCG_1 that correspond to pixels 24 that behave as shown in graph 63, which are higher than the gradation values ​​of the surrounding pixels. Therefore, the bright spot correction image data BCG_1 input to the input unit 21 IN If the gradation value of the bright spot correction image data BCG_1 is increased, it becomes difficult to detect the pixel 24 that behaves as shown in the graph 63. IN If the gradation value is lowered, graph 63 becomes closer to graph 61, making it difficult to detect pixel 24 that behaves as shown in graph 63. For the reasons described above, it is difficult to detect pixel 24 that behaves as shown in graph 63 with method M1.

[0160] As described above, by performing both method M1 and method M2, it is possible to detect, for example, pixels 24 that are bright spots with high accuracy, as well as to detect pixels 24 that are likely to become bright spots as the display device 20 is used. In this way, by performing both method M1 and method M2, it is possible to comprehensively detect, for example, pixels 24 that should be dark spots.

[0161] FIG. 14 shows the actual measured gradation values ​​of the bright spot correction image data BCG_1 and the IN12A is a graph showing the relationship between the gradation value and the line 57, and differs from the graph shown in FIG. 12A in that lines 57A and 57B are shown instead of line 57.

[0162] 14, of the gradation values ​​of the bright spot correction image data BCG_1, the gradation value indicated by line 57A is set to the first threshold value, and the gradation value indicated by line 57B is set to the second threshold value. The first threshold value is smaller than the value indicated by line 56, and the second threshold value is smaller than the first threshold value. Lines 57A and 57B can be expressed by linear equations with positive slopes, just like line 57.

[0163] An example of an image processing method using the image processing system 10 will be described with reference to Fig. 14. First, the graph shown in Fig. 14 is created by method M1. In the graph shown in Fig. 14, for example, among the gradation values ​​of the m rows and n columns of bright spot correction image data BCG_1, the coordinates of the gradation values ​​that are less than a first threshold value and greater than or equal to a second threshold value are defined as first bright spot coordinates. Furthermore, the coordinates of the gradation values ​​that are less than the second threshold value are defined as second bright spot coordinates.

[0164] Furthermore, the bright spot coordinates are detected by the method M2, and the detected bright spot coordinates are set as the third bright spot coordinates.

[0165] And content image data CG ML is input to the bright spot correction unit 50, the content image data CG ML Among the gradation values ​​of m rows and n columns of the pixel 22, for example, the gradation value of the coordinates that are the same as the first bright spot coordinates and also the third bright spot coordinates is reduced. Also, for example, the gradation value of the coordinates that are the same as the second bright spot coordinates is reduced regardless of whether they are the same as the third bright spot coordinates. This makes it possible to correct the bright spots and display a high-quality image on the display unit 22.

[0166] By performing image processing in the above manner, it is possible to prevent pixels 24 that do not have a significant effect on visibility even if they are not made into dark points from being made into dark points. This makes it possible to prevent the quality of the image displayed on the display unit 22 from being reduced by making pixels 24 into dark points. In the example shown in FIG. 14, among the pixels 24 in m rows and n columns, for example, the pixel 24 with coordinates that are the same as the first bright point coordinates but different from the third bright point coordinates can be made into a pixel that does not have a significant effect on visibility even if it is not made into a dark point. Therefore, for example, content image data CG ML Among the gradation values ​​of m rows and n columns of the image, the gradation values ​​of the coordinates that are the same as the first bright spot coordinates but different from the third bright spot coordinates can be left uncorrected.

[0167] The above is an example of the bright spot correction method, which is an image processing method according to one embodiment of the present invention.

[0168] <Example of machine learning model configuration> FIG. 15A is a diagram showing an example configuration of a machine learning model MLM. As shown in FIG. 15A, the machine learning model MLM can be a neural network model having an input layer IL, intermediate layers ML1, ML2, ML3, and an output layer OL. The input layer IL, intermediate layers ML1, ML3, and output layer OL each have a plurality of layers made up of neurons, and the neurons provided in each layer are connected to each other. Image data can be input to the input layer IL.

[0169] The image data input to the input layer IL can have m rows and n columns of matrices, the number of which is the same as the number of types of sub-pixels included in the display unit 22. For example, if the pixel 24 has sub-pixels that emit red (R) light, sub-pixels that emit green (G) light, and sub-pixels that emit blue (B) light, the image data can have an m rows and n column matrix with red gradation values ​​as components, an m rows and n column matrix with green gradation values ​​as components, and an m rows and n column matrix with blue gradation values ​​as components. In other words, the image data can be configured to have three matrices.

[0170] If the image data includes matrices as described above, the number of neurons in the input layer IL can be set to the same number as the number of components in the matrices. For example, if the image data includes three matrices with 1200 rows and 1920 columns, the number of neurons in the input layer IL can be set to 1920 × 1200 × 3. If the image data includes matrices, the number of neurons in the output layer OL can be set to the same number as the number of components in the matrices. For example, if the image data includes three matrices with 1200 rows and 1920 columns as described above, the number of neurons in the output layer OL can be set to 1920 × 1200 × 3.

[0171] The hidden layer ML1 has a function of generating data D1 to be supplied to the hidden layer ML2. The data D1 can be a matrix having h elements x (h is an integer equal to or greater than 2).

[0172] In this specification, for example, h components x are referred to as components x1 to x h The same notation is used for other ingredients.

[0173] The number of neurons in the hidden layer ML1 is set to be greater than the number of neurons in the input layer IL. This allows the number of components in the data D1 to be greater than the number of components in the image data input to the input layer IL. Details of the calculation processes that can be performed by the hidden layer ML1 will be described later.

[0174] The hidden layer ML2 has a function of converting the component x into the component y. For example, the hidden layer ML2 converts the components x1 to x2 into a component y using a nonlinear polynomial function of one variable. h are the components y1 to y h An example of this function is shown below.

[0175]

number

[0176] Here, i can be an integer between 1 and h. The above formula is a function with component x as an independent variable, component y as a dependent variable, and a as a coefficient. The function has a term including x raised to the dth power (d is an integer greater than or equal to 2). In FIG. 15A, it is assumed that the hidden layer ML2 performs the arithmetic processing shown in the above formula.

[0177] Another example of a nonlinear polynomial function of one variable is shown below.

[0178]

number

[0179] The above formula is a function with x as the independent variable, y as the dependent variable, and a and b as coefficients. The function has a term including the cosine of the component x and a term including the sine of the component x. Note that the function does not necessarily have to have a term including the cosine of the component x. Also, the function does not necessarily have to have a term including the sine of the component x.

[0180] As a result, the hidden layer ML2 is composed of components y1 to y h This data is referred to as data D2. Data D2 can be a matrix, similar to data D1.

[0181] The hidden layer ML3 has the function of generating data to be supplied to the output layer OL. The number of neurons in the hidden layer ML3 is set to be greater than the number of neurons in the output layer OL. This allows the number of components in the image data output from the output layer OL to be less than the number of components in the data D2. Details of the arithmetic processing that can be performed by the hidden layer ML3 will be described later.

[0182] Two or more hidden layers may be provided between the input layer IL and the hidden layer ML2. Also, two or more hidden layers may be provided between the hidden layer ML2 and the output layer OL.

[0183] Figure 15B is a diagram showing an example of a method for generating the machine learning model MLM when the machine learning model MLM has the configuration shown in Figure 15A. As shown in Figure 7, the machine learning model MLM is generated in step S16. Therefore, Figure 15B can be said to be a diagram showing an example of the operation of step S16 when the machine learning model MLM has the configuration shown in Figure 15A.

[0184] As described above, the machine learning model MLM can be generated by the learning unit 46. Specifically, for example, the learning image data LG IN and learning image data LG GEN For example, a machine learning model MLM can be generated using the learning image data LG IN The image data output when input is the learning image data LG GEN The coefficient a is set to match 1,0 or coefficient a n,k By acquiring the values ​​of coefficient a and the like through learning, the learning unit 46 can generate the machine learning model MLM. When the hidden layer ML2 performs the calculation shown in Equation 2, the value of coefficient b is also acquired through learning in addition to the value of coefficient a.

[0185] 16A and 16B are diagrams showing an example of calculations performed by the machine learning processing unit 23 to which the machine learning model MLM is applied. As shown in FIG. 16A, the hidden layer ML1 calculates the content image data CG IN In the example shown in FIG. 16A, the content image data CG IN contains three matrices with 1200 rows and 1920 columns. In other words, the content image data CG IN is data with a width of 1920, a height of 1200, and a channel count of 3. The channel count of the filter fa is 3, and the content image data CG IN and nine filters fa (filters fa1 to fa9). By performing such a multiplication and addition operation, data D1 with a height of 1200, a width of 1920, and nine channels can be output from the hidden layer ML1. The data D1 is made up of components x1 to x1920×1200×9 This includes:

[0186] The components x1 to x2 of the data D1 1920×1200×9 In the hidden layer ML2, the components y1 to y 1920×1200×9 The components y1 to y 1920×1200×9 The data including this is referred to as data D2.

[0187] As shown in FIG. 16B, the intermediate layer ML3 can perform a product-sum operation between data D2 and filter fb. Here, data D2 can be data with a height of 1200, a width of 1920, and a number of channels of 9, similar to data D1. Furthermore, the number of channels of filter fb is set to 9, and the intermediate layer ML3 performs a product-sum operation between data D2 and three filters fb (filters fb1 to fb3). By performing such a product-sum operation, data with a width of 1920, a height of 1200, and a number of channels of 3 can be output from the intermediate layer ML3. This data is represented by content image data CG ML It can be said that:

[0188] From the above, using the machine learning model MLM, for example, content image data CG IN Content image data CG ML can be converted to

[0189] As described above, in the machine learning model MLM having the configuration shown in FIG. 15A, in the intermediate layer ML2, the components x1 to x2 included in the data D1 are h are calculated by using a nonlinear polynomial function of one variable to calculate the components y1 to y h This allows us to convert the components x1 to x2 using a linear function of one variable or a monomial function of one variable. h are the components y1 to y hInference using the machine learning model MLM can be performed with higher accuracy than when converting to . Furthermore, since the number of filters fa shown in Fig. 16A and the number of channels of filter fb shown in Fig. 16B can be reduced, the amount of calculation required to generate the machine learning model MLM through learning and for inference using the machine learning model MLM can be reduced. Therefore, learning and inference can be performed at high speed. [Example]

[0190] In this example, the results of learning performed to obtain the machine learning model MLM shown in FIG. 15A will be described.

[0191] In this example, a machine learning model MLM was generated by supervised learning using image data with a width of 1920, a height of 1200, and three channels as training data and ground truth data. The component x contained in data D1 was converted to a component y by the hidden layer ML2 using Formula 1, Formula 2, or the formula "y = ax + b." In Formula 1 and Formula 2, d = 5.

[0192] Moreover, the hidden layer ML1 performs the calculation shown in Fig. 16A, and the hidden layer ML3 performs the calculation shown in Fig. 16B. When converting a component x included in data D1 into a component y using Equation 1 or Equation 2, the hidden layer ML1 performs a product-sum calculation between the image data and the filters fa1 to fa9 with three channels, and the hidden layer ML3 performs a product-sum calculation between the data D1 and the filters fa1 to fa3 with nine channels. When converting a component x included in data D1 into a component y using Equation 1 or Equation 2, the hidden layer ML1 performs a product-sum calculation between the image data and the filters fa1 to fa9 with three channels. 162 The hidden layer ML3 performs a multiply-and-accumulate operation between the data D2 and the filters fb1 to fb3, each having 162 channels.

[0193] Figure 17 is a graph showing the relationship between SSIM and the number of learning epochs. SSIM was calculated using test data and correct answer data. The larger the SSIM, the higher the similarity between the test data and correct answer data, indicating that the machine learning model MLM can make inferences with high accuracy. The test data was image data with a width of 1920, a height of 1200, and three channels, just like the training data and correct answer data.

[0194] As described above, when the component x included in the data D1 is converted into the component y using Formula 1 or Formula 2, the number of filters fa and the number of channels of the filter fb are smaller than when the component x is converted into the component y using the formula "y=ax+b." Despite this, as shown in Fig. 17, when the number of learning times is 200 or more, when the component x is converted into the component y using Formula 1 or Formula 2, the SSIM is larger than when the component x is converted into the component y using the formula "y=ax+b." [Explanation of symbols]

[0195] 10: Image processing system, 20: Display device, 21: Input unit, 22: Display unit, 23: Machine learning processing unit, 24: Pixel, 26: Region, 29: Region, 30: Imaging device, 33: Pixel, 40: Generation device, 42: Database, 43: Image extraction unit, 44: Image processing unit, 45: Image generation unit, 46: Learning unit, 50: Bright spot correction unit, 51: Bright spot, 52: Region, 53: Gradation value, 54: Gradation value, 55: gradation value, 56: line, 57: line, 57A: line, 57B: line, 61: graph, 63: graph, 63A: graph, 65: gradation value, 126: wiring, 134: wiring, 161: transistor, 162: transistor, 170: light-emitting element, 171: transistor, 173: capacitance, 174: wiring, 175: wiring, 180: liquid crystal element, 181: capacitance, 182: wiring, 183: wiring

Claims

[Claim 1] The image processing system includes an input unit, a machine learning processing unit, a display unit in which pixels are arranged in a matrix of m rows and n columns (m and n are integers of 2 or more), a database, an image extraction unit, an image processing unit, an image generation unit, and a learning unit, the database stores a table generated based on first image data input to the input unit and second image data acquired by capturing an image of the display unit displaying an image corresponding to the first image data; the first image data has first gradation values ​​arranged in m rows and n columns; the second image data has second gradation values ​​arranged in m rows and n columns, the table represents the first gradation value and the second gradation value at a coordinate corresponding to the coordinate of the first gradation value; the image processing unit has a function of generating third learning image data by performing image processing on the first learning image data input to the input unit based on the second learning image data; the second learning image data is acquired by capturing an image of the display unit that displays an image corresponding to the first learning image data; the third learning image data has third gradation values ​​arranged in m rows and n columns, the image extraction unit has a function of acquiring the second image data by extracting data of a portion representing the image displayed on the display unit from first imaging data acquired by imaging the display unit on which an image corresponding to the first image data is displayed, The image extraction unit has a function of acquiring the second learning image data by extracting data of a portion representing the image displayed on the display unit from second imaging data acquired by imaging the display unit that displays an image corresponding to the first learning image data, and the image generation unit has a function of generating fourth learning image data including the first gradation value corresponding to the second gradation value selected based on the third gradation value; The learning unit has a function of generating a machine learning model such that image data output when the first learning image data is input matches the fourth learning image data, and outputting the machine learning model to the machine learning processing unit; The machine learning processing unit is an image processing system having a function of performing processing using the machine learning model on content image data input to the input unit.

Images