Image contrast enhancement method and information processing device

Abstract

The image contrast enhancement method is a method for enhancing the contrast of an image obtained by capturing an image display surface of a display device. It includes the step of enhancing the contrast of the image by an information processing device. The line graph showing the relationship between the gray values ​​of the pixels contained in the image before contrast enhancement and the values ​​after contrast enhancement is a curve with two inflection points.

Description

Technical Field

[0001] This disclosure relates to a method for enhancing image contrast and an information processing apparatus. Background Technology

[0002] A method for evaluating the brightness unevenness of a display device displaying an image is known (e.g., Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2009-180583 Summary of the Invention

[0006] In evaluations related to display defects in display devices, such as uneven brightness, contrast enhancement is performed to make features caused by display defects clearer. However, simply enhancing contrast is insufficient to make features caused by display defects clearer. Therefore, a contrast enhancement method for images that achieves more appropriate parameter settings for making features caused by display defects clearer is sought.

[0007] This disclosure is made in view of the above-mentioned problems, and its purpose is to provide a method and information processing apparatus for enhancing the contrast of an image by setting more appropriate parameters to make features caused by display defects clearer.

[0008] One aspect of this disclosure is an image contrast enhancement method that involves capturing an image on the image display surface of a display device. The information processing apparatus includes a step of performing the image contrast enhancement. The line graph showing the relationship between the grayscale values ​​of the pixels contained in the image before and after contrast enhancement is a curve with two inflection points. Attached Figure Description

[0009] Figure 1 It is a block diagram representing the main structure of an information processing device.

[0010] Figure 2 It is a block diagram representing the main functional structure of an information processing device.

[0011] Figure 3 This is a diagram illustrating the classification of images and a schematic example of the images in each classification during the judgment process performed by the judgment unit.

[0012] Figure 4 It is a line graph that shows the relationship between the gray values ​​of pixels before contrast enhancement and the gray values ​​of pixels after contrast enhancement.

[0013] Figure 5This is a block diagram illustrating a more specific functional structure example of the decision-making section.

[0014] Figure 6 This is a diagram illustrating the considerations for leaving room for verification.

[0015] Figure 7 This is a diagram illustrating the considerations for cross-validation.

[0016] Figure 8 This is a diagram illustrating the considerations related to the accuracy of judgments made using machine learning.

[0017] Figure 9 It is a flowchart representing a machine learning process that employs multi-stage processing.

[0018] Figure 10 This is a diagram illustrating the process of machine learning that employs multi-stage processing.

[0019] Figure 11 This indicates the actual category of the verification data contained in the reference data and the category obtained through reference. Figure 9 The diagram illustrates the correspondence between the output (predicted category) of the first verification process performed in step S3.

[0020] Figure 12 It means in reference Figure 9 The diagram illustrates the correspondence between the actual category of the validation data contained in the "reference data corresponding to a specific output" input to the machine learning department in step S6, and the output (predicted category) of the second validation process (first time) performed through step S6.

[0021] Figure 13 It indicates a reference. Figure 9 The diagram illustrates the correspondence between the actual category of the validation data extracted in step S4 and input into the machine learning department in step S7, and the output (predicted category) of the second validation process (second time) performed through step S7.

[0022] Figure 14 This indicates the actual category of the verification data contained in the reference data and the category obtained through reference. Figure 9 The diagram illustrates the correspondence between the final output (predicted category) obtained from step S8.

[0023] Figure 15 This is a flowchart illustrating the judgment process performed by the machine learning department, which applies multi-stage processing to image data classification judgments that utilize machine learning.

[0024] Figure 16It is a line graph that shows the relationship between the gray values ​​of pixels before contrast enhancement and the gray values ​​of pixels after contrast enhancement.

[0025] Figure 17 This is a graph showing the relationship between the grayscale distribution of an image taken from a display panel with "uneven tilt" and the grayscale distribution of an image taken from a display panel with "qualified" grayscale.

[0026] Figure 18 It is a graph showing the difference between equation (1) and equation (2), as well as the line graph corresponding to equation (2) and the relationship between gain_1, gain_2, inflection_1, inflection_2, height_1, and height_2.

[0027] Figure 19 This diagram illustrates an image taken of a display panel classified as "qualified" and an example of applicable image processing for that image.

[0028] Figure 20 This diagram illustrates an image captured from a display panel that meets both the "scan line defect" and "signal line defect" criteria, and an example of applicable image processing for that image.

[0029] Figure 21 This diagram illustrates an image taken of a display panel classified as "unevenly tilted" and an example of applicable image processing for that image.

[0030] Figure 22 This diagram illustrates an image taken of a display panel classified as "uneven whiteness" and an example of applicable image processing for that image.

[0031] Figure 23 It is a graph showing the relationship between the histogram of the distribution of gray values ​​of pixels in an image captured from a display panel and the position of the inflection point (b). Detailed Implementation

[0032] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, the disclosure is merely an example, and appropriate modifications that can be readily conceived by those skilled in the art while maintaining the spirit of the invention are naturally included within the scope of this disclosure. Additionally, to make the description clearer, there are instances where the width, thickness, shape, etc., of various parts are schematically represented in the drawings compared to the actual form; however, these are merely examples and do not limit the interpretation of the present disclosure. Furthermore, in this specification and the various drawings, sometimes the same reference numerals are used for elements that are the same as those described above with respect to existing drawings, and detailed descriptions are appropriately omitted.

[0033] Figure 1This is a block diagram showing the main structure of the information processing apparatus 1. The information processing apparatus 1 includes a communication unit 11, an input unit 12, an arithmetic unit 21, a storage unit 22, and an output unit 30. The information processing apparatus 1 is capable of using machine learning to determine which of j (e.g., j=8) categories an image obtained from the image display surface of an image capture display device (e.g., an image of the image data contained in the judgment object image data 100 or reference data 223, described later) conforms to. Furthermore, the information processing apparatus 1 is capable of enhancing the contrast of the image.

[0034] The communication unit 11 communicates with external devices. Specifically, the communication unit 11 may have circuitry that functions as a NIC (Network Interface Controller). The communication unit 11 outputs data received from the external device to the arithmetic unit 21. Furthermore, when the information processing device 1 processes data to be sent to the external device, the communication unit 11 sends that data to the external device. Moreover, the external device may be a fixed information processing device such as a server or PC (Personal Computer) different from the information processing device 1, a mobile terminal such as a smartphone, or other types of information processing devices not illustrated herein.

[0035] The input unit 12 has an interface configured to input data. This interface may be, for example, a USB (Universal Serial Bus) or a bus interface corresponding to other standards, but is not limited to these; the specific standard can be appropriately modified. An external device may be connected to this bus interface, for example, via a cable for inputting data from an external device. Input devices such as a keyboard and mouse for operator operation of the information processing device 1 may also be connected to this bus interface. At least one of the communication unit 11 and the input unit 12 functions as the acquisition unit 40.

[0036] The arithmetic unit 21 has arithmetic circuitry that functions as a CPU (Central Processing Unit). The arithmetic unit 21 performs arithmetic processing corresponding to the various functions of the information processing device 1 by reading software / programs and data referenced during the execution of the software / programs from the storage unit 22 and executing them. Hereinafter, when described as programs, this refers to the software / programs read and executed by the arithmetic unit 21, as well as the data referenced during the execution of the software / programs.

[0037] Storage unit 22 includes a storage device for storing programs, etc. This storage device includes, for example, one or more of a solid-state drive (SSD), hard disk drive (HDD), and other rewritable storage devices, but may also include a non-rewritable storage device containing programs, etc. Figure 1 In the diagram, the storage unit 22 contains programs such as a machine learning program 221, an image processing program 222, reference data 223, and setting data 224. The arithmetic unit 21 and the storage unit 22, which stores programs read from the arithmetic unit 21, function as the processing unit 50. The machine learning program 221 is a program for performing machine learning. The image processing program 222 is a program for performing image processing on the image data 100 of the judgment object. The reference data 223 contains training data and validation data for machine learning. The setting data 224 contains various parameters related to the execution of the machine learning program 221 and the image processing program 222.

[0038] The output unit 30 outputs data corresponding to the processing content of the processing unit 50. The output unit 30 is, for example, a monitor displaying images, but is not limited to this. The output unit 30 may also include a structure that outputs data in other ways, such as a speaker for voice output. In the case where the communication unit 11 transmits data to an external device, the communication unit 11 can be said to function as the output unit 30. Furthermore, if the input unit 12 is a standard capable of bidirectional data transmission, the input unit 12 also functions as the output unit 30.

[0039] Figure 2 This is a block diagram showing the main functional structure of the information processing device 1. The information processing device 1 functions as an acquisition unit 40, a processing unit 50, and an output unit 30. The acquisition unit 40 acquires image data 100 of the judgment object. The communication unit 11 can acquire the image data 100 of the judgment object from an external device via communication. The input unit 12 can acquire the image data 100 of the judgment object input from an external device.

[0040] The processing unit 50 functions as both the image processing unit 51 and the judgment unit 60. The image processing unit 51 performs image processing on the judgment object image data 100. The program used by the arithmetic unit 21, which functions as the image processing unit 51, is an image processing program 222. The judgment unit 60 performs judgment processing by classifying the image using a neural network (NN). The program used by the arithmetic unit 21, which functions as the judgment unit 60, is a machine learning program 221. The image that becomes the judgment object for the judgment unit 60 is obtained by reading image data contained in the judgment object image data 100.

[0041] The output unit 30 outputs information related to the result of the judgment process of the judgment unit 60. For example, the output unit 30 displays the result of the judgment process of the judgment unit 60.

[0042] Figure 3 This diagram illustrates the classification of images and a schematic example of the images in the judgment process performed by the judgment unit 60. For example... Figure 3 As shown, the image is classified into one of the following categories by the judgment processing of the judgment unit 60: qualified product, circuit defect, signal line defect, scan line defect, tilt unevenness, vertical stripes, clustering, and white unevenness.

[0043] In the aforementioned defect patterns, tilt unevenness refers to a defect where an area with a brightness difference relative to its surroundings appears to extend along a long axis. Although described as "tilted," it also includes situations where it extends horizontally or vertically relative to the screen. Vertical stripes refer to defects that appear to produce a brightness difference corresponding to signal line segments, etc. Clustering refers to defects that are identified as concentrated and clustered in areas with slight defects. White unevenness refers to a defect where, in the case of uniform grayscale display within a plane, areas appear irregularly brighter / darker than their surroundings. In this invention, the detection is not limited to these types of unevenness themselves; defect patterns not included above can also be further defined.

[0044] The image data 100 for judging objects includes multiple image data. This image data is obtained by capturing images of the display surface of a display panel in a state where it is displaying a pure white or mid-tone image. Here, the state of displaying a pure white image refers to a state where all pixels are displayed at the highest grayscale value of white. The state of displaying a pure mid-tone image refers to a state where all pixels are displayed at a grayscale value lower than the highest grayscale value and higher than the lowest grayscale value. The display panel is, for example, an OLED (Organic Light Emitting Diode) display panel, but is not limited to this and may be other types of display panels. Each of the multiple image data included in the image data 100 for judging objects is obtained by capturing images of its own different display panel. In the embodiment, the judgment process performed by the judgment unit 60 is intended to automatically check whether the multiple display panels that are the subjects of the photographs during the generation of the image data 100 for judging objects can each display with the intended quality. Hereinafter, when only display panels are described, it refers to one of the multiple display panels that are the subjects of the photographs during the generation of the image data 100 for judging objects. In addition, if it is only recorded as an image, unless otherwise specified, it refers to an image obtained by reading out image data.

[0045] The image data 100 and reference data 223 can be either an image captured in the state of the Mother Glass substrate and an image cropped from each panel, or an image captured from each panel after the Mother Glass substrate has been cut into sections.

[0046] The image data contained in the object image data 100 and the reference data 223 is determined to be a grayscale image or a black-and-white image. Furthermore, if the image data input to the information processing device 1 is in color, the image processing unit 51, described later, can sequentially perform grayscale processing based on each primary color.

[0047] A qualified product refers to a display panel that can output images with the intended quality. In other words, a display panel that captures an image that is judged to be a qualified product is a qualified product.

[0048] Circuit malfunction refers to a characteristic display defect caused by a malfunction in part or all of the driving circuitry that supplies image or scan signals to the pixels constituting the display screen. Figure 3 The illustration shows a characteristic display defect where part or all of the image becomes a noticeably dark black area BA or an excessively bright white area WA for the display of a pure color image with white or mid-tones. The display panel that captured the image was determined to have a circuit defect exhibited a circuit defect.

[0049] A signal line malfunction refers to a characteristic display defect caused by a defect in a portion of the multiple signal lines transmitting image signals to multiple pixels on the display panel. Figure 3 In the example shown, a characteristic display defect is illustrated by a linear pattern along the long side of a rectangular image, where a black area BA is noticeably dark for displaying a pure color image with white or mid-tones, or an overly bright white area WA. Furthermore, in the rightmost example, a black area BA is also generated along a direction orthogonal to this long side.

[0050] Scan line defects refer to characteristic display defects caused by a defect in a portion of the multiple scan lines that transmit drive signals to multiple pixels on the display panel. Figure 3 In the example shown, as a characteristic display defect, a schematic example is shown where a black area BA that is significantly darker than the black area BA but darker than the surrounding area GA, which is brighter than the black area BA but darker than the surrounding area, is generated in a straight line along a direction orthogonal to the long side of the rectangular image.

[0051] As described above, the determination of object image data 100 conforms to the reference. Figure 3 The image illustrates a certain category. Reference data 223 is prepared in advance for classifying the images contained in the image data 100 of the judgment object. Specifically, reference data 223 is, for example, for classifying images that conform to... Figure 3 Image data is obtained by capturing an image display surface of a display panel of a certain category shown (and the category of the display panel is predetermined). In addition, the image data contained in reference data 223 includes information that can be determined which category it belongs to when read by the arithmetic unit 21.

[0052] The image processing unit 51 performs image processing on each of the multiple image data contained in the object image data 100. Specifically, the image processing unit 51 further performs processing to enhance the contrast of the images.

[0053] Figure 4 This is a line graph showing the relationship between the gray values ​​of pixels before contrast enhancement and the gray values ​​of pixels after contrast enhancement. In the case where the gray values ​​of each of the multiple pixels in the image before processing by the image processing unit 51 are represented by 8 bits, such as... Figure 4 As shown on the horizontal axis, the grayscale value takes a value within the range of 0 to 255. If the contrast is not changed through image processing by the image processing unit 51, the object image data 100 is determined to be... Figure 4 As shown in line graph L4, there is no difference in the grayscale values ​​of the pixels before and after contrast enhancement. The grayscale values ​​after contrast enhancement are... Figure 4 It is shown on the vertical axis.

[0054] Image processing unit 51, for example, can become Figure 4 The contrast enhancement is further enhanced by changing the gray values ​​of multiple pixels in the image by representing the relationship before and after contrast enhancement in one of the line graphs L1, L2, and L3 shown in the figure. Line graphs L1, L2, and L3 are S-shaped growth curves represented by the S-shaped growth function shown in Equation (1). In Equation (1), y represents the gray value of the pixel after image processing by the image processing unit 51. In Equation (1), x represents the gray value of the pixel before image processing by the image processing unit 51. In Equation (1), b represents the value of x corresponding to the inflection point of the S-shaped growth curve, that is, x in the coordinate (x, y) where the slope of the S-shaped growth curve becomes the largest. With b as the boundary, the S-shaped growth curve is convex downward when x b. In Equation (1), a is a coefficient corresponding to the magnitude of the slope of the S-shaped growth curve. In addition, exp[] in Equation (1) and Equation (2) described later represents an exponential function.

[0055]

Number 1

[0056]

[0057] Furthermore, in the description of the embodiment and in equations (1) and (2) described later, it is assumed that the image data contained in the reference data 223 and the image data of the judgment object image data 100 is an 8-bit image. That is, in the description of the embodiment and in equations (1) and (2), it is assumed that the highest gray value of the pixels contained in the image is 255. The value of "255" recorded as the numerator of equations (1) and (2) is assumed to be the highest gray value of the pixels contained in the 8-bit image. However, the images that can be processed by the information processing device 1 are not limited to 8-bit images. The information processing device 1 can process r-bit images. r is any natural number. The highest gray value of the r-bit image is 2. r -1. Therefore, when the image processed in the information processing device 1 is an r-bit image, the value of "255" recorded as the numerator of equations (1) and (2) is replaced with "2". r -1". In addition, the image data contained in the reference data 223 of the embodiment has been pre-processed by the image processing unit 51.

[0058] Figure 5 This is a block diagram illustrating a more specific functional structure example of the judgment unit 60. The judgment unit 60 includes a machine learning unit 70, a verification unit 61, and a limitation classification processing unit 62. The machine learning unit 70 performs machine learning according to a predetermined algorithm. In this embodiment, a CNN (Convolutional Neural Network) is used as the predetermined algorithm.

[0059] The machine learning unit 70 includes a feature extraction unit 71 and a recognition unit 72. The processing contained in the CNN is generally divided into two processes. One of these processes is the extraction of features from the image that is the object of the CNN processing, which is performed by the feature extraction unit 71. The other of these processes is the processing of taking data representing the image's features as input to a neural network NN and obtaining the output of that neural network NN, which is performed by the recognition unit 72. In this embodiment, the output of the neural network NN conforms to a reference... Figure 3 The description specifies the probability value (in the range of 0 to 1) for each category.

[0060] The feature extraction unit 71 includes a convolution processing unit 711 and a pooling processing unit 712. The convolution processing unit 711 performs convolution processing on the image. The convolution processing described here is the same as that used in a typical CNN.

[0061] In convolution processing, the gray values ​​of individual pixels in an image are treated as matrix values. For example, consider a two-dimensional image with p pixels arranged in one direction (X-direction) and q pixels arranged in the other (Y-direction). This two-dimensional image can be viewed as a matrix with p rows and q columns. The gray values ​​of the individual pixels in this two-dimensional image can be considered as components of the matrix. If this matrix is ​​denoted as α, and the components of matrix α are denoted as β, then the component at position e in the X-direction and position f in the Y-direction of matrix α is represented as β. ef This refers to the grayscale value of the pixel located at position e in the X direction and position f in the Y direction in the two-dimensional image. Furthermore, in the convolution processing, a reference matrix, called a kernel or filter, is set as a matrix different from the two-dimensional image described above. For ease of explanation, this reference matrix will be referred to as the kernel. The number of rows in the kernel is smaller than the number of rows in the image to be convolved (e.g., the two-dimensional image described above). The number of columns in the kernel is smaller than the number of columns in the image to be convolved (e.g., the two-dimensional image described above). The components of the matrix contained in the kernel are predetermined weighted values ​​(generally integers). The convolution processing unit 711 performs extraction processing, treating one of the images contained in the target image data 100 as a matrix and extracting a portion of that matrix corresponding to the number of rows and columns of the kernel. That is, this portion can be treated as a matrix with the same number of rows and columns as the kernel. The convolution processing unit 711 performs multiplication processing to obtain the product of this portion and the kernel. The convolution processing unit 711 performs addition processing, summing the values ​​of the components contained in the product of the matrix to obtain a single value. The convolution processing unit 711 treats this single value as a feature quantity. The convolution processing unit 711 sets multiple ranges within an image treated as a matrix to be the objects of the extraction processing. For example, when p = q = 30, and the extraction processing ranges are set at intervals of three in the X and Y directions, the range X × Y = 10 × 10 = 100 is set. The convolution processing unit 711 performs extraction processing, multiplication processing, and addition processing on each range to obtain the feature quantity for each range. Thus, an image is converted into data representing the feature quantity values ​​arranged in a matrix (the first feature quantity matrix). Furthermore, more specifically, padding (pixel supplementation), etc., is sometimes performed during the extraction processing; however, since these are known techniques, detailed explanations are omitted here.

[0062] The pooling processing unit 712 performs pooling processing on the image that has undergone convolution processing by the convolution processing unit 711. The pooling processing described here is the same as the pooling processing used in a typical CNN.

[0063] In the pooling process, the components of the first eigenvalue matrix obtained through the convolution process described above are divided into multiple groups. For example, the first eigenvalue matrix is ​​set as a matrix with 10 rows and 10 columns. Here, if the first eigenvalue matrix is ​​divided into ranges of 2 rows and 2 columns, a 5×5 range can be set for the first eigenvalue matrix. Here, each range is considered as a separate group. In the pooling process, with this consideration, the components of the first eigenvalue matrix obtained through the convolution process can be divided into multiple groups. In the pooling process, a value is derived for each group. Examples of methods for deriving this single value include max pooling and average pooling. Max pooling is a method that derives the maximum value among the components of the matrix contained in the group. Average pooling is a method that derives the average value among the components of the matrix contained in the group. By treating the value derived for each group as a component of the matrix, a matrix (the second eigenvalue matrix) with a smaller number of rows and columns than the first eigenvalue matrix can be derived using the pooling process. The second feature matrix can be regarded as data representing the feature quantities of an image (e.g., one of the multiple images contained in the object image data 100) that is viewed as a matrix by the convolution processing unit 711.

[0064] Furthermore, the convolution processing unit 711 and the pooling processing unit 712 for an image can be performed once or more, or repeatedly for a predetermined number of times. Additionally, the processing performed in the feature extraction unit 71 may include the inverse operation of convolution (deconvolution) and / or the inverse operation of pooling (unpooling).

[0065] The recognition unit 72 includes a neural network generation unit 721 and a probability conversion unit 722. The neural network generation unit 721 generates a neural network NN. The probability conversion unit 722 performs calculations to convert the output of the neural network NN generated by the neural network generation unit 721 into probability values ​​(values ​​in the range of 0 to 1).

[0066] In the case of a CNN, the input to the neural network (NN) is a component of the matrix (the second feature matrix) obtained as the output of the pooling process described above. The number of components in this matrix corresponds to the number of input nodes in the neural network (NN). The neural network (NN), like typical neural networks in machine learning, contains one or more fully connected layers. Weighting of the input is performed in these fully connected layers. Typically, the bias of the fully connected layer (final segment) closest to the output in the one or more fully connected layers of the neural network (NN) corresponds to the number of discriminations performed using that neural network (e.g., referring to...). Figure 3The number of categories described is the same. Typically, the output of a neural network (NN) can take any value (a number greater than 1). Therefore, since the output of the neural network (NN) is processed as a "deterministic value that can be used to determine which category it belongs to," the probability transformation unit 722 performs the operation. The operation of the probability transformation unit 722 is, for example, performed using a normalized exponential function (Softmax function), but it is not limited to this and can be appropriately modified.

[0067] The validation unit 61 provides validation functionality for the machine learning performed by the machine learning unit 70. Through the validation unit 61, overlearning in machine learning is suppressed, making it easier to further improve the judgment accuracy of the machine learning. Examples of validation functions of the validation unit 61 include leave-out validation and cross-validation.

[0068] Figure 6 This is a diagram illustrating the considerations for leaving room for verification. In Figure 6 And then Figure 7 In the image, a rectangle represents a dataset DS containing m image data points. Figure 6 and Figure 7 In the rectangle, let m be the first m-th image data, arranged sequentially from one end to the other along the long side of the rectangle. m is a natural number.

[0069] In the validation process, such as Figure 6 As shown, the dataset DS is divided into training data TD and validation data CD. The training data TD is used as training data in machine learning, and the validation data CD is used as validation data in machine learning.

[0070] A dataset DS contains multiple image datasets corresponding to a particular judgment result obtained through machine learning. For example, it illustrates... Figure 3The dataset DS, representing "qualified products," contains m images of m display panels that meet the "qualified product" criteria, captured individually. Here, n images from the m images are extracted as training data TD, and the judgment unit 60 performs machine learning using this training data TD. n is a natural number less than m. Then, the judgment unit 60, having learned from the training data TD, judges the (mn) images in the dataset DS excluding the training data TD, i.e., the verification data CD. Here, the verification data CD is a part of the dataset DS and contains only the image data corresponding to a specific judgment result obtained based on machine learning. Therefore, ideally, the judgment unit 60, having learned from the training data TD, expects to judge all the image data contained in the verification data CD as corresponding to that single judgment result. However, in reality, some of the verification data CD is judged as image data corresponding to other judgment results. Therefore, in the judgment unit 60, which has learned from the training data TD, the judgment accuracy of the judgment unit 60 is improved by judging the verification data CD, further enhancing the judgment accuracy learned from the training data TD.

[0071] In the case of leave-out verification, the verification unit 61 divides the dataset DS into training data TD and verification data CD, and the machine learning unit 70 performs learning based on the training data TD and verification of the judgment accuracy based on the verification data CD.

[0072] Figure 7 This is a schematic diagram illustrating the considerations for cross-validation. In the case of cross-validation, the validation unit 61, for example... Figure 7As shown, dataset DS is treated as dataset DSS1. Dataset DSS1 contains datasets DSa, DSb, DSC, DSd, and DSe. Dataset DSa is the data from dataset DS processed by dividing it into validation data CD1 and training data TD1. Dataset DSb is the data from dataset DS processed by dividing it into validation data CD2 and training data TD21 and TD22. Dataset DSC is the data from dataset DS processed by dividing it into validation data CD3 and training data TD31 and TD32. Dataset DSD is the data from dataset DS processed by dividing it into validation data CD4 and training data TD41 and TD42. Dataset DSe is the data from dataset DS processed by dividing it into validation data CD5 and training data TD5. Training data TD1, TD21, TD22, TD31, TD32, TD41, TD42, and TD5 are used as training data in machine learning. Validation data CD1, CD2, CD3, CD4, and CD5 are used as validation data in machine learning. Validation data CD1, CD2, CD3, CD4, and CD5 each contain distinct image data. That is, cross-validation defines multiple patterns (e.g., datasets DSa, DSb, DSc, DSd, DSe) for which parts of the m image data in dataset DS should be considered as validation data, and repeatedly performs hold-out validation corresponding to the number of these patterns (see [reference]). Figure 6 ).

[0073] In this implementation, a verification unit 61 is used to perform cross-validation, but it is not limited to this. The specific verification algorithm performed by the verification unit 61 can be appropriately changed.

[0074] Reference data 223 in the implementation method includes references to Figure 3 The description specifies the number of categories corresponding to multiple datasets. Specifically, reference data 223 includes datasets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 (see reference). Figure 10Datasets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 are presented in three datasets. Dataset DS1 contains m images of m display panels that meet the "qualified" criteria, taken individually from their display surfaces. Dataset DS2 contains m images of m display panels that meet the "tilt unevenness" criteria. Dataset DS3 contains m images of m display panels that meet the "signal line defect" criteria. Dataset DS4 contains m images of m display panels that meet the "scan line defect" criteria. Dataset DS5 contains m images of m display panels that meet the "circuit defect" criteria. Dataset DS6 contains m images of m display panels that meet the "vertical stripe" criteria. Dataset DS7 contains m images of m display panels that meet the "clustering" criteria. Dataset DS8 is a dataset DS containing m image data obtained by individually capturing the image display surfaces of m display panels that conform to "white unevenness". The verification unit 61 of the embodiment performs cross-validation on each of the datasets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8.

[0075] Figure 8 This diagram illustrates the considerations related to the accuracy of judgments using machine learning. Assume there exists a condition where an image is set to 1 (Positive) if it "matches the criteria for capturing a specific object," and -1 (Negative) if it "does not match the criteria for capturing a specific object." Under this condition, all images are considered to produce a judgment of 1 or -1. In machine learning, the output is a judgment result indicating whether the image, which forms the basis of the input data (features), "matches the criteria for capturing a specific object." The machine learning unit 70 outputs a judgment result indicating whether the image "matches the criteria for capturing a specific object" in accordance with the input. To give a more specific example, the machine learning unit 70 in this embodiment outputs an image indicating which category the image, which forms the basis of the input data (features), falls under: "qualified," "uneven tilt," "bad signal line," "bad scan line," "bad circuitry," "vertical stripes," "clustering," or "uneven whiteness," in accordance with the input of the judgment object image data 100 or verification data (e.g., verification data CD). Here, it can be said, for example, that an image judged to fall into a category other than "qualified product" is an image judged to be "unqualified product".

[0076] Figure 8The "Actual Category" column shows the results of the visual inspection conducted by the inspector. For example, an image that actually "captured a specific object" is an image that "conforms to an image capturing a specific object" and is assigned 1. Conversely, an image that "did not capture a specific object" is an image that "does not conform to an image capturing a specific object" and is assigned -1. On the other hand, Figure 8 The "Predicted Category" shown indicates the judgment result based on machine learning. Specifically, images judged as "matching the criteria for capturing a specific object" are assigned 1. Conversely, images judged as "not matching the criteria for capturing a specific object" are assigned -1.

[0077] In practice, if an image that actually "captures a specific object" is correctly identified as such by machine learning, the value assigned to both the "actual category" and the "predicted category" is 1. This is called a True Positive (TP). Conversely, if an image that actually "does not capture a specific object" is correctly identified as such by machine learning, the value assigned to both the "actual category" and the "predicted category" is -1. This is called a True Negative (TN). On the other hand, if an image that actually "captures a specific object" is incorrectly identified as such by machine learning, the value assigned to both the "actual category" is 1 and the "predicted category" is -1. This is called a False Negative (FN). Finally, if an image that actually "does not capture a specific object" is incorrectly identified as such by machine learning, the value assigned to both the "actual category" and the "predicted category" is -1. The case where the "actual class" is assigned a value of -1 and the "predicted class" is assigned a value of 1 is called FP (False Positive).

[0078] As concepts representing the precision of machine learning, we know of, for example, accuracy, recall, precision, false positive rate, and F-score. Accuracy is represented by the value (TP+TN) / (TP+TN+FN+FP). Recall is represented by the value TP / (TP+FN). Precision is represented by the value TP / (TP+FP). False positive rate is represented by the value FP / (TN+FP). If we set the detection rate as Q and the precision as R, then the F-score is 2×Q×R / (Q+R).

[0079] The limited classification processing unit 62 performs multi-stage processing. The multi-stage processing referred to here means that a portion of the input data corresponding to a portion of the judgment results shown in the output of the machine learning unit 70 for multiple input data is extracted from the multiple images, and the extracted portion of input data is input into the machine learning unit 70, so that the machine learning unit 70 can further determine which of the partial judgment results it belongs to.

[0080] To give a specific example, the image data 100 for judging the object contains multiple image data (e.g., 1600 image data). First, by inputting these multiple image data into the machine learning unit 70, a result is obtained indicating that each of the multiple image data is judged to meet the reference. Figure 3 The output describes which category of classification. Here, the output is judged to conform to the reference. Figure 3 In the case of a portion of image data (e.g., 220 images) representing a part of the described classification (e.g., "qualified product" or "uneven tilt"), the limiting classification processing unit 62 extracts this portion of image data from the plurality of image data. The limiting classification processing unit 62 then inputs the extracted portion of input data back into the machine learning unit 70, causing the machine learning unit 70 to again display the output of the judgment. However, in the judgment performed in response to the input of this portion of input data, the classification based on the judgment is limited to a part of that classification (e.g., "qualified product" or "uneven tilt"). Furthermore, if only this portion of input data is input into the machine learning unit 70, the convolution processing unit 711 and the pooling processing unit 712 process this portion of input data again. Here, it is also possible to directly use the data corresponding to this portion of input data from the data processed by the convolution processing unit 711 and the pooling processing unit 712 on the plurality of image data before the extraction of this portion of input data, without repeating the processing of the convolution processing unit 711 and the pooling processing unit 712.

[0081] The following is for reference Figure 9 and Figure 10 This describes a machine learning process based on the machine learning unit 70, using a multi-stage processing method based on the constraint classification processing unit 62. (See reference...) Figure 9 and Figure 10 The process described above involves using the image data contained in reference data 223 as training data for training and learning.

[0082] Figure 9 It is a flowchart representing a machine learning process that employs multi-stage processing. Figure 10 This is a diagram illustrating the workflow of machine learning that employs multi-stage processing. For example... Figure 9As shown, firstly, reference data 223 is acquired (step S1). Specifically, the computation unit 21 in the processing of the machine learning program 221 reads the reference data 223 from the storage unit 22. Here, as described above, a portion of the image data contained in the reference data 223 is used as training data, and another portion is used as validation data. Figure 10 In the example, the datasets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 contained in reference data 223 are schematically shown as TD, which is used as part of the training data processing, and as CD, which is used as another part of the validation data processing.

[0083] After the processing in step S1, the first stage of machine learning processing (step S2) of the machine learning unit 70 is performed. Step S2 is a machine learning process using the training data contained in the datasets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 included in the reference data 223. This machine learning process, for example, involves using data representing the feature values ​​of the images of the validation data included in the reference data 223 as input to the neural network NN, so that the machine learning unit 70 determines that the image of the data is a reference image. Figure 3 Which of the eight categories described is the objective to determine the weighted values ​​of neurons in a neural network (NN). Figure 10 In the text, the first stage of machine learning processing is recorded as "8-class CNN" and the symbol for step ST1 is marked.

[0084] After the processing in step S2, the first stage of verification processing (step S3) is performed. Step S3 involves the following process: the computation unit 21 takes the verification data contained in the datasets DS1, DS2, DS3, DS4, DS5, DS6, DS7, and DS8 included in the reference data 223 as input to the machine learning unit 70, and obtains the machine learning output from the machine learning unit 70 reflecting the machine learning performed based on the processing in step S2. That is, the processing in step S3 is to obtain the image that the machine learning unit 70 determines is the reference image. Figure 3 The description specifies which of the eight categories' outputs is processed. The output obtained through step S3 is... Figure 10 The value shown in the middle is output OP1.

[0085] After the processing in step S3, the limitation classification processing unit 62 extracts the verification data that yielded a specific output (verification result) in the verification processing of the first stage (step S4). The verification data extracted in step S4 is part of the verification data processed as input to the machine learning unit 70 in step S3, and is used to obtain the determination of whether it is a reference. Figure 3The validation data for the output of a subset of the eight categories described (e.g., "qualified" or "uneven"). Specifically, "specific output" refers to the output that indicates the product is classified as that subset of the categories.

[0086] After the processing in step S4, a second stage of machine learning processing (step S5) is performed using image data corresponding to a specific output from the image data contained in reference data 223. "Image data corresponding to a specific output from the image data contained in reference data 223" or "reference data corresponding to a specific output" is a part of reference data 223, referring to a dataset (e.g., datasets DS1 and DS2) containing image data corresponding to the classification (e.g., "qualified" or "uneven tilt") as part of the "specific output" processing in step S4. The processing in step S5 is the aforementioned machine learning processing using the training data contained in the "reference data corresponding to a specific output." This machine learning processing, for example, involves setting the feature data representing the verification data contained in reference data 223 as input to a neural network NN, causing the machine learning unit 70 to determine that the image of the data is a reference image. Figure 3 This section describes a subset of the eight categories corresponding to a "specific output" in machine learning processing, focusing on determining the weighted values ​​of neurons in a neural network (NN). Figure 10 In the text, the second stage of machine learning processing is recorded as "2-class CNN" and the symbol for step ST2 is marked.

[0087] After the processing in step S5, the second stage of verification processing (first time) (step S6) is performed. Step S6 involves the computation unit 21 taking the verification data contained in the aforementioned "reference data corresponding to a specific output" as input to the machine learning unit 70 and obtaining an output from the machine learning unit 70 reflecting the machine learning performed based on the processing in step S5. That is, step S6 involves obtaining an image of the verification data contained in the aforementioned "reference data corresponding to a specific output" as determined by the machine learning unit 70. Figure 3 The processing of which of the eight categories described corresponds to the "specific output" category in step S4.

[0088] Furthermore, before the processing in step S7 described later, feedback can be provided to the machine learning process based on the output obtained from the processing in step S6. That is, the accuracy of the machine learning process based on the processing in step S5 can be improved based on the processing in step S6.

[0089] After the processing in step S6, a second stage of verification processing (second time) (step S7) is performed using the verification data extracted in step S4 to obtain a specific output (verification result). Step S7 involves the computation unit 21 using the verification data extracted in step S4 as input to the machine learning unit 70 and obtaining the output of the machine learning unit 70 after the machine learning process based on step S5 and the verification process (first time) based on step S6. That is, step S7 is the process where the machine learning unit 70 determines that the image of the verification data from step S4, which yields a specific output (verification result), is a reference image. Figure 3 This describes the processing of which of the eight categories, specifically the category corresponding to "specific output," is being processed in step S4. The output obtained through step S7 is... Figure 10 The output shown is OP2.

[0090] After the processing in step S7, a process is performed whereby the output obtained in the second stage of verification processing (the second time) and the outputs that are not specific outputs (verification results) obtained in the first stage of verification processing are added together to obtain the final output (step S8). In step S8, the computation unit 21 integrates the outputs obtained in step S7 and the verification data from the outputs obtained in step S3 that are not the extraction targets of step S4, and uses this integrated output as the final output of the machine learning unit 70. The output obtained through step S8... Figure 10 The output shown is OP3.

[0091] Furthermore, when the validation function of the restricted classification processing unit 62 is cross-validation, steps S2 to S7 are repeated accordingly to the number of datasets generated in the cross-validation. For example, in the case of dataset DSS1, datasets DSa, ..., dataset DSe become objects, so the number of datasets is 5.

[0092] The following is for reference Figure 11 Figure 14 This explains the changes in machine learning accuracy caused by the application of multi-stage processing. (Referring to...) Figures 11 to 14 The explanation uses the example of datasets DS1, DS2, ..., DS8, each containing 1000 images (m = 1000), with 800 images used as training data and 200 used as validation data (number of judgment objects). Additionally, Figures 11 to 14 The number of correct answers in the table and Figure 8 The TP corresponds to this. Additionally... Figures 11 to 14The “Prediction Precision” in the table is the ratio of the number of correct answers to the number of validation data that fit each category (the actual number of categories), which is consistent with the recall rate mentioned above.

[0093] Figure 11 This indicates that the actual category of the verification data contained in reference data 223 is related to the reference. Figure 9 This diagram illustrates the correspondence between the outputs (predicted categories) of the first verification process obtained from step S3. Firstly, in... Figure 11 In the example shown, there are FNs (44 / 200) where the actual category is "qualified" but the predicted category is something other than "qualified". The number of FNs (21 / 44) with predicted categories of "skewed" in this FN is sometimes close to half of the total number of FNs, significantly larger than the number of FNs with other categories in this FN. Next, in Figure 11 The example shown illustrates the TP (True Value) where the actual class matches the predicted class. The TP for the "skewed" class (111 / 200) is significantly smaller compared to the TPs for other classes. Next, in... Figure 11 In the example shown, there are 89 / 200 FNs whose actual category is "skewed" but whose predicted category is something other than "skewed". The number of FNs (62 / 89) whose predicted category is "qualified" is more than half of the total number of FNs, significantly larger than the number of FNs in that category that are for other categories.

[0094] Figure 12 It indicates a reference. Figure 9 This diagram illustrates the correspondence between the actual category of the validation data contained in the "reference data corresponding to a specific output" input to the machine learning unit 70 during step S6 and the output (predicted category) of the second validation process (first step) obtained based on step S6. (See diagram for example.) Figure 11 and Figure 12 The comparison shows that by limiting the actual and predicted categories to "qualified" and "skewed," the consistency between the actual and predicted categories is further improved. Specifically, in Figure 11 In the example shown, 156 out of 200 cases had both the actual and predicted categories as "qualified," which translates to a prediction accuracy of 78.0%. In contrast, in... Figure 12 In the example shown, the ratio of actual to predicted "qualified" is 180 / 200, which translates to a prediction accuracy of 90.0%. Additionally, in Figure 11 In the example shown, 111 / 200 of the actual and predicted categories are both "skewed," which translates to a prediction accuracy of 55.5%. In contrast, in... Figure 12In the example shown, the ratio of actual category to predicted category is 160 / 200, which translates to a prediction accuracy of 80.0%.

[0095] Figure 13 It indicates a reference. Figure 9 The diagram illustrates the correspondence between the actual category of the validation data extracted in step S4 and input into the machine learning unit 70 in step S7, and the output (predicted category) of the second validation process (second time) obtained based on step S7. (See diagram for example.) Figure 11 and Figure 13 The comparison shows that limiting both the actual and predicted categories to "qualified" and "skewed" further improves the consistency between the actual and predicted categories. Specifically, in Figure 11 In the example shown, 156 out of 200 cases had both the actual and predicted categories as "qualified," which translates to a prediction accuracy of 78.0%. In contrast, in... Figure 13 In the example shown, 159 out of 177 cases had both the actual and predicted categories as "qualified," which translates to a prediction accuracy of 89.8%. Furthermore, in... Figure 11 In the example shown, 111 out of 200 cases have both the actual and predicted categories exhibiting "skewed" behavior, which translates to a prediction accuracy of 55.5%. In contrast, in... Figure 13 In the example shown, the number of cases where both the actual and predicted categories are "qualified" is 140 / 173, which translates to a prediction accuracy of 80.9%.

[0096] Figure 14 This indicates that the actual category of the verification data contained in reference data 223 is related to the category passed by reference. Figure 9 The diagram illustrates the correspondence between the final output (predicted category) obtained from step S8. Figure 14 The table shown is through Figure 13 The values ​​contained in the table shown will Figure 11 The table shown is obtained by updating a subset of the various values ​​contained therein. This update of the subset of values ​​reflects... Figure 13 The values ​​shown are TP (159) for “Qualified”, FN (i.e., the actual category is “Qualified” but the predicted category is “skewed”) (18), TP (140) for “skewed”, and FN (i.e., the actual category is “skewed” but the predicted category is “Qualified”) (33).

[0097] like Figure 11 and Figure 14The comparison shows that the prediction accuracy of "qualified" and "skewed unevenness" is improved in the final output obtained through step S8 compared to the output of the first verification process. Therefore, considering the overall prediction accuracy using machine learning, the prediction accuracy is indeed improved in the final output obtained through step S8 compared to the output of the first verification process.

[0098] For reference Figures 9 to 14 As explained, by applying the multi-stage processing of the restricted classification processing unit 62 in machine learning, the prediction accuracy, i.e., the degree of consistency between the actual category and the predicted category, is further improved. The multi-stage processing of the restricted classification processing unit 62 is applicable not only to machine learning but also to classification judgments using image data derived from machine learning. The following refers to… Figure 15 This describes the judgment process of the machine learning unit 70, which employs multi-stage processing in image data classification using machine learning. Furthermore, the classification of image data using machine learning is performed with reference to... Figures 9 to 14 This will be done after the explained machine learning. Additionally, in Figure 10 In the middle, the dashed lines indicate the basis Figure 15 The input process for judging the image data 100 of the object.

[0099] Figure 15 This is a flowchart illustrating the judgment process of the machine learning unit 70, which applies multi-stage processing in the classification judgment of image data using machine learning. First, the image data 100 of the judgment object is acquired (step S11). Specifically, the acquisition unit 40 acquires the image data 100 of the judgment object from an external device and outputs it to the calculation unit 21.

[0100] After the processing in step S11, image processing based on the image processing unit 51 is performed (step S12). Specifically, as shown in reference... Figure 4 As explained, the image processing unit 51 further enhances the contrast of the image data contained in the image data 100 of the judgment object image data through image processing.

[0101] After the processing in step S12, the first stage of the judgment process of the machine learning unit 70 is performed (step S13). In step S13, the calculation unit 21 takes the image data contained in the judgment object image data 100 as input to the machine learning unit 70 and obtains the response from the machine learning unit 70. Figure 9 The processing of the machine learning output is explained. Specifically, step S13 involves obtaining an image of the image data contained in the object image data 100, as determined by the machine learning unit 70, which is a reference image. Figure 3 This describes the output processing for which of the eight categories is being described.

[0102] After the processing in step S13, the limitation classification processing unit 62 extracts the judgment object image data 100 that yielded a specific output (judgment result) in the judgment processing of the first stage (step S14). The verification data extracted in step S14 is part or all of the image data contained in the judgment object image data 100 processed as input to the machine learning unit 70 in step S13, and is used to obtain a reference representing the judgment. Figure 3 Image data of the output for a subset of the eight categories described (e.g., "qualified" or "uneven tilt"). That is, "specific output" refers to the output that indicates the product is classified as that subset of the product.

[0103] After the processing in step S14, a second-stage judgment process (step S15) is performed using the image data corresponding to the specific output extracted in step S14. Step S15 is performed as follows: The image data from which the specific output (judgment result) is obtained by the machine learning unit 70 is used as a reference. Figure 3 Which of the eight categories described refers to the output of a subset of the categories corresponding to "specific output" in the processing of step S14?

[0104] After the processing in step S15, a process is performed to combine the output obtained in the second stage of the judgment process and the outputs that are not specific outputs (verification results) obtained in the first stage of the judgment process, and set the combined output as the final output (step S16). In the processing of step S16, the arithmetic unit 21 combines the output obtained in the processing of step S15 and the image data that are not the extraction object of the processing in step S14 as input, and uses the combined output as the final output of the machine learning unit 70.

[0105] Through reference Figure 15 The described processing flow, compared to the case of simply using machine learning to classify the images contained in the image data 100 of the judgment object, is different from the reference process. Figures 11 to 14 Similarly, this approach further improves classification accuracy.

[0106] In addition, in reference Figure 4 The description shows line graphs L1, L2, and L3 as examples of the relationship between gray values ​​before and after contrast enhancement, but the relationship between gray values ​​before and after contrast enhancement is not limited to these.

[0107] Figure 16 This is a line graph showing the relationship between the grayscale values ​​of pixels before contrast enhancement and the grayscale values ​​of pixels after contrast enhancement. The image processing unit 51 can, for example, be configured as follows: Figure 16The line graphs L11, L12, and L13 shown represent the relationship before and after contrast enhancement by changing the gray values ​​of the individual pixels in the image, thereby further enhancing the contrast of the image.

[0108] Furthermore, the image processing performed by the image processing unit 51 in step S12, namely contrast enhancement, is not limited to the formula (1) described above. Hereinafter, a more advanced approach to contrast enhancement performed by the image processing unit 51 will be explained, with reference to... Figure 17 This section explains the relationship between the grayscale distribution of images obtained from photographing a display panel with "uneven tilt" and the grayscale distribution of images obtained from photographing a display panel with "qualified" grayscale.

[0109] Figure 17 This is a graph showing the relationship between the distribution of grayscale values ​​in an image obtained from a display panel with "uneven tilt" and the distribution of grayscale values ​​in an image obtained from a display panel with "qualified" display. Histogram Hs1 is a histogram showing the grayscale values ​​of the pixels in an image obtained from a display panel with "uneven tilt". Including histogram Hs1, the histograms referred to in the description of the implementation embodiment show the range of grayscale values ​​of the pixels in the image (from 0 to the maximum value expressed in bits) on the horizontal axis and the number of pixels for each grayscale value on the vertical axis. Histogram Hs2 is a histogram showing the grayscale values ​​of the pixels in an image obtained from a display panel with "qualified" display. Histogram Hs3 is a histogram showing the grayscale values ​​of the pixels in an image obtained from a display panel with "qualified" display (a different display panel than the one in histogram Hs2). Furthermore, Figure 17 The histograms shown are histograms representing the grayscale values ​​of the image before image processing by the image processing unit 51.

[0110] like Figure 17 As shown in histograms Hs2 and Hs3, the distribution of grayscale values ​​in images captured from a display panel of a "qualified" product displaying a pure color image of white or mid-tones significantly produces a histogram ridge corresponding to the highest frequency of occurrence of each grayscale value. The grayscale value fluctuation range (the difference between the lowest and highest grayscale values ​​in the ridge) of this histogram is generally less than 10% of the pixel's bit count (the difference between the lowest and highest grayscale values ​​a pixel can take). For example, if the pixel has 8 bits, the grayscale values ​​range from 0 to 255. On the other hand, grayscale value fluctuations exceeding 25 are rare in the ridge of this histogram.

[0111] On the other hand, in images captured of display panels classified as "tilted" as shown in histogram Hs1, the number of pixels containing gray values ​​in the low gray value band F11, which corresponds to a relatively low gray value relative to the peak of the histogram corresponding to the highest frequency of gray value occurrence, is clearly present to a non-negligible degree. Furthermore, in histogram Hs1, on the side opposite to the low gray value band F11 across the peak of the histogram—that is, the high gray value band F12, which contains relatively high gray values ​​relative to the peak of the histogram—is essentially or completely absent. However, in cases where a tilted display panel is visually perceived as having a relatively higher brightness than its surroundings, pixels contained in the high gray value band F12 are produced.

[0112] Furthermore, the low grayscale bands F21 and F22 in histogram Hs2, and the low grayscale bands F31 and F32 in histogram Hs3 contain virtually no pixels. In other words, the difference between histogram Hs1 and histogram Hs2 is that histogram Hs1 clearly contains pixels contained in the low grayscale band F11, but histogram Hs2 contains virtually no pixels contained in the low grayscale band F21. The "foothills" of the histogram, such as the low grayscale bands F11, F21, F31 and the high grayscale bands F12, F22, F32, are grayscale value ranges centered on the "mountain" of the histogram, encompassing less than 30% of the number of bits per pixel.

[0113] exist Figure 17 To illustrate that low grayscale bands F11 and F21 are approximately the same grayscale band, the diagram shows a dashed line F1a connecting the lowest grayscale value of low grayscale band F11 to the lowest grayscale value of low grayscale band F21, and a dashed line F1b connecting the highest grayscale value of low grayscale band F11 to the highest grayscale value of low grayscale band F21. Similarly, to illustrate that high grayscale bands F12 and F22 are approximately the same grayscale band, the diagram shows a dashed line F2a connecting the lowest grayscale value of high grayscale band F12 to the lowest grayscale value of high grayscale band F22, and a dashed line F2b connecting the highest grayscale value of high grayscale band F12 to the highest grayscale value of high grayscale band F22. The dashed lines F1a, F1b, F2a, and F2b are generally along the vertical axis of the line graph.

[0114] Here, if we compare histograms Hs2 and Hs3, the overall trend of histogram Hs3 shifts towards lower grayscale values ​​compared to histogram Hs2. Thus, even display panels treated as the same "qualified" products in classification exhibit individual differences in their grayscale value tendencies. In contrast enhancement of multiple images obtained from multiple images of display panels exhibiting such individual differences, when calculating the modified grayscale value (y) based on equation (1), depending on the value of b, it is sometimes difficult to clearly distinguish between images containing pixels in the low grayscale value band F11 in histogram Hs1 and images where pixels in the low grayscale value band F31 are not present in histogram Hs3.

[0115] Therefore, the image processing unit 51 can replace the above-mentioned equation (1) and perform contrast enhancement based on the following equation (2). According to the contrast enhancement based on equation (2), compared with equation (1), it is easier to more clearly distinguish whether the pixels contained in the above-mentioned "historical base" are clearly present. Regarding the differences between equation (1) and equation (2) and the gain_1, gain_2, inflection_1, inflection_2, and height_1 of equation (2), refer to... Figure 18 Please provide an explanation.

[0116]

Number 2

[0117]

[0118] Figure 18 It is a graph showing the difference between equation (1) and equation (2), as well as the line graph corresponding to equation (2) and the relationship between gain_1, gain_2, inflection_1, inflection_2, height_1, and height_2. Figure 18 Line graph L21 shown is a line graph representing the relationship between grayscale values ​​before and after contrast enhancement based on equation (1), and is the line graph corresponding to equation (1). Line graph L21 is a line graph that applies a = 0.05 and b = 128 to equation (1). In addition, line graph L22 is a line graph representing the relationship between grayscale values ​​before and after contrast enhancement based on equation (2), and is the line graph corresponding to equation (2).

[0119] like Figure 18As shown, line graph L22 has two inflection points. One of the two inflection points is located at Inflection_1 with respect to the horizontal axis. The other of the two inflection points is located at Inflection_2 with respect to the horizontal axis. In other words, the values ​​of Inflection_1 and Inflection_2 reflect the setting of these two inflection points in the contrast enhancement based on equation (2). Here, Inflection_2 is greater than Inflection_1. gain_1 indicates the magnitude of the slope of one of the two inflection points. gain_2 indicates the magnitude of the slope of the other of the two inflection points. gain_1 and gain_2 are the same values ​​as 'a' mentioned above. height_1 indicates the upper limit of the grayscale value after contrast enhancement based on one of the two inflection points and the lower limit of the grayscale value after contrast enhancement based on the other of the two inflection points. Furthermore, the lower limit of the grayscale value after contrast enhancement based on one of the two inflection points is 0. In addition, the upper limit of the grayscale value after contrast enhancement based on the other of the two inflection points is the highest grayscale value of the pixel. Inflection_1, Inflection_2, and height_1 are the grayscale values ​​that can be taken for a pixel, and are usually integers above 0.

[0120] exist Figure 18 In the equation, gain_1 = 0.5, gain_2 = 0.5, inflection_1 = 90, inflection_2 = 135, and height_1 = 128. Additionally, in... Figure 8 In the diagram, height_2 = (255 - height_1) = 127.

[0121] Figure 18 The histogram Hs4 shown is a histogram of images taken from display panels classified as "qualified". For example... Figure 18 As shown, it is desired that Inflection_1 and Inflection_2 be set within a gray value range with a lower limit corresponding to the gray value of Inflection_1 and an upper limit corresponding to the gray value of Inflection_2, so that the gray value range of the histogram of the image obtained by taking a picture of the display panel classified as "qualified" is sufficiently converged.

[0122] Figure 19 This diagram illustrates an image obtained by taking a picture of a display panel that is classified as "qualified" and an applicable example of image processing applied to that image. Figure 19The image Be1 shown before processing is the image before image processing by the image processing unit 51. Histogram Hs5 is a histogram showing the distribution of gray values ​​of the pixels contained in the image Be1 before processing. The image Af11 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be1 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that is shown in line graph L21. The image Af12 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be1 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that is shown in line graph L31. Line graph L31 is a line graph corresponding to equation (2), where gain_1 = 0.5, gain_2 = 0.5, inflection_1 = 80, inflection_2 = 140, and height_1 = 128.

[0123] Figure 20 This diagram illustrates an example of image processing applied to a display panel that exhibits both "scan line defect" and "signal line defect". Figure 20 The image Be2 shown before processing is the image before image processing by the image processing unit 51. Histogram Hs6 is a histogram showing the distribution of gray values ​​of the pixels contained in the image Be2 before processing. The image Af21 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be2 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that is shown as the relationship before and after contrast enhancement in line graph L21. The image Af22 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be2 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that is shown as the relationship before and after contrast enhancement in line graph L41. Line graph L41 is a line graph corresponding to equation (2), where gain_1 = 0.5, gain_2 = 0.5, Inflection_1 = 90, Inflection_2 = 140, and height_1 = 128.

[0124] Figure 20 The processed image Af21 is generally darker, making it more difficult to identify the black area BA, which indicates scan line defects, compared to the unprocessed image Be2 before contrast enhancement. On the other hand, the processed image Af22 is generally brighter, making it easier to identify the black area BA, which indicates scan line defects. Furthermore, the processed image Af22 allows for thorough identification of the white area WA, which indicates signal line defects. Additionally, as... Figure 19As shown, there is no problem with Equation (2) regarding the confirmation of "qualified products". In this way, by performing contrast enhancement based on Equation (2), the classification of "qualified products" and other products based on image brightness becomes easier.

[0125] Furthermore, when the value of height_1 is set to the same value as the highest grayscale value of the pixel, gain_2 and Inflection_2 essentially do not function. That is, in this case, equation (2) essentially functions the same as equation (1).

[0126] Figure 21 This diagram illustrates an image taken from a display panel classified as "unevenly tilted" and an example of applicable image processing applied to that image. Figure 21 The image Be3 shown before processing is the image before image processing by the image processing unit 51. Histogram Hs7 is a histogram showing the distribution of gray values ​​of the pixels contained in the image Be3 before processing. The image Af31 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be3 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that shows the relationship before and after contrast enhancement as shown in line graph L21. The image Af32 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be3 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that shows the relationship before and after contrast enhancement as shown in line graph L51. Line graph L51 is a line graph corresponding to equation (2), where gain_1 = 1, inflection_1 = 100, and height_1 = 255.

[0127] Figure 22 This diagram illustrates an image obtained by taking a picture of a display panel classified as "uneven whiteness" and an applicable example of image processing applied to that image. Figure 22 The image Be4 shown before processing is the image before image processing by the image processing unit 51. Histogram Hs8 is a histogram showing the distribution of gray values ​​of the pixels contained in the image Be4 before processing. The image Af41 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be4 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that is shown as the relationship before and after contrast enhancement in line graph L21. The image Af42 after processing is an image obtained by setting the gray values ​​of the pixels contained in the image Be4 before processing to the gray values ​​before contrast enhancement, and performing contrast enhancement based on the image processing unit 51 in a manner that is shown as the relationship before and after contrast enhancement in line graph L61. Line graph L61 is a line graph corresponding to equation (2), where gain_1 = 0.1, inflection_1 = 105, and height_1 = 255. Figure 21 and Figure 22 As shown, by appropriately setting the position (Inflection_1) and slope (gain_1) of the inflection point, the unevenness can be made more obvious.

[0128] The following discussion on how to determine the value of b in equation (1) and the value of Inflection_1 in equation (2) will refer to [reference needed]. Figure 23 Please provide an explanation.

[0129] Figure 23 This is a graph showing the relationship between a histogram Hs9, which represents the distribution of grayscale values ​​of pixels in an image captured from a display panel, and the position of the inflection point (b). Furthermore, histogram Hs9 is a histogram of an image captured from a display panel classified as "unevenly tilted".

[0130] Figure 23 The range UN shown is a grayscale value range that is less than 2000 pixels, starting from the number of pixels with lower grayscale values. Here, by setting the width of the grayscale values ​​of the range UN along the horizontal axis with the lowest grayscale value (0) of the pixel as the origin, i.e., by setting the highest grayscale value of the range UN as the value of b, the inflection point of the line graph when formula (1) is used in the contrast enhancement of the image that derives the range UN (e.g., histogram Hs9) is set more appropriately. In addition, when formula (2) is used for the contrast enhancement of the image, the highest grayscale value of the range UN can also be set as the value of Inflection_1. Furthermore, by setting such a b or Inflection_1 determination routine in the image processing unit 51, the inflection point in the contrast enhancement processing of the image processing unit 51 can be automatically set.

[0131] Furthermore, the fact that the range UN contains fewer than 2000 pixels is just one example, and is not limited to this. The number of pixels contained in the range UN is expected to be defined as a proportion relative to the number of pixels in the image obtained by capturing the display panel (e.g., 15%).

[0132] Settings related to various parameters such as a and b in equation (1), gain_1, gain_2, Inflection_1, Inflection_2, height_1, and the number of pixels contained in the range UN in equation (2) are included in setting data 224 for example. Rules related to image processing algorithms such as equations (1) and (2) are included in image processing program 222 for example.

[0133] As described above, the method of the embodiment includes a display device inspection method that uses machine learning to determine which of j (e.g., j=8) categories an image obtained by capturing an image display surface of a display device (e.g., an image containing image data in object image data 100 or reference data 223) conforms to. The method of the embodiment includes a first machine learning step (e.g., step S2), wherein an information processing device (e.g., information processing device 1) performs machine learning to determine which of the j categories an image conforms to based on pre-prepared training data (e.g., reference data 223) containing data of images corresponding to the j categories. The method of the embodiment includes a second machine learning step (e.g., step S5), wherein the information processing device performs machine learning to determine which of the k categories an image conforms to based on training data (e.g., datasets DS1 and DS2 in reference data 223) containing data of images corresponding to k (e.g., k=2) of the j categories. j and k are natural numbers greater than 2. k is less than j. Thus, for the k categories of the j categories, the classification accuracy of the image is further improved. Therefore, it is possible to classify images on display devices with higher accuracy.

[0134] Furthermore, one of the k categories (e.g., k=2) is a category for display devices that are processed as qualified products and do not produce display output considered as display defects. Therefore, for the k categories that include more than one category other than qualified products, the classification accuracy of the image is further improved. Thus, for the j categories (e.g., j=8) such as uneven tilt, which are more difficult to classify than qualified products, the classification accuracy between qualified products and non-qualified products is also further improved.

[0135] Furthermore, the training data used in the second machine learning step (e.g., step S5) (e.g., datasets DS1 and DS2 in reference data 223) is a part of the training data used in the first machine learning step (e.g., step S2) (e.g., reference data 223). Therefore, the training data used in the first and second machine learning steps can be shared.

[0136] Additionally, the method of the embodiment includes a first checking step (e.g., step S3), wherein the information processing device (e.g., information processing device 1) determines which of j (e.g., j=8) categories an image (e.g., the image of verification data contained in reference data 223) captured from the image display surface of the display device conforms to after the first machine learning step (e.g., step S2) and before the second machine learning step (e.g., step S5). The method of the embodiment includes a first extraction step (e.g., step S4), wherein the information processing device extracts data from the image data (e.g., the verification data contained in reference data 223) that was judged to conform to one of k (e.g., k=2) categories in the first checking step from the image data (e.g., the verification data contained in reference data 223) that was judged in the first checking step. The method of the embodiment includes a second checking step (e.g., step S7), wherein the information processing device determines which of the k categories the image data extracted in the first extraction step conforms to after the second machine learning step. Therefore, it is possible to confirm the classification accuracy of an image that was classified into one of the k types in the first machine learning step and is reclassified after the second machine learning step. Thus, by reflecting the classification result of the second inspection step, the classification accuracy of the image is further improved for the k types compared to the case with only the first machine learning step.

[0137] Furthermore, the method of the embodiment includes a first determination step (e.g., step S13), wherein the information processing device (e.g., information processing device 1) determines which of the following j categories (e.g., j=8) the data of an image obtained by capturing an image of the image display surface of the display device and which differs from the training data (e.g., reference data 223) belongs to. The method of the embodiment includes a second extraction step (e.g., step S14), wherein the data of the image that was determined to belong to one of the k categories (e.g., k=2) in the first determination step is extracted from the data of the image that became the object of determination in the first determination step. The method of the embodiment includes a second determination step (e.g., step S15), wherein the information processing device determines which of the k categories the data of the image extracted in the second extraction step belongs to. Thus, for the k categories out of the j categories, the classification accuracy of the image is further improved. Therefore, images of the display device can be classified with higher accuracy.

[0138] Furthermore, the information processing apparatus 1 in this embodiment is an information processing apparatus capable of using machine learning to determine which of j (e.g., j=8) categories an image obtained by capturing an image display surface of a display device (e.g., an image of the object image data 100 or the image data contained in the reference data 223) conforms to. The information processing apparatus 1 includes a storage unit (e.g., storage unit 22) storing pre-prepared training data (e.g., reference data 223) containing data of images corresponding to the j categories, and a computation unit (e.g., computation unit 21) reading the training data from the storage unit and performing machine learning. Based on the training data, the computation unit performs machine learning to determine which of the j categories an image conforms to; based on a portion of the training data containing data of images corresponding to k (e.g., k=2) of the j categories (e.g., datasets DS1 and DS2 in the reference data 223), it performs machine learning to determine which of the k categories an image conforms to. j and k are natural numbers greater than or equal to 2. k is less than j. Therefore, regarding the image classification performed by the information processing device 1, for the k-th classification among the j types, the image classification accuracy is further improved. Thus, it is possible to classify images of the display device with higher accuracy.

[0139] Furthermore, the method of the embodiment includes a contrast enhancement method for an image obtained by capturing an image display surface of a display device (e.g., an image containing image data of the object image data 100 or reference data 223). The method of the embodiment includes a step (e.g., step S12) in which an information processing device (e.g., information processing device 1) performs contrast enhancement on the image. A line graph (e.g., line graph L31, etc.) representing the relationship between the grayscale values ​​of the pixels contained in the image before and after contrast enhancement is a curve having two inflection points (e.g., Inflection_1, Inflection_2). Therefore, in image contrast enhancement, a more appropriate parameter setting can be achieved to make features caused by poor display clearer. That is, by setting the two inflection points, features caused by poor display can be made clearer.

[0140] Furthermore, the curve containing one of the two inflection points (e.g., Inflection_1) and the curve containing the other of the two inflection points (e.g., Inflection_2) are S-shaped growth curves. Therefore, the grayscale values ​​of the contrast-enhanced image increase in stages at each inflection point, resulting in a contrast-enhanced image with three stages of grayscale value levels as a whole.

[0141] Furthermore, one of the two inflection points (e.g., Inflection_1) is located in the line graph (e.g., line graph L31, etc.) at a gray value lower than the highest frequency gray value of the plurality of pixels contained in the image (e.g., the image of the image data contained in the object image data 100 or reference data 223), while the other of the two inflection points (e.g., Inflection_2) is located in the line graph at a gray value higher than the highest frequency gray value of the plurality of pixels contained in the image. Therefore, by contrast enhancement, pixels with gray values ​​lower than the highest frequency gray value and pixels with gray values ​​higher than the highest frequency gray value, which contain more pixels than those obtained by the display device for photographing qualified products, can be made clearer.

[0142] Furthermore, the information processing apparatus 1 in this embodiment is an information processing apparatus capable of enhancing the contrast of an image obtained from the image display surface of an image capture display device (e.g., an image containing image data from the object image data 100 or reference data 223). The information processing apparatus 1 includes an image processing unit (e.g., image processing unit 51) that performs contrast enhancement. The line graph showing the relationship between the grayscale values ​​of the pixels contained in the image before and after contrast enhancement is a curve with two inflection points (e.g., Inflection_1, Inflection_2). Therefore, the information processing apparatus 1 can provide a more suitable parameter setting for making features caused by poor display clearer during image contrast enhancement. That is, by setting the two inflection points, features caused by poor display can be made clearer.

[0143] Furthermore, in the above implementation, j = 8 and k = 2, but this is only one example and is not limited to this. j and k can be natural numbers greater than 2, and k can be less than j. It can be said that j must be 3 or greater. Additionally, the second check step and the second judgment step, which target types less than j among j, can be performed multiple times. In this case, in each of the multiple second check steps and the second judgment step, a classification of k types, which is limited to being different from j, is performed. Furthermore, the number of classifications in each of the multiple second check steps (or second judgment steps) can be different, with one focusing on k types of j and the other on h types of j.

[0144] Furthermore, in the above-described embodiments, the training data used in the second machine learning step is a part of the training data used in the first machine learning step, but the training data used in the first machine learning step and the training data used in the second machine learning step may be different.

[0145] Furthermore, in the above-described embodiment, the object image data 100 and reference data 223 include multiple image data captured by a device capable of simultaneously capturing images of multiple display panels, referred to as a large-scale illumination inspection device. This is merely one example of image data obtained by capturing images of the display surface of a display device, and is not limited to this. Images obtained by capturing images of the display surface of a display device may also be, for example, images obtained by capturing images of the display surface of a smartphone, or images obtained by capturing images of the display surface of a display device that is not merely a display panel.

[0146] Furthermore, while the above implementation uses CNN, it is merely a specific example of a machine learning algorithm and is not limited to it. Machine learning algorithms based on other methods can also be used instead of CNN in the above implementation.

[0147] Furthermore, any other effects resulting from the manner described in this embodiment, including effects clearly defined by the description in this specification or effects that can be reasonably conceived by those skilled in the art, are naturally interpreted as effects resulting from this disclosure.

[0148] Explanation of reference numerals in the attached figures

[0149] 1. Information processing device

[0150] 21. Arithmetic Department

[0151] 22 Storage Department

[0152] 30 Output Section

[0153] 40 Acquisition Department

[0154] 50 Processing Department

[0155] 51 Image Processing Department

[0156] 60 Judgment Department

[0157] 62. Limited Classification Processing Department

[0158] 70 Machine Learning Department

[0159] 100 Determine the object image data

[0160] 221 Machine Learning Program

[0161] 222 Image Processing Program

[0162] 223 Reference Data

[0163] 224 Setting Data

[0164] NN Neural Network.

Claims

1. A method for enhancing image contrast, which is a method for enhancing the contrast of an image obtained by capturing an image on the image display surface of a display device, wherein, The process includes the step of enhancing the contrast of the image using an information processing device. The line graph showing the relationship between the grayscale values ​​x before contrast enhancement and y after contrast enhancement of the pixels contained in the image is a curve with two inflection points. Wherein, inflection_1 represents the position of one of the two inflection points, inflection_2 represents the position of the other of the two inflection points, gain_1 represents the slope of one of the two inflection points, gain_2 represents the slope of the other of the two inflection points, and height_1 represents the upper limit of the grayscale value after contrast enhancement based on one of the two inflection points and the lower limit of the grayscale value after contrast enhancement based on the other of the two inflection points.

2. The image contrast enhancement method according to claim 1, wherein, The curve containing one of the two inflection points and the curve containing the other of the two inflection points are S-shaped growth curves.

3. The image contrast enhancement method according to claim 1 or 2, wherein, One of the two inflection points is located in the line graph at a position with a gray value lower than the highest frequency gray value of the multiple pixels contained in the image. The other of the two inflection points is located in the line graph at a position with a gray value higher than the highest frequency gray value of the multiple pixels contained in the image.

4. An information processing apparatus for enhancing the contrast of an image obtained from the image display surface of an image-capturing display device, wherein, Equipped with an image processing unit for contrast enhancement. The line graph showing the relationship between the grayscale values ​​x before contrast enhancement and y after contrast enhancement of the pixels contained in the image is a curve with two inflection points. Wherein, inflection_1 represents the position of one of the two inflection points, inflection_2 represents the position of the other of the two inflection points, gain_1 represents the slope of one of the two inflection points, gain_2 represents the slope of the other of the two inflection points, and height_1 represents the upper limit of the grayscale value after contrast enhancement based on one of the two inflection points and the lower limit of the grayscale value after contrast enhancement based on the other of the two inflection points.

Images