Defect inspection apparatus, defect inspection method, and manufacturing method

Abstract

A defect inspection apparatus inspects defects generated in an object to be inspected based on an image of the object to be inspected, wherein a plurality of defect discriminators are provided, the defect discriminators discriminate respective different kinds of defects using a prescribed machine learning model based on the image, and the kinds of defects discriminated by each of the defect discriminators are part of a prescribed number of kinds of defects that are the discrimination targets of the defect inspection apparatus. The present embodiment can be implemented in any one of a defect inspection apparatus, a defect inspection method, and a manufacturing method. According to the present embodiment, system management in defect inspection can be easily performed.

Description

Technical Field

[0001] This invention relates to defect inspection apparatus, defect inspection method, and manufacturing method. For example, this invention relates to a technique for determining the state of defects produced in an inspected object using image data representing an image of the inspected object.

[0002] This application claims priority to Japanese Patent Application No. 2020-207561, filed on December 15, 2020, the contents of which are incorporated herein by reference. Background Technology

[0003] When managing product quality, determining the product's condition is crucial. For example, tiny electronic components are sometimes formed on the surface of glass substrates covered by thin films of metal or oxides. These glass substrates can be used in various displays such as liquid crystal displays, photomasks, electronic device supports, information recording media, and planar antennas. Defects on the surface of the glass substrate are often exemplified by broken lines. Other defects include scratches and dirt. Therefore, high cleanliness and flatness are required for the surface of the glass substrate.

[0004] To reduce defects such as broken wires, it is possible to: analyze the surface condition of the glass substrate, determine the state of defects, identify the causes of defects as needed, and implement countermeasures for the manufacturing process. Therefore, it is possible to use machine learning models to classify defect types from images of the inspected object. For example, Patent Document 1 describes a defect inspection method that uses a deep learning model to classify defects generated on a wafer that serves as a substrate.

[0005] Generally, deep learning models require a large amount of training data to learn the model parameters that represent the input-output relationship. Depending on the amount of training data used for learning, misclassification may occur frequently. For example, when classifying the types of defects represented by an image, the probability of misclassification into a specific type may increase. In such cases, users may wish to adjust the model parameters to avoid misclassification.

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-124591

[0007] However, when adjusting model parameters in machine learning models, such as deep learning models, it's difficult to correct misclassifications of only specific species; it can also affect classification results for other species. That is, even if the model parameters are adjusted to accurately classify a specific species, the classification of other species may become inaccurate. Because model parameters must be adjusted to account for the classification of all species, system management sometimes requires significant labor and time. Summary of the Invention

[0008] The present invention was made in view of the above-mentioned problems. One of the objectives of the present invention is to provide a defect inspection apparatus, defect inspection method and manufacturing method that make system management easier in defect inspection.

[0009] (1) The present invention was made to solve the above-mentioned problem. One aspect of the present invention is a defect inspection device that inspects defects generated in the inspected object based on an image of the inspected object. The device includes a plurality of defect discriminators that use a predetermined machine learning model to identify different types of defects based on the image. The types of defects identified by each defect discriminator are part of a predetermined number of defect types that are the objects of the defect inspection device.

[0010] (2) Another aspect of the present invention is a defect inspection method that inspects defects generated in the inspected object based on an image of the inspected object, wherein multiple defect identification steps are provided, and in the defect identification steps, a predetermined machine learning model is used to identify different types of defects based on the image, and the types of defects identified in each defect identification step are part of a predetermined number of defect types that are identified in the defect inspection method.

[0011] (3) Another aspect of the present invention may be a glass manufacturing method, wherein the object to be inspected is glass, and has an inspection process using the defect inspection device of (1) or the defect inspection method of (2).

[0012] According to the present invention, system management in defect inspection can be made easier. For example, the frequency of incorrect judgments for certain types of defects can be reduced or eliminated without affecting the judgment results for other types of defects. Attached Figure Description

[0013] Figure 1 This is a schematic block diagram illustrating a structural example of the defect inspection device according to this embodiment.

[0014] Figure 2 This is a flowchart illustrating a first example of the inspection process involved in this embodiment.

[0015] Figure 3 This is an explanatory diagram used to illustrate the shooting conditions in the shooting unit involved in this embodiment.

[0016] Figure 4 This is an example of an image representing the object being examined.

[0017] Figure 5 This is another example of an image representing the object being examined.

[0018] Figure 6 This diagram illustrates a connection example of the defect detector involved in this embodiment.

[0019] Figure 7 This is a diagram illustrating an example of the processing time for defect identification processing according to this embodiment.

[0020] Figure 8 This is a graph showing an example of the false response rate for defect categories caused by the machine learning model involved in this embodiment.

[0021] Figure 9 This is a flowchart illustrating a second example of the inspection process involved in this embodiment.

[0022] Figure 10 This is an explanatory diagram used to illustrate the distances in the feature space involved in this embodiment.

[0023] Figure 11 This is a flowchart illustrating a third example of the inspection process involved in this embodiment.

[0024] Figure 12 This is an explanatory diagram illustrating an example of feedback processing from the inspection process to the manufacturing process involved in this embodiment.

[0025] Figure 13 This is a flowchart illustrating an example of the glass manufacturing process involved in this embodiment.

[0026] Figure 14 This is a diagram illustrating an example of the machine learning model involved in this embodiment. Detailed Implementation

[0027] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0028] First, the structure of this embodiment will be explained. Figure 1 This is a schematic block diagram illustrating a structural example of the defect inspection device according to this embodiment.

[0029] The defect inspection apparatus 100 according to this embodiment is an inspection apparatus for inspecting defects that may occur in an inspected object using image data representing an image of the inspected object. The defect inspection apparatus 100 performs multiple defect identification steps, which are as follows: acquiring image data representing an image of the inspected object; and, based on the acquired image data, using a predetermined machine learning model to identify the type of defect occurring in the inspected object. In each defect identification step, the candidate for one or more defects to be identified is part of a predetermined number (M types, where M is a predetermined integer of 2 or more) of defect types (hereinafter, identifiable types) that are the identification targets of the defect inspection apparatus 100 as a whole. Furthermore, the candidate for one or more defects to be identified differs in each defect identification step.

[0030] The defect inspection device 100 is configured to include: a control unit 110, an imaging unit 130, an input / output unit 140, an operation unit 150, a display unit 160, and a storage unit 170.

[0031] The control unit 110 performs processing for implementing the functions of the defect inspection device 100 and processing for controlling those functions. The control unit 110 may also include general-purpose components such as a processor, thus being configured as a computer. The processor reads a program pre-stored in the storage unit 170 and performs processing instructed by the instructions described in the read program to implement the function. In this application, performing processing instructed by the instructions described in the program is sometimes referred to as program execution, program execution, etc. Part or all of the control unit 110 may be configured as dedicated hardware including LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit), and is not limited to general-purpose hardware such as a processor. The functional units that implement the functions of the control unit 110 will be described later.

[0032] The imaging unit 130 captures images of various objects present within its field of view around the component and outputs image data representing the captured images to the control unit 110. The imaging unit 130 may also have a mechanism that allows the imaging conditions to be varied for a single object under inspection, or it may have a mechanism that allows for simultaneous imaging under multiple imaging conditions. Imaging conditions significantly affect the captured images. Depending on the type of defect present in the object under inspection, the brightness and shape of the image of that defect may vary significantly depending on the imaging conditions. Imaging conditions are specified, for example, by any one of brightness, shooting direction, illumination method, or a combination thereof.

[0033] Figure 3This example illustrates the case of capturing an image of a single subject under four different shooting conditions. The four shooting conditions are categorized into two types based on combinations of brightness and shooting direction, and into two types based on illumination methods: reflection and transmission. The combinations of brightness and shooting direction include bright field of view and dark field of view. A bright field of view is a shooting condition where the light emitted from the illumination source is directed towards the subject Sb, and the shooting unit 130 (camera) is positioned and oriented such that the incident direction of the reflected light or transmitted light from the subject Sb is included within the field of view. In a bright field of view, the reflected or transmitted light from the subject Sb directly enters the camera, thus capturing a bright image. A dark field of view is a shooting condition where the light emitted from the illumination source is directed towards the subject Sb, and the shooting unit 130 is positioned and oriented such that the incident direction of the reflected light or transmitted light from the subject Sb is not included within the field of view, but the subject Sb itself is included within the field of view. In dark field, reflected or transmitted light from the subject Sb does not directly strike the camera, but scattered light generated on the surface of the subject Sb strikes it, thus capturing an image darker than that in bright field.

[0034] The imaging unit 130 can also be configured to include either a digital camera that captures still images or a digital video camera that captures moving images. The moving images are configured to include still images obtained by repeatedly shooting at constant time intervals (e.g., 1 / 120 to 1 / 12 of a second).

[0035] return Figure 1 The input / output unit 140 can be connected to other devices wirelessly or via wired connection to input and output various data. The input / output unit 140 may have, for example, an input / output interface or a communication interface. The input / output unit 140 may be connected to, for example, various control devices, measuring devices, and other devices used in the manufacturing process.

[0036] The operation unit 150 accepts user operations and generates operation signals corresponding to the received operations. The operation unit 150 may be configured as dedicated components including buttons, knobs, and dials, or as general-purpose components including a mouse and keyboard. The operation unit 150 may also be an input interface that receives operation signals from other devices wirelessly or via a wired connection. These other devices may be mobile devices such as remote controls or multi-function mobile phones. The operation unit 150 outputs the acquired operation signals to the control unit 110.

[0037] The display unit 160 displays display information such as images, text, and symbols based on display data input from the control unit 110. The display unit 160 may also include, for example, any one of a liquid crystal display or an organic light-emitting diode display.

[0038] In addition to the procedures described above, the storage unit 170 stores various data used for the processing performed by the control unit 110, as well as various data acquired by the control unit 110. The storage unit 170 may be configured as a non-volatile (non-temporary) storage medium, such as ROM (Read Only Memory), flash memory, or HDD (Hard Disk Drive). Alternatively, the storage unit 170 may be configured as a volatile storage medium, such as RAM (Random Access Memory) or registers.

[0039] The control unit 110 is configured to include a defect detection unit 112, a defect identification unit 114, a comprehensive judgment unit 116, a model learning unit 118, a manufacturing process management unit 120, a new type judgment unit 122, and a judgment input unit 124 as functional units to perform its functions. These functional units may also be configured to include dedicated components, or they may perform their functions by executing a predetermined program through a processor. In the following description, the inspected object is mainly a glass substrate for a flat panel display (FPD), but the inspected object may also be other objects.

[0040] The control unit 110 is configured to include a defect detection unit 112, a defect identification unit 114, a comprehensive judgment unit 116, a model learning unit 118, a manufacturing process management unit 120, a new type judgment unit 122, and a judgment input unit 124 as functional units to perform its functions. These functional units may also be configured to include dedicated components, or they may perform their functions by executing a predetermined program through a processor. In the following description, the inspected object is mainly a glass substrate for a flat panel display (FPD), but the inspected object may also be other objects.

[0041] return Figure 1 The defect detection unit 112 outputs image data to the defect identification unit 114, whereby the image data represents a portion of the image containing one defect region at a time as a defect image. If the defect detection unit 112 does not detect any defect regions in the image of the object being inspected, it determines the image of the object being inspected as a qualified image and outputs image data representing the qualified image to the comprehensive determination unit 116. Alternatively, in this case, the defect detection unit 112 may also output instruction information to the determination input unit 124, indicating the display of the input screen.

[0042] Furthermore, if the size of a region identified as a defect area is less than the prescribed detection threshold in either the horizontal or vertical direction, the defect detection unit 112 may ignore the defect area's determination and classify it as a normal area. A detection threshold that is sufficiently small than the size of a typical defect is pre-set in the defect detection unit 112.

[0043] The defect identification unit 114 has multiple defect detectors. In the following description, the number of defect detectors is set to N (N is a predetermined integer of 2 or more). Sometimes, each defect detector is labeled with a sub-number such as defect detector 114-1, 114-2, etc., for distinction. Similarly, sometimes the structural parts of each defect detector are also labeled with sub-numbers for distinction. Each defect detector may not necessarily be constructed solely by hardware; its function may also be implemented through the execution of a specified program.

[0044] N defect discriminators 114-1 to 114-N process each defect generated in the inspected object as follows: They identify the type of defect shown in the image represented by image data input from the defect detection unit 112 as any one of one or more candidate defects. The candidate defects for one or more identifiable defects can differ among the N defect discriminators 114-1 to 114-N. These candidates each form part of the M identifiable types that the defect detection unit 112 can identify as a whole. Therefore, M is equal to N, or more than N. When M and N are equal, the N defect discriminators 114-1 to 114-N function as primary discriminators, which determine whether the type of defect generated in the inspected object is a distinct type of defect, or the probability of that type. When M is greater than N, at least one defect discriminator functions as a multi-level discriminator. This multi-level discriminator determines which of the multiple types of defects exists in the inspected object, or the probability of each type. In the following explanation, we mainly use the case where M and N are equal as an example.

[0045] Next, the functional structure of defect detector 114-1 will be explained. As for defect detectors 114-2 to 114-N, unless otherwise specified, they have the same functional structure as defect detector 114-1. The description of defect detector 114-1 will be used as a reference.

[0046] Defect detector 114-1 performs the following steps: pre-processing, inference, defect determination, and good / bad determination. Figure 2 This corresponds to steps S112-1, S114-1, S116-1, and S118-1. The preprocessing step includes a process for integrating the image data input to the device into a form required as input to the inference step (e.g., the number of features of the input value). The preprocessing step includes one or both of dimensionality reduction and size adjustment.

[0047] Size adjustment is a process that changes the number of pixels in the horizontal and vertical directions of a portion of the processed area. Size adjustment can be either enlargement or reduction. The defect detector 114-1 interpolates the signal value of each pixel in the input image to determine the signal value of each pixel in the enlarged or reduced image. The defect detector 114-1 can use known interpolation methods such as bilinear interpolation and bicubic interpolation in the interpolation. The defect detector 114-1 can also determine, for each defect area, that the maximum value of its horizontal or vertical diameter, including the entire defect area, is a constant r (e.g., a real number greater than 0.5 and less than 1) times the size of the horizontal or vertical direction of the portion of the area, rather than simply changing the size of the image by a predetermined ratio. The diameter is equivalent to the length of a line segment that runs through all directions of the defect area. For example, when the detected defect shape is elliptical, the maximum value of the defect diameter is represented by the length of the major axis. When the detected defect shape is rectangular, the maximum value of the defect diameter is represented by the length of the long side.

[0048] Dimension reduction is the process of combining multiple images of a partial region captured under different shooting conditions into a smaller number of images. Dimension reduction is achieved through image synthesis. Image synthesis includes the following process: calculating a weighted sum of the luminance values ​​of multiple images for each pixel, which serves as a new signal value. The weighted sum is equivalent to the sum of the product of the luminance value of each image and its corresponding weight coefficient, i.e., the sum of the multiplicative values ​​among the images. The minimum number of images generated through image synthesis is one, but multiple (e.g., three) can also be used. Alpha mixing can also be used in image synthesis, for example. Alpha mixing is a method where the sum of the weight coefficients (alpha values) of each image used for synthesis is normalized to 1. The ratio of the weight coefficients between images can also be different between defect discriminators 114-1 to 114-N. Therefore, a value larger than the weight coefficient for other images can be set as the weight coefficient for images captured under shooting conditions where the defect is easily detected, based on the type of defect for each defect discriminator. For example, regarding scratches, the settings are configured such that the weighting coefficient for images captured in bright field is relatively small, while the weighting coefficient for images captured in dark field is large. This allows for more reliable detection of scratches from images obtained in dark field.

[0049] When the number of images generated by dimension reduction is set to three, the defect detector 114-1 can also generate image data representing a single color image. This single color image is formed by using the signal value of each generated image as the color signal value for each pixel as its own different hue, and merging the images of each hue of the three hues. As a color system for representing the color image, the RGB color system, YCrCb color system, etc., can be used. An example is the case where images Im01 to Im04, representing four different shooting conditions, are combined into a single color image Im05 through dimension reduction. First, the defect detector 114-1 acquires images Im01 and Im02 captured under bright field and images Im03 and Im04 captured under dark field. At this time, images Im01 and Im03 are images obtained by capturing transmitted light, while images Im02 and Im04 are images obtained by capturing reflected light. Depending on the shooting conditions, the shape and brightness of the defects differ. Image Im05 is represented in defect discriminator 114-1 by synthesizing the signal values ​​of four images using different weighting coefficients for red, green, and blue. For example, the signal value of the red channel of image Im05 is obtained by synthesizing the luminance values ​​of image Im01 and image Im02 using a weighting coefficient ratio of 0.7:0.3. The signal value of the green channel of image Im05 is obtained by synthesizing the luminance values ​​of image Im03 and image Im04 using a weighting coefficient ratio of 0.4:0.6. The signal value of the blue channel of image Im05 is obtained by synthesizing the luminance values ​​of image Im02 and image Im03 using a weighting coefficient ratio of 0.5:0.5.

[0050] The inference process is a process of determining the probability that the type of defect produced in the inspected object is a specified type from image data representing the image being processed. The defect discriminator 114-1 takes the signal value of each pixel constituting the image data as input values, and uses a specified machine learning model to calculate the probability as the output value. The output value is a real number between 0 and 1. A parameter set (model parameters) is pre-set in the defect discriminator 114-1 for calculating the output value based on the input values.

[0051] As machine learning models, neural networks such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) can be used. However, the available machine learning models are not limited to neural networks; methods such as Random Forests (RFs) and Support Vector Machines (SVMs) can also be employed.

[0052] The defect determination process is a process of determining whether a defect conforms to the type of defect to be determined based on the probability calculated in the inference process. For example, the defect discriminator 114-1 determines that a defect conforms to the type if the calculated probability is greater than a predetermined defect determination threshold, and determines that a defect does not conform to the type if the calculated probability is less than the defect determination threshold. The defect determination threshold can also be set independently according to the type of defect to be determined. For example, in each defect discriminator, the defect determination threshold is set to a smaller value for types of defects with a higher risk of occurrence, and a larger value for types of defects with a lower risk of occurrence. A high risk of occurrence means a high probability of material or economic loss due to its occurrence, or a large potential loss. That is, there is a trend where defects with a higher risk of occurrence (i.e., high-risk defects) are less likely to be missed and rejected, while defects with a lower risk of occurrence (i.e., low-risk defects) are more likely to be over-detected. For example, the risk of scratches, bubbles, foreign matter, and dirt decreases in this order, with scratches having a higher risk. Therefore, by decreasing the defect determination threshold for higher-risk defects, the recall rate, which can detect defects with relatively few omissions, can be improved. Conversely, by increasing the defect determination threshold for lower-risk defects, the precision rate, which can reliably detect defects, can be improved. The defect discriminator 114-1 generates defect flags indicating whether the object region being judged conforms to or does not conform to a defect, and stores the defect flags generated in correspondence with the object region in the storage unit 170.

[0053] Furthermore, in this application, as mentioned above, "foreign matter" and "dirt" are sometimes used as the names of the objects to refer to the state in which a specified object is attached or mixed in, as a type of defect.

[0054] The good / bad determination process is as follows: when a defect is determined to conform to a specified defect type in the defect determination process, the good / bad status of the detected defect area is determined based on a specified determination criterion value. If a defect is determined to be non-compliant in the defect determination process, the good / bad determination process is not performed. For example, if the size (dimension) of a detected defect is larger than the determination criterion value shown in the pre-set determination data in this device, the defect detector 114-1 determines the area as unqualified; if the size of a detected defect is less than or equal to the determination criterion value, the area is determined as acceptable. Generally, each defect is represented graphically on a two-dimensional plane, but the defect detector 114-1 determines the size of the defect by the maximum diameter of the detected defect. The determination criterion value can also be set independently according to the type of defect being determined. For example, the determination criterion value for scratches is 100–300 μm, and the determination criterion value for foreign objects is 50–120 μm. The defect identifier 114-1 generates a non-conforming flag indicating whether a defect is qualified or unqualified based on the defect and its type, and stores the non-conforming flag in the storage unit 170 in correspondence with the object area.

[0055] The comprehensive judgment unit 116 determines the overall condition of the inspected body by referring to the good / bad status of each detected defect. For example, regardless of the defect type, the comprehensive judgment unit 116 counts the number of defects deemed unqualified for each defect of the inspected body, referring to all good / bad judgments indicated by the good / bad markings. If the total number of defects of all types exceeds the prescribed number of judgment criteria, the comprehensive judgment unit 116 determines the inspected body to be unqualified; if the total number of defects of all types is less than or equal to the prescribed number of judgment criteria, the inspected body is determined to be qualified.

[0056] In the defect identification unit 114, the good / bad determination process runs in parallel across defect types, so there may be cases where a single defect is associated with multiple defect types and thus indicates non-conformity. The comprehensive determination unit 116 can also process defects associated with these multiple non-conformity indicators as belonging to multiple types. For example, when three defects are detected from the inspected object, the comprehensive determination unit 116 determines that defect 001 conforms to both scratches and foreign matter, defect 002 conforms to dirt, and defect 003 conforms to both foreign matter and dirt. The defect identification unit 114 can also determine the number of defects as the actual number of defects, i.e., three, or it can determine the total number of items of each type, i.e., five, as the number of defects.

[0057] Furthermore, in the comprehensive judgment unit 116, judgment data representing the number of judgment criteria for each defect type can be pre-set, and the judgment data is used to determine the status (rule base) of the inspected object. The comprehensive judgment unit 116 counts the number of defects for each defect type, referring to the good / bad judgment indicated by the good / bad flag. If a defect type exists where the counted number of defects exceeds the number of judgment criteria shown in the judgment data, the comprehensive judgment unit 116 determines the inspected object to be unqualified; if such a defect type does not exist, the inspected object is determined to be qualified. Alternatively, the number of judgment criteria can be set to be fewer for defect types with higher risk of occurrence, and more for defect types with lower risk of occurrence. For example, in the case of defects such as scratches and dust, the number of judgment criteria can be set to 1 and 5 respectively. Figure 5 Example: One scratch and two specks of dust are detected on the inspected object. Assuming that only the specific type of defect, namely dust, is considered, and other types of defects are ignored, the number of detected dust particles is two, which is less than the number of criteria pieces, so the inspected object is judged as acceptable. However, the number of scratches is one, which is equal to the number of criteria pieces, so the comprehensive judgment unit 116 judges the inspected object as unacceptable. Therefore, by comparing the number of defects for each type with the number of criteria pieces, the risk of judging an inspected object with unacceptable defects as acceptable can be avoided.

[0058] Alternatively, a weighting coefficient can be pre-set for each defect type in the comprehensive judgment unit 116, and the sum of the product of the weighting coefficient and the number of defects for each defect type can be calculated as the effective number. The comprehensive judgment unit 116 can also judge the inspected object as unqualified if the effective number is more than the prescribed judgment standard number, and judge the inspected object as qualified if the effective number is less than the judgment standard number.

[0059] Alternatively, when image data representing a qualified image is input from the defect detection unit 112 to the comprehensive judgment unit 116, the inspected object can be judged as qualified.

[0060] Alternatively, regardless of the type of defect, if all defects and types indicated by the good or bad mark are judged as qualified for each defect of the inspected object, the comprehensive judgment unit 116 outputs instruction information indicating the display of the input screen to the judgment input unit 124.

[0061] The inspected objects determined to be qualified become the objects for shipment, and the inspected objects determined to be unqualified become the objects for disposal or for return to the manufacturing process. The comprehensive determination unit 116 outputs determination result information indicating the determination result to the manufacturing process management unit 120. The manufacturing process management unit 120 refers to the determination result information input from the comprehensive determination unit 116, and disposes of or returns the inspected objects determined to be unqualified to the manufacturing process. The manufacturing process management unit 120 outputs, for example, a control signal indicating return to the manufacturing process or disposal to the manufacturing equipment.

[0062] In addition, the comprehensive determination unit 116 may include information on the number of defects of each defect type or the total number of defects in the determination result information.

[0063] The model learning unit 118 calculates a parameter set of a machine learning model for identifying the defect type by the defect discrimination unit 114 as model parameters. The model learning unit 118 uses a prescribed machine learning model for the training data ((training data), also referred to as learning data (learning data), supervised data (supervised data), etc.) according to the type of the defect to be discriminated, and performs a learning process to calculate the model parameters. In the model learning unit 118, the training data is set before the learning process, and the training data includes a set of known input values and the output values corresponding to the input values (typically, 1000 to 10000 or more), that is, a data set. As the input value, image data including the signal value of each pixel is used. As the output value of each set included in the training data, 1 is assigned when the image represented by the image data as the input value shows a defect of the type to be discriminated, and 0 is assigned in other cases. In the learning process, the model learning unit 118 can use, for example, the training data formed by adding (annotating (annotation)) the type of the defect and its output value to each image data as the input value.

[0064] In the learning process, the model learning unit 118 updates the model parameters until the magnitude of the difference between the calculated value and the output value obtained by using the prescribed machine learning model for the input value converges to approximately zero for the entire set. When the change amount of the model parameters before and after the update or the change amount of the magnitude of the difference before and after the update is less than the prescribed convergence determination threshold, it can be determined that the model parameters have converged.

[0065] In updating model parameters, methods such as steepest descent, stochastic gradient descent, conjugate gradient method, and back propagation can be used.

[0066] As an indicator of the magnitude of the difference, error functions such as mean squared difference (SSD) and cross entropy error can be used. The model learning unit 118 sets the model parameters calculated according to the defect type to the defect discriminator involved in the identification of that defect type.

[0067] Furthermore, the model learning unit 118 can also add a dataset to the training data related to the defect type input from the decision input unit 124, where the output value for that defect type is set to 1 and the image data used as the decision object is used as the input value. Alternatively, the model learning unit 118 can also add a dataset to the training data related to the defect type, where the output value for other types is set to 0 and the image data used as the decision object is used as the input value. Then, the model learning unit 118 can use the newly added training data to update the model parameters related to each defect type (transfer learning).

[0068] The manufacturing process management unit 120 controls the manufacturing process of a product based on the state of defects generated in the inspected product as determined by the defect identification unit 114. For example, control data is pre-set in the manufacturing process management unit 120, which includes information on the state of defects and correction conditions, and these are correspondingly represented. Correction conditions are conditions used to correct the manufacturing conditions used in the manufacturing process at that point in time and to assign corrected manufacturing conditions. As an example of defect state information, the manufacturing process management unit 120 can use the number of defects generated in the inspected product, categorized by type. The number of each defect type is transmitted through the judgment result information input from the comprehensive judgment unit 116. Manufacturing conditions can include action parameters for performing the product's manufacturing process and environmental parameters representing the environment. Action parameters can include rotational speed, power consumption, etc., which power the manufacturing equipment. Environmental parameters can include temperature, pressure, etc. Correction conditions can be represented by the change in any parameter used to assign modified manufacturing conditions. The manufacturing process management unit 120 uses the control data to determine information on correction conditions corresponding to the referenced defect state. The manufacturing process management unit 120 generates control information indicating changes to manufacturing conditions under determined modification conditions and outputs the generated control information to the manufacturing equipment. The manufacturing equipment uses the modification conditions represented by the manufacturing information input from the manufacturing process management unit 120 to modify the manufacturing conditions and executes the manufacturing process under the modified manufacturing conditions.

[0069] The new type determination unit 122 determines whether the type of defect detected from the inspected object is a new type that is different from any of the known defect types. For example, if the new type determination unit 122 determines that the type of defect detected by the defect detection unit 112 is not any of the defect types identified by the defect identification unit 114, it determines that the defect type is a new type. For example, for each defect type corresponding to the defect identifiers 114-1 to 114-N, the new type determination unit 122 can determine that defects whose defect symbols all do not conform are new types. The new type determination unit 122 can also display a notification screen indicating that the defect type is new on the display unit 160 when it determines that the defect type is new. In addition, in this case, the new type determination unit 122 can also output instruction information indicating the display of the input screen to the determination input unit 124. Thus, the user, i.e., the operator, can be notified that the defect type is new, prompting the input of the defect category and whether it is good or bad.

[0070] The determination input unit 124 receives an operation signal from the operation unit 150. This operation signal indicates the type of defect in the inspected object shown in the image data or whether the inspected object is good or bad. The determination input unit 124 may also generate an input screen, which is then displayed on the display unit 160. This input screen includes an image of the inspected object shown in the image data and a screen component that, when pressed, can indicate either the type of defect or the good or bad condition of the inspected object, or both. "Pressing" refers not only to physically pressing the screen but also to receiving an operation signal from the operation unit 150 or another device, indicating a position within the display area. Screen components can include, for example, buttons, checkboxes, or menu bars.

[0071] Furthermore, when instruction information indicating the display of the input screen is input from the defect detection unit 112, the comprehensive judgment unit 116, or the new type judgment unit 122, the input screen can be displayed on the display unit 160. This prompts the user to input a judgment on the category or quality.

[0072] The judgment input unit 124 outputs the input defect category and the quality determination information of the inspected object to the comprehensive judgment unit 116. Alternatively, the judgment input unit 124 can also output the input defect category to the model learning unit 118.

[0073] (Inspection and processing)

[0074] Next, examples of the inspection process involved in this embodiment will be described. Figure 2 This is a flowchart illustrating an example of the inspection process involved in this embodiment.

[0075] (Step S102) The imaging unit 130 captures an image of the FPD glass substrate, which is an example of the object under inspection. Then, the process proceeds to step S104.

[0076] (Step S104) The defect detection unit 112 detects the defective parts, i.e., the defective areas, from the image of the object to be inspected captured by the imaging unit 130. Then, the process proceeds to step S110.

[0077] The processing in step S110 includes the processing in steps S110-1 to S110-N. For the processing in steps S110-1 to S110-N, the defect detectors 114-1 to 114-N perform processing in parallel for each common partial region, including the various defect regions detected from the captured image. The processing in steps S110-2 to S110-N is the same as that in step S110-1, so its description is provided below.

[0078] (Step S112-1) The defect detector 114-1 performs a preprocessing step on the image within a certain region. The defect detector 114-1 includes: dimension reduction of images captured under different shooting conditions; and dimension adjustment of the number of features in the dimension-reduced image as the input value of the image inference step. In one example, when the number of horizontal pixels, the number of vertical pixels, and the dimension (number of frames) of the image being processed are 200, 200, and 4 respectively, the number of horizontal features, the number of vertical features, and the dimension of the preprocessed image can be 224, 224, and 3 respectively. The defect detector 114-1 can also reconstruct a two-dimensional color image with 224 horizontal pixels and 224 vertical pixels. Then, the processing in step S114-1 is performed.

[0079] (Step S114-1) The defect detector 114-1 performs inference processing on the signal value of each pixel representing the preprocessed image. The defect detector 114-1 takes the signal value of each pixel as input and uses a prescribed machine learning model to calculate the probability that the type of defect represented by a portion of the region matches the type of defect specified in this device, as the output value. Then, the process proceeds to step S116-1.

[0080] (Step S116-1) Defect identifier 114-1 performs the defect judgment process. Defect identifier 114-1 determines whether the type of defect conforms to the defect type specified in the equipment based on whether the calculated probability is greater than the specified defect judgment threshold set in the equipment. Then, it proceeds to the processing in step S118-1.

[0081] (Step S118-1) Defect detector 114-1 performs a good / bad determination process. For example, defect detector 114-1 determines good / bad based on whether the size of the detected defect is larger than the judgment benchmark value set in this device. After that, the process proceeds to step S122.

[0082] (Step S122) The comprehensive judgment unit 116 refers to the good or bad status of each type of defect detected and determines whether it is qualified or unqualified, as the overall status of the inspected body. For example, the comprehensive judgment unit 116 determines whether it is qualified or unqualified based on whether the number of defects judged as unqualified is greater than the prescribed number of judgment criteria. Afterwards, the manufacturing process management unit 120 uses the inspected bodies that have been judged as qualified as the products to be shipped, and discards or returns the inspected bodies that have been judged as unqualified to the manufacturing process. Then, the process ends. Figure 2 The processing.

[0083] As described above, the defect inspection apparatus 100 according to this embodiment includes defect discriminators 114-1 to 114-N, which determine whether a detected defect conforms to a specific defect type. Therefore, corrections to the model parameters used for determining one type of defect do not affect the determination of other types of defects. Furthermore, suitable imaging conditions, imaging units 130, or pre-processing can be used for each defect type. Moreover, the N defect discriminators 114-1 to 114-N are arranged in parallel. That is, each defect discriminator determines whether the detected defect conforms to a predetermined defect type, independent of the determination results of other defect discriminators. Therefore, even if the number of defect types to be determined increases, the processing time does not change. Furthermore, each defect discriminator selects information specific to that type of defect, such as characteristic quantities inherent in that type, regardless of the defect quality determination criterion. However, the more types of defects to be determined, the more computational resources are required.

[0084] In the defect inspection device 100, defect detectors 114-1 to 114-N can also be arranged in series. More specifically, when defect detector 114-n (where n is an integer between 1 and N-1) determines that the type of defect detected in the image of a partial area does not conform to the defect types specified in this device, defect detector 114-n+1 begins processing to determine whether the type of the detected defect conforms to the defect types specified in this device. When defect detector 114-n determines that the type of defect detected from the image of a partial area conforms to the types specified in this device, it identifies the type of the detected defect as a conforming type and does not perform the processing of defect detector 114-n+1 and thereafter. Figure 6 In the example shown, in step S110-1, the defect detector 114-1 determines whether the detected defect is a scratch. If it is determined to be a scratch, the process ends. Figure 6 The processing continues. If the defect is determined to be inconsistent with the scratch pattern, the defect detector 114-1 proceeds to step S110-2. In step S110-2, the defect detector 114-2 determines whether the detected defect type is an air bubble. If it is determined to be consistent with an air bubble, the process ends. Figure 6 The processing begins when a bubble is determined to be non-compliant. Defect detector 114-2 then proceeds to the next stage of processing.

[0085] Therefore, by connecting defect discriminators 114-1 to 114-N in series, the number of defect discriminators executing the defect discrimination process at one time is limited to one, thus effectively utilizing limited computing resources. Furthermore, the learning of model parameters used in subsequent defect discriminators can be performed independently of the learning of model parameters used in earlier defect discriminators, thus avoiding the problem of increased learning time. However, the execution of the defect discrimination process implemented using subsequent defect discriminators begins only after the defect discrimination process implemented using earlier defect discriminators has been completed. Therefore, as... Figure 7 As illustrated, the defect inspection device 100 generally exhibits the following trend: the more types of defects an object identifies, the slower the processing time. Figure 7 In the diagram, the vertical axis and the horizontal axis represent processing time and the number of models, respectively. The number of models corresponds to the types of defects being detected, i.e., the number of defect discriminators. Figure 7 The illustrated processing time includes a component proportional to the number of models and a constant component independent of the number of models (approximately 0.043 seconds, see dashed line). The former corresponds to the time required to determine whether there is compliance for each defect type. The latter corresponds to the time spent on imaging, preprocessing, etc., independent of the defect type. When the number of defect discriminators is 10 or less, the processing time is almost identical to that using the previously used multi-level model. Regarding accuracy, the results are equivalent to, or the 2-level model has higher accuracy, when comparing the machine learning model used by the defect discriminator as a 2-level model with that of a multi-level model. A 2-level model is a machine learning model used to determine whether a defect type conforms to a specification. A multi-level model is a machine learning model used to determine which of the specified multiple categories the defect type belongs to. Figure 8 In the example shown, the error rates for each of the multi-level model and the level 2 model are 4.0% and 2.0%, respectively, with the level 2 model exhibiting higher accuracy.

[0086] Furthermore, the defect discriminators 114-1 to 114-N connected in series can also be determined in descending order of the frequency of occurrence of the first to Nth defects, which are the objects of discrimination. The more frequently occurring types of defects are detected earlier, thus minimizing processing time. Additionally, the defect discriminators 114-1 to 114-N can also be determined in descending order of the risk level of the first to Nth defects, which are the objects of discrimination. The higher the risk of a defect, the earlier it is detected, thus reducing damage caused by delays in defect detection.

[0087] Next, use Figure 9Another structural example of the new species determination unit 122 will be described. The new species determination unit 122 performs a new species determination process (step S122-n). The new species determination process includes a preprocessing process (step S122-a), an inference process (step S122-b), a new species degree calculation process (step S122-c), and a new species degree determination process (step S122-d). Among them, in Figure 9 The illustration of step S110-3 is omitted, but this does not mean that the processing of step S110-3 must be omitted.

[0088] In step S122-a, the new type determination unit 122 performs the same pre-processing steps as in step S112-1 or step S112-2.

[0089] In step S122-b, the new type determination unit 122 performs inference using an N-level machine learning model on a portion of the image input to the unit, and extracts the feature quantities calculated in the inference. The N-level machine learning model can be, for example, an N-level neural network (e.g., a CNN), which calculates the probability of each of the N types of defects as the output value. The new type determination unit 122 sets model parameters learned using training data. For each set, the training data assigns 1 as the output value for the defect type shown in the image data as input, and appends 0 as the output value for the dimensions corresponding to other types. Furthermore, the new type determination unit 122 can obtain the calculated values ​​output from predetermined intermediate layers other than the input and output layers of the neural network constituting the unit, as feature quantities representing the features of the image. In addition, using the set model parameters, for each image where the defect type is known, the feature quantities representing the features of the image (defect) are calculated using the method described later. For each defect type, a representative value (e.g., centroid) of the feature quantity is pre-set in the new type determination unit 122.

[0090] In step S122-c, the new type determination unit 122 calculates the distance between the representative value of the feature quantity determined according to the defect type and the obtained feature quantity, and determines the minimum value among the distances calculated according to the type as the new type degree. For Figure 10 In this context, the two-dimensional vector representing the feature quantity of the detected defect (detected defect) is represented by an asterisk in the feature quantity space expanded from the feature quantities X and Y as elements. Furthermore, the distance between the representative value (×print) of each defect type (dirt, scratch, bubble, dust) and the calculated feature quantity, and the distance between the representative value of the bubble and the feature quantity, are calculated as a new type degree.

[0091] return Figure 9In step S122-d, the new type determination unit 122 determines that the defect type is a new type if the calculated new type degree is greater than a predetermined distance threshold, and determines that the defect type is not a new type if the calculated new type degree is less than the predetermined distance threshold. The new type determination unit 122 determines the distance threshold as a value that is sufficiently larger than the variance of any feature quantity. As an index value representing distance, for example, Mahalanobis' distance, Minkowski's distance, Manhattan distance, cosine similarity, etc., can be used.

[0092] Furthermore, when the new type determination unit 122 determines that the defect type is new, it can output new type detection information to the comprehensive determination unit 116. This new type detection information indicates that the defect type being determined is new. Additionally, when the comprehensive determination unit 116 receives new type detection information from the new type determination unit 122, it can also discard the good / bad flag for defects indicated by the new type detection information. This avoids the erroneous use of existing determination results.

[0093] The new type determination unit 122 can also determine the feature quantity of the image representing the defect that has been determined to be a new type as the representative value of the new type. The new type determination unit 122 can also determine that the defect type shown in the image data is a new type if the distance between the feature quantity calculated based on another image data and the representative value of the new type is within a predetermined distance. Furthermore, the model learning unit 118 can perform learning processing using training data to determine model parameters for identifying new types of defects based on image data. The training data includes multiple sets of image data representing images of new types of defects as input values ​​and 1 as the output value; and multiple sets of image data representing images of other types of defects as input values ​​and 0 as the output value. The defect identification unit 114 can also be configured as a new defect determiner, which uses the determined model parameters to identify new types of defects.

[0094] Next, use Figure 11 and Figure 12 An example of manufacturing process control using the Manufacturing Process Management Department 120 will be explained. Figure 11 , Figure 12 The illustrated process includes steps S102-S126. Regarding steps S102-S122, the above description is referenced. Specifically, in step S122, as an example of the state of defects generated in the product being inspected, the comprehensive determination unit 116 outputs determination result information indicating the number of each defect type to the manufacturing process management unit 120.

[0095] The processing in steps S102-S122 consists of a series of inspection procedures, which are obtained as the defect status of each product of the inspected body, or their time series as quality trends.

[0096] In step S124, the manufacturing process management unit 120 refers to the control data preset in this component and determines the correction conditions corresponding to the defect status shown in the judgment result information (described later).

[0097] In step S126, the manufacturing process management unit 120 generates control information indicating changes to the manufacturing conditions under the determined correction conditions, and outputs (feedbacks) the generated control information to the manufacturing equipment. The manufacturing equipment corrects the manufacturing conditions using the correction conditions shown by the manufacturing information input from the manufacturing process management unit 120. The corrected manufacturing conditions for each product, or their time series, are obtained as control trends. In the manufacturing process, the manufacturing equipment performs the manufacturing process using the corrected manufacturing conditions (step S200). This embodiment can be implemented as a manufacturing method including an inspection process in steps S102-S122.

[0098] Furthermore, the processing of steps S102-S126 can be integrated, for example, midway through manufacturing process S200, such as between the cutting / cutting process S203 and the grinding process S204, rather than after manufacturing process S200. In this case, in step S126, the manufacturing process management unit 120 can output (feedback) the generated control information to the upstream manufacturing equipment performing the process preceding steps S102-S126, or, further, the manufacturing process management unit 120 can output (feedforward) the generated control information to the downstream manufacturing equipment performing the process following steps S102-S126. This makes the manufacturing conditions in the downstream processes following steps S102-S126 more efficient. For example, if steps S102-S126 are integrated before the grinding process S204, the grinding amount is adjusted in the grinding process S204 based on the control information generated in step S126.

[0099] Next, an application example of the manufacturing process for the glass substrate will be described. For details regarding the overall manufacturing process, please refer to International Publication No. 2012 / 090766 and Japanese Patent No. 5983406. For details regarding the polishing process, please refer to Japanese Patent No. 4862404 and Japanese Patent No. 4207153.

[0100] Glass substrates are manufactured using methods such as float glass and fusion glass. Figure 13The illustrated glass substrate manufacturing process S200 includes, for example, a melting process S201, a forming process S202, a cutting / splitting process S203, a grinding process S204, a product shedding process S205, and a crushing process S208. The melting mechanism, forming mechanism, cutting / splitting mechanism, grinding mechanism, product shedding mechanism, and crushing mechanism that respectively perform the melting process S201, forming process S202, cutting / splitting process S203, grinding process S204, product shedding process S205, and crushing process S208 constitute a manufacturing equipment (not shown). The defect inspection device 100 involved in this embodiment may also be included in the manufacturing equipment.

[0101] As a melting mechanism, a melting furnace is used, for example. In the melting process S201, the melting furnace melts the glass raw material by heating to form molten glass. As a forming mechanism, a forming device is used, for example. The forming device has a molten tin bath, and in the forming process S202, the molten glass transferred from the melting furnace is spread on the tin in the molten tin bath to form a strip of glass with a specified width into a glass strip. The glass strip can also be placed on the main conveying path of the conveying rollers and conveyed towards the packaging mechanism (not shown) as a glass substrate that becomes a product. During the period until reaching the packaging mechanism, the glass substrate is subjected to a cutting / cutting process S203, a grinding process S204, and a product unloading process S205. As a cutting / cutting mechanism that performs the cutting / cutting process S203, a cutting and folding device is used, for example. The cutting and folding device forms the glass strip flowing in the main conveying path into a glass substrate of a specified size. A cutting line processing device is provided upstream in the conveying direction, and a folding device is provided downstream of the cutting line processing device. In the cutting / cutting process S203, the cutting line processing device is equipped with a cutting tool. By pressing the tip of the cutting tool against the surface of the glass strip with a specified pressure, a cutting line is formed on the glass strip. The folding device cuts the glass strip along the cutting line, dividing it into glass substrates of a specified size.

[0102] The polishing mechanism performing polishing step S204 uses a polishing apparatus for polishing the surface of a segmented glass substrate. The polishing apparatus, for example, includes multiple circular polishing tools that rotate and revolve, continuously polishing the glass substrate (continuous type) while moving in the transport direction of the glass substrate. The polishing apparatus forms a pair of circular polishing tools with a diameter smaller than the width of the glass substrate, with the center line of movement of the glass substrate as a reference, and arranges two pairs in a serrated pattern along this movement direction. The circular polishing tools polish the surface of the glass substrate beyond the center line of movement.

[0103] The polishing apparatus may also have a structure for polishing the surface of a glass substrate in a static state (discontinuous type). The polishing apparatus may include, for example, a substrate bonding stage, a film frame mounting stage, a polishing stage, a film frame removal stage, and a substrate removal stage. The substrate bonding stage is used to bond the glass substrate to the film frame. The film frame mounting stage is used to mount the film frame to the lower part of the carrier. The polishing stage is used to, after the film frame is mounted to the carrier, bring the carrier and the polishing platform closer together, press the polishing surface of the substrate bonded to the film frame against the polishing platform, and polish it. The film frame removal stage is used to remove the film frame from the carrier. The substrate removal stage is used to remove the polished glass substrate from the film frame.

[0104] A dropping device is used as the product dropping mechanism for the product dropping process S205. The dropping device includes a control unit and a rotating mechanism, which controls whether the manufactured glass substrates fall from the main transport path in the product dropping process S205. When the control unit receives a control signal indicating that the product is being returned from the defect inspection device 100 to the manufacturing process, it rotates the rotating mechanism to form the connecting member of the main transport path. As a result, glass substrates deemed defective fall and are returned to the manufacturing process without going through the packaging mechanism. The falling glass substrates break upon impact with an obstacle, becoming broken pieces. On the other hand, when the control unit does not receive a control signal, the main transport path is maintained. Therefore, glass substrates deemed acceptable are transported by the packaging mechanism and become shipment items.

[0105] A crusher is used as the crushing mechanism for performing the crushing process S208. The crusher is equipped with rotating blades that crush the broken material, forming glass shards as raw material in the crushing process S208. The glass shards are conveyed to the melting furnace via a conveyor belt equipped in the manufacturing equipment.

[0106] In the inspection process described in this embodiment, typically, a polished glass substrate produced by the polishing process S204 is used as the object to be inspected. Depending on the type of defect in the object to be inspected, molten glass produced by the melting process S201, glass ribbon produced by the forming process S202, or a glass substrate before polishing produced by the cutting / cutting process S203 may also be used.

[0107] The defect inspection device 100 according to this embodiment can identify the types of defects that may occur in each process and achieve control corresponding to the identified defects. For example, regarding bubbles generated in a glass substrate, there is a tendency for them to occur more frequently in the melting process S201 as the melting temperature of the glass is lower than a predetermined reference temperature. Therefore, the control data used for controlling the manufacturing process is set such that, as a correction condition, the more bubbles are detected, the greater the increase in the temperature of the melting furnace. Thus, for the manufacturing process management unit 120, the more bubbles that can be detected, the greater the increase in the temperature of the melting furnace.

[0108] For foreign matter adhering to the surface of the glass substrate, there is a tendency for more residue to remain if the grinding time is shorter than the reference time in the grinding process S204. Therefore, the control data is configured such that the more foreign matter detected as a defect, the greater the increase in grinding time as a correction condition. Thus, for the manufacturing process management unit 120, the more foreign matter that can be detected, the longer the grinding time will be.

[0109] For scratches on the glass substrate surface, there is a tendency for them to occur due to the deterioration of the abrasive used in the polishing process S204. Therefore, the control data is set in conjunction with a reference quantity for the amount of scratches detected as a defect state and the replacement of the polishing tool as a correction condition. As a result, the manufacturing process management unit 120 can replace the polishing tool when the amount of scratches detected exceeds the reference quantity.

[0110] Regarding scratches, there is a tendency for them to occur due to insufficient pressure applied by the polishing tool to the glass substrate surface during polishing process S204. Therefore, the control data can be configured such that the more scratches are detected as defective conditions, the greater the increase in the pressure applied by the polishing tool as a correction condition. Consequently, the more scratches the manufacturing process management unit 120 can detect, the greater the pressure applied by the polishing tool to the glass substrate surface.

[0111] Furthermore, the manufacturing process management unit 120 can also accept input from the operation unit 150, which represents operation signals indicating modified conditions for manufacturing conditions, and save the information that establishes the association between the modified conditions and the defect status shown by the judgment result information in the control data. Thus, the control data is updated using information that establishes the association between the manufacturing conditions instructed by the user (i.e., the operator) to the manufacturing equipment and the detected defect status. Therefore, the updated control data can be used for control of manufacturing conditions based on the detected defect status.

[0112] (Machine Learning Model)

[0113] Next, examples of machine learning models involved in this implementation will be described. Figure 14 A CNN is shown as an example of a machine learning model involved in this implementation. Figure 14 In the example shown, the input value to the CNN is two-dimensional image data, and a one-dimensional probability (scalar) is calculated as the output value from the CNN. This is a level 2 model.

[0114] CNN is a type of artificial neural network that has one input layer, multiple intermediate layers, and an output layer. Figure 14 The illustrated CNN has an input layer In02, 6 intermediate layers, and an output layer Out16. The 6 intermediate layers include 3 convolutional layers Cv04, Cv08, and Cn12, 2 pooling layers Pl06 and Pl10, and a fully connected layer Fc14. Specifically, after alternating between one convolutional layer and one pooling layer twice, a convolutional layer Cv12 is placed, followed by a fully connected layer Fc14.

[0115] Each layer has one or more nodes (also called nodes, neurons, etc.). Each node outputs a specified function value relative to the input value as its output value.

[0116] The input layer In02 outputs the signal value of each sample point, represented by the measurement signal input as the input value, to the next layer. Each sample point corresponds to one pixel. At each node of the input layer In02, the signal value of the sample point corresponding to that node is input, and the input signal value is output to the corresponding node in the next layer. In the convolutional layer, the number of kernels is pre-defined. The number of kernels refers to the number of kernels used for processing (e.g., computation) the input values. Conventionally, the number of kernels is less than the number of input values. A kernel is a processing unit used to calculate one output value at a time. The output value calculated in a layer is used as the input value for the next layer. Kernels are also called filters. The kernel size represents the number of input values ​​processed in a kernel at one time. Conventionally, the kernel size is an integer greater than or equal to 2.

[0117] Pooling and convolutional layers calculate feature quantities representing their characteristics based on multiple input values. As feature quantities, the output values ​​from any one of the specified convolutional layers Cv04, Cv08, Cv12 and pooling layers Pl06, Pl10 can be used to determine new types of defects.

[0118] A convolutional layer is a layer that performs convolution operations on each input value from the preceding layer to multiple nodes, using each kernel to calculate the convolution value. The output value is then calculated as the value of a predefined activation function with a correction value, and the calculated output value is output to the next layer. The correction value is obtained by adding the calculated convolution value and the bias value. Furthermore, in the convolution operation, one or more input values ​​are input to each node from the preceding layer, and independent convolution coefficients are used for each input value. The convolution coefficients, bias values, and activation function parameters are part of a set of model parameters.

[0119] As activation functions, for example, normalized linear units (Rectified Linear Units) and double-bending functions can be used. A Rectified Linear Unit is a function that outputs values ​​below a specified threshold (e.g., 0) and outputs values ​​above that threshold as is. Therefore, this threshold can be part of a set of model parameters. Furthermore, regarding convolutional layers, whether to reference input values ​​from nodes in the preceding layer and whether to output values ​​to nodes in the next layer can also be part of a set of model parameters. Therefore, unlike fully connected layers described later, the nodes in a convolutional layer are not necessarily combined with all nodes in the preceding layer as input values, nor are they necessarily combined with all nodes in the next layer as output values.

[0120] A pooling layer is a layer that determines a representative value based on input values ​​from multiple nodes in the preceding layer, and outputs this representative value as the output value to the next layer. The representative value can be statistically represented by values ​​such as the maximum, average, or most frequent value. A predefined span is set in the pooling layer. The span represents the range of adjacent nodes in the preceding layer relative to a given input value. Therefore, a pooling layer can be viewed as a layer that shrinks (downsamples) the input values ​​from the preceding layer to a lower dimension and provides the output value to the next layer.

[0121] A fully connected layer works as follows: it performs convolution operations on each input value from the preceding layer to multiple nodes to calculate the convolution value, and then calculates the output value by adding the calculated convolution value and bias values. This output value is then passed to the next layer. In other words, a fully connected layer performs convolution operations on all multiple input values ​​from the preceding layer using a set of parameters (kernels) smaller than the number of input values, and outputs the resulting calculated value. Therefore, in a fully connected layer, the convolution coefficients, bias values, and activation function parameters are part of a set of model parameters. Thus, by placing a fully connected layer immediately before the output layer, it is possible to comprehensively consider the components that significantly influence the characteristic values ​​assigned to the preceding layer while reducing the degrees of freedom to derive the final output value.

[0122] Furthermore, the number of layers, the categories in each layer, and the number of nodes in each layer of a CNN are not limited to these parameters. Figure 14 The example shown. The CNN involved in this embodiment only needs to have the following structure: capable of calculating the probability of each defect category as the output value for a measurement signal with signal values ​​at each of multiple sampling points as input values. However, as... Figure 14 As illustrated, the CNN according to this embodiment preferably includes an intermediate layer consisting of alternating convolutional layers and pooling layers repeated for one or more cycles and stacked sequentially. This is because the repetition of convolutional layers filters out components that significantly influence the feature values. Furthermore, the pooling layer may be omitted in the repetition of the convolutional layers.

[0123] Furthermore, for multi-level models with 3 or more levels, any machine learning model that uses vector values ​​of 3 or more computational elements as output values ​​is sufficient. Taking the determination of defect types as an example, the output value is obtained as the probability of the defect type corresponding to that element. Figure 14 In the example shown, in the fully connected layer Fc14, it is necessary to set a parameter set corresponding to the elements of each output value. Therefore, in a multi-level model, the entire parameter set is optimized based on a prescribed specification during model learning. Therefore, as mentioned above, even if the intention is only to modify the parameter set for the determination of one type of defect, it may affect the determination results of other types of defects. Therefore, the defect identification unit 114 according to this embodiment preferably uses multiple two-level models for determining compliance / non-compliance with each type of defect.

[0124] Furthermore, while the above examples primarily use the signal value of each pixel as the input value to the machine learning model, they are not limited to this. The control unit 110 of the defect inspection device 100 may also include a feature analysis unit (not shown), which calculates feature quantities representing the features of patterns (including defects) appearing in the image based on image data. Moreover, all or part of the defect discriminators 114-1 to 114-N may replace the signal value of each pixel, or use the feature quantities calculated by the feature analysis unit together with the signal value, as the input value to the machine learning model. As input values ​​for the training data calculated to form model parameters for these machine learning models, the model learning unit 118 replaces the signal value of each pixel, or uses the feature quantities calculated by the feature analysis unit together with the signal value, as the input value to the machine learning model. As such a feature quantity, any one or a combination of these can be used, such as shape feature parameters like roundness, Euler number, and Fretter diameter, HOG (Histograms of Oriented Gradients) features, SIFT (Scaled Invariance Feature Transform) features, or other features used for image recognition. In other words, the calculated feature quantity can be used as a feature quantity representing the state of the inspected object's defects. The new type determination unit 122 can also use the feature quantity calculated by the feature analysis unit for determining a new type.

[0125] Furthermore, in the determination data, information representing the state of the inspected object can be replaced by image feature quantities instead of the presence or number of defects for each defect type. Moreover, the defect identification unit 114 or the comprehensive determination unit 116 may also include a defect detector, which determines the type of defect based on the image feature quantities analyzed by the feature analysis unit, referring to the determination data.

[0126] As explained above, the defect inspection apparatus 100 of this embodiment is a defect inspection apparatus that inspects defects generated in an inspected object based on an image of the inspected object. It includes multiple defect discriminators, which use a predetermined machine learning model to identify different types of defects based on the image. The types of defects identified by each defect discriminator are part of a predetermined number of defect types that are the object of the defect inspection apparatus.

[0127] In addition, multiple defect detectors can also determine whether the type of defect generated in the inspected object conforms to a specified type of defect.

[0128] Alternatively, it can be implemented as follows: the object to be inspected is glass, and an inspection process using the aforementioned defect inspection device is performed.

[0129] This structure allows each defect detector to determine whether a defect detected in an image belongs to a predetermined number of defect types. Changing the parameter set used by a specific defect detector for defect type determination does not affect the parameter sets used by other defect detectors, unlike when all predetermined defect types are considered. Furthermore, it prevents degradation in determination accuracy when each defect detector determines whether a single defect conforms to a predetermined category. Therefore, system management becomes easier.

[0130] In addition, multiple defect discriminators can also identify the types of defects generated in the inspected object independently and in parallel, regardless of the judgment results of other defect discriminators.

[0131] This structure allows for parallel judgments of compliance / non-compliance for each type of defect, so even if the number of defect types being identified increases, the processing time does not increase, thus enabling rapid processing.

[0132] Alternatively, the number of defect detectors can be N. The nth defect detector determines whether the type of defect generated in the inspected object conforms to the nth defect type. When the nth defect detector determines that the type of defect generated in the inspected object does not conform to the nth defect type, the (n+1)th defect detector begins to determine whether the type of defect generated in the inspected object conforms to the (n+1)th defect type.

[0133] This structure allows each defect identifier to determine whether a defect conforms to a specified type in a series of steps. This avoids excessive processing volume and thus contributes to cost-effectiveness.

[0134] Alternatively, n can be determined by descending order of the frequency or risk of the nth defect type.

[0135] This structure prioritizes the identification of defect types that occur more frequently or pose a greater risk of occurrence, thus enabling the system as a whole to suppress damage caused by defect occurrence.

[0136] In addition, multiple defect discriminators can also use machine learning models to extract the feature quantities of defects.

[0137] This structure allows for the expression of characteristics corresponding to each type of defect, even without pre-defining specific feature quantities.

[0138] Alternatively, a model learning unit may be included, which determines the model parameters for using a machine learning model to identify the specific type of defect based on the image data representing an image containing a specific type of defect.

[0139] This structure allows the use of training data representing the relationship between image data as input and defect types as output to determine the model parameters for identifying the type of defect. Therefore, by using model parameters appropriate to the usage environment in defect type identification, the accuracy of the determination can be improved.

[0140] In addition, the defect detector can also identify the type of defect based on image data representing multiple images taken under different conditions for the inspected object.

[0141] This structure allows us to use the differences in each shooting condition of the image features as clues to more accurately determine the type of defect.

[0142] Alternatively, a new type determination unit may be provided. This new type determination unit calculates the distance in space of the feature quantity representing the defect characteristics, that is, the distance between the representative feature quantity predetermined according to the type of defect and the feature quantity extracted from the image, that is, the extracted feature quantity. If the calculated distance is greater than a predetermined distance threshold relative to any type of defect, the type of defect detected from the image is determined to be a new type.

[0143] This structure allows defects whose characteristics differ from known defect types to be identified as new types of defects. Therefore, it facilitates process management corresponding to defects with different characteristics.

[0144] Alternatively, a new type determination unit may be provided, which determines the type of defect detected from the above image as a new type when there is no defect discriminator that successfully identifies the type of defect.

[0145] This structure enables the identification of defects of unknown type as new types. Therefore, it facilitates process management that is independent of known defect types.

[0146] In addition, a manufacturing process management department may also be included, which determines the correction conditions for correcting the manufacturing conditions of the inspected object based on the status of defects identified by multiple defect detectors.

[0147] This structure enables efficient control of the manufacturing process of the inspected object, corresponding to the state of the identified defects, without relying on manual intervention.

[0148] Alternatively, it may include: a feature analysis unit that analyzes the feature quantity of a defect based on an image; and a second defect discriminator that uses determination data representing the relationship between the feature quantity of a defect and the type of the defect to determine the type of the defect based on the feature quantity of the defect analyzed by the feature analysis unit.

[0149] This structure determines the defect type corresponding to the known feature values ​​of a detected defect, based on the relationship between the known defect feature values ​​and the defect type. It can determine the defect type completely independently of machine learning models, thus reducing processing load.

[0150] Alternatively, a determination input unit may be provided, which determines the type of defect generated in the inspected body as the type of defect indicated by the obtained operation input.

[0151] This structure allows users to know the types of defects that have been identified. It enables defect identification entirely independently of machine learning models, thus reducing the risk of incorrect identification.

[0152] Alternatively, the defects in the inspected body can be determined by a second defect detector or a judgment input unit before multiple defect detectors determine the type of defects in the inspected body.

[0153] This structure allows for the determination of the relationship between known defect features and defect types, or the use of user judgment. Therefore, even defect types that cannot be determined by machine learning models can be identified.

[0154] Alternatively, the glass manufacturing method described in this embodiment may also be configured such that the object to be inspected is glass, and an inspection process using the aforementioned defect inspection device is performed.

[0155] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the specific structure is not limited to the above structure, and various design changes can be made without departing from the spirit of the present invention.

[0156] For example, the defect inspection device 100 can be implemented as part of the manufacturing equipment of the inspected object, or it can be a separate device independent of the inspected object. The defect inspection device 100 can also acquire image data from other devices such as data storage devices or PCs, and is not limited to manufacturing equipment.

[0157] Alternatively, the defect inspection device 100 may include an imaging unit 130, an operation unit 150, and a display unit 160, or some or all of them may be omitted. The imaging unit 130, the operation unit 150, and the display unit 160 may also be connected via an input / output unit 140.

[0158] In the defect inspection device 100, some or all of the model learning unit 118, manufacturing process management unit 120, new type determination unit 122, and determination input unit 124 may be omitted.

[0159] Depending on the type of the object being inspected, the type and number of defects being detected, one or both of the defect detection unit 112 and the comprehensive judgment unit 116 may be omitted.

[0160] The width, length, thickness, and other dimensions of the glass being inspected are arbitrary. Furthermore, the defect inspection device 100 can also be used to inspect other types of objects besides glass, such as circuit boards and wafers, to determine the presence or absence of defects and the types of defects.

[0161] Alternatively, part or all of the defect inspection device 100 in the above embodiments can be implemented as an integrated circuit such as LSI (Large Scale Integration). Each functional module of the defect inspection device 100 can be independently processorized, or part or all can be integrated into a processor. Furthermore, the method of integrated circuit implementation is not limited to LSI; it can also be implemented using dedicated circuits or general-purpose processors. Additionally, where advancements in semiconductor technology have led to the development of integrated circuit technologies that replace LSI, integrated circuits utilizing this technology can also be used.

[0162] Industrial availability

[0163] Based on the aforementioned defect inspection apparatus, defect inspection method, and manufacturing method, each defect discriminator determines whether the types of defects detected from the image are part of a predetermined number of defect types. When the parameter set used by a specific defect discriminator for determining defect types is changed, unlike the case where all predetermined number of defect types are considered for determination, the parameter sets used by other defect discriminators are not affected, and therefore the determination of the defect type is not affected. Thus, system management becomes easier.

[0164] Explanation of reference numerals in the attached figures

[0165] 100…Defect inspection device; 110…Control unit; 112…Defect detection unit; 114…Defect identification unit; 116…Comprehensive judgment unit; 118…Model learning unit; 120…Manufacturing process management unit; 122…New type judgment unit; 124…Judgment input unit; 130…Photography unit; 140…Input / output unit; 150…Operation unit; 160…Display unit; 170…Storage unit.

Claims

1. A defect inspection apparatus for inspecting defects in an inspected object based on an image of the inspected object, characterized in that, The system has N defect detectors, each using a prescribed machine learning model to identify different types of defects based on the image, where N is a predetermined integer greater than or equal to 2. The types of defects identified by each defect detector are part of a predetermined number of defect types that are the objects of identification by the defect inspection device. The nth defect detector takes as input a weighted sum of the luminance values ​​of each pixel across multiple images of the inspected object under different imaging conditions. It calculates the probability that the defect matches the nth defect type. Based on whether this probability is greater than a pre-set defect judgment threshold, it determines whether the defect type in the inspected object matches the nth defect type, where n is an integer greater than 1 and less than N-1. As the weighting coefficients involved in the weighted sum, the weighting coefficients for images captured under imaging conditions where the nth defect is easily detected are set to larger values ​​than the weighting coefficients for other images. When the nth defect discriminator determines that the type of defect generated in the inspected object does not conform to the nth defect type, the (n+1)th defect discriminator begins to determine whether the type of defect generated in the inspected object conforms to the (n+1)th defect type. The number n is determined in descending order of the frequency of occurrence or the risk of occurrence of the nth type of defect. The more specific the type of defect with a high risk of occurrence, the smaller the defect determination threshold is set to.

2. The defect inspection device according to claim 1, characterized in that, The system includes a model learning unit that, for image data representing images containing a specific type of defect, determines model parameters for using the machine learning model to identify that specific type of defect.

3. The defect inspection device according to claim 1, characterized in that, The system includes a new type determination unit that calculates the distance between a representative feature quantity in the feature quantity space representing defect features, predetermined according to the type of the defect, and the feature quantity extracted from the image, i.e., the extracted feature quantity. If the calculated distance is greater than a predetermined distance threshold for any of the types of defects, the type of defect detected from the image will be determined as a new type.

4. The defect inspection device according to claim 1, characterized in that, The new type determination unit is equipped with a new type determination unit. When there is no defect discriminator that can successfully identify the type of defect, the new type determination unit will determine the type of defect detected in the image as a new type.

5. The defect inspection device according to claim 1, characterized in that, It also includes a manufacturing process management department, which determines correction conditions for correcting the manufacturing conditions of the inspected object based on the status of defects identified by the multiple defect detectors.

6. The defect inspection device according to claim 1, characterized in that, have: The feature analysis department analyzes the feature quantities of defects based on image data; The second defect identifier uses judgment data representing the relationship between the feature quantity of a defect and the type of the defect, and determines the type of the defect based on the feature quantity of the defect analyzed by the feature analysis unit. as well as The determination input unit determines the type of defect generated in the inspected object as indicated by the obtained operation input.

7. The defect inspection device according to claim 6, characterized in that, Before the multiple defect discriminators determine the type of defect generated in the inspected object, the second defect discriminator or the determination input unit determines the defect generated in the inspected object.

8. A defect inspection method, which inspects defects in an inspected object based on an image of the inspected object, characterized in that, The system comprises N defect identification steps, in which a predetermined machine learning model is used to identify the different types of defects based on the image, where N is a predetermined integer greater than or equal to 2. The types of defects identified in each defect identification process are part of a predetermined number of defect types that are the objects of identification in the defect inspection method. In the nth defect identification step, the weighted sum of the luminance values ​​of each pixel among multiple images of the inspected object under different imaging conditions is input. The probability of conforming to the nth type of defect is calculated. Based on whether this probability is greater than a preset defect judgment threshold, it is determined whether the type of defect generated in the inspected object conforms to the nth type of defect, where n is an integer greater than 1 and less than N-1. As the weighting coefficients involved in the weighted sum, the weighting coefficients for images captured under imaging conditions where the nth defect is easily detected are set to larger values ​​than the weighting coefficients for other images. In the nth defect identification process, if it is determined that the type of defect generated in the inspected object does not conform to the nth defect type, the (n+1)th defect identification process begins. The number n is determined in descending order of the frequency of occurrence or the risk of occurrence of the nth type of defect. The more specific the type of defect with a high risk of occurrence, the smaller the defect determination threshold is set to.

9. A method for manufacturing glass, characterized in that, The object being inspected is glass. The defect inspection method described in claim 8 was used.

10. A method for manufacturing glass, characterized in that, The object being inspected is glass. An inspection process using the defect inspection device as described in claim 1 is provided.

Images