Defect inspection device, defect inspection method, and manufacturing method

Abstract

This defect inspection device for inspecting for a defect in an object under inspection on the basis of an image of the object under inspection comprises a plurality of defect identifiers that use prescribed machine learning models to identify different types of defects on the basis of an image. The types of defects identified by the individual defect identifiers are some of a prescribed number of defect types subject to identification by the defect inspection device. This configuration may be embodied as a defect inspection device, defect inspection method, or manufacturing method. This configuration makes it possible to facilitate defect inspection system management.

Description

[Technical Field]

[0001] The present invention relates to a defect inspection apparatus, a defect inspection method, and a manufacturing method, and more particularly to a technique for determining the state of a defect occurring in an object to be inspected, for example, by using image data representing an image of the object to be inspected. This application claims priority from Japanese Patent Application No. 2020-207561, filed on December 15, 2020, the contents of which are incorporated herein by reference. [Background technology]

[0002] When managing product quality, it is important to determine the condition of the product. For example, in the case of a glass substrate whose surface is covered with a thin film of metal or oxide, fine electronic components may be formed on the surface. Such glass substrates can be used for, for example, various displays such as liquid crystal displays, photomasks, electronic device supports, information recording media, and planar antennas. Defects that occur on the surface of the glass substrate can cause defects such as disconnections. Examples of defects include scratches and dirt. Therefore, high cleanliness and flatness are required for the surface of the glass substrate.

[0003] In order to reduce defects such as wire breaks, it is conceivable to analyze the surface condition of the glass substrate, determine the state of the defect, identify the cause of the defect as necessary, and take measures in the manufacturing process. Therefore, attempts have been made to classify the type of defect using a machine learning model for an image of the object to be inspected. For example, Patent Document 1 describes a defect inspection method that uses a deep learning model to classify defects that occur on a wafer, which is a substrate.

[0004] Generally, deep learning models require a large amount of training data to learn model parameters that represent input-output relationships. Depending on the amount of training data used for learning, misclassification may occur frequently. For example, when classifying the type of defect shown in an image, there may be a high probability of misclassification to a specific type. In such cases, users may want to modify the model parameters to avoid misclassification. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-124591 Summary of the Invention [Problem to be solved by the invention]

[0006] However, when modifying model parameters in machine learning models, including deep learning models, it is difficult to correct misclassifications for only a specific type, and the classification results for other types tend to be affected as well. In other words, even if the model parameters are modified to accurately classify a specific type, there is a risk that classifications for other types will be inaccurate. Since the model parameters must be modified taking into account the classifications for all types, a lot of effort and time is required for system management.

[0007] The present invention has been made in view of the above points, and one of the objects of the present invention is to provide a defect inspection apparatus, a defect inspection method, and a manufacturing method that make it easier to manage a system in defect inspection. [Means for solving the problem]

[0008] (1) The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is a defect inspection device that inspects an object to be inspected for defects occurring on the object to be inspected based on an image of the object to be inspected, the defect inspection device includes N (N is a predetermined integer of 2 or more) defect classifiers that use a predetermined machine learning model to classify different types of defects based on the image, the types of defects classified by each defect classifier are a portion of a predetermined number of types of defects that the defect inspection device is to classify, and the nth (n is an integer of 1 to N-1) defect classifier inputs a weighted sum of brightness values ​​for each pixel between a plurality of images of the object to be inspected under different imaging conditions, calculates a probability that the defect corresponds to the nth type of defect, and determines whether the probability corresponds to a predetermined defect judgment method. Threshold whether or not the type of defect occurring in the object to be inspected corresponds to the n-th type of defect, and as a weighting coefficient for the weighted sum, a weighting coefficient for an image under imaging conditions under which the n-th type of defect is easily detected is set to a value greater than a weighting coefficient for other images, and when the n-th defect classifier determines that the type of defect occurring in the object to be inspected does not correspond to the n-th type of defect, an (n+1)-th defect classifier starts a process of determining whether or not the type of defect occurring in the object to be inspected corresponds to the (n+1)-th type of defect, wherein n is the occurrence frequency of the n-th type of defect or occurrence Determined in descending order of risk, The aforementioned Defect determination for types of defects with a high risk of occurrence Threshold This is a defect inspection device in which the value is set as small as possible.

[0009] (2) Another aspect of the present invention is a defect inspection method for inspecting defects occurring in an object to be inspected based on an image of the object, the method comprising: N (N is a predetermined integer of 2 or more) defect discrimination steps for discriminating different types of defects based on the image using a predetermined machine learning model; the defect types discriminated in each defect discrimination step are a part of a predetermined number of defect types to be discriminated in the defect inspection method; and the nth (n is an integer of 1 to N-1) defect discrimination step inputs a weighted sum of brightness values ​​for each pixel among a plurality of images of the object to be inspected under different imaging conditions, calculates a probability that the defect corresponds to the nth type of defect, and determines whether the probability corresponds to a predetermined defect judgment value. Threshold whether the type of defect occurring in the object to be inspected corresponds to the nth type of defect or not, and as a weighting coefficient for the weighted sum, a weighting coefficient for an image under imaging conditions under which the nth type of defect is easily detected is set to a value greater than a weighting coefficient for other images, and when it is determined in the nth defect discrimination step that the type of defect occurring in the object to be inspected does not correspond to the nth type of defect, an n+1th defect discrimination step is started, where n is the occurrence frequency of the nth type of defect or occurrence Determined in descending order of risk, The aforementioned Defect determination for types of defects with a high risk of occurrence Threshold This is a defect inspection method in which the value is set as small as possible.

[0010] (3) Another aspect of the present invention may be a glass manufacturing method, in which the object to be inspected is glass, and the method includes an inspection step using the defect inspection device of (1) or a defect inspection method of (2). [Effects of the Invention]

[0011] According to the present invention, system management in defect inspection can be made easier, for example, the frequency of erroneous determinations for a specific type of defect can be reduced or eliminated without affecting the determination results for other types of defects. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a schematic block diagram showing an example of the configuration of a defect inspection apparatus according to an embodiment of the present invention. [Figure 2] 4 is a flowchart showing a first example of an inspection process according to the present embodiment. [Figure 3] 5A and 5B are explanatory diagrams for explaining imaging conditions in an imaging section according to the present embodiment. [Figure 4] FIG. 10 is a diagram showing an example of an image of an object to be inspected. [Figure 5] FIG. 10 is a diagram showing another example of an image of an object under inspection. [Figure 6] FIG. 2 is a diagram illustrating an example of connection of defect discriminators according to the present embodiment. [Figure 7] FIG. 10 is a diagram showing an example of processing time for defect determination processing according to the present embodiment. [Figure 8] FIG. 10 is a diagram showing an example of an error rate for each defect type by the machine learning model according to the present embodiment. [Figure 9] 10 is a flowchart showing a second example of the inspection process according to the present embodiment. [Figure 10] FIG. 10 is an explanatory diagram for explaining a distance in a feature amount space according to the present embodiment. [Figure 11] 10 is a flowchart showing a third example of the inspection process according to the present embodiment. [Figure 12] FIG. 10 is an explanatory diagram of an example of feedback processing from an inspection process to a manufacturing process according to the present embodiment. [Figure 13] 2 is a flowchart showing an example of a manufacturing process of glass according to the present embodiment. [Figure 14] FIG. 1 is a diagram illustrating an example of a machine learning model according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. First, the configuration of this embodiment will be described. Fig. 1 is a schematic block diagram showing an example of the configuration of a defect inspection device according to this embodiment. The defect inspection apparatus 100 according to this embodiment is an inspection apparatus for inspecting an object to be inspected for defects that may occur in the object to be inspected, using image data representing the image of the object to be inspected. The defect inspection apparatus 100 acquires image data representing the image of the object to be inspected, and executes a plurality of defect discrimination processes, which are processes for discriminating the type of defect that may occur in the object to be inspected, based on the acquired image data, using a predetermined machine learning model. The candidate defect types to be discriminated in each defect discrimination process are a portion of a predetermined number (M types, where M is a predetermined integer equal to or greater than 2) of defect types (hereinafter, discriminable types) to be discriminated by the defect inspection apparatus 100 as a whole. Furthermore, the candidate defect types to be discriminated differ for each defect discrimination process.

[0014] The defect inspection device 100 includes a control unit 110 , an imaging unit 130 , an input / output unit 140 , an operation unit 150 , a display unit 160 , and a storage unit 170 . The control unit 110 executes processes for realizing the functions of the defect inspection apparatus 100 and processes for controlling those functions. The control unit 110 may include general-purpose components such as a processor and be configured as a computer. The processor reads a program previously stored in the storage unit 170 and performs processes instructed by instructions written in the read program to realize the functions. In this application, performing processes instructed by instructions written in a program may be referred to as executing a program, program execution, etc. Part or all of the control unit 110 is not limited to general-purpose hardware such as a processor, and may be configured to include dedicated hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit). Functional units that realize the functions of the control unit 110 will be described later.

[0015] The imaging unit 130 captures images showing various objects present within its surrounding field of view and outputs image data showing the captured images to the control unit 110. The imaging unit 130 may be equipped with a mechanism for varying the imaging conditions for one object under inspection, or may have a mechanism for capturing images under multiple imaging conditions at once. The imaging conditions are conditions that significantly affect the captured image. Depending on the type of defect occurring in the object under inspection, the brightness and shape of the defect image may differ significantly depending on the imaging conditions. The imaging conditions are specified by, for example, brightness, imaging direction, lighting method, etc., or a combination of these.

[0016] FIG. 3 illustrates an example of capturing images of a common object under four different imaging conditions at the same time. The four imaging conditions are classified into two combinations of brightness and imaging direction and two illumination methods: reflection and transmission. The combinations of brightness and imaging direction include bright field and dark field. Bright field is an imaging condition in which light emitted from an illumination source is irradiated onto the object under inspection Sb, and the imaging unit 130 (camera) is positioned and oriented so that the direction of incidence of the reflected light reflected from or transmitted through the object under inspection Sb is included in the field of view. In bright field, the reflected light or transmitted light from the object under inspection Sb is directly incident on the camera, resulting in a bright image. Dark field is an imaging condition in which light emitted from an illumination source is irradiated onto the object under inspection Sb, and the imaging unit 130 is positioned and oriented so that the direction of incidence of the reflected light reflected from or transmitted through the object under inspection Sb is not included in the field of view, but the object under inspection Sb is included in the field of view. In the dark field, the reflected or transmitted light from the object Sb is not directly incident on the camera, but scattered light generated on the surface of the object Sb is incident, so that an image darker than that in the bright field is captured. The imaging unit 130 may be configured to include either a digital still camera that captures still images or a digital video camera that captures moving images. Moving images are configured to include still images obtained by repeatedly capturing images at regular time intervals (for example, 1 / 120 to 1 / 12 seconds).

[0017] 1, the input / output unit 140 is connected to other devices wirelessly or via a wire to input and output various types of data. The input / output unit 140 includes, for example, an input / output interface or a communication interface. The input / output unit 140 is connected to, for example, various control devices, measuring devices, and other devices used in the manufacturing process.

[0018] The operation unit 150 receives a user operation and generates an operation signal corresponding to the received operation. The operation unit 150 may include dedicated components such as buttons, knobs, and dials, or general-purpose components such as a mouse and keyboard. The operation unit 150 may be an input interface that receives an operation signal wirelessly or via a cable from another device. The other device may be, for example, a portable device such as a remote controller or a multi-function mobile phone. The operation unit 150 outputs the acquired operation signal to the control unit 110.

[0019] Display unit 160 displays display information such as images, characters, and symbols based on display data input from control unit 110. Display unit 160 may include, for example, a liquid crystal display, an organic electroluminescence display, or the like.

[0020] In addition to the above programs, the storage unit 170 stores various data used in the processes executed by the control unit 110 and various data acquired by the control unit 110. The storage unit 170 includes, for example, a non-volatile (non-temporary) storage medium such as a ROM (Read Only Memory), a flash memory, or an HDD (Hard Disk Drive). The storage unit 170 also includes a volatile storage medium such as a RAM (Random Access Memory), a register, or the like.

[0021] The control unit 110 includes functional units for realizing its functions, such as a defect detection unit 112, a defect discrimination unit 114, a comprehensive judgment unit 116, a model learning unit 118, a manufacturing process management unit 120, a new species judgment unit 122, and a judgment input unit 124. These functional units may each be configured with a dedicated component, or the function may be achieved by a processor executing a predetermined program. In the following description, the object under inspection is mainly a glass substrate for a flat panel display (FPD), but the object under inspection may be other objects.

[0022] The control unit 110 includes functional units for realizing its functions, such as a defect detection unit 112, a defect discrimination unit 114, a comprehensive judgment unit 116, a model learning unit 118, a manufacturing process management unit 120, a new species judgment unit 122, and a judgment input unit 124. These functional units may each be configured with a dedicated component, or the function may be achieved by a processor executing a predetermined program. In the following description, the object under inspection is mainly a glass substrate for a flat panel display (FPD), but the object under inspection may be other objects.

[0023] 1, the defect detection unit 112 outputs image data indicating an image of a portion including each defect area as a defect image to the defect discrimination unit 114. If no defect area is detected from the image of the object to be detected, the defect detection unit 112 determines that the image of the object to be detected is a non-defective image, and outputs image data indicating the non-defective image to the overall judgment unit 116. In this case, the defect detection unit 112 may also output instruction information to the judgment input unit 124 instructing the display of an input screen. If the size of an area determined as a defective area is less than a predetermined detection threshold in both the horizontal and vertical directions, the defect detection unit 112 may ignore the determination result of the defective area and determine it as a normal area. The detection threshold is set in advance in the defect detection unit 112 to a size that is sufficiently smaller than the size of a typical defect.

[0024] The defect discrimination unit 114 includes a plurality of defect discriminators. In the following description, the number of defect discriminators is assumed to be N (N is a predetermined integer equal to or greater than 2), and the individual defect discriminators may be distinguished by being assigned sub-numbers such as defect discriminator 114-1, 114-2, etc. Similarly, the components of each defect discriminator may also be distinguished by being assigned sub-numbers. Each defect discriminator does not necessarily have to be configured solely by hardware, and its function may be realized by executing a predetermined program.

[0025] Each of the N defect classifiers 114-1 to 114-N performs a process for classifying, for each defect occurring on the object under inspection, the type of defect shown in the image represented by the image data input from the defect detection unit 112 into one or more types of defect candidates. The one or more types of defect candidates that can be distinguished may differ among the N defect classifiers 114-1 to 114-N. These candidates each form a part of the M types of distinguishable types that can be distinguished by the defect detection unit 112 as a whole. Therefore, M is equal to or greater than N. When M and N are equal, the N defect classifiers 114-1 to 114-N function as one-class classifiers that determine whether the type of a defect occurring on the object under inspection is one of different types of defect, or the probability that it is one of those types. When M is greater than N, at least one defect classifier functions as a multi-class classifier that determines whether the type of a defect occurring on the object under inspection is one of multiple types, or the probability that it is each of those types. In the following explanation, the case where M and N are equal will be mainly taken as an example.

[0026] Next, the functional configuration of the defect discriminator 114-1 will be described. Unless otherwise specified, the defect discriminators 114-2 to 114-N have the same functional configuration as the defect discriminator 114-1, and the description of the defect discriminator 114-1 will be used. The defect classifier 114-1 executes a preprocessing step, an inference step, a defect determination step, and a pass / fail determination step (corresponding to steps S112-1, S114-1, S116-1, and S118-1 in FIG. 2). The preprocessing step includes a process for matching the format of the image data input to the defect classifier with the format required for the input to the inference step (e.g., the number of elements of the input value). The preprocessing step includes one or both of dimension reduction and resizing.

[0027] Resizing is a process of changing the number of pixels in the horizontal and vertical directions of the partial area to be processed. Resizing can be either enlargement or reduction. The defect classifier 114-1 interpolates the signal values ​​of each pixel in the input image to determine the signal values ​​of each pixel in the enlarged or reduced image. The defect classifier 114-1 can use known interpolation methods, such as bilinear interpolation and bicubic interpolation. Instead of simply resizing the image by a predetermined magnification, the defect classifier 114-1 may determine a size for each defect area so that the entire defect area is included and the maximum value of its horizontal or vertical diameter is a constant r (e.g., a real number greater than or equal to 0.5 and less than 1) times the horizontal or vertical size of the partial area. The diameter corresponds to the length of a line segment that crosses the defect area in any direction. For example, if the detected defect has an elliptical shape, the maximum diameter of the defect is represented by the length of its major axis. If the detected defect has a rectangular shape, the maximum diameter of the defect is represented by the length of its long side.

[0028] Dimensionality reduction is a process of aggregating multiple images of a single partial region, each captured under different imaging conditions, into a smaller number of images. Dimensionality reduction is achieved by image synthesis. Image synthesis includes a process of calculating, for each pixel, a weighted sum of the luminance values ​​of multiple images as a new signal value. The weighted sum corresponds to the sum of the multiplication values, which are the product of the luminance values ​​of each image and the weighting coefficient corresponding to that image, across the images. The number of images generated by image synthesis needs to be at least one, but may be multiple (e.g., three). For example, a technique called alpha blending may be used in image synthesis. Alpha blending is a technique in which the sum of the weighting coefficients (alpha values) for each image to be synthesized is normalized to 1. The ratio of weighting coefficients between images may differ among defect classifiers 114-1 to 114-N. As a result, a weighting coefficient for an image captured under imaging conditions that make it easy to detect the defect may be set to a larger value than the weighting coefficients for other images, depending on the type of defect for each defect classifier. For example, for scratches, the weighting coefficient for the image captured in the bright field is set relatively small, and the weighting coefficient for the image captured in the dark field is set relatively large, so that scratches can be detected more reliably from the image captured in the dark field.

[0029] When the number of images generated by dimension reduction is three, the defect discriminator 114-1 may use the signal values ​​of each generated image as color signal values ​​of different color tones for each pixel, and generate image data representing a single color image obtained by integrating the images for the three different color tones. Color systems that can be used to represent color images include the RGB color system and the YCrCb color system. An example is shown in which images Im01 to Im04 of four partial regions captured under different imaging conditions are combined into a single color image Im05 by dimension reduction. First, the defect discriminator 114-1 acquires images Im01 and Im02 captured in a bright field and images Im03 and Im04 captured in a dark field. Images Im01 and Im03 are images obtained by capturing transmitted light, and images Im02 and Im04 are images obtained by capturing reflected light. The shape and brightness of defects vary depending on the imaging conditions. Image Im05 is represented by a composite value obtained by combining the signal values ​​of four images in defect classifier 114-1 using different weighting factors for red, green, and blue. For example, the signal value of the red channel of image Im05 is obtained by combining the luminance value of image Im01 with the luminance value of image Im02 at a weighting factor ratio of 0.7:0.3. The signal value of the green channel of image Im05 is obtained by combining the luminance value of image Im03 with the luminance value of image Im04 at a weighting factor ratio of 0.4:0.6. The signal value of the blue channel of image Im05 is obtained by combining the luminance value of image Im02 with the luminance value of image Im03 at a weighting factor ratio of 0.5:0.5.

[0030] The inference process is a process for determining the probability that the type of defect occurring in the object to be inspected will be a predetermined type for image data representing the image to be processed. The defect classifier 114-1 receives the signal value of each pixel constituting the image data as an input value, and calculates the probability as an output value using a predetermined machine learning model for the input value. The output value is a real value between 0 and 1. A parameter set (model parameters) for calculating the output value from the input value is set in advance in the defect classifier 114-1. As machine learning models, neural networks such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs) can be used. Usable machine learning models are not limited to neural networks; other methods such as random forests (RFs) and support vector machines (SVMs) can also be used.

[0031] The defect determination process is a process for determining whether a defect corresponds to a target defect type based on the probability calculated in the inference process. For example, if the calculated probability is greater than a predetermined defect determination threshold, the defect classifier 114-1 determines that the defect corresponds to that type. If the calculated probability is equal to or less than the defect determination threshold, the defect classifier 114-1 determines that the defect does not correspond to that type. The defect determination threshold may be set independently for each target defect type. For example, each defect classifier may be set with a lower defect determination threshold for a defect type with a higher occurrence risk, and a higher defect determination threshold for a defect type with a lower occurrence risk. A high occurrence risk means that there is a high possibility that material or economic losses will occur due to the occurrence of the defect, or that such losses will be significant. In other words, high-risk defects, which are types of defects with a higher occurrence risk, are rejected and are not allowed to be overlooked, while low-risk defects, which are types of defects with a lower occurrence risk, tend to be allowed to be overdetected. For example, the risks associated with scratches, bubbles, foreign matter, and dirt decrease in that order, with scratches being the highest risk. Therefore, by decreasing the defect determination threshold for higher-risk defects, the recall rate (recall) at which defects can be detected relatively without omission can be increased. By increasing the defect determination threshold for lower-risk defects, which are types of defects with a low risk of occurrence, the precision rate (precision) at which defects can be detected relatively reliably can be increased. The defect classifier 114-1 generates a defect flag for the target area to be judged, indicating whether or not the area is a defect, and stores the generated defect flag in association with the target area in the storage unit 170. In this application, a state in which a specific object is attached or mixed in as a type of defect, such as the above-mentioned "foreign matter" and "dirt," may be referred to using the name of the object.

[0032] The pass / fail judgment process is a process for judging whether a detected defect area is pass / fail based on a predetermined judgment criterion value when the defect judgment process judges that the detected defect corresponds to a predetermined defect type. If the defect judgment process judges that the defect does not correspond to a defect, the pass / fail judgment process is not executed. For example, if the size of the detected defect is larger than a judgment criterion value indicated in judgment data previously set in the defect discriminator 114-1, the defect discriminator 114-1 judges the area as defective, and if the size of the detected defect is equal to or smaller than the judgment criterion value, the defect discriminator 114-1 judges the area as pass / fail. Generally, individual defects are represented by figures on a two-dimensional plane, and the defect discriminator 114-1 determines the maximum diameter of the detected defects as the defect size. The judgment criterion value may be set independently for each type of defect to be judged. For example, the judgment criterion value for scratches is 100 to 300 μm, and the judgment criterion value for foreign matter is 50 to 120 μm. The defect discriminator 114-1 generates a defect flag indicating whether the defect is a pass or fail product for the defect and its type to be discriminated, and stores the defect flag in the storage unit 170 in association with the target region.

[0033] The overall judgment unit 116 judges the overall condition of the object under inspection by referring to the pass / fail status of each detected defect. For example, the overall judgment unit 116 refers to all pass / fail judgments indicated by the pass / fail flags for each defect in the object under inspection, regardless of the type of defect, and counts the number of defects judged to be defective. If the number of all types of defects is greater than a predetermined judgment criterion number, the overall judgment unit 116 judges the object under inspection to be defective, and if the number of all types of defects is equal to or less than the predetermined judgment criterion number, the overall judgment unit 116 judges the object under inspection to be good. Because the defect discrimination unit 114 performs pass / fail judgment processes for different defect types in parallel, a single defect may be associated with defective product flags indicating a defective product for multiple defect types. The overall judgment unit 116 may treat a defect associated with multiple defective product flags as corresponding to those multiple types. For example, when three defects are detected in an inspected object, the overall judgment unit 116 may determine that defect 001 corresponds to both a scratch and a foreign substance, defect 002 corresponds to dirt, and defect 003 corresponds to a foreign substance and dirt. The defect discrimination unit 114 may determine the number of defects to be three, which is the number of defects that actually occurred, or may determine the number of defects to be five, which is the sum of the number of defects for each type.

[0034] The overall judgment unit 116 may be configured with, for example, preset judgment data indicating the judgment reference number for the number of defects for each defect type, and may use the judgment data to judge the state of the object under inspection (rule-based). The overall judgment unit 116 counts the number of defects for each defect type by referring to the pass / fail judgment indicated in the pass / fail flag for each defect. If there is a defect type for which the counted number of defects is equal to or greater than the judgment reference number indicated in the judgment data, the overall judgment unit 116 judges the object under inspection as defective. If there is no such defect type, the overall judgment unit 116 judges the object under inspection as pass / fail. The judgment reference number for a defect type with a high risk of occurrence may be smaller, and the judgment reference number for a defect type with a low risk of occurrence may be larger. For example, if the defect types are scratches and dust, the judgment reference numbers may be one and five, respectively. FIG. 5 illustrates an example in which one scratch and two dust particles are detected on the object under inspection. If attention is focused only on dust, a specific type of defect, and other types of defects are ignored, the number of detected dust particles would be two, which is less than the criterion number, and the object under inspection would be judged as a good product. However, since the number of scratches is equal to the criterion number of one, the comprehensive judgment unit 116 judges the object under inspection as a defective product. Therefore, by comparing the number of each type of defect with the criterion number, it is possible to avoid the risk of an object under inspection with unacceptable defects being judged as a good product.

[0035] A weighting coefficient for each defect type may be preset in the overall judgment unit 116, and the effective number may be calculated as a weighted sum, which is the sum of the product values ​​of the weighting coefficient for each defect type and the number of defects for each defect type. The overall judgment unit 116 may judge the object to be inspected as defective if the effective number is greater than a predetermined judgment reference number, and may judge the object to be inspected as non-defective if the effective number is equal to or less than the judgment reference number. When image data showing a non-defective product image is input from the defect detection unit 112 to the comprehensive judgment unit 116, the object to be inspected may be judged as a non-defective product. The overall judgment unit 116 may output instruction information indicating the display of an input screen to the judgment input unit 124 when the pass / fail judgment for all defects and types indicated in the pass / fail flag for each defect in the inspected object is judged to be pass, regardless of the type of defect.

[0036] An inspected object determined to be a non-defective product is to be shipped, while an inspected object determined to be a defective product is to be discarded or returned to the manufacturing process. The overall judgment unit 116 outputs judgment result information indicating the judgment result to the manufacturing process management unit 120. The manufacturing process management unit 120 refers to the judgment result information input from the overall judgment unit 116 and discards or returns to the manufacturing process any inspected object determined to be a defective product. The manufacturing process management unit 120 outputs, for example, a control signal to the manufacturing equipment indicating return to the manufacturing process or discard. The comprehensive judgment unit 116 may include information on the number of defects for each type of defect or the number of all defects in the judgment result information.

[0037] The model learning unit 118 calculates, as model parameters, a set of parameters of a machine learning model used by the defect discrimination unit 114 to discriminate the type of defect. The model learning unit 118 calculates model parameters by performing a learning process on training data (also referred to as training data, learning data, supervised data, etc.) using a predetermined machine learning model for each type of defect to be discriminated. Before performing the learning process, training data including multiple sets (typically 1,000 to 10,000 or more) of data sets, each of which is a pair of known input values ​​and an output value corresponding to the input value, is set in the model learning unit 118. Image data including signal values ​​for each pixel is used as input values. As an output value for each set included in the training data, 1 is assigned if the image represented by the image data used as input values ​​represents an image of the type of defect to be discriminated, and 0 is assigned otherwise. In the learning process, the model learning unit 118 can use, for example, training data configured by adding (annotating) the type of defect and its output value to each image data used as input values.

[0038] In the learning process, the model learning unit 118 updates the model parameters until the magnitude of the difference between the output value and the calculated value calculated for the input values ​​using a predetermined machine learning model for the entire set of input values ​​converges to approximately zero. When the amount of change in the model parameters before and after the update, or the amount of change in the magnitude of the difference between before and after the update, is less than a predetermined convergence determination threshold, it can be determined that the model parameters have converged. In updating the model parameters, methods such as steepest descent, stochastic gradient descent, conjugate gradient method, and back propagation can be used. As an index value of the magnitude of the difference, for example, an error function such as the Sum of Squared Differences (SSD) or the cross entropy error can be used. The model learning unit 118 sets the model parameters calculated for each defect type in the defect classifier related to the classification of the defect type. Note that for a defect type input from the judgment input unit 124, the model learning unit 118 may set the output value for that defect type to 1 and add a data set having the image data to be judged as an input value to the training data for that defect type. Alternatively, the model learning unit 118 may set the output value for other types to 0 and add a data set having the image data to be judged as an input value to the training data for that defect type. Then, the model learning unit 118 may update the model parameters for each defect type using the newly added training data (transfer learning).

[0039] The manufacturing process control unit 120 controls the manufacturing process of a product based on the defect status of the product being inspected, as determined by the defect determination unit 114. The manufacturing process control unit 120, for example, stores control data that includes information on the defect status and repair conditions and indicates their correspondence. Repair conditions are conditions for modifying the manufacturing conditions currently used in the manufacturing process to provide the modified manufacturing conditions. The manufacturing process control unit 120 can use the number of each type of defect in the product being inspected as an example of information on the defect status. The number of each type of defect is transmitted by the determination result information input from the comprehensive determination unit 116. The manufacturing conditions may include operating parameters for executing the product manufacturing process and environmental parameters that indicate the environment. Operating parameters may include the rotational speed and power consumption of the power source constituting the manufacturing equipment. Environmental parameters may include temperature, pressure, etc. The repair conditions may be expressed as the amount of change in any type of parameter required to provide the modified manufacturing conditions. The manufacturing process control unit 120 uses the control data to determine information on repair conditions corresponding to the referenced defect status. The manufacturing process management unit 120 generates control information that instructs changes to the manufacturing conditions under the determined modified conditions, and outputs the generated control information to the manufacturing equipment. The manufacturing equipment modifies the manufacturing conditions using the modified conditions indicated in the manufacturing information input from the manufacturing process management unit 120, and executes the manufacturing process under the modified manufacturing conditions.

[0040] The new type determination unit 122 determines whether the type of defect detected in the object to be inspected is a new type that is different from any of the known defect types. For example, when the new type determination unit 122 determines that the type of defect detected by the defect detection unit 112 is different from any of the defect types determined by the defect determination unit 114, the new type determination unit 122 determines that the type of defect is a new type. For example, the new type determination unit 122 can determine that the type of defect is a new type for a defect whose defect flag indicates "Non-applicable" for each of the defect types corresponding to the defect classifiers 114-1 to 114-N. When the new type determination unit 122 determines that the type of defect is a new type, the new type determination unit 122 may cause the display unit 160 to display a notification screen indicating that the type of defect is a new type. In this case, the new type determination unit 122 may output instruction information to the determination input unit 124 instructing the determination input unit 124 to display an input screen. This makes it possible to notify the user, i.e., an operator, that the type of defect is a new type and prompts the operator to input the defect type and a pass / fail determination.

[0041] The judgment input unit 124 receives an operation signal from the operation unit 150 indicating the type of defect occurring in the object to be inspected shown in the image data or the pass / fail of the object to be inspected. The judgment input unit 124 may, for example, generate an input screen including an image of the object to be inspected shown in the image data and screen components that enable the type of defect, the pass / fail of the object to be inspected, or both, to be indicated by pressing, and display the generated input screen on the display unit 160. Pressing refers not only to an actual pressing, but also to an operation signal indicating a position included in the display area being input from the operation unit 150 or another device in response to an operation. Examples of screen components that can be used include buttons, check boxes, menu bars, and the like. When instruction information for displaying an input screen is input from the defect detection unit 112, the overall judgment unit 116, or the new type judgment unit 122, the input screen may be displayed on the display unit 160. This prompts the user to input a type or a pass / fail judgment. The judgment input unit 124 outputs the input defect type and information on the judgment of the pass / fail of the object to the overall judgment unit 116. The judgment input unit 124 may also output the input defect type to the model learning unit 118.

[0042] (Inspection processing) Next, an example of the inspection process according to this embodiment will be described with reference to a flowchart shown in FIG. (Step S102) The imaging unit 130 captures an image of an FPD glass substrate as an example of an object to be inspected, and then proceeds to the processing of step S104. (Step S104) The defect detection section 112 detects a defect area, which is a portion where a defect occurs, from the image of the object to be detected captured by the imaging section 130. Then, the process proceeds to step S110.

[0043] The process of step S110 includes the processes of steps S110-1 to S110-N. The processes of steps S110-1 to S110-N are executed in parallel for each common partial area including each defect area detected from the captured image by defect classifiers 114-1 to 114-N, respectively. The processes of steps S110-2 to S110-N are similar to the process of step S110-1, and therefore the description thereof is incorporated herein. (Step S112-1) The defect classifier 114-1 performs a preprocessing process on the image within the partial region. The defect classifier 114-1 performs dimensionality reduction on the images captured under different imaging conditions and resizing the number of elements of the image after dimensional reduction to the number of elements of the input value to the image inference process. For example, if the number of elements of the image to be processed is 200, 200, and 4 in the horizontal direction, the number of pixels in the vertical direction, and the number of dimensions (number of frames), the number of elements in the horizontal direction, the number of elements in the vertical direction, and the number of dimensions of the image after preprocessing can be 224, 224, and 3, respectively. The defect classifier 114-1 may reconstruct the image into a two-dimensional color image with a horizontal number of 224 and a vertical number of 224 pixels. Then, the process proceeds to step S114-1.

[0044] (Step S114-1) The defect classifier 114-1 performs inference processing on the signal value of each pixel representing the preprocessed image. The defect classifier 114-1 uses the signal value of each pixel as an input value and calculates the probability that the type of defect represented in the partial region as an output value corresponds to the predetermined defect type using a predetermined machine learning model. Then, the process proceeds to step S116-1. (Step S116-1) The defect discriminator 114-1 executes a defect determination process. The defect discriminator 114-1 determines whether the type of defect corresponds to a predetermined defect type set for the device, depending on whether the calculated probability is greater than a predetermined defect determination threshold set for the device. Then, the process proceeds to step S118-1. (Step S118-1) The defect discriminator 114-1 executes a pass / fail judgment process. The defect discriminator 114-1 judges pass / fail based on, for example, whether the size of the detected defect is larger than a judgment reference value set in the discriminator. Then, the process proceeds to step S122.

[0045] (Step S122) The overall judgment unit 116 judges whether the inspected object is good or defective as a whole, by referring to the pass / fail status of each type of defect detected. The overall judgment unit 116 judges whether the inspected object is good or defective, for example, based on whether the number of defects judged to be defective is greater than a predetermined judgment reference number. Thereafter, the manufacturing process management unit 120 adopts the inspected object judged to be good as a product to be shipped, and discards or returns to the manufacturing process the inspected object judged to be defective. Thereafter, the processing of FIG. 2 ends.

[0046] As described above, the defect inspection apparatus 100 according to this embodiment includes defect classifiers 114-1 to 114-N, each of which determines whether a detected defect corresponds to a specific defect type. Therefore, modifying model parameters used to determine one type of defect does not affect the determination of other types of defects. Furthermore, imaging conditions, imaging units 130, or preprocessing appropriate for each defect type can be used. The N defect classifiers 114-1 to 114-N are arranged in parallel. That is, each defect classifier determines whether a detected defect corresponds to a specific defect type, regardless of the determination results of the other defect classifiers. Therefore, the processing time remains constant even if the number of defect types to be determined increases. Furthermore, each defect classifier selects information about that type of defect, such as a feature value specific to that type, regardless of the defect pass / fail criteria. However, the more types of defects to be determined, the more computational resources are required.

[0047] In the defect inspection apparatus 100, the defect classifiers 114-1 to 114-N may be arranged in series. More specifically, when a defect classifier 114-n (n is an integer between 1 and N-1) determines that the type of defect detected in an image within a partial region does not correspond to a predetermined defect type for that device, the defect classifier 114-n+1 starts a process of determining whether the type of the detected defect corresponds to a predetermined defect type for that device. When the defect classifier 114-n determines that the type of defect detected in an image within a partial region corresponds to a predetermined defect type for that device, the defect classifier 114-n establishes the detected defect type as the determined type, and does not perform the processes after the defect classifier 114-n+1. In the example shown in FIG. 6, in step S110-1, the defect classifier 114-1 determines whether the type of the detected defect is a scratch, and if it determines that the detected defect corresponds to a scratch, the process of FIG. 6 ends. If it is determined that the defect is not a scratch, the defect discriminator 114-1 proceeds to the process of step S110-2. In step S110-2, the defect discriminator 114-2 determines whether the type of the detected defect is a bubble, and if it is determined that the defect is a bubble, the process of Fig. 6 ends. If it is determined that the defect is not a bubble, the defect discriminator 114-2 proceeds to the subsequent process.

[0048] Therefore, by connecting the defect classifiers 114-1 to 114-N in series, the number of defect classifiers performing the defect discrimination process at one time is limited to one, thereby enabling efficient use of limited computational resources. Furthermore, learning of model parameters used by subsequent defect classifiers can be performed independently of learning of model parameters used by preceding defect classifiers, eliminating the problem of increased learning time. However, the timing at which subsequent defect classifiers begin performing the defect discrimination process is after the preceding defect classifier has completed its process. Therefore, as illustrated in FIG. 7 , the processing time tends to be longer as the number of defect types to be discriminated increases across the entire defect inspection apparatus 100. In FIG. 7 , the vertical and horizontal axes represent the processing time and the number of models, respectively. The number of models corresponds to the types of defects to be detected, i.e., the number of defect classifiers. The processing time illustrated in FIG. 7 includes a component proportional to the number of models and a constant component (approximately 0.043 seconds, see the dashed line) regardless of the number of models. The former corresponds to the time required to determine whether or not a defect corresponds to an individual defect type. The latter corresponds to the time spent on photography, preprocessing, etc., regardless of the defect type. However, when the number of defect classifiers is 10 or less, the processing time is almost the same as when using a conventional multi-class model. In terms of accuracy, when the machine learning model adopted by the defect classifier is a two-class model, compared to a multi-class model, the results are comparable or the two-class model has higher accuracy. A two-class model is a machine learning model used to determine whether a defect type corresponds to one of a predetermined type. A multi-class model is a machine learning model used to determine which of multiple predetermined types a defect type corresponds to. In the example shown in Figure 8, the error rates for the multi-class model and the two-class model are 4.0% and 2.0%, respectively, meaning the two-class model has higher accuracy.

[0049] The first to Nth types of defects to be discriminated by the serially connected defect discriminators 114-1 to 114-N may be arranged in descending order of their occurrence frequency. Since the more frequently occurring types of defects are detected earlier, the processing time does not become excessive. Furthermore, the first to Nth types of defects to be discriminated by the defect discriminators 114-1 to 114-N may be arranged in descending order of their risk. Since the higher the risk of a defect type is detected earlier, damage due to delays in defect detection can be reduced.

[0050] Next, another example configuration of the new species determination unit 122 will be described with reference to Fig. 9. The new species determination unit 122 executes a new species determination step (step S122-n). The new species determination step includes a preprocessing step (step S122-a), an inference step (step S122-b), a new species degree calculation step (step S122-c), and a new species degree determination step (step S122-d). However, although step S110-3 is not shown in Fig. 9, this does not necessarily mean that the processing of step S110-3 is omitted.

[0051] In step S122-a, the new species determination unit 122 performs a pre-processing step similar to that in step S112-1 or step S112-2. In step S122-b, the new species determination unit 122 performs inference on the partial image input thereto using an N-class machine learning model and extracts feature quantities calculated in the inference. The N-class machine learning model can be, for example, an N-class neural network (e.g., CNN) that calculates the probability of each of N defect types as an output value. For each set, the new species determination unit 122 sets model parameters learned using training data in which, for each defect type shown in the image represented by the image data used as input values, 1 is assigned as the output value of the dimension corresponding to that type, and 0 is assigned as the output value of the dimension corresponding to the other defect types. The new species determination unit 122 can then acquire, as feature quantities indicating the characteristics of the image, calculated values ​​output from predetermined intermediate layers, excluding the input and output layers, of the neural network constituting the new species determination unit. Using the set model parameters, feature quantities indicating the characteristics of the image (defect) are calculated for each image whose defect type is known using a method described below, and a representative value (e.g., center of gravity) of the feature quantities for each defect type is preset in the new species determination unit 122.

[0052] In step S122-c, the new type determination unit 122 calculates the distance between the representative value of the feature amount determined for each defect type and the acquired feature amount, and determines the smallest value of the distances calculated for each type as the new type. In Fig. 10, a two-dimensional vector indicating the feature amount for a detected defect (detected defect) is represented by a star in a feature amount space spanned by feature amounts X and Y as elements. Then, of the distances between the representative value (marked with an x) of each of the defect types, dirt, scratches, bubbles, and dust, and the calculated feature amount, the distance between the representative value of bubbles and the feature amount is calculated as the new type.

[0053] 9, in step S122-d, the new type determination unit 122 determines that the defect type is new if the calculated new type likelihood is greater than a predetermined distance threshold, and determines that the defect type is not new if the calculated new type likelihood is equal to or less than the predetermined distance threshold. The new type determination unit 122 sets a distance threshold value that is sufficiently greater than the variance of any of the feature quantities. As an index value indicating the distance, for example, Mahalanobis' distance, Minkowski's distance, Manhattan distance, cosine similarity, etc. can be used. When the new type determination unit 122 determines that the type of defect is a new type, it may output new type detection information indicating that the type of the defect being determined is a new type to the overall determination unit 116. When the new type detection information is input from the new type determination unit 122, the overall determination unit 116 may reject the pass / fail flag for the defect indicated by the new type detection information. This prevents an existing determination result from being erroneously adopted.

[0054] The new type determination unit 122 may determine a feature quantity of an image indicating a defect determined to be a new type as a representative value of the new type. If a feature quantity calculated from separate image data is within a predetermined distance from the representative value of the new type, the new type determination unit 122 may determine that the type of defect indicated in the image data is a new type. The model learning unit 118 may then perform a learning process using training data including a plurality of sets in which image data indicating an image of the new type of defect is given as an input value and an output value of 1, and a plurality of sets in which image data indicating images of other types of defects is given as an input value and an output value of 0, thereby determining model parameters for discriminating the new type of defect from the image data. The defect discrimination unit 114 may configure a new defect discriminator for discriminating the new type of defect using the determined model parameters.

[0055] Next, an example of manufacturing process control by the manufacturing process control unit 120 will be described with reference to Figures 11 and 12. The processing illustrated in Figures 11 and 12 includes steps S102 to S126. The above description applies to steps S102 to S122. However, in step S122, the comprehensive judgment unit 116 outputs judgment result information indicating the number of defects of each type to the manufacturing process control unit 120 as an example of the state of defects occurring in the product to be inspected. The processing of steps S102 to S122 is included in a series of inspection processes, and the defect status of each product to be inspected or the time series thereof is obtained as a quality trend. In step S124, the manufacturing process management unit 120 determines a repair condition (described later) corresponding to the state of the defect indicated in the determination result information, by referring to control data previously set in the manufacturing process management unit 120. In step S126, the manufacturing process control unit 120 generates control information that instructs changes to the manufacturing conditions under the determined modified conditions, and outputs the generated control information to the manufacturing equipment (feedback). The manufacturing equipment modifies the manufacturing conditions using the modified conditions indicated in the manufacturing information input from the manufacturing process control unit 120. The modified manufacturing conditions for each product, or a time series of these, are obtained as a control trend. In the manufacturing process, the manufacturing equipment executes the manufacturing process (step S200) using the modified manufacturing conditions. This embodiment can be realized as a manufacturing method including inspection processes of steps S102-S122.

[0056] Furthermore, the processing of steps S102-S126 may be incorporated into the manufacturing process S200, for example, between the notching / cutting step S203 and the polishing step S204, rather than after the manufacturing process S200. In this case, in step S126, the manufacturing process management unit 120 may output the generated control information to upstream manufacturing equipment that executes processes prior to steps S102-S126 (feedback). Alternatively, or in addition, the manufacturing process management unit 120 may output the generated control information to downstream manufacturing equipment that executes processes subsequent to steps S102-S126 (feedforward). This improves the efficiency of the manufacturing conditions in the downstream processes subsequent to steps S102-S126. For example, if steps S102-S126 are incorporated before the polishing step S204, the polishing amount in the polishing step S204 is adjusted based on the control information generated in step S126.

[0057] Next, an example of application to a manufacturing process of a glass substrate will be described. The entire manufacturing process is described in more detail in, for example, WO 2012 / 090766 and Japanese Patent No. 5983406, and the polishing process is described in more detail in, for example, Japanese Patent No. 4862404 and Japanese Patent No. 4207153. Glass substrates are manufactured using a manufacturing method such as a float method or a fusion method. The glass substrate manufacturing process S200 illustrated in FIG. 13 includes, for example, a melting step S201, a shaping step S202, a notching / cutting step S203, a polishing step S204, a product part dropping step S205, and a crushing step S208. The melting means, the shaping means, the notching / cutting means, the polishing means, the product part dropping means, and the crushing means that respectively perform the melting step S201, the shaping step S202, the notching / cutting step S203, the polishing step S204, the product part dropping step S205, and the crushing step S208 constitute a manufacturing facility (not shown). The defect inspection device 100 according to this embodiment may also be included in the manufacturing facility.

[0058] As the melting means, for example, a melting furnace is used. In the melting step S201, the melting furnace heats and melts glass raw materials to form molten glass. As the forming means, for example, a forming device is used. The forming device is equipped with a molten tin bath, and in the forming step S202, molten glass transferred from the melting furnace is spread on tin in the molten tin bath, and a ribbon-shaped glass having a predetermined width is formed into a glass ribbon. The glass ribbon may be placed on a main conveying path of conveying rollers and conveyed toward a packaging means (not shown) as a glass substrate to be manufactured. Before reaching the packaging means, the glass substrate is subjected to a slitting / cutting step S203, a polishing step S204, and a product part dropping step S205. As the slitting / cutting means for performing the slitting / cutting step S203, for example, a cutting / folding device is used. The cutting / folding device forms the glass ribbon flowing in the main conveying path into a glass substrate of a predetermined size, and is equipped with a score line processing device upstream in the conveying direction and a folding device downstream of the score line processing device. In the scoring / cutting step S203, the scoring device has a cutter and forms score lines in the glass ribbon by pressing the tip of the cutter against the surface of the glass ribbon with a predetermined pressure. The folding device divides the glass ribbon along the score lines into glass substrates of a predetermined size.

[0059] As the polishing means for performing the polishing step S204, a polishing apparatus that polishes the surfaces of the divided glass substrates is used. The polishing apparatus, for example, is equipped with a plurality of rotating and revolving circular polishing tools, and continuously polishes the glass substrates while moving in the conveying direction of the glass substrates (continuous type). The polishing apparatus has two pairs of circular polishing tools, each with a diameter smaller than the width of the glass substrate, arranged in a staggered pattern along the movement direction, with the circular polishing tools paired based on the center line of movement of the glass substrate, and polishes the surfaces of the glass substrates so that the circular polishing tools extend beyond the center line of movement.

[0060] The polishing apparatus may be configured to polish the surface of a glass substrate while stationary (discontinuous type). The polishing apparatus may comprise, for example, a substrate adhering stage, a film frame mounting stage, a polishing stage, a film frame removal stage, and a substrate removal stage. The substrate adhering stage adheres the glass substrate to the film frame. The film frame mounting stage attaches the film frame to the lower part of the carrier. After attaching the film frame to the carrier, the polishing stage brings the carrier and polishing platen closer to each other and presses the polished surface of the substrate attached to the film body against the polishing platen to polish it. The film frame removal stage removes the film frame from the carrier. The substrate removal stage removes the polished glass substrate from the film frame.

[0061] A dropping device is used as a product dropping means for performing the product dropping step S205. The dropping device includes a control unit and a rotation mechanism, and controls whether or not the manufactured glass substrate needs to be dropped from the main transport path in the product dropping step S205. When the control unit receives a control signal from the defect inspection device 100 indicating a return to the manufacturing process, the control unit causes the rotation mechanism to rotate the connecting member that forms the main transport path. As a result, the glass substrate determined to be defective falls and is returned to the manufacturing process via the packaging means without being shipped. The dropped glass substrate breaks upon colliding with an obstacle and becomes a broken piece. On the other hand, when the control unit does not receive a control signal, the main transport path is maintained. Therefore, the glass substrate determined to be non-defective is transported to the packaging means and becomes a target for shipping. A crusher is used as the crushing means for performing the crushing step S208. The crusher has rotary blades that crush the broken material, and in the crushing step S208, cullet is formed as a glass raw material. The cullet is transported by a belt conveyor provided in the manufacturing facility and supplied to a melting furnace.

[0062] In the inspection process according to this embodiment, typically, the glass substrate after polishing produced in the polishing process S204 is used as the object to be inspected. Depending on the type of defect to be inspected, the molten glass produced in the melting process S201, the glass ribbon produced in the forming process S202, or the glass substrate before polishing produced in the notching and cutting process S203 may be used.

[0063] The defect inspection device 100 according to this embodiment can distinguish between types of defects that may occur in each process and implement control in accordance with the distinguished defects. For example, bubbles tend to form in a glass substrate when the temperature of the molten glass in the melting process S201 is lower than a predetermined reference temperature. Therefore, the control data used to control the manufacturing process is set so that the greater the amount of bubbles detected as a defect state, the greater the amount of increase in the melting furnace temperature as a correction condition. This allows the manufacturing process management unit 120 to increase the temperature of the melting furnace as the number of detected bubbles increases.

[0064] Foreign matter adhering to the glass substrate surface tends to remain longer the shorter the polishing time in the polishing step S204 is compared to the reference time. Therefore, the control data is set so that the greater the amount of foreign matter detected as a defect state, the greater the increase in polishing time as a correction condition. This allows the manufacturing process management unit 120 to extend the polishing time as the amount of detected foreign matter increases. Scratches on the surface of the glass substrate tend to occur due to deterioration of the abrasive used in the polishing step S204. Therefore, the control data is set in association with a reference amount of scratches as a detected defect state and replacement of the polishing tool as a correction condition. This allows the manufacturing process management unit 120 to replace the polishing tool when the amount of detected scratches exceeds the reference amount. Scratches also tend to occur due to insufficient pressure applied to the glass substrate surface by the polishing tool used in the polishing step S204. Therefore, the control data may be set so that the greater the amount of scratches as a detected defect state, the greater the increase in the pressure applied to the polishing tool as a repair condition. This allows the manufacturing process management unit 120 to increase the pressure applied to the glass substrate surface by the polishing tool as the number of detected scratches increases.

[0065] The manufacturing process management unit 120 may receive input from the operation unit 150 of an operation signal indicating a modification condition for the manufacturing conditions, and store information correlating the modification condition with the defect status indicated in the judgment result information, including the information in the control data. As a result, the control data is updated with information correlating the manufacturing conditions instructed to the manufacturing equipment by the user (operator) with the detected defect status. Therefore, the updated control data can be used to control the manufacturing conditions based on the detected defect status.

[0066] (machine learning model) Next, an example of a machine learning model according to this embodiment will be described. Fig. 14 shows a CNN as an example of a machine learning model according to this embodiment. The example shown in Fig. 14 is a two-class model in which an input value to the CNN is two-dimensional image data, and a one-dimensional probability (scalar) is calculated as an output value from the CNN. CNN is a type of artificial neural network and includes one input layer, multiple hidden layers, and an output layer. The CNN illustrated in FIG. 14 includes an input layer In02, six hidden layers, and an output layer Out16. The six hidden layers include three convolutional layers Cv04, Cv08, and Cn12, two pooling layers Pl06 and Pl10, and a fully connected layer Fc14. However, after one convolutional layer and one pooling layer are alternately repeated twice, one convolutional layer Cv12 is placed afterwards, and another fully connected layer Fc14 is placed afterwards. Each layer has one or more nodes (also called nodes or neurons). Each node outputs a function value of a predetermined function for an input value.

[0067] The input layer In02 outputs signal values ​​for each sample point indicated by the measurement signal input as an input value to the next layer. Each sample point corresponds to one pixel. The signal value of the sample point corresponding to each node of the input layer In02 is input, and the input signal value is output to the corresponding node of the next layer. The number of kernels is set in advance in the convolution layer. The number of kernels corresponds to the number of kernels used to process (e.g., calculate) each input value. The number of kernels is usually smaller than the number of input values. A kernel refers to a processing unit for calculating one output value at a time. The output value calculated in a layer is used as an input value for the next layer. A kernel is also called a filter. The kernel size indicates the number of input values ​​used in one processing in the kernel. The kernel size is usually an integer greater than or equal to 2.

[0068] The pooling layer and convolutional layer calculate features that indicate the characteristics of multiple input values. The output values ​​from any specified layer among the convolutional layers Cv04, Cv08, and Cv12 and the pooling layers Pl06 and Pl10 can be used as features to identify new types of defects. A convolutional layer is a layer that calculates convolution values ​​by performing a convolution operation for each kernel on input values ​​input to multiple nodes from the previous layer, calculates the function value of a predetermined activation function for a correction value obtained by adding the calculated convolution value and a bias value as an output value, and outputs the calculated output value to the next layer. In the convolution operation, each node receives one or more input values ​​from the previous layer, and an independent convolution coefficient is used for each input value. The convolution coefficients, bias values, and activation function parameters are part of one set of model parameters.

[0069] Examples of activation functions that can be used include the rectified linear unit (RLU) and the sigmoid function. The RLU is a function that sets a threshold (e.g., 0) as the output value for input values ​​below that threshold and outputs input values ​​above the threshold as is. Therefore, this threshold can be part of a set of model parameters. For convolutional layers, the need to reference input values ​​from nodes in the previous layer and the need to output output values ​​to nodes in the next layer can also be part of a set of model parameters. Therefore, unlike fully connected layers (described later), each node in a convolutional layer is not necessarily connected to all nodes in the previous layer so that it receives input values, nor is it necessarily connected to all nodes in the next layer so that it outputs output values.

[0070] A pooling layer is a layer that has nodes that determine a representative value from the input values ​​received from multiple nodes in the previous layer and output the determined representative value as an output value to the next layer. The representative value is, for example, a value that statistically represents the multiple input values, such as the maximum value, average value, or mode. A stride is set in advance for the pooling layer. The stride indicates the range of adjacent nodes in the previous layer that refer to the input value for a single node. Therefore, a pooling layer can also be considered as a layer that downsamples the input values ​​from the previous layer to a lower dimension and provides the output value to the next layer.

[0071] A fully connected layer is a layer that performs a convolution operation on input values ​​input from the previous layer to each of multiple nodes to calculate a convolution value, calculates an output value by adding the calculated convolution value and a bias value, and outputs the calculated output value to the next layer. In other words, a fully connected layer is a layer that outputs an output value obtained by performing a convolution operation on all of the multiple input values ​​input from the previous layer using parameter sets (kernels) whose number is less than the number of input values. Therefore, in a fully connected layer, the convolution coefficients, bias values, and activation function parameters are part of one set of model parameters. In this way, by placing a fully connected layer immediately before the output layer, it is possible to reduce the degrees of freedom while taking into consideration all components that significantly affect the characteristic values ​​provided by the previous layer, and derive the final output value.

[0072] The number of layers of the CNN, the types of each layer, the number of nodes of each layer, etc. are not limited to those shown in FIG. 14. The CNN according to this embodiment may have a configuration capable of calculating the probability of each defect type as an output value for a measurement signal having signal values ​​for multiple sample points as an input value. However, as exemplified in FIG. 14, the CNN according to this embodiment preferably includes an intermediate layer configured by alternately stacking one or more convolutional layers and pooling layers in a repeated cycle or more. This is because the repetition of the convolutional layers narrows down the components that significantly affect the characteristic value. Note that the pooling layer may be omitted in this repetition of the convolutional layers.

[0073] Furthermore, a machine learning model that calculates a vector value with three or more elements as an output value may be used as a multi-class model with three or more classes. Taking the example of application to defect type determination, the elements of the output value are obtained as the probability of the defect type corresponding to that element. In the example shown in FIG. 14, the fully connected layer Fc14 must be set with parameter sets corresponding to the individual output value elements. Therefore, in the multi-class model, optimization based on a predetermined standard is performed for all parameter sets in model learning. Therefore, as described above, even if a parameter set intended for determining only one type of defect is intended, it may affect the determination results for other types of defects. Therefore, the defect determination unit 114 according to this embodiment preferably uses multiple two-class models for determining whether each type of defect corresponds to the other type of defect.

[0074] In the above example, the signal value for each pixel is mainly used as an input value to the machine learning model, but this is not limiting. The control unit 110 of the defect inspection apparatus 100 may include a feature analysis unit (not shown) that calculates, from image data, feature quantities that indicate the characteristics of a pattern (including defects) appearing in the image. All or some of the defect classifiers 114-1 to 114-N may use the feature quantities calculated by the feature analysis unit as input values ​​to the machine learning model, instead of or together with the signal values ​​for each pixel. The model learning unit 118 uses the feature quantities calculated by the feature analysis unit as input values ​​to the machine learning model, instead of or together with the signal values ​​for each pixel, as input values ​​that constitute training data used to calculate model parameters of the machine learning model. Such feature quantities may be, for example, shape feature parameters such as circularity, Euler number, and Feret diameter, feature quantities used in image recognition such as HOG (Histograms of Oriented Gradients) feature quantities and SIFT (Scaled Invariance Feature Transform) feature quantities, or a combination thereof. In other words, the calculated feature quantities can be used as defect feature quantities that indicate the state of the object under inspection. The new species determination unit 122 may use the feature quantities calculated by the feature analysis unit to determine whether the object is a new species. The judgment data may include image feature quantities as information indicating the state of the object under inspection, instead of the presence or absence and number of defects for each type of defect. The defect judgment unit 114 or the comprehensive judgment unit 116 may further include a defect classifier that refers to the judgment data and determines the type of defect from the image feature quantities analyzed by the feature analysis unit.

[0075] As described above, the defect inspection apparatus 100 according to this embodiment is a defect inspection apparatus that inspects an object for defects based on an image of the object, and includes a plurality of defect classifiers that each classify a different defect type based on the image using a predetermined machine learning model. The defect types classified by each defect classifier are a portion of a predetermined number of defect types that are the object of classification by the defect inspection apparatus. Furthermore, each of the plurality of defect discriminators may determine whether or not the type of defect occurring in the object to be inspected corresponds to one predetermined type of defect. Furthermore, the object to be inspected may be glass, and the manufacturing method may include an inspection step using the defect inspection device described above. With this configuration, each defect classifier determines whether the type of defect detected from the image is part of a predetermined number of defect types. When a parameter set used by a specific defect classifier to determine the defect type is changed, unlike when all of the predetermined number of defect types are subject to determination, this does not affect the parameter sets used by other defect classifiers, and thus the determination of that defect type. In particular, when each defect classifier determines whether a defect corresponds to one predetermined type of defect, deterioration in determination accuracy is further avoided. This makes system management easier.

[0076] Furthermore, a plurality of defect discriminators may discriminate the types of defects occurring in the object to be inspected in parallel, regardless of the results of the judgments made by the other defect discriminators. With this configuration, the determination of whether each defect type is applicable or not is performed in parallel, so that even if the number of defect types to be determined increases, the processing time does not increase, and rapid processing can be achieved.

[0077] Furthermore, the number of defect classifiers may be N, and the nth defect classifier may determine whether the type of defect occurring in the object to be inspected corresponds to the nth defect type, and when the nth defect classifier determines that the type of defect occurring in the object to be inspected does not correspond to the nth defect type, the n+1th defect classifier may start a process of determining whether the type of defect occurring in the object to be inspected corresponds to the n+1th defect type. This configuration allows each defect classifier to serially execute a process of determining whether or not a defect corresponds to a predetermined defect type, thereby avoiding an excessively large processing load and contributing to economical implementation.

[0078] Furthermore, n may be determined in descending order of occurrence frequency or risk of the nth defect type. With this configuration, a type of defect that occurs more frequently or a type of defect that has a greater risk of occurrence is given priority in discrimination, so that damage caused by the occurrence of defects can be suppressed for the entire system.

[0079] Furthermore, each of the plurality of defect classifiers may extract a feature amount of a defect using a machine learning model. With this configuration, features according to the type of each defect can be expressed without predefining specific feature amounts.

[0080] The image processing device may also include a model learning unit that determines model parameters for identifying a specific type of defect using a machine learning model for image data showing an image including the specific type of defect. With this configuration, model parameters for discriminating the defect type can be determined using training data that indicates the relationship between image data as input values ​​and defect types as output values. Therefore, by using model parameters that correspond to the usage environment to discriminate the defect type, the accuracy of the determination can be improved.

[0081] The defect discriminator may also discriminate the type of defect based on image data showing a plurality of images of the object to be inspected taken under different imaging conditions. This configuration allows the type of defect to be determined more accurately using the differences in image features for each imaging condition as a clue.

[0082] The apparatus may also include a new type determination unit that calculates the distance in space between a feature value that indicates the characteristics of a defect, the distance between a representative feature value that is predetermined for each type of defect and an extracted feature value that is a feature value extracted from the image, and determines that the type of defect detected from the image is a new type if the calculated distance is greater than a predetermined distance threshold for any of the types of defect. This configuration makes it possible to identify defects with characteristics different from known defect types as new types of defects, thereby promoting process management according to defects with different characteristics.

[0083] The image processing device may further include a new type determining unit that determines the type of defect detected from the image as a new type when there is no defect classifier that has succeeded in determining the type of defect. This configuration allows defects whose types could not be identified to be identified as new types of defects, thereby promoting process management that does not rely on known defect types.

[0084] The inspection system may further include a manufacturing process management section that determines correction conditions for correcting manufacturing conditions for the object to be inspected based on the defect states determined by the plurality of defect determiners. This configuration makes it possible to efficiently control the manufacturing process of the object to be inspected in accordance with the determined defect state without relying on human labor.

[0085] The apparatus may also include a feature analysis unit that analyzes the feature amounts of defects from the image, and a second defect discriminator that determines the type of defect from the feature amounts of the defect analyzed by the feature analysis unit using judgment data that indicates the relationship between the feature amounts of the defect and the type of the defect. This configuration determines the type of defect according to the feature of the detected defect based on the relationship between the feature of a known defect and the defect type. Since the defect type can be determined without relying entirely on machine learning models, the amount of processing can be reduced.

[0086] The inspection device may also include a judgment input unit that determines the type of defect occurring in the object to be inspected as the type of defect indicated by the acquired operation input. This configuration allows the user to know the type of defect determined by the user. Since the type of defect can be determined without relying entirely on machine learning models, the risk of misjudgment can be reduced.

[0087] Furthermore, before the plurality of defect discriminators determine the types of defects occurring in the object under inspection, the second defect discriminator or the judgment input unit may determine the type of defect occurring in the object under inspection. With this configuration, a judgment is made using the relationship between the features of a known defect and the type of defect, or by the user, so that the type of defect can be determined even if it is a type of defect that cannot be determined by a machine learning model.

[0088] Furthermore, the glass manufacturing method according to the present embodiment may include an inspection step in which the object to be inspected is glass and the above-described defect inspection device is used.

[0089] The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to that described above, and various design modifications can be made within the scope of the gist of the present invention.

[0090] For example, the defect inspection apparatus 100 may be realized as part of a manufacturing facility for the object to be inspected, or may be a separate device independent of the object to be inspected. The defect inspection apparatus 100 is not limited to the manufacturing facility, and may acquire image data from other devices such as a data storage device or a PC. The defect inspection apparatus 100 may include the imaging unit 130, the operation unit 150, and the display unit 160, or may omit some or all of them. The imaging unit 130, the operation unit 150, and the display unit 160 may be connected to each other via the input / output unit 140.

[0091] In the defect inspection apparatus 100, some or all of the model learning unit 118, the manufacturing process management unit 120, the new species determination unit 122, and the determination input unit 124 may be omitted. Depending on the type of object to be inspected and the types and number of defects to be detected, one or both of the defect detection unit 112 and the comprehensive judgment unit 116 may be omitted. The glass to be inspected may have any size, such as width, length, thickness, etc. Furthermore, the defect inspection apparatus 100 may be applied to inspecting objects other than glass, such as circuit boards and wafers, to determine the presence or absence of defects and the type of defects.

[0092] Furthermore, part or all of the defect inspection apparatus 100 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the defect inspection apparatus 100 may be individually implemented as a processor, or part or all of the functional blocks may be integrated into a processor. Furthermore, the integrated circuit implementation method is not limited to LSI, and may be implemented using a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit implementation technology that can replace LSI emerges due to advances in semiconductor technology, an integrated circuit based on that technology may be used. [Industrial Applicability]

[0093] According to the defect inspection device, defect inspection method, and manufacturing method of each of the above aspects, each defect classifier determines whether the type of defect detected from an image is part of a predetermined number of defect types. Changing the parameter set used by a specific defect classifier to determine the defect type does not affect the parameter sets used by other defect classifiers, nor does it affect the defect type determination, as opposed to when all of the predetermined number of defect types are subject to determination. This makes system management easier. [Explanation of symbols]

[0094] 100... defect inspection device, 110... control unit, 112... defect detection unit, 114... defect discrimination unit, 116... overall judgment unit, 118... model learning unit, 120... manufacturing process management unit, 122... new species judgment unit, 124... judgment input unit, 130... imaging unit, 140... input / output unit, 150... operation unit, 160... display unit, 170... storage unit

Claims

1. 1. A defect inspection apparatus for inspecting an object to be inspected for defects occurring in the object to be inspected based on an image of the object, N (N is a predetermined integer equal to or greater than 2) defect classifiers are provided, each classifying a different type of defect based on the image using a predetermined machine learning model; the types of defects discriminated by each defect discriminator are a part of a predetermined number of types of defects to be discriminated by the defect inspection device, an nth (n is an integer between 1 and N-1) defect classifier inputs a weighted sum of pixel brightness values ​​between a plurality of images of the object under inspection taken under different imaging conditions, calculates the probability that the defect corresponds to the nth type of defect, and determines whether the type of defect occurring in the object under inspection corresponds to the nth type of defect depending on whether the probability is greater than a preset defect determination threshold; As a weighting coefficient for the weighted sum, a weighting coefficient for an image captured under an imaging condition under which the n-type defect is easily detected is set to a value greater than a weighting coefficient for other images; when the nth defect classifier determines that the type of defect occurring in the object to be inspected does not correspond to the nth type of defect, the (n+1)th defect classifier starts a process of determining whether the type of defect occurring in the object to be inspected corresponds to the (n+1)th type of defect; The n is determined in descending order of occurrence frequency or occurrence risk of the n-th type of defect, The defect determination threshold for a type of defect with a high risk of occurrence is set to a smaller value. Defect inspection equipment.

2. A model learning unit is provided that determines model parameters for discriminating a specific type of defect using the machine learning model for image data showing an image including the specific type of defect. The defect inspection device according to claim 1 .

3. calculating a distance in space between a feature quantity indicating a feature of a defect, the distance being between a representative feature quantity predetermined for each type of defect and an extracted feature quantity which is a feature quantity extracted from the image; and a new type determination unit that determines the type of defect detected from the image as a new type when the calculated distance is greater than a predetermined distance threshold for any of the defect types.

3. The defect inspection device according to claim 1.

4. When there is no defect classifier that has succeeded in classifying the type of defect, a new type determination unit is provided that determines the type of defect detected from the image as a new type. The defect inspection device according to any one of claims 1 to 3.

5. The manufacturing process control unit determines correction conditions for correcting manufacturing conditions of the object to be inspected based on the defect states determined by the plurality of defect determiners. The defect inspection device according to any one of claims 1 to 4.

6. a feature analysis unit that analyzes the feature amount of a defect from the image data; a second defect discriminator that determines the type of the defect from the feature amount of the defect analyzed by the feature analysis unit using determination data indicating the relationship between the feature amount of the defect and the type of the defect; a judgment input unit for determining the type of defect occurring in the object to be inspected as the type of defect indicated by the acquired operation input; The defect inspection device according to any one of claims 1 to 5.

7. Before the plurality of defect discriminators determine the type of defect occurring in the object to be inspected, the second defect discriminator or the judgment input unit determines the type of defect occurring in the object to be inspected. The defect inspection device according to claim 6.

8. 1. A defect inspection method for inspecting defects occurring in an object to be inspected based on an image of the object to be inspected, comprising: N (N is a predetermined integer equal to or greater than 2) defect discrimination processes are provided, each of which discriminates a different type of defect based on the image using a predetermined machine learning model; the type of defect determined in each defect determination step is a part of a predetermined number of types of defects to be determined in the defect inspection method, The nth (n is an integer of 1 to N-1) defect discrimination step inputs a weighted sum of luminance values ​​for each pixel between a plurality of images of the object under different imaging conditions, calculates the probability that the defect corresponds to the nth type, and determines whether the type of defect occurring in the object under inspection corresponds to the nth type of defect depending on whether the probability is greater than a preset defect determination threshold value; As a weighting coefficient for the weighted sum, a weighting coefficient for an image captured under an imaging condition under which the n-type defect is easily detected is set to a value greater than a weighting coefficient for other images; when it is determined in the n-th defect discrimination step that the type of defect occurring in the object to be inspected does not correspond to the n-th type of defect, starting an (n+1)-th defect discrimination step; The n is determined in descending order of occurrence frequency or occurrence risk of the n-th type of defect, The defect determination threshold for a type of defect with a high risk of occurrence is set to a smaller value. Defect inspection methods.

9. The test object is glass, A glass manufacturing method using the defect inspection method according to claim 8.

10. The test object is glass, An inspection process using the defect inspection device according to any one of claims 1 to 7. A method for producing glass having the above formula.

Images