Method for performing appearance inspection, and appearance inspection system

Abstract

To provide a method for performing appearance inspection and an appearance inspection system that make it possible to determine acceptability of defects in a GRAY region in appearance inspection of an inspection target by using a machine learning model.SOLUTION: The present invention is a method for performing appearance inspection executed by an appearance inspection system, and includes the steps of: selecting, by a rule-base model, a defect image for enabling a machine learning model to determine a defect of an inspection target; and determining the defect of the inspection target by using an ensemble model including multiple machine learning models, the multiple machine learning models having been trained with images captured by multiple imaging devices using different methods for imaging the defect image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an appearance inspection method and an appearance inspection system. [Background technology]

[0002] In the manufacturing industry, in order to provide the market with high-quality, safe and secure products, it is necessary to properly distinguish between good and defective products through inspection processes. Generally, in appearance inspection, a combination of imaging optical systems such as cameras and lighting captures an image of the appearance of the product (inspected object), and then performs appropriate image processing to determine the presence and size of defects on the product surface and determine whether it is good or bad.

[0003] Patent Document 1 discloses a configuration for the purpose of determining the quality of the appearance of an object to be inspected with high accuracy and in a short time, in which a plurality of feature amounts are extracted from each of at least two images based on images captured under at least two or more different imaging conditions, a feature amount for determining the quality of the object is selected from the feature amounts that span the extracted feature amounts of each image, and the quality of the object is determined based on the selected feature amounts. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the visual inspection of the above-mentioned object to be inspected (for example, a photosensitive body), when a rule-based threshold for OK or NG is drawn, there is an area (gray area) where good and defective products are mixed, and in order to prevent defective products from being released, the threshold line is drawn so that all defective products are judged as NG (i.e., all gray areas are judged as NG), which may result in some good products being judged as NG and defective products being over-detected.

[0005] The present invention has been made in consideration of the above, and aims to provide an appearance inspection method and an appearance inspection system that can determine whether defects in the gray area are pass or fail using a machine learning model during the appearance inspection of an object to be inspected. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the present invention provides an appearance inspection method executed in an appearance inspection system, which includes the steps of selecting defect images that can be used to determine defects in an object to be inspected using a machine learning model based on a rule-based model, and determining defects in the object to be inspected using an ensemble model that uses multiple machine learning models trained using images from multiple imaging devices with different imaging methods for the defect images. [Effects of the Invention]

[0007] According to the present invention, it is possible to determine whether defects in the gray area are good or bad using a machine learning model in the visual inspection of an object to be inspected. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a functional configuration of a visual inspection system according to the present embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of the visual inspection system according to the present embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of the flow of control processing in the visual inspection system according to this embodiment. [Figure 4A] FIG. 4A is a flowchart showing an example of the flow of a process for optimizing the selection of feature quantities used for learning a machine learning model in the visual inspection system according to this embodiment. [Figure 4B] FIG. 4B is a diagram illustrating an example of a functional configuration related to a process for optimizing the selection of feature amounts used for learning a machine learning model in the visual inspection system according to this embodiment. [Figure 4C] FIG. 4C is a diagram showing an example of division of learning data and test data in the visual inspection system according to this embodiment. [Figure 4D] FIG. 4D is a diagram showing an example of division of training data and verification data in the visual inspection system according to this embodiment. [Figure 4E] FIG. 4E is a diagram showing an example of distribution of defect sizes in each fold in the visual inspection system according to this embodiment. [Figure 5] FIG. 5 is a diagram showing a list of feature amounts extracted in the visual inspection system according to this embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of learning of a machine learning model in the device of this embodiment. [Figure 7A] FIG. 7A is a flowchart showing an example of the flow of a pass / fail determination process during inspection of an object to be inspected in the visual inspection system according to this embodiment. [Figure 7B] FIG. 7B is a diagram showing an example of a functional configuration related to the pass / fail determination process during inspection of an object to be inspected in the visual inspection system according to this embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of the flow of the rule-based model determination process in the visual inspection system according to this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of a visual inspection method and a visual inspection system to which the visual inspection system is applied will be described in detail with reference to the accompanying drawings. Although the embodiment of the present invention will be specifically described below, the present invention is not limited thereto.

[0010] 1 is a diagram showing an example of the functional configuration of a visual inspection system according to this embodiment. Details of the various functions will be explained using other drawings, and an outline of the embodiment will be explained using FIG. 1.

[0011] The visual inspection system according to this embodiment includes a rule-based model, a first machine learning model, a second machine learning model, and a weighted ensemble, as shown in Fig. 1. In the following description, when there is no need to distinguish between the first machine learning model and the second machine learning model, they will be referred to as machine learning models.

[0012] In this embodiment, the visual inspection system first captures an image of an object to be inspected using the visual inspection system to obtain visual inspection data including a plurality of images. Next, the visual inspection system performs a judgment of OK, GRAY, or NG using a rule-based judgment method on the acquired visual inspection data by itself, and performs a judgment using a machine learning model on objects judged as GRAY (hereinafter also referred to as GRAY judgment). Here, GRAY judgment is a judgment criterion established to prevent frequent oversights and overdetections of objects that are difficult to clearly judge as OK or NG by the visual inspection system by itself.

[0013] The visual inspection system then determines whether an object under inspection has been previously determined to be gray by the rule-based model. Specifically, the visual inspection system calculates feature quantities by converting image data of the object under inspection determined to be gray into numerical data using a predefined processing method. The resulting feature quantities are used to perform a determination process using the rule-based model, and the object is determined to be either OK or NG, based on whether it can be determined by the machine learning model. That is, the visual inspection system uses the rule-based model to select, from the image data, images (defect images) in which the machine learning model can determine defects in the object under inspection. By performing a determination using the rule-based model, defects that are difficult for the machine learning model to predict can be determined to be NG, thereby reducing the number of oversights in which an object under inspection is determined to be OK. Here, the machine learning model may predict the size of the defect in the object under inspection.

[0014] For images for which the machine learning model can determine defects, different sets of features are input to the first machine learning model and the second machine learning model, and the predicted results of the defect size are calculated by each machine learning model. Here, the first machine learning model and the second machine learning model are examples of multiple machine learning models trained using images from multiple imaging devices with different imaging methods.

[0015] Furthermore, the visual inspection system uses a weighted ensemble to predict the defect size using predefined weights for each machine learning model, and compares this to a predefined threshold to determine whether the defect is OK or NG. In other words, the visual inspection system uses a weighted ensemble model (an example of an ensemble model) that uses multiple machine learning models to determine defects in the object being inspected for defect images. In this way, using multiple machine learning models makes it possible to determine whether the object is OK or NG by taking into account the multiple features of the same object being inspected that are contained in multiple image data, thereby achieving high judgment accuracy.

[0016] FIG. 2 is a diagram showing an example of the hardware configuration of a visual inspection system according to this embodiment. FIG. 3 is a flowchart showing an example of the flow of control processing in the visual inspection system according to this embodiment. In this embodiment, the visual inspection system rotates an inspection object 1 carried to the visual inspection system, and captures multiple images to be used for judgment using two imaging means 3 and 8. The first imaging device 12 has a light source 2, imaging means 3, inspection means 4, stage 5, and control unit 6. The second imaging device 13 has a light source 7, imaging means 8, stage 9, and slits 10 and 11. Here, the first imaging device 12 and the second imaging device 13 are examples of multiple imaging devices with different imaging methods.

[0017] In this embodiment, the visual inspection system first moves the imaging means 3 installed on the movable stage 5 to a first imaging position by imaging using the first imaging device 12, then illuminates the inspected object 1 with the light source 2, images the object 1 with the imaging means 3, and stores the image in the PC of the inspection means 4. Similarly, the visual inspection system performs imaging under different imaging conditions by moving the stage 5 to a second imaging position or a third imaging position by imaging using the first imaging device 12.

[0018] Next, the visual inspection system captures images using the second imaging device 13 with slits 10 and 11 placed between the light source 7 and the object under inspection 1, respectively, and saves the images in the PC (Personal Computer) of the inspection means 4. The visual inspection system similarly captures images using grids with different angles. By capturing images of the same defect under different camera positions and grids, multiple features of the defect can be captured as images. The image capture method is not limited to the first imaging device 12 and the second imaging device 13, and can be appropriately selected from known imaging methods that can capture multiple features of the defect as images.

[0019] FIG. 4A is a flowchart showing an example of the flow of a process for optimizing selection of feature quantities used for training a machine learning model in the visual inspection system according to the present embodiment. FIG. 4B is a diagram showing an example of a functional configuration related to the process for optimizing selection of feature quantities used for training a machine learning model in the visual inspection system according to the present embodiment. FIG. 4C is a diagram showing an example of division of learning data and test data in the visual inspection system according to the present embodiment. FIG. 4D is a diagram showing an example of division of training data and verification data in the visual inspection system according to the present embodiment. FIG. 4E is a diagram showing an example of the distribution of defect sizes in each fold in the visual inspection system according to the present embodiment. In FIG. 4E, the vertical axis represents the number of data, and the horizontal axis represents the size of the defect. FIG. 5 is a diagram showing a list of feature quantities extracted in the visual inspection system according to the present embodiment. FIG. 6 is a diagram for explaining an example of training of a machine learning model in the apparatus according to the present embodiment.

[0020] The image acquisition unit 401 acquires images of defects captured by a visual inspection system and measurement results of the defect sizes from a learning database (step S101). Next, the image acquisition unit 401 divides the acquired data (i.e., data usable for learning the machine learning model) into learning data to be used for learning the machine learning model and test data that is not used for learning the machine learning model and is used for accuracy verification (step S102), so that the accuracy of the machine learning model can be verified (e.g., verification of the prediction accuracy of the defect sizes) using the acquired data (i.e., data usable for learning the machine learning model).

[0021] As a method of dividing the data into training data and test data, for example, as shown in Fig. 4C, by dividing the data into chronologically most recent data from April as test data and past data from January to March as training data, it becomes possible to evaluate the prediction accuracy simulating a situation similar to the visual inspection of an actual inspected object 1. There is no restriction on the ratio of the acquisition periods of the data used for training data and test data, and an appropriate number of divisions can be selected depending on the number and distribution of data.

[0022] Next, the learning unit 402 divides the training data into training data used to train the machine learning model and validation data used to validate the accuracy of the machine learning model so that the accuracy of the machine learning model can be verified using the entire divided training data, and then performs training and accuracy verification (evaluation) of the machine learning model (step S103). To verify the accuracy of the machine learning model using all the training data, the learning unit 402 divides the training data into four folds as shown in FIG. 4D, and uses one fold as validation data and the remaining three folds as training data to train and evaluate the machine learning model. The learning unit 402 can verify all the training data by repeating the same operation four times. As shown in FIG. 4D, the folds are divided so that the distribution of defect sizes contained in each fold is as uniform as possible. There are no restrictions on the number of folds or the division index, and any known division method can be selected depending on the number and distribution of data.

[0023] Next, the feature extraction unit 403 calculates features as shown in Fig. 5 from the acquired images (training data for each fold) (step S104). The feature extraction unit 403 calculates each feature by changing parameters for 15 types of images, thereby creating several hundred types of features in total. In this way, the learning unit 402 uses the features extracted from images of the object 1 captured using different imaging methods for training the machine learning model.

[0024] Furthermore, the feature selection unit 404 selects approximately 5 to 10 features from all of the features created by the feature extraction unit 403 in step S104 as features to be used in training the machine learning model (step S105). In other words, the machine learning model may be a model trained using a set of multiple features. Since there are an extremely large number of combinations of feature selections and it is difficult to try all of them, Bayesian optimization is used to determine a combination of features that will achieve the highest possible prediction accuracy within a limited trial time. Note that there are no limitations on the method for searching for the optimal combination of features to be used by the machine learning model, and any known optimization method can be selected. When there are only a few features, a method of trying all combinations can be considered.

[0025] The learning unit 402 then uses the selected features to train the machine learning model, calculates prediction results for the defect sizes in the training data and test data, and outputs evaluation indices for Bayesian optimization (step S106). In this embodiment, an optimal set of features is determined for each of the two machine learning models, the Ridge regression model and the support vector regression model, but there are no restrictions on the selection of the machine learning model, and it can be selected from known models.

[0026] The method for calculating the predicted results of the defect size using the trained machine learning model is as shown in Figure 6. The selected feature values ​​are input into the machine learning model for each fold, which is trained using different validation data for each fold into which the training data is divided, to obtain the predicted results of the defect size for each machine learning model. The average value of the prediction results for each machine learning model is output as the prediction result of the machine learning model.

[0027] The feature selection unit 404 also determines whether a specified trial time has elapsed (step S107). If the specified trial time has elapsed (step S107: True), the feature selection unit 404 selects a feature combination that is considered to have the highest prediction accuracy during the trial period, selects a machine learning model to be used in actual testing, and outputs the feature combination and the trained machine learning model (step S108).

[0028] Fig. 7A is a flowchart showing an example of the flow of a pass / fail determination process when an object to be inspected is inspected in the visual inspection system according to this embodiment. Fig. 7B is a diagram showing an example of a functional configuration related to the pass / fail determination process when an object to be inspected is inspected in the visual inspection system according to this embodiment. Fig. 8 is a flowchart showing an example of the flow of a determination process of a rule-based model in the visual inspection system according to this embodiment.

[0029] First, the image acquisition unit 701 acquires images captured under various conditions by the visual inspection system (step S201). Next, the feature extraction unit 702 calculates, from the acquired images, a set of features to be used in the machine learning model determined by the process shown in Fig. 4A and features for determining the rule-based model (step S202).

[0030] Next, the determination unit 703 uses the rule-based model to determine whether the defect is one that the machine learning model can predict with high accuracy (OK) or not (step S203). As a determination method, as shown in Fig. 8, the feature amounts for determination of the rule-based model (for example, feature amounts 1 to 3) are sequentially compared with preset thresholds (steps S801 to S803), and if all feature amounts are True (step S203: True), the defect is determined to be one that the machine learning model can predict with high accuracy, and otherwise (step S203: False), the product is determined to be not good.

[0031] Judgment using the rule-based model mainly uses feature quantities based on the statistics shown in Fig. 5 to determine whether or not a defect has been clearly photographed, thereby reducing the number of oversights that often occur when blurry images are judged using a machine learning model.

[0032] Furthermore, the prediction unit 704 inputs the set of feature values ​​calculated in step S202 into each of the first machine learning model and the second machine learning model, and calculates a prediction result (predicted value) of the defect size (step S204). As a method for calculating the prediction result, as shown in Fig. 5, the feature values ​​are input into each of 12 machine learning models with different trained folds, and the prediction results are simply averaged to obtain the prediction result of the first machine learning model. Similarly, a prediction result is calculated for the second machine learning model.

[0033] In addition, the prediction unit 704 obtains a final prediction result by weighting the prediction results of the first machine learning model and the second machine learning model using a weighted ensemble model and calculating the average, and compares it with a pre-set threshold to output a judgment result of OK or NG (step S205).

[0034] As described above, according to the visual inspection system of this embodiment, rather than using a rule-based model to make a pass / fail judgment, a machine learning model (AI model) is used to make a judgment between defects that can be predicted with high accuracy and defects that cannot be predicted with high accuracy. For defects that the rule-based model determines can be predicted with high accuracy by the machine learning model, multifaceted imaging results are extracted from multiple imaging results obtained using different imaging methods, and the multiple machine learning models make a pass / fail judgment, thereby achieving high judgment accuracy. As a result, it is possible to reduce overdetection, in which a non-defective inspected object (e.g., a photosensitive body) is judged as NG, without releasing defective products. In other words, during the visual inspection of an inspected object, defects in the gray area can be judged as pass / fail using a machine learning model.

[0035] The program executed by the visual inspection system of this embodiment is provided in advance in a ROM (Read Only Memory) etc. The program executed by the visual inspection system of this embodiment may also be provided by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disk).

[0036] Furthermore, the program executed by the visual inspection system of this embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the program executed by the visual inspection system of this embodiment may be provided or distributed via a network such as the Internet.

[0037] The program executed by the visual inspection system of this embodiment (e.g., inspection means 4) has a modular structure including the above-mentioned units (image acquisition units 401, 701, learning unit 402, feature extraction units 403, 702, feature selection unit 404, judgment unit 703, prediction unit 704), and in actual hardware, a processor such as a CPU (Central Processing Unit) reads and executes the program from the above-mentioned ROM, thereby loading the above-mentioned units onto the main memory, and the image acquisition units 401, 701, learning unit 402, feature extraction units 403, 702, feature selection unit 404, judgment unit 703, and prediction unit 704 are generated on the main memory.

[0038] For example, aspects of the present invention are as follows. <1> A visual inspection method performed in a visual inspection system, comprising: A step of selecting defect images that can be determined as defects in the object to be inspected using a machine learning model by using a rule-based model; determining defects of the object to be inspected contained in the defect image using an ensemble model using a plurality of the machine learning models trained using images from a plurality of imaging devices with different imaging methods; A visual inspection method including: <2> The imaging device includes a light source that irradiates the surface of the object under inspection with light, and imaging means that images the object under inspection illuminated by the light source. <1> The appearance inspection method according to claim 1. <3> Utilizing feature amounts extracted from images of the object to be inspected using different imaging methods for training the machine learning model. <1> or <2> The appearance inspection method according to claim 1. <4> The machine learning model is a model trained using a set of multiple feature quantities. <1> from <3> 10. The appearance inspection method according to claim 9, wherein <5> The machine learning model predicts the size of the defect in the inspected object. <1> from <4> 10. The appearance inspection method according to claim 9, wherein <6> a rule-based model that selects defect images that can be used to determine defects in the object to be inspected using a machine learning model; an ensemble model that determines defects of the object to be inspected that are included in the defect image by using a plurality of the machine learning models trained using images from a plurality of imaging devices that use different imaging methods; A visual inspection system comprising: [Explanation of symbols]

[0039] 2,7 light source 3,8 Imaging Method 4. Inspection methods Stages 5 and 9 6 Control Unit 10,11 Slit 12 First imaging device 13 Second imaging device 401,701 Image acquisition unit 402 Learning Department 403,702 Feature Extraction Unit 404 Feature Selection Unit 703 Judgment section 704 Prediction Department [Prior art documents] [Patent documents]

[0040] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-049974

Claims

1. A visual inspection method performed in a visual inspection system, comprising: A step of selecting defect images that can be determined as defects in the object to be inspected using a machine learning model by using a rule-based model; determining defects of the object to be inspected contained in the defect image using an ensemble model using a plurality of the machine learning models trained using images from a plurality of imaging devices with different imaging methods; A visual inspection method including:

2. 2. The visual inspection method according to claim 1, wherein the imaging device comprises a light source that irradiates the surface of the object under inspection with light, and imaging means that images the object under inspection illuminated by the light source.

3. 3. The visual inspection method according to claim 1, wherein feature amounts extracted from images of the object to be inspected obtained by different imaging methods are used for training the machine learning model.

4. The visual inspection method according to claim 1 , wherein the machine learning model is a model trained using a set of a plurality of feature quantities.

5. The visual inspection method according to claim 1 , wherein the machine learning model predicts the size of the defect in the object to be inspected.

6. a rule-based model that selects defect images that can be used to determine defects in the object to be inspected using a machine learning model; an ensemble model that determines defects of the object to be inspected that are included in the defect image by using a plurality of the machine learning models trained using images of a plurality of imaging devices that have different imaging methods; A visual inspection system comprising:

Images