IMAGE JUDGMENT DEVICE, IMAGE JUDGMENT METHOD, AND PROGRAM
By combining old and new machine learning models trained on different datasets, the system enhances image determination accuracy, addressing issues from imaging variations and maintaining precision across varying image conditions.
Patent Information
- Application Number
- JP2023512885
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-04-05
- Filing Date
- 2022-03-14
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-03-14
AI Technical Summary
Existing image determination systems using machine learning models face accuracy issues due to differences in images captured by imaging means, which are difficult to detect and adjust for, leading to reduced detection precision.
The system employs a combination of old and new machine learning models trained on different datasets to improve accuracy by integrating determination results using a preset rule, allowing for improved detection even when image quality or capture conditions change.
This approach maintains high detection accuracy for both old and new image datasets, reducing oversights and overdetections by leveraging multiple models' strengths.
Smart Images

Figure 0007584073000001 
Figure 0007584073000002 
Figure 0007584073000003
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image judgment device, an image judgment method, and a program using a machine learning model. [Background technology]
[0002] Patent Document 1 discloses a technology that can detect an image showing a detection target such as a pedestrian with high accuracy using a learning model. According to Patent Document 1, by selecting one learning model corresponding to the gain of an image captured by an imaging means from among a plurality of learning models that have been trained in advance, it is possible to detect a detection target shown in the image with high accuracy. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2012-068965 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, due to various reasons such as maintenance of the imaging means, differences may occur in the images captured by the imaging means. In such cases, even if the detection target is detected using one learning model corresponding to the gain of the image captured by the imaging means, the accuracy may be significantly reduced. Furthermore, when it is difficult to adjust the criteria for evaluating the difference in the images, or when the difference is not noticeable to a person at a glance, it is not possible to prepare multiple learning models or to select a learning model during inspection, so that the accuracy of detecting the detection target cannot be improved.
[0005] In other words, in the task of having a machine learning model determine the label of a target image, if differences occur in the image that are not immediately noticeable to a human, the accuracy of the determination cannot be improved.
[0006] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide an image assessment device, etc. that can improve accuracy even when the accuracy of image assessment using a machine learning model decreases. [Means for solving the problem]
[0007] In order to achieve the above object, an image judgment device according to one embodiment of the present disclosure includes: a learning unit that obtains one or more first models by learning one or more types of machine learning models using a first teacher dataset including a plurality of first images and a first label associated with each of the plurality of first images; and obtains one or more second models by learning one or more types of machine learning models using one or more second teacher datasets including a plurality of second images different from the plurality of first images, a second label associated with each of the plurality of second images, and at least a portion of the first teacher dataset; an image acquisition unit that acquires a target image; and a judgment unit that outputs a judgment result of the label of the target image obtained by using at least two of one of the one or more first models and one of the one or more second models for the target image acquired by the image acquisition unit.
[0008] This makes it possible to improve accuracy even when the accuracy of image judgment using a machine learning model decreases.
[0009] These general or specific aspects may be realized by an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention
[0010] The present disclosure makes it possible to provide an image assessment device or the like that can improve accuracy even when the accuracy of image assessment using a machine learning model decreases. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of an image assessment device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer that realizes the functions of the image assessment device according to the embodiment by software. [Diagram 3] FIG. 3 is a diagram illustrating an example of old and new data sets serving as a teacher data set according to an embodiment. [Figure 4] FIG. 4 is a diagram for explaining another example of old and new data sets serving as a teacher data set according to an embodiment. [Diagram 5] FIG. 5 is a diagram for explaining yet another example of old and new data sets serving as a teacher data set according to an embodiment. [Figure 6A] FIG. 6A is a diagram illustrating an example of rule information according to an embodiment. [Figure 6B] FIG. 6B is a diagram for explaining another example of the rule information according to the embodiment. [Figure 6C] FIG. 6C is a diagram for explaining still another example of the rule information according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating the concept of a method for selecting a combination of an old machine learning model and a new machine learning model by machine learning according to an embodiment. [Figure 8A] FIG. 8A is a diagram illustrating an example of machine learning models to be combined and a validation dataset according to an embodiment. [Figure 8B] FIG. 8B is a diagram showing an example of the output of each machine learning model to be combined for each image included in the validation dataset. [Figure 9] FIG. 9 is a diagram illustrating an example of a list for allowing a user to select an optimal combination of an old machine learning model and a new machine learning model according to an embodiment. [Figure 10] FIG. 10 is a flowchart showing an outline of the operation of the image assessment device in this embodiment. [Figure 11] FIG. 11 is a diagram showing teacher data 1 and teacher data 2 according to the embodiment. [Figure 12] FIG. 12 is a diagram qualitatively illustrating the accuracy of the machine learning model for the training data (data set 1) and the most recent data according to the embodiment. [Figure 13] FIG. 13 shows an example of training data used for learning Model 1 and Model 2 shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, standards, components, the arrangement and connection forms of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims showing the highest concept of the present disclosure are described as optional components. In addition, each figure is not necessarily a strict illustration. In each figure, substantially the same configuration is given the same symbol, and duplicated explanations may be omitted or simplified.
[0013] (Embodiment) First, an image assessment device and an image assessment method according to the present embodiment will be described.
[0014] [1. Image assessment device 10] The configuration of the image assessment device 10 according to this embodiment will be described below. Fig. 1 is a block diagram showing the functional configuration of the image assessment device 10 according to this embodiment.
[0015] The image assessment device 10 is realized by a computer or the like, and is a device that can improve accuracy even when the accuracy of image assessment using a machine learning model decreases.
[0016] 1, the image assessment device 10 includes a learning unit 101, a storage unit 102, an image acquisition unit 103, and a judgment unit 104. The image acquisition unit 103 and the judgment unit 104 may be separate devices from the learning unit 101. In this case, the image assessment device 10 may include a memory or storage unit that stores the models 1-1 to 2-2 stored in the storage unit 102.
[0017] [1-1. Hardware configuration] FIG. 2 is a diagram showing an example of a hardware configuration of a computer 1000 that realizes the functions of the image assessment device 10 according to the present embodiment by software.
[0018] Before describing the functional configuration of the image assessment device 10 according to the present embodiment, an example of the hardware configuration of the image assessment device 10 according to the present embodiment will be described with reference to FIG.
[0019] 2, the computer 1000 is a computer including an input device 1001, an output device 1002, a CPU 1003, an internal storage 1004, a RAM 1005, a reading device 1007, a transmission / reception device 1008, and a bus 1009. The input device 1001, the output device 1002, the CPU 1003, the internal storage 1004, the RAM 1005, the reading device 1007, and the transmission / reception device 1008 are connected by the bus 1009.
[0020] The input device 1001 is a user interface device such as an input button, a touch pad, a touch panel display, etc., and accepts user operations. The input device 1001 may be configured to accept voice operations and remote operations using a remote control or the like in addition to accepting touch operations by the user.
[0021] The output device 1002 is used in combination with the input device 1001, and is configured with a touch pad or a touch panel display, etc., to notify the user of information that should be made known to the user.
[0022] The built-in storage 1004 is a flash memory or the like. The built-in storage 1004 may also store in advance at least one of a program for implementing the functions of the image assessment device 10 and an application that utilizes the functional configuration of the image assessment device 10. The built-in storage 1004 may also store models 1-1 to 2-2, etc.
[0023] The RAM 1005 is a random access memory, and is used to store data and the like when a program or application is executed.
[0024] The reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory. The reading device 1007 reads the above-mentioned programs and applications from a recording medium on which the programs and applications are recorded, and stores the programs and applications in the built-in storage 1004.
[0025] The transmitting / receiving device 1008 is a communication circuit for performing wireless or wired communication. The transmitting / receiving device 1008 may communicate with, for example, a server device connected to a network, download the above-mentioned programs and applications from the server device, and store them in the built-in storage 1004.
[0026] The CPU 1003 is a central processing unit that copies programs and applications stored in the internal storage 1004 to the RAM 1005, and sequentially reads out instructions included in the programs and applications from the RAM 1005 and executes them.
[0027] Next, each functional configuration of the image assessment device 10 according to the present embodiment will be described.
[0028] In this embodiment, the image assessment device 10 is described as constituting an inspection device that assesses an inspection image of a manufactured product using machine learning. In addition, in the following, an example of an inspection image of a manufactured product is described as an image optically captured of a wafer on which a semiconductor circuit is formed, but is not limited thereto. The inspection image of a manufactured product may be, for example, an image optically captured of a cross-section of a secondary battery, or may be a two-dimensional image obtained by optically capturing an image of a manufactured product.
[0029] [1-2. Learning Section 101] The learning unit 101 is a computing device that trains a machine learning model using a data set 1 or the like that serves as training data.
[0030] More specifically, the learning unit 101 obtains one or more first models by learning one or more types of machine learning models using a first teacher data set including a plurality of first images and first labels associated with each of the plurality of first images. In this embodiment, the learning unit 101 obtains model 1-1, model 1-2, ... by learning one or more types of models using a previously prepared dataset 1 as teacher data.
[0031] Moreover, the learning unit 101 obtains one or more second models by learning one or more types of machine learning models using one or more second teacher datasets. Here, each of the one or more second teacher datasets includes a plurality of second images different from the plurality of first images, a second label linked to each of the plurality of second images, and at least a portion of the first teacher dataset. In this embodiment, the learning unit 101 obtains models 2-1, 2-2, ... by learning one or more types of models using an updated dataset 2 including at least a portion of the data of dataset 1 and new data (updated data) as teacher data.
[0032] Note that Model 2-1, Model 2-2, ... are machine learning models obtained after Model 1-1, Model 1-2, .... For this reason, as shown in Figure 1, Model 1-1, Model 1-2, ... can be collectively referred to as old machine learning models, and Model 2-1, Model 2-2, ... can be collectively referred to as new machine learning models.
[0033] <Dataset> The data set 1 and the updated data set 2 will be described below.
[0034] Data set 1 is a teacher dataset including a plurality of images collected before the image assessment device 10 is introduced into the inspection process, i.e., before the inspection is performed. The updated dataset 2 is a teacher dataset including a plurality of images collected after a predetermined period of time has elapsed since the image assessment device 10 was introduced into the inspection process. In this way, since the updated dataset 2 is a dataset acquired chronologically after the dataset 1, the updated dataset 2 can also be referred to as a new dataset, and the dataset 1 can also be referred to as an old dataset.
[0035] In this embodiment, the images included in the data set 1 and the updated data set 2 are, for example, inspection images of manufactured products during a predetermined period. Here, for example, each of the multiple images (first images) included in the data set 1 may be an inspection image obtained during a first period of the predetermined period, and each of the multiple images (second images) included in the data set 2 may be an inspection image obtained after the first period of the predetermined period.
[0036] FIG. 3 is a diagram illustrating an example of old and new data sets serving as a teacher data set according to an embodiment.
[0037] Dataset 1 is an old dataset that includes a large number of inspection images collected before inspection and labels associated with each of the large number of inspection images. In the example of Dataset 1 shown in Figure 3, N1 inspection images are associated with a label indicating a dent defect, N2 inspection images are associated with a label indicating a scratch defect, N3 inspection images are associated with a label indicating a non-defective product, and N4 inspection images are associated with a label indicating a chip defect. Dataset 1 also includes images collected in a balanced manner under various conditions. For example, N5 images are collected from manufacturing condition A, N6 images are collected from manufacturing condition B, and N7 images are collected from manufacturing condition C, with N5, N6, and N7 all being equal to or greater than a predetermined number.
[0038] Updated Data Set 2 is a new dataset that includes multiple inspection images and labels associated with each of the multiple inspection images, which were hastily collected after the introduction of the inspection, for example, on the 100th day after the inspection, when the judgment accuracy of Model 1 decreased. In the example of Updated Data Set 2 shown in Figure 3, M1 inspection images are associated with a label indicating a defective product, and M2 inspection images are associated with a label indicating a non-defective product.
[0039] 3 further illustrates an updated dataset 3. The updated dataset 3 is a new dataset including a plurality of inspection images hastily collected because the determination accuracy by Model 1 and Model 2 has decreased on the 300th day since the inspection was performed, and labels associated with each of the plurality of inspection images. In the example of the updated dataset 3 illustrated in FIG 3, a label indicating a defective product is associated with L1 inspection images, and a label indicating a non-defective product is associated with L2 inspection images.
[0040] Thus, updated data set 2 includes inspection images taken after data set 1, and updated data set 3 includes inspection images taken after data set 1 and after updated data set 2.
[0041] In addition, FIG. 3 shows that dataset 1 is used as the teacher dataset for model 1, and dataset 1 and updated dataset 2 are used as the teacher dataset for model 2. Also, it is shown that dataset 1 and updated dataset 3 are used as the teacher dataset for model 3 obtained after model 2. Also, FIG. 3 shows that when the test is introduced, the test is performed using model 1, and after the 100th day of the test, the test is performed using models 1 and 2 in combination, and after the 300th day of the test, the test is performed using models 1 and 3 in combination. Of course, after the 300th day of the test, the test may be performed using models 1 to 3 in combination.
[0042] FIG. 4 is a diagram for explaining another example of old and new data sets serving as a teacher data set according to an embodiment.
[0043] In FIG. 4, compared to FIG. 3, an example of updated dataset 3 is not shown, while dataset 1 is the same as that shown in FIG. 3. Updated dataset 2 is a new dataset that includes 40 inspection images and labels associated with each of the 40 inspection images, which were hastily collected because the judgment accuracy of model 1 decreased, for example, on the 100th day after the introduction of the inspection. In the example of updated dataset 2 shown in FIG. 4, 20 inspection images are associated with labels indicating defective products, and 20 inspection images are associated with labels indicating non-defective products.
[0044] Thus, in the example shown in FIG. 4, the updated dataset 2 includes the inspection images obtained after the dataset 1. Also, in the example shown in FIG. 4, the dataset 1 is used as the teacher dataset of the model 1, and a part of the dataset 1 and the updated dataset 2 are used as the teacher dataset of the model 2. This is because, when the number of inspection images included in the updated dataset 2 is small, the number of inspection images included in the dataset 1 is reduced and then added to the teacher dataset of the model 2. Note that, in FIG. 4, an example is shown in which 10% (×0.1) of the dataset 1 is used as part of the dataset 1 as the teacher dataset of the model 2, but this is not limited to this and can be determined arbitrarily. For example, it is also possible to reduce the number of images other than the dent defect images in the dataset 1 before adding them. Also, in FIG. 4, similar to FIG. 3, when the inspection is introduced, the inspection is performed using the model 1, and after the 100th day of the inspection, the inspection is performed using both the models 1 and 2.
[0045] As described in FIG. 3 and FIG. 4, in the old and new datasets that serve as the teacher dataset, the number of inspection images included in the new dataset is smaller than the number of inspection images included in the old dataset. This relationship can also be utilized when multiple production lines for manufactured products are deployed. In other words, when the same manufactured product is produced on each of multiple production lines, it is difficult and requires a huge amount of man-hours to collect a number of inspection images from each of all production lines that can ensure sufficient accuracy in learning a machine learning model. For this reason, inspection images collected evenly from multiple production lines can be used as the old dataset, and inspection images collected from a specific production line that do not cover various variations can be used as the new dataset for Model 1 and Model 2. An example of this case will be described with reference to FIG. 5.
[0046] Fig. 5 is a diagram for explaining yet another example of old and new datasets serving as a teacher dataset according to an embodiment. Fig. 5 shows an example in which the relationship between the old and new datasets is used for inspection images that can be collected from a production line of manufactured products that are deployed in multiple stages.
[0047] Dataset 1, as the old dataset, includes a large number of inspection images collected evenly from multiple production lines and labels associated with each of the large number of inspection images. In the example of Dataset 1 shown in Fig. 5, N1 inspection images are associated with a label indicating a dent defect, N2 inspection images are associated with a label indicating a scratch defect, N3 inspection images are associated with a label indicating a non-defective product, and N4 inspection images are associated with a label indicating a chip defect. These images are collected, for example, from production line A, N5 from production line B, and N7 from production line C, so that N5, N6, and N7 are all equal to or greater than a predetermined number.
[0048] The updated dataset 2A includes, as a new dataset, a plurality of inspection images collected from, for example, a production line A and labels associated with each of the plurality of inspection images. In the example of the updated dataset 2A shown in FIG. 5, M 1a The inspection images have a label indicating that the product is good, 2a Each inspection image is associated with a label indicating that it is a defective product.
[0049] Similarly, the updated dataset 2B includes, as a new dataset, a plurality of inspection images collected from, for example, a production line B and labels associated with each of the plurality of inspection images. In the example of the updated dataset 2B shown in FIG. 5, M 1b The inspection images have a label indicating that the product is good, 2b Each inspection image is associated with a label indicating that it is a good product.
[0050] The updated data sets 2C and 2D (not shown) collected from the production lines C and D are similar, and therefore will not be described here.
[0051] In this way, data set 1 includes inspection images collected evenly from all production lines, while updated data sets 2A to 2D include inspection images obtained from a specific production line.
[0052] 5 shows that dataset 1 is used as the teacher dataset for model 1, and dataset 1 and updated dataset 2A are used as the teacher dataset for model 2A used in production line 2A. Similarly, FIG. 5 shows that dataset 1 and updated dataset 2B are used as the teacher dataset for model 2B used in production line 2B. The same is true for the teacher datasets for models 2C and 2D used in production lines 2C and 2D, so a description will be omitted. However, this is because the teacher dataset for model 2A, etc. used in a specific production line has a larger proportion of inspection images obtained in inspection of the specific production line compared to the teacher dataset for model 1.
[0053] In this way, when testing is introduced, model 1 is used in combination with each of models 2A to 2D to perform testing on the production line.
[0054] [1-3. Storage section 102] The storage unit 102 is configured with a HDD (Hard Disk Drive) or a memory, etc., and stores the old machine learning model and the new machine learning model learned by the learning unit 101. In this embodiment, the storage unit 102 stores, for example, model 1-1, model 1-2, . . ., model 2-1, model 2-2, . . . learned by the learning unit 101.
[0055] <Machine learning model> The machine learning model used in this embodiment will be described below.
[0056] Model 1-1, Model 1-2, ... are one or more types of machine learning models trained using the same dataset 1 as a teacher dataset. Model 2-1, Model 2-2, ... are one or more types of machine learning models trained using the same teacher dataset including at least a portion of dataset 1 and updated dataset 2.
[0057] Here, the type of machine learning model may be a supervised learning model such as logistic regression, support vector machine, random forest, neural network, DNN (Deep Neural Network), etc. The type of machine learning model may also include an autoencoder, or a good product model that is generated using good product data and outputs outliers as an anomaly degree. In other words, the machine learning model according to the present embodiment may be one or more of the types given as examples.
[0058] In addition, the memory unit 102 is not limited to storing the trained models 1-1, 1-2, ..., 2-1, 2-2, ..., etc. shown in Figure 1, but may also store models 3-1, 3-2, ..., and even model NM (N and M are positive integers).
[0059] Furthermore, when only one type of machine learning model is applied, the trained models 1-1, 1-2, ..., 2-1, 2-2, ... shown in Figure 1 are represented as Model 1 and Model 2. When multiple types of machine learning models are applied, in model NM, M indicates the type and N indicates whether the teacher datasets used are the same. In other words, models NM showing the same M value indicate that they are the same type of machine learning model, and models NM showing the same N value indicate that they have been trained using the same teacher dataset.
[0060] [1-4. Image acquisition unit 103] The image acquisition unit 103 acquires a target image.
[0061] In this embodiment, the image acquisition unit 103 acquires, as a target image, an inspection image obtained by inspecting an inspection image of a manufactured product on a production line, for example.
[0062] [1-5. Judgment section 104] The determination unit 104 outputs a determination result of the label of the target image obtained by using at least two of one of the one or more first models and one of the one or more second models for the target image acquired by the image acquisition unit 103. Here, the determination unit 104 may integrate the determination results of each of the one or more first models and the determination results of each of the one or more second models according to a rule set in advance, and output the result as a determination result of the label of the target image.
[0063] For example, the determination unit 104 uses (combines) two models, for example, Model 1-1 and Model 2-1, as the old machine learning model and the new machine learning model shown in Fig. 1. In this case, the determination unit 104 uses Model 1-1 and Model 2-1 to determine the quality (OK / NG) of the manufactured product shown in the target image as a label of the target image, and outputs a combined determination result according to a rule set in advance.
[0064] <Decision results integrated with pre-defined rules> Here, examples of preset rules and the determination results obtained by integrating these rules will be described.
[0065] Fig. 6A is a diagram for explaining an example of rule information according to an embodiment. Fig. 6A shows an example in which rules for reducing oversights in inspection are set in advance. Fig. 6A also shows a judgment result in which the judgment unit 104 judges the pass / fail (OK / NG) of a manufactured product shown in a target image using, for example, two models, model 1-1 and model 2-1, and these are integrated according to the rule for reducing oversights set in advance.
[0066] That is, FIG. 6A shows an example of a case where the integrated judgment result is determined to be OK only when the outputs (judgment results) of the model 1-1 and the model 2-1 are both OK.
[0067] Fig. 6B is a diagram for explaining another example of rule information according to the embodiment. Fig. 6B shows an example in which a rule for reducing overdetection in an inspection is set in advance. Fig. 6B also shows a judgment result in which the judgment unit 104 judges the pass / fail (OK / NG) of a manufactured product shown in a target image using, for example, two models, model 1-1 and model 2-1, and these are integrated according to the rule for reducing overdetection set in advance.
[0068] That is, FIG. 6B shows an example in which if the output (judgment result) of either model 1-1 or model 2-1 is OK, the integrated judgment result is determined to be OK.
[0069] Fig. 6C is a diagram for explaining yet another example of rule information according to the embodiment. Fig. 6C shows an example in which a rule is set in advance to reduce cases in which a non-defective product is judged as defective according to the rule shown in Fig. 6A. Fig. 6C also shows a judgment result in which the judgment unit 104 judges the quality (OK / NG) of a manufactured product shown in a target image using two models, for example, model 1-1 and model 2-1, and combines them according to the previously set rule.
[0070] In FIG. 6C, the integrated judgment result is OK only when the judgment results of both Model 1-1 and Model 2-1 are OK, whereas when the output (judgment result) of either Model 1-1 or Model 2-1 is NG, the result is marked gray, i.e., a human is required to confirm the result.
[0071] <Method 1 for selecting a combination of old and new machine learning models> In the above, the determination unit 104 has been described using an example in which two models, for example model 1-1 and model 2-1, are used (used together) as a combination of the old machine learning model and the new machine learning model shown in Figure 1, but examples of combinations are not limited to this.
[0072] For example, a combination of an old machine learning model and a new machine learning model may be selected by machine learning.
[0073] FIG. 7 is a diagram illustrating the concept of a method for selecting a combination of an old machine learning model and a new machine learning model by machine learning according to an embodiment.
[0074] 7(a) shows an example in which trained models 1-1, 1-2, and 1-3 can be used as old machine learning models, and trained models 2-1, 2-2, and 2-3 can be used as new machine learning models. And, FIG. 7(b) shows that the combination obtained by selecting the optimal combination from all these models is model 1-1, model 2-1, and model 2-3.
[0075] For example, logistic regression may be performed on a combination of two or three of the old and new machine learning models shown in (a) of Figure 7, and the best combination may be selected from the obtained judgment accuracy. Note that machine learning such as support vector machines, random forests, gradient boost, neural networks, and deep learning may be used instead of logistic regression.
[0076] Also, for example, the determination unit 104 may select an optimal combination of the old machine learning model and the new machine learning model using a machine learning model trained separately. More specifically, the determination unit 104 may further cause a third machine learning model trained using the outputs of each of the one or more first models and each of the one or more second models as input to select a combination consisting of at least one of the one or more first models and at least one of the one or more second models. Then, the determination unit 104 may output a determination result of the label of the target image obtained by using the selected combination.
[0077] Here, a method for learning a machine learning model for selecting the optimal combination of an old machine learning model and a new machine learning model (i.e., for selecting a combination) and a method for optimally combining an old machine learning model and a new machine learning model will be described with reference to Figures 8A, 8B, and 9.
[0078] Fig. 8A is a diagram showing an example of machine learning models to be combined and a validation dataset according to an embodiment. Fig. 8B is a diagram showing an example of the output of each machine learning model to be combined for each image included in the validation dataset. The validation dataset is composed of a part of the teacher data including, for example, the above-mentioned dataset 1 and updated dataset 1.
[0079] 8A shows trained models 1-1 and 1-2, which are old machine learning models, and trained models 2-1 and 2-2, which are new machine learning models, as examples of machine learning models to be combined. The validation dataset includes a plurality of images and is used to input to each machine learning model.
[0080] In other words, by inputting each image included in the validation dataset into each of the machine learning models to be combined, it is possible to obtain an output (judgment result) for each image from each machine learning model, as shown in Figure 8B.
[0081] Next, a machine learning model for selecting combinations is created that selects n items from the outputs of each machine learning model as shown in Fig. 8B and uses them as explanatory variables to predict labels such as 0 (OK) or 1 (NG). In the example of Fig. 8A, the combinations when n=2 are (model 1-1, model 2-1), (model 1-1, model 2-2), (model 1-2, model 2-1), and (model 1-2, model 2-2).
[0082] Note that the number of explanatory variables used may be reduced by utilizing regularization such as L1 regularization and L2 regularization. In this case, it is possible to create a machine learning model for combination selection by omitting outputs that are not used as a result. In addition, the machine learning model for combination selection may be created using a combination of only new machine learning models or old machine learning models, not limited to the case where the machine learning model for combination selection is created using a combination that includes at least one of the new machine learning model and the old machine learning model.
[0083] This makes it possible to evaluate the accuracy of the machine learning model created for selecting combinations and select combinations with high accuracy.
[0084] Therefore, the determination unit 104 can determine and select a combination of the old and new machine learning models with high accuracy by using a machine learning model for selecting a combination to which the outputs of the old and new machine learning models are input.
[0085] <Method 2 for selecting a combination of old and new machine learning models> Although the above describes a method of selecting a combination of an old machine learning model and a new machine learning model by machine learning, the present invention is not limited to this. Information such as the accuracy of the combination may be displayed on a GUI (Graphical User Interface) to allow the user to make a selection.
[0086] More specifically, the image assessment device 10 may include a display unit that displays the assessment accuracy of the label assessment result of the target image obtained by using a combination of at least one of one or more first models and at least one of one or more second models for the validation dataset. The validation dataset is, for example, composed of a part of the first teacher dataset and a part of each of one or more second teacher datasets.
[0087] That is, the image assessment device 10 may further include a display unit or a display device. Of course, the image assessment device 10 may simply be connected to an external display unit or display device. The image assessment device 10 may then display on the display unit or the display device a list of the accuracy of each of the old machine learning models, the accuracy of each of the new machine learning models, and the accuracy of the combination of the old machine learning model and the new machine learning model, allowing the user to select the optimal combination.
[0088] 9 is a diagram showing an example of a list for allowing a user according to an embodiment to select an optimal combination of an old machine learning model and a new machine learning model. In FIG. 9, the accuracy (%) is shown when two types of data sets, data set 1 and updated data set 2, are used as examples of validation data sets. In addition, the respective determination speeds (takt times) are shown as an example of a criterion for selecting a combination.
[0089] Users can look at a list like the one in Figure 9 and select the optimal combination of the old and new machine learning models.
[0090] 9 may be able to perform simple operations such as searching, narrowing down, or sorting. In this case, the determination unit 104, rather than the user, may sort the above list by accuracy for data set 1 and select a combination that has a predetermined accuracy or higher, such as an accuracy of 90% or higher for data set 1, and has the best accuracy for updated data set 2. The determination unit 104 may also sort the above list by determination speed (takt time) and select a combination with the best accuracy within a predetermined determination speed.
[0091] [2. Operation of image assessment device 10] An example of the operation of the image assessment device 10 configured as above will be described below.
[0092] FIG. 10 is a flowchart showing an outline of the operation of the image assessment device 10 in this embodiment.
[0093] First, the image assessment device 10 uses a dataset 1 as teacher data to learn and obtain one or more models 1, and also uses a dataset including an updated dataset 2 as teacher data to learn and obtain one or more models 2 (S1). More specifically, the learning unit 101 of the image assessment device 10 learns one or more types of machine learning models using a first teacher dataset including a plurality of first images and a first label associated with each of the plurality of first images to obtain one or more first models. In addition, the learning unit 101 learns one or more types of machine learning models using one or more second teacher datasets to obtain one or more second models. Here, each of the one or more second teacher datasets includes a plurality of second images different from the plurality of first images, a second label associated with each of the plurality of second images, and at least a portion of the first teacher dataset.
[0094] Next, the image assessment device 10 acquires a target image (S2). In this embodiment, the image acquisition unit 103 acquires, as the target image, an inspection image obtained by inspecting an inspection image of a manufactured product on a production line, for example.
[0095] Next, the image judgment device 10 outputs a judgment result of the label of the target image obtained by using at least two of one of one or more models 1 and one of one or more models 2 for the target image (S3). More specifically, the judgment unit 104 of the image judgment device 10 outputs a judgment result of the label of the target image obtained by using at least two of one of one or more first models and one of one or more second models for the target image acquired by the image acquisition unit 103. Note that the judgment unit 104 may integrate the judgment results of one or more first models and the judgment results of one or more second models according to a rule set in advance, and output the result as a judgment result of the label of the target image. In this way, in this embodiment, the image judgment device 10 can judge the quality (OK / NG) of the manufactured product shown in the target image by using at least two machine learning models in combination.
[0096] [3. Effects, etc.] For example, if model 1 is trained on an old dataset before the introduction of the test in order to carry out the test, but the judgment accuracy of model 1 declines after the test is carried out due to various reasons such as maintenance, model 2 is trained on a new dataset that is hastily collected. The test images in the new dataset in which the judgment accuracy of model 1 has declined have differences that are difficult for people to notice at a glance compared to the test images in the old dataset.
[0097] In the comparative example, inspection is performed using only model 2, which is obtained by retraining model 1 with a new dataset. However, there are cases where the accuracy is lower than that of model 1 for inspection images similar to the old dataset.
[0098] Therefore, according to this embodiment, since inspection is performed by using Model 1 and Model 2 in combination, high accuracy can be maintained for inspection images similar to the old data set and inspection images similar to the new data set.
[0099] In this way, according to this embodiment, even if the accuracy of image judgment using a machine learning model decreases, the accuracy can be improved.
[0100] The accuracy is not limited to the accuracy rate, but may be at least one combination of the precision rate, the recall rate, the F value calculated by the harmonic mean of the precision rate and the recall rate, and the accuracy rate.
[0101] 11 is a diagram showing teacher data 1 and teacher data 2 according to an embodiment. Teacher data 1 is, for example, the above-mentioned data set 1. In the following, teacher data 1 is, for example, 100,000 inspection images for inspecting a wafer on which a semiconductor circuit is formed, and is assumed to be composed of data set 1 including inspection images collected before the introduction of inspection using a machine learning model. In this case, model 1 is a machine learning model trained using data set 1 as teacher data 1.
[0102] Moreover, the teacher data 2 is composed of the most recent data and a part of the teacher data 1. The model 2 is a machine learning model trained using the teacher data 2. The most recent data is, for example, the above-mentioned updated data set, and is composed of inspection images collected after the introduction of inspection using the machine learning model (after the introduction of the inspection). Here, the most recent data is assumed to be inspection images collected after a certain period of time, such as the 100th day after the introduction of the inspection, when the judgment accuracy has decreased. Note that the judgment accuracy decreases due to various factors, such as maintenance of the inspection device performed after the introduction of the inspection.
[0103] Fig. 12 is a diagram qualitatively showing the accuracy of the machine learning model for the learning data (dataset 1) and the most recent data according to the embodiment. Fig. 12 calculates the accuracy of Model 1 and Model 2 using Dataset 1 used as Teacher Data 1 and the most recent data included in Teacher Data 2 as validation data sets, respectively.
[0104] As shown in FIG. 12, the accuracy of model 1 for the most recent data is lower, whereas the accuracy of model 2 for the most recent data is higher. However, since model 2 is trained using training data 2 in which the proportion of data set 1 is lower than that of training data 1, the accuracy of model 2 for data set 1 may be slightly lower than that of model 1. In other words, the accuracy of model 1 for the training data, i.e., data set 1, may be high as shown in FIG. 12, whereas model 2 may be moderate and lower than that of model 1. Therefore, if the most recent data at the time when the judgment accuracy has decreased is collected and model 1 is retrained using training data 2 added to the training data, the accuracy of the most recent data may be improved, but the accuracy of the most recent data may be lower. Since the difference that has occurred in the most recent data cannot be seen at a glance by a person, the cause of the occurrence cannot be identified, and it is not possible to predict whether the cause will be removed in the future. Therefore, it is necessary to maintain high judgment accuracy not only for the most recent data but also for data set 1.
[0105] Therefore, in this embodiment, for example, by using (combining) Model 1 and Model 2 together, it is possible to maintain high accuracy for the learning data, that is, the accuracy for Data Set 1 and the accuracy for the most recent data.
[0106] Next, we verified the extent to which accuracy could be improved by training Model 2 with teacher data 2, which was training data 1 (training data) containing a large number of test images plus recent data, which was a small number of test images. We will explain the results of this.
[0107] FIG. 13 shows an example of training data used for learning Model 1 and Model 2 shown in FIG.
[0108] In Fig. 13, teacher data 1a is an example of the teacher data 1, and teacher data 2a is an example of the teacher data 2. In the example shown in Fig. 13, teacher data 1a is composed of 2444+1942=4386 inspection images as data determined to be good products, and 1237+968=2205 inspection images as data determined to be defective products. Teacher data 2a is obtained by adding 269+71=340 inspection images as data determined to be good products and 57+16=73 inspection images as data determined to be defective products to teacher data 1a.
[0109] In addition, as verification data for verifying the judgment accuracy of the trained models 1 and 2, the most recent data was used that was used in the above-mentioned teacher data 2a.
[0110] As a result, Model 1 overdetected 1,967 of the 7,481 inspection images of good products included in the validation data, i.e., determined them to be defective. On the other hand, Model 2 overdetected 155 of the 7,481 inspection images of good products included in the validation data, i.e., determined them to be defective.
[0111] In other words, for the most recent data that was not used for training data 2a, it was confirmed that the overdetection rate of Model 1 was 26.3%, while the overdetection rate of Model 2 was 2.07%, a significant improvement.
[0112] In this way, it can be seen that accuracy is improved by training model 2 with training data 2, which is training data 1 (training data) containing a large number of test images plus recent data, which is a small number of test images.
[0113] (Other embodiments) The image assessment device 10 according to the present disclosure has been described above based on each embodiment, but the present disclosure is not limited to these embodiments. As long as it does not deviate from the gist of the present disclosure, various modifications conceived by a person skilled in the art to each embodiment and other forms constructed by combining some of the components in each embodiment are also included within the scope of the present disclosure.
[0114] (1) For example, although it has been described that the old dataset used for the first training data, i.e., dataset 1 itself, is not updated, this is not limited thereto. The model 1 may be retrained by using dataset 1 updated using updated dataset 2 hastily collected as the first training data at the next retraining.
[0115] (2) In the above embodiment, the updated dataset 2 is created by adding data to the first training data, but this is not limited to the above. If there is a sufficient amount of dataset 2 collected in a hurry, this dataset 2 itself may be used as the updated dataset 2 to train the model 2, instead of adding it to the first training data.
[0116] (3) In the above embodiment, the judgment unit 104 constituting the image judgment device 10 outputs the judgment result of the label of the target image by using the machine learning model in combination with the target image, but this is not limited to this. Before or after judging the label of the target image by using the machine learning model in combination, a rule judgment may be further performed. As the rule judgment, a judgment to detect whether or not the target image includes a judgment object may be considered. In this case, if the target image does not include the judgment object, the judgment unit 104 may not perform the judgment to the target image by using the machine learning model in combination. In addition, as the judgment to detect whether or not the target image includes a judgment object, it may be judged whether or not the judgment object is reflected in the target image, or it may be judged whether or not information such as the luminance of the target image is included in the applicable range in which the judgment may be performed.
[0117] The following forms may also be included within the scope of one or more aspects of the present disclosure.
[0118] (4) Some of the components constituting the image judgment device 10 may be a computer system composed of a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, etc. A computer program is stored in the RAM or the hard disk unit. The microprocessor achieves its function by operating according to the computer program. Here, the computer program is composed of a combination of multiple instruction codes that indicate commands to a computer to achieve a predetermined function.
[0119] (5) Some of the components constituting the image assessment device 10 may be composed of one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on one chip, and specifically, is a computer system including a microprocessor, a ROM, a RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating in accordance with the computer program.
[0120] (6) Some of the components constituting the image assessment device 10 may be composed of an IC card or a standalone module that can be attached to each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may include the ultra-multifunctional LSI. The microprocessor operates according to a computer program, causing the IC card or the module to achieve its function. The IC card or the module may be tamper-resistant.
[0121] (7) In addition, some of the components constituting the image judgment device 10 may be the computer program or the digital signal recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. Also, it may be the digital signal recorded on these recording media.
[0122] In addition, some of the components constituting the image judgment device 10 may transmit the computer program or the digital signal via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc.
[0123] (8) The present disclosure may be embodied as any of the methods described above. It may also be embodied as a computer program for implementing these methods by a computer, or as a digital signal comprising the computer program.
[0124] (9) The present disclosure may also be a computer system having a microprocessor and a memory, the memory storing the above-mentioned computer program, and the microprocessor operating in accordance with the computer program.
[0125] (10) The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network, etc.
[0126] (11) The above-described embodiments and modifications may be combined with each other. [Industrial Applicability]
[0127] The present disclosure can be used in image assessment devices, image assessment methods, and programs that use machine learning models for determining pass / fail products in an inspection process. [Explanation of symbols]
[0128] 10. Image Judgment Device 101 Learning Department 102 Storage section 103 Image acquisition unit 104 Judgment section 1000 Computers 1001 Input Device 1002 Output device 1003 CPU 1004 Internal Storage 1005 RAM 1007 Reading device 1008 Transmitting and receiving device 1009 Bus
Claims
1. a learning unit that obtains one or more first models by learning one or more types of machine learning models using a first teacher dataset including a plurality of first images and a first label associated with each of the plurality of first images, and obtains one or more second models by learning one or more types of machine learning models using one or more second teacher datasets including a plurality of second images different from the plurality of first images, a second label associated with each of the plurality of second images, and at least a portion of the first teacher dataset; an image acquisition unit that acquires a target image; a determination unit that outputs a determination result of a label of the target image obtained by using at least two of the one or more first models and the one or more second models for the target image acquired by the image acquisition unit, Image assessment device.
2. Each of the plurality of first images and each of the plurality of second images are inspection images of a manufactured product during a predetermined period of time. The image assessment device according to claim 1 .
3. each of the plurality of first images is an inspection image obtained during a first period of the predetermined period; the plurality of second images are inspection images obtained after the first period of the predetermined period; The image assessment device according to claim 2.
4. One second teacher data set among the one or more second teacher data sets has a larger proportion of inspection images at a specific date and time than the first teacher data set. The image assessment device according to claim 2.
5. One second teacher data set among the one or more second teacher data sets has a larger proportion of inspection images obtained by inspection of a specific production line compared to the first teacher data set. The image assessment device according to claim 2.
6. The determination unit further a third machine learning model that has been trained using the outputs of each of the one or more first models and the outputs of each of the one or more second models as inputs, and selects a combination consisting of at least one of the one or more first models and at least one of the one or more second models; outputting a label determination result of the target image obtained by using the selected combination; The image assessment device according to any one of claims 1 to 5.
7. The determination unit integrates the determination results of the one or more first models and the determination results of the one or more second models in accordance with a preset rule, and outputs the integrated result as a determination result of the label of the target image. The image assessment device according to any one of claims 1 to 5.
8. moreover, a display unit that displays a determination accuracy of a determination result of a label of the target image obtained by using a combination of at least one of the one or more first models and at least one of the one or more second models for a validation dataset including a part of the first teacher dataset and a part of each of the one or more second teacher datasets; The image assessment device according to any one of claims 1 to 7.
9. a learning step of obtaining one or more first models by learning one or more types of machine learning models using a first teacher data set including a plurality of first images and a first label associated with each of the plurality of first images, and obtaining one or more second models by learning one or more types of machine learning models using one or more second teacher data sets including a plurality of second images different from the plurality of first images, a second label associated with each of the plurality of second images, and at least a portion of the first teacher data set; an image acquisition step of acquiring a target image; a determination step of outputting a determination result of a label of the target image obtained by using at least two of the one or more first models and the one or more second models for the target image acquired in the image acquisition step, Image assessment method.
10. a learning step of obtaining one or more first models by learning one or more types of machine learning models using a first teacher data set including a plurality of first images and a first label associated with each of the plurality of first images, and obtaining one or more second models by learning one or more types of machine learning models using one or more second teacher data sets including a plurality of second images different from the plurality of first images, a second label associated with each of the plurality of second images, and at least a portion of the first teacher data set; an image acquisition step of acquiring a target image; a determination step of outputting a determination result of a label of the target image obtained by using at least two of the one or more first models and one or more second models for the target image acquired in the image acquisition step; A program that a computer runs.
Citation Information
Patent Citations
Image recognition device
JP2012068965A
Visual inspection device
JP2019095217A
JPP6687962B
Device and method of data recognition model construction, and data recognition devicce
KR1020170061016A
Inspection device, image discriminating device, discriminating device, inspection method, and inspection program
WO2019176990A1
