Training device and training method

Abstract

The present invention is provided with: an image acquisition unit (1101) for training that acquires an image data set to be inspected; an existing feature amount acquisition unit (1102) that acquires a feature amount of an image data set that is an executed task; a feature amount extraction unit (1103) that extracts a feature amount of an image data set to be inspected; a feature amount comparison unit (1104) that calculates the similarity between the feature amount of the image data set to be inspected and the feature amount of the image data set that is the executed task; a model selection unit (1105) that selects, on the basis of the similarity in the feature amount, at least one trained model corresponding to an image data set, which is an executed task, having a high similarity of the feature amount with respect to the image data set to be inspected among image data sets that are the executed tasks; a model training unit (1106) that generates a new trained model that is a distribution of non-defective products by inputting the image data set to be inspected to the trained model selected by the model selection unit (1105); a model evaluation unit (1107) that determines whether an evaluation for the new trained model is equal to or greater than a threshold value; and a model output unit (1108) that outputs information indicating the new trained model in which it is determined that the evaluation is equal to or greater than the threshold value.

Description

Learning device and learning method

[0001] The present disclosure relates to a learning device and a learning method for obtaining a trained model.

[0002] When applying AI-based automated visual inspection to a product at a production site, it is necessary to collect an image dataset of the product to be inspected as training data to realize the desired functionality. Therefore, there has been a demand for small-data learning using past models, such as transfer learning.

[0003] On the other hand, when multiple past models exist, in order to obtain the optimal model for the image dataset to be inspected, it is necessary to train and evaluate all past models and select the model with the highest accuracy. Therefore, when the number of past models is enormous, a huge training cost is incurred to obtain the optimal model.

[0004] Therefore, a method has been proposed in which a past model is selected based on the characteristics of the image dataset, and transfer learning is performed.

[0005] An example of the transfer learning technique described above is the technique disclosed in Patent Document 1. This technique combines multiple AI models to improve the accuracy of product inspection for determining whether a product is good or bad. This technique first inputs inspection data into multiple past models, and selects past models whose intermediate or final outputs have a correlation below a certain level. The selected models are then used to create multiple hybrid model candidates. The hybrid model with the highest accuracy is then adopted. In this way, this technique inputs inspection data into multiple hybrid models, and the models learn to correctly determine labels based on the weighted sum of the outputs of each model.

[0006] International Publication No. 2022 / 215559

[0007] As mentioned above, existing transfer learning techniques select multiple pairs of models that have low correlation between intermediate or final outputs when test data (image datasets to be tested) is input to past models. In this case, although the models are independent, it may not be possible to select an appropriate model for the test data.

[0008] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a learning device that, unlike conventional devices, makes it possible to obtain a trained model that is appropriate for the image dataset to be inspected.

[0009] The learning device according to the present disclosure includes a training image acquisition unit that acquires an image dataset to be inspected, an existing feature acquisition unit that acquires features of the image dataset that is an executed task, the features being based on outputs from a plurality of intermediate layers in a trained model that corresponds to the image dataset that is the executed task, a feature extraction unit that extracts features of the image dataset that is an inspection target, the features being based on outputs from a plurality of intermediate layers in the trained model, based on the trained model that corresponds to the image dataset that is the executed task and the image dataset that is the inspection target acquired by the training image acquisition unit, a feature comparison unit that calculates similarity of features based on the features of the image dataset that is the inspection target extracted by the feature extraction unit and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit, and The system is characterized by comprising: a model selection unit that selects, from among image datasets that are executed tasks, one or more trained models that correspond to an image dataset that is an executed task and has high similarity in features to an image dataset that is an inspection target acquired by the training image acquisition unit, based on the similarity of features; a model training unit that generates a new trained model that is a non-defective product distribution by inputting the image dataset that is the inspection target acquired by the training image acquisition unit and the trained model selected by the model selection unit; a model evaluation unit that determines, based on the new trained model generated by the model training unit, whether the evaluation of the new trained model is equal to or greater than a threshold; and a model output unit that outputs information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold, based on the determination result by the model evaluation unit.

[0010] According to the present disclosure, with the above-described configuration, it is possible to obtain a trained model that is more appropriate for the image dataset to be inspected than in the past.

[0011] 14A and 14B are block diagrams showing an example of the configuration of a learning system according to embodiment 1. FIG. 14A is a block diagram showing an example of the configuration of a learning device according to embodiment 1. FIG. 14B is a flowchart showing an example of the operation of the learning device according to embodiment 1. FIG. 14C is a diagram showing an overview of the overall operation of the learning device according to embodiment 1. FIG. 14D is a diagram showing an overview of the operation of the learning device according to embodiment 1. FIG. 14E is a diagram showing an overview of the operation of the learning device according to embodiment 1. FIG. 14F is a diagram showing an example of the configuration of a learning device according to embodiment 2. FIG. 14F is a flowchart showing an example of the operation of the learning device according to embodiment 2. FIG. 14D is a diagram showing an overview of the operation of the learning device according to embodiment 2. FIG. 14E is a diagram showing an overview of the operation of the learning device according to embodiment 2. FIG. 14F is a diagram showing an overview of the operation of the learning device according to embodiment 2. FIG. 14B is a block diagram showing an example of the hardware configuration of the learning devices according to embodiments 1 and 2.

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings. Embodiment 1. Fig. 1 is a diagram showing an example of the configuration of a learning system 1 according to embodiment 1. As shown in Fig. 1, the learning system 1 includes a learning device 11, an operation input device 12, a storage device 13, and a display output device 14.

[0013] The learning device 11 outputs information indicating a new trained model based on the image dataset that is the inspection target input via the operation input device 12, the feature quantities of the image dataset that is the executed task indicated by the information stored in the storage device 13, and the trained model corresponding to the image dataset that is the executed task. An example configuration of this learning device 11 will be described later.

[0014] The operation input device 12 is a device that accepts operations by a user. For example, the operation input device 12 outputs an image dataset that is an inspection target input by the user to the learning device 11.

[0015] The storage device 13 stores various types of data handled by the learning system 1. For example, the storage device 13 stores information indicating the feature quantities of an image dataset that is an executed task and a trained model corresponding to the image dataset that is the executed task. Here, examples of the storage device 13 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0016] The display output device 14 displays information indicating the new trained model output by the training device 11.

[0017] Next, an example of the configuration of the learning device 11 will be described with reference to Fig. 2. As shown in Fig. 2, the learning device 11 includes a learning image acquisition unit 1101, an existing feature acquisition unit 1102, a feature extraction unit 1103, a feature comparison unit 1104, a model selection unit 1105, a model learning unit 1106, a model evaluation unit 1107, and a model output unit 1108.

[0018] The learning image acquisition unit 1101 acquires an image data set to be inspected. At this time, the learning image acquisition unit 1101 acquires the image data set to be inspected that is input via the operation input device 12.

[0019] The existing feature acquisition unit 1102 acquires the feature amounts of the image dataset that is the executed task. At this time, the existing feature acquisition unit 1102 acquires the feature amounts of the image dataset that is the executed task indicated by the information stored in the storage device 13. Furthermore, the feature amounts of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102 are feature amounts based on outputs from multiple intermediate layers in the trained model that corresponds to the image dataset that is the executed task.

[0020] The feature extraction unit 1103 extracts features of the image dataset that is the inspection target, based on a trained model corresponding to the image dataset that is the executed task, and the image dataset that is the inspection target acquired by the training image acquisition unit 1101. The features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 are features based on outputs from the multiple intermediate layers in the trained model.

[0021] In this case, for example, the feature extraction unit 1103 may extract, as features, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed. Furthermore, for example, the feature extraction unit 1103 may extract, as features, a group of vectors obtained by averaging, for each channel, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed. Furthermore, for example, the feature extraction unit 1103 may extract, as features, a vector obtained by averaging, as a whole, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed.

[0022] At this time, the feature extraction unit 1103 obtains a trained model corresponding to the image dataset, which is the executed task indicated by the information stored in the storage device 13, and then performs the above-mentioned feature extraction.

[0023] The feature comparison unit 1104 calculates the similarity of features based on the features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102. Here, in the learning device 11 according to embodiment 1, the feature comparison unit 1104 calculates the similarity between the overall features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the overall features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102.

[0024] In this case, for example, the feature comparison unit 1104 may calculate the similarity between the overall feature of the image dataset to be inspected and the overall feature of the image dataset to be executed, using a distribution difference. Also, for example, the feature comparison unit 1104 may calculate the similarity between the overall feature of the image dataset to be inspected and the overall feature of the image dataset to be executed, using a common area between the distributions.

[0025] Based on the feature similarity calculated by the feature comparison unit 1104, the model selection unit 1105 selects, from the image datasets that are executed tasks, one or more trained models corresponding to the image datasets that are executed tasks and have high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101. In this case, for example, from the image datasets that are executed tasks, the model selection unit 1105 selects one or more trained models corresponding to the image datasets in the order of image datasets that are executed tasks and have high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101.

[0026] The model learning unit 1106 generates a new trained model by inputting the image data set to be inspected into the trained model based on the image data set to be inspected acquired by the training image acquisition unit 1101 and the trained model selected by the model selection unit 1105. Note that the new trained model generated by the model learning unit 1106 is a non-defective product distribution.

[0027] In this case, for example, the model learning unit 1106 inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and combines all of the outputs from each intermediate layer to generate a new trained model.

[0028] The model evaluation unit 1107 determines whether the accuracy of the new trained model is equal to or greater than a threshold, based on the new trained model generated by the model training unit 1106. The threshold can be set as appropriate to a value for evaluating the trained model.

[0029] The model output unit 1108 outputs to the outside information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold based on the determination result by the model evaluation unit 1107.

[0030] Next, an example of the operation of the learning device 11 according to the first embodiment shown in FIGS. 1 and 2 will be described with reference to FIGS. 3 to 8. In the learning device 11 according to the first embodiment, as shown in FIG. 4, for example, first, feature quantities are extracted from an image dataset that is an inspection target (new task) (step ST1). In the example of FIG. 4, the image dataset that is an inspection target is image dataset X (bottle mouth), and the feature quantities of image dataset X are feature quantities X1 to X3. The learning device 11 also acquires feature quantities of image datasets that are executed tasks (step ST2). In the example of FIG. 4, the image datasets that are executed tasks include three types of image datasets: image dataset A (nut), image dataset B (cable cross section), and image dataset C (skin surface). Furthermore, feature quantities and trained models are associated with the image datasets that are executed tasks. In the example of FIG. 4, feature quantity A and trained model A are associated with image dataset A, feature quantity B and trained model B are associated with image dataset B, and feature quantity C and trained model C are associated with image dataset C. Information indicating the features and trained models corresponding to the image datasets that are the executed tasks is stored in advance in the storage device 13. The learning device 11 then finds, from the image datasets that are the executed tasks, an image dataset whose features are similar to the image dataset to be inspected, and selects a trained model corresponding to the found image dataset (step ST3). The example in FIG. 4 illustrates a case in which feature A is similar to feature X1 and feature B is similar to feature X2, and the learning device 11 selects trained model A corresponding to image dataset A and trained model B corresponding to image dataset B. The learning device 11 then generates a new trained model (non-defective product distribution) based on the selected trained models (step ST4). The example in FIG. 4 illustrates a case in which the learning device 11 generates a new trained model X based on the selected trained models A and B.

[0031] Furthermore, as shown in FIG. 5, for example, the learning device 11 according to the first embodiment is assumed to extract features using multiple intermediate layers from among the intermediate layers of a trained model to generate a new trained model (non-defective product distribution). The example in FIG. 5 illustrates a case where the multiple intermediate layers are three layers (layer a, layer b, and layer c). In this case, the learning device 11 according to the first embodiment utilizes a previous model (a trained model corresponding to an image dataset that is an executed task) to extract features. Furthermore, given the premise of generating a non-defective product distribution, it is desirable to utilize multiple intermediate layers useful for feature extraction from the previous model.

[0032] Therefore, as shown in Figures 6 and 7, the learning device 11 according to the first embodiment uses features based on the outputs of the multiple intermediate layers and selects one or more past models with high overall feature similarity. The example in Figure 6 illustrates a case in which the learning device 11 compares feature X1 of image dataset X with feature A of image dataset A. In this case, the learning device 11 inputs image dataset X into trained model A corresponding to image dataset A to extract feature X1. Note that the example in Figure 6 illustrates a case in which the similarity between feature X1 and feature A is high. Furthermore, the example in Figure 7 illustrates a case in which the learning device 11 compares feature X3 of image dataset X with feature C of image dataset C. In this case, the learning device 11 inputs image dataset X into trained model C corresponding to image dataset C to extract feature X3. Note that the example in Figure 7 illustrates a case in which the similarity between feature X3 and feature C is low. Then, as shown in Fig. 8, for example, the learning device 11 inputs the image dataset to be inspected into the selected past model and combines all of the outputs from each intermediate layer to generate a new trained model. The example in Fig. 8 shows a case where the learning device 11 selects trained model A and trained model B and generates a new trained model X.

[0033] 1 and 2, the learning image acquisition unit 1101 first acquires an image data set to be inspected (step ST101), as shown in Fig. 3. At this time, the learning image acquisition unit 1101 acquires the image data set to be inspected, which is input via the operation input device 12.

[0034] In the examples of FIGS. 4 to 8, the learning image acquisition unit 1101 acquires an image data set X.

[0035] Furthermore, the existing feature acquisition unit 1102 acquires the feature of the image dataset that is the executed task (step ST102). At this time, the existing feature acquisition unit 1102 acquires the feature of the image dataset that is the executed task indicated by the information stored in the storage device 13. Furthermore, the feature of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102 is the feature based on outputs from multiple intermediate layers in the trained model corresponding to the image dataset that is the executed task.

[0036] 4 to 8, the existing feature acquisition unit 1102 acquires feature A of image dataset A, feature B of image dataset B, and feature C of image dataset C. In the examples of Fig. 6 and Fig. 7, the feature acquired by the existing feature acquisition unit 1102 is feature based on the outputs of layers a, b, and c of the past model.

[0037] Next, the feature extraction unit 1103 extracts features of the image dataset that is the inspection target based on the trained model corresponding to the image dataset that is the executed task and the image dataset that is the inspection target acquired by the training image acquisition unit 1101 (step ST103). The features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 are features based on outputs from the multiple intermediate layers in the trained model.

[0038] In this case, for example, the feature extraction unit 1103 may extract, as features, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed. Furthermore, for example, the feature extraction unit 1103 may extract, as features, a group of vectors obtained by averaging, for each channel, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed. Furthermore, for example, the feature extraction unit 1103 may extract, as features, a vector obtained by averaging, as a whole, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed.

[0039] At this time, the feature extraction unit 1103 obtains a trained model corresponding to the image dataset, which is the executed task indicated by the information stored in the storage device 13, and then performs the above-mentioned feature extraction.

[0040] In the examples of FIGS. 4 to 8, the feature extraction unit 1103 extracts features X1 to X3. Feature X1 is a feature extracted by inputting image dataset X to trained model A. Feature X2 is a feature extracted by inputting image dataset X to trained model B. Feature X3 is a feature extracted by inputting image dataset X to trained model C. In the examples of FIGS. 6 and 7, the features extracted by the feature extraction unit 1103 are features based on the outputs of layers a, b, and c of the past model.

[0041] Next, the feature comparison unit 1104 calculates the similarity of the features based on the features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102 (step ST104). Here, in the learning device 11 according to embodiment 1, the feature comparison unit 1104 calculates the similarity between the overall features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the overall features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102.

[0042] In this case, for example, the feature comparison unit 1104 may calculate the similarity between the overall feature amounts of the image dataset to be inspected and the overall feature amounts of the image dataset to be executed using a distribution difference. In this case, the feature comparison unit 1104 may employ, for example, the Fréchet inception distance, the KL divergence, the JS divergence, or the Mahalanobis distance as the distribution difference. Furthermore, for example, the feature comparison unit 1104 may calculate the similarity between the overall feature amounts of the image dataset to be inspected and the overall feature amounts of the image dataset to be executed using a common area between the distributions. In this case, the feature comparison unit 1104 may employ, for example, histogram intersection as the common area between the distributions.

[0043] Next, based on the feature similarity calculated by the feature comparison unit 1104, the model selection unit 1105 selects one or more trained models corresponding to the image datasets that are executed tasks and have high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from the image datasets that are executed tasks (step ST105). At this time, for example, the model selection unit 1105 selects one or more trained models corresponding to the image datasets in the order of high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from the image datasets that are executed tasks.

[0044] 4 to 8, the similarity between feature X1 (the overall feature for layers a, b, and c) and feature A (the overall feature for layers a, b, and c), and between feature X2 (the overall feature for layers a, b, and c) and feature B (the overall feature for layers a, b, and c), is high, so the feature extraction unit 1103 selects trained model A and trained model B. On the other hand, in the examples of FIGS. 4 to 8, the similarity between feature X3 (the overall feature for layers a, b, and c) and feature C (the overall feature for layers a, b, and c) is low, so the feature extraction unit 1103 does not select trained model C.

[0045] Next, the model learning unit 1106 generates a new trained model by inputting the image data set of the inspection target acquired by the learning image acquisition unit 1101 and the trained model selected by the model selection unit 1105 into the trained model (step ST106). Note that the new trained model generated by the model learning unit 1106 is a non-defective product distribution.

[0046] In this case, for example, the model learning unit 1106 inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and combines all of the outputs from each intermediate layer to generate a new trained model.

[0047] 4 to 8, the model training unit 1106 generates a new trained model X from trained model A and trained model B. In this case, the model training unit 1106 inputs image dataset X to trained model A and trained model B, respectively, and generates a new trained model X (non-defective product distribution) by combining all of the outputs from the intermediate layers (layers a, b, and c) of trained model A and the outputs from the intermediate layers (layers a, b, and c) of trained model B.

[0048] Next, the model evaluation unit 1107 determines whether the accuracy of the new trained model generated by the model training unit 1106 is equal to or greater than a threshold (step ST107). Note that the threshold can be set as appropriate to a value for evaluating the trained model.

[0049] Next, the model output unit 1108 outputs to the outside information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold based on the determination result by the model evaluation unit 1107 (step ST108).

[0050] Here, in the conventional technology, when selecting from multiple past models, it may be impossible to select a trained model appropriate for the image dataset to be inspected. In contrast, the learning device 11 according to the first embodiment compares the features of the image dataset to be inspected (features based on the intermediate model output) with the features of the image dataset that is the executed task (features based on the intermediate model output), thereby enabling the selection of a trained model using data similar to the image dataset to be inspected, with consideration given to generating a new trained model for anomaly detection, and is expected to achieve higher accuracy.

[0051] As described above, according to the first embodiment, the learning device 11 includes a training image acquisition unit 1101 that acquires an image dataset to be inspected; an existing feature acquisition unit 1102 that acquires features of the image dataset that is an executed task, the features being based on outputs from a plurality of intermediate layers in a trained model corresponding to the image dataset that is the executed task; a feature extraction unit 1103 that extracts features of the image dataset that is an inspection target, the features being based on outputs from a plurality of intermediate layers in the trained model, based on the trained model corresponding to the image dataset that is the executed task and the image dataset that is the inspection target acquired by the training image acquisition unit 1101; a feature comparison unit 1104 that calculates similarities of features based on the features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102; The apparatus includes a model selection unit 1105 that selects one or more trained models corresponding to an image dataset that is an executed task and has high feature similarity with the image dataset that is an inspection target acquired by the training image acquisition unit 1101, from among the image datasets that are executed tasks, based on the feature similarity calculated by the trainer 1104; a model training unit 1106 that generates a new trained model that is a non-defective product distribution by inputting the image dataset that is the inspection target acquired by the training image acquisition unit 1101 and the trained model selected by the model selection unit 1105; a model evaluation unit 1107 that determines, based on the new trained model generated by the model training unit 1106, whether the evaluation of the new trained model is equal to or greater than a threshold; and a model output unit 1108 that outputs information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold, based on the determination result by the model evaluation unit 1107.In particular, in the learning device 11 according to the first embodiment, the feature comparison unit 1104 calculates the similarity between the overall feature of the image dataset to be inspected, extracted by the feature extraction unit 1103, and the overall feature of the image dataset to be executed, acquired by the existing feature acquisition unit 1102, and selects a trained model. This enables the learning device 11 according to the first embodiment to obtain a trained model that is more appropriate for the image dataset to be inspected than conventional methods.

[0052] Embodiment 2. The learning device 11 according to embodiment 1 calculates the similarity between the overall feature amount of an image dataset to be inspected and the overall feature amount of an image dataset that is an executed task, and selects a trained model. In contrast, the learning device 11 according to embodiment 2 calculates the similarity between the feature amount of each intermediate layer of an image dataset to be inspected and the feature amount of each intermediate layer of an image dataset that is an executed task, and selects a trained model (intermediate layer).

[0053] Figure 9 is a diagram showing an example configuration of a learning device 11 according to embodiment 2. In the learning device 11 according to embodiment 2 shown in Figure 9, the feature comparison unit 1104 is changed to a feature comparison unit 1104b, and the model learning unit 1106 is changed to a model learning unit 1106b, compared to the learning device 11 according to embodiment 1 shown in Figure 2. The other example configuration of the learning device 11 according to embodiment 2 shown in Figure 9 is the same as the example configuration of the learning device 11 according to embodiment 1 shown in Figure 2, and the same reference numerals are used, and only the different parts will be described.

[0054] The feature comparison unit 1104b calculates the similarity of features based on the features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102. Here, in the learning device 11 according to the second embodiment, the feature comparison unit 1104b calculates the similarity between the features of each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the overall features of each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102.

[0055] In this case, for example, the feature comparing unit 1104b may calculate the similarity between the feature for each intermediate layer of the image dataset to be inspected and the feature for each intermediate layer of the image dataset that is the executed task, using a distribution difference. Also, for example, the feature comparing unit 1104b may calculate the similarity between the feature for each intermediate layer of the image dataset to be inspected and the feature for each intermediate layer of the image dataset that is the executed task, using a common area between the distributions.

[0056] The model learning unit 1106b generates a new trained model by inputting the image data set to be inspected into the trained model based on the image data set to be inspected acquired by the training image acquisition unit 1101 and the trained model selected by the model selection unit 1105. Note that the new trained model generated by the model learning unit 1106b is a non-defective product distribution.

[0057] In this case, for example, the model learning unit 1106b inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and generates a new trained model by selectively combining tensors from the outputs of each intermediate layer.

[0058] Next, an example of the operation of the learning device 11 according to the second embodiment shown in FIGS. 1 and 9 will be described with reference to FIGS. 10 to 13. For example, as shown in FIGS. 11 and 12, the learning device 11 according to the second embodiment uses features based on the outputs of the multiple intermediate layers and selects one or more past models with high similarity between the features of each intermediate layer. The example in FIG. 11 illustrates a case in which the learning device 11 compares a feature X1 of an image dataset X with a feature A of an image dataset A. In this case, the learning device 11 extracts the feature X1 by inputting the image dataset X into a trained model A corresponding to the image dataset A. Note that the example in FIG. 11 illustrates a case in which the similarity between the feature X1 and the feature A is high for the a and b layers. Furthermore, the example in FIG. 12 illustrates a case in which the learning device 11 compares a feature X2 of the image dataset X with a feature B of the image dataset B. In this case, the learning device 11 extracts the feature X2 by inputting the image dataset X into a trained model B corresponding to the image dataset B. Note that the example of Figure 12 shows a case where, for layer c, feature X2 and feature B have a high similarity. Then, as shown in Figure 13, for example, the learning device 11 inputs the image dataset to be inspected into the selected past model and selectively combines tensors from the outputs of each intermediate layer to generate a new trained model. The example of Figure 13 shows a case where the learning device 11 selects trained model A (layers a and b) and trained model B (layer c) to generate a new trained model X.

[0059] 1 and 9, the learning image acquisition unit 1101 first acquires an image data set to be inspected (step ST201), as shown in Fig. 10. At this time, the learning image acquisition unit 1101 acquires the image data set to be inspected, which is input via the operation input device 12.

[0060] In the examples of FIGS. 4, 5, and 11 to 13, the learning image acquisition unit 1101 acquires an image data set X.

[0061] Furthermore, the existing feature acquisition unit 1102 acquires the feature of the image dataset that is the executed task (step ST202). At this time, the existing feature acquisition unit 1102 acquires the feature of the image dataset that is the executed task indicated by the information stored in the storage device 13. Furthermore, the feature of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102 is the feature based on outputs from multiple intermediate layers in the trained model corresponding to the image dataset that is the executed task.

[0062] 4, 5, and 11 to 13, the existing feature acquisition unit 1102 acquires feature A of image dataset A, feature B of image dataset B, and feature C of image dataset C. Note that in the examples of Fig. 11 and Fig. 12, the feature acquired by the existing feature acquisition unit 1102 is feature based on the outputs of layers a, b, and c of the past model.

[0063] Next, the feature extraction unit 1103 extracts features of the image dataset that is the inspection target based on the trained model corresponding to the image dataset that is the executed task and the image dataset that is the inspection target acquired by the training image acquisition unit 1101 (step ST203). The features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 are features based on outputs from the multiple intermediate layers in the trained model.

[0064] In this case, for example, the feature extraction unit 1103 may extract, as features, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed. Furthermore, for example, the feature extraction unit 1103 may extract, as features, a group of vectors obtained by averaging, for each channel, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed. Furthermore, for example, the feature extraction unit 1103 may extract, as features, a vector obtained by averaging, as a whole, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset to be executed.

[0065] At this time, the feature extraction unit 1103 obtains a trained model corresponding to the image dataset, which is the executed task indicated by the information stored in the storage device 13, and then performs the above-mentioned feature extraction.

[0066] In the examples of Figures 4, 5, and 11 to 13, the feature extraction unit 1103 extracts features X1 to X3. Feature X1 is a feature extracted by inputting image dataset X to trained model A. Feature X2 is a feature extracted by inputting image dataset X to trained model B. Feature X3 is a feature extracted by inputting image dataset X to trained model C. In the examples of Figures 11 and 12, the features extracted by the feature extraction unit 1103 are features based on the outputs of layers a, b, and c of the past model.

[0067] Next, the feature comparison unit 1104b calculates the similarity of the features based on the features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102 (step ST204). Here, in the learning device 11 according to the second embodiment, the feature comparison unit 1104b calculates the similarity between the features for each intermediate layer of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102.

[0068] In this case, for example, the feature comparison unit 1104b may calculate the similarity between the feature for each intermediate layer of the image dataset to be inspected and the feature for each intermediate layer of the image dataset that is the executed task using a distribution difference. In this case, the feature comparison unit 1104b may employ, for example, Fréchet inception distance, KL divergence, JS divergence, or Mahalanobis distance as the distribution difference. Furthermore, for example, the feature comparison unit 1104b may calculate the similarity between the feature for each intermediate layer of the image dataset to be inspected and the feature for each intermediate layer of the image dataset that is the executed task using a common area between the distributions. In this case, the feature comparison unit 1104b may employ, for example, histogram intersection as the common area between the distributions.

[0069] Next, based on the feature similarity calculated by the feature comparison unit 1104b, the model selection unit 1105 selects one or more trained models corresponding to the image datasets that are executed tasks and have high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from the image datasets that are executed tasks (step ST205). At this time, for example, the model selection unit 1105 selects one or more trained models corresponding to the image datasets in the order of high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from the image datasets that are executed tasks.

[0070] 4, 5, and 11 to 13, the similarity between feature X1 and feature A for layers a and b, and between feature X2 and feature B for layer c are high, so the feature extraction unit 1103 selects trained model A and trained model B. On the other hand, in the examples of FIGS. 4, 5, and 11 to 13, the similarity between feature X3 and feature C for layers a, b, and c is low, so the feature extraction unit 1103 does not select trained model C.

[0071] Next, the model training unit 1106b generates a new trained model by inputting the image data set of the inspection target acquired by the training image acquisition unit 1101 and the trained model selected by the model selection unit 1105 into the trained model (step ST206). Note that the new trained model generated by the model training unit 1106b is a non-defective product distribution.

[0072] In this case, for example, the model learning unit 1106b inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and generates a new trained model by selectively combining tensors from the outputs of each intermediate layer.

[0073] 4, 5, and 11 to 13, the model training unit 1106b generates a new trained model X from trained model A and trained model B. In this case, the model training unit 1106 inputs image dataset X to trained model A and trained model B, respectively, and generates a new trained model X (non-defective product distribution) by combining the output of the intermediate layers (layers a and b) with high similarity of trained model A with the output of the intermediate layer (layer c) with high similarity of trained model B.

[0074] Next, the model evaluation unit 1107 determines whether the accuracy of the new trained model generated by the model training unit 1106b is equal to or greater than a threshold (step ST207). Note that the threshold can be set as appropriate to a value for evaluating the trained model.

[0075] Next, the model output unit 1108 outputs to the outside information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold based on the determination result by the model evaluation unit 1107 (step ST208).

[0076] In the prior art, when selecting from multiple past models, there is a risk that a trained model appropriate for the image dataset being inspected may not be selected. In contrast, the learning device 11 according to the second embodiment compares the features of the image dataset being inspected (features based on the model intermediate output) with the features of the image dataset being an executed task (features based on the model intermediate output), thereby enabling the selection of a trained model (intermediate layer) using data similar to the image dataset being inspected, with consideration given to generating a new trained model for anomaly detection, and is expected to achieve higher accuracy.

[0077] As described above, in the learning device 11 according to the second embodiment, the feature comparison unit 1104 calculates the similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102, and selects a trained model (intermediate layer). This enables the learning device 11 according to the second embodiment to obtain a trained model that is more appropriate for new data than conventional methods.

[0078] Finally, with reference to FIG. 14 , an example of the hardware configuration of the learning device 11 according to the first and second embodiments will be described. Here, an example of the hardware configuration of the learning device 11 according to the first embodiment will be described, but the same applies to the example of the hardware configuration of the learning device 11 according to the second embodiment. The functions of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 in the learning device 11 are realized by a processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in FIG. 14A , or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in FIG. 14B .

[0079] When the processing circuitry 51 is dedicated hardware, the processing circuitry 51 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 may be realized individually by the processing circuitry 51, or the functions of each unit may be realized collectively by the processing circuitry 51.

[0080] When the processing circuitry 51 is a CPU 52, the functions of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 53. The processing circuitry 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. That is, the learning device 11 includes a memory 53 for storing a program that, when executed by the processing circuitry 51, results in the execution of, for example, each step shown in FIG. 3 . In addition, these programs can also be said to cause a computer to execute the procedures and methods of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), as well as magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0081] Note that the functions of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 may be partially implemented by dedicated hardware and partially implemented by software or firmware. For example, the function of the learning image acquisition unit 1101 may be implemented by the processing circuitry 51 as dedicated hardware, and the functions of the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 may be implemented by the processing circuitry 51 reading out and executing programs stored in the memory 53.

[0082] In this way, the processing circuitry 51 can realize each of the above-described functions by hardware, software, firmware, or a combination of these.

[0083] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0084] The learning device according to the present disclosure is capable of obtaining a trained model that is appropriate for the image dataset to be inspected, unlike conventional devices, and is suitable for use in learning devices that obtain trained models.

[0085] 1 Learning system, 11 Learning device, 12 Operation input device, 13 Storage device, 14 Display output device, 51 Processing circuit, 52 CPU, 53 Memory, 1101 Learning image acquisition unit, 1102 Existing feature acquisition unit, 1103 Feature extraction unit, 1104, 1104b Feature comparison unit, 1105 Model selection unit, 1106, 1106b Model learning unit, 1107 Model evaluation unit, 1108 Model output unit.

Claims

1. A learning image acquisition unit that acquires an image dataset to be inspected; an existing feature acquisition unit that acquires features of the image dataset to be inspected, the features being based on outputs from a plurality of intermediate layers in a trained model corresponding to the image dataset to be inspected; a feature extraction unit that extracts features of the image dataset to be inspected, the features being based on outputs from the plurality of intermediate layers in the trained model, based on the trained model corresponding to the image dataset to be inspected and the image dataset to be inspected acquired by the learning image acquisition unit; a feature comparison unit that calculates a similarity of features based on the features of the image dataset to be inspected extracted by the feature extraction unit and the features of the image dataset to be inspected acquired by the existing feature acquisition unit; and a model selection unit that selects, from the image dataset to be inspected, one or more trained models corresponding to the image dataset to be inspected, the image dataset to be inspected having a high similarity of features to the image dataset to be inspected acquired by the learning image acquisition unit, based on the similarity of features calculated by the feature comparison unit; a model learning unit that generates a new trained model which is a non-defective product distribution by inputting the image dataset of the inspection target acquired by the learning image acquisition unit and the trained model selected by the model selection unit, based on the image dataset of the inspection target to the trained model; a model evaluation unit that determines whether the accuracy of the new trained model generated by the model learning unit is equal to or greater than a threshold; and a model output unit that outputs information indicating the new trained model whose accuracy is determined to be equal to or greater than the threshold, based on the determination result by the model evaluation unit.

2. The learning device according to claim 1, wherein the feature extraction unit extracts, as features, outputs from a plurality of intermediate layers when an image dataset of an inspection target acquired by the learning image acquisition unit or a good product image dataset from among the image dataset of the inspection target is input to a trained model corresponding to the image dataset of the executed task.

3. The learning device according to claim 1, wherein the feature extraction unit extracts, as features, a group of vectors obtained by averaging, for each channel, outputs from a plurality of intermediate layers when an image dataset of an inspection target acquired by the learning image acquisition unit or a good product image dataset from among the image dataset of the inspection target is input to a trained model corresponding to the image dataset of the executed task.

4. The learning device according to claim 1, wherein the feature extraction unit extracts as a feature a vector obtained by averaging outputs from a plurality of intermediate layers when an image dataset of an inspection target acquired by the learning image acquisition unit or a good product image dataset from among the image dataset of the inspection target is input to a trained model corresponding to the image dataset of the executed task.

5. A learning device as claimed in claim 2 or 3, wherein the feature comparison unit calculates a similarity between the overall feature of the image dataset being the inspection target extracted by the feature extraction unit and the overall feature of the image dataset being the executed task acquired by the existing feature acquisition unit, using a distribution difference as a feature.

6. The learning device according to claim 4, wherein the feature comparison unit calculates a similarity between the overall feature of the image dataset being the inspection target extracted by the feature extraction unit and the overall feature of the image dataset being the executed task acquired by the existing feature acquisition unit, using a common area between the distributions.

7. A learning device as claimed in claim 2 or claim 3, wherein the feature comparison unit calculates a similarity between the feature for each intermediate layer of the image dataset being the inspection target extracted by the feature extraction unit and the feature for each intermediate layer of the image dataset being the executed task acquired by the existing feature acquisition unit, using a distribution difference.

8. The learning device according to claim 4, wherein the feature comparison unit calculates the similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit and the feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit, using a common area between the distributions.

9. The learning device according to claim 1, characterized in that the model learning unit generates a new trained model which is a non-defective product distribution by inputting a non-defective product image data set from the image data set of the inspection target acquired by the learning image acquisition unit into the trained model selected by the model selection unit and combining all of the outputs from each intermediate layer.

10. The learning device according to claim 1, characterized in that the model learning unit inputs a good-product image data set from the image data set of the inspection target acquired by the learning image acquisition unit to the trained model selected by the model selection unit, and selectively combines tensors from the output of each intermediate layer to generate a new trained model which is a good-product distribution.

11. A step in which a learning image acquisition unit acquires an image dataset to be inspected; a step in which an existing feature acquisition unit acquires features of the image dataset, which is an executed task, based on outputs from a plurality of intermediate layers in a trained model corresponding to the image dataset, which is the executed task; a step in which a feature extraction unit extracts features of the image dataset to be inspected, which are based on outputs from the plurality of intermediate layers in the trained model, based on the trained model corresponding to the image dataset, which is the executed task, and the image dataset to be inspected acquired by the learning image acquisition unit; a step in which a feature comparison unit calculates a similarity of features based on the features of the image dataset to be inspected extracted by the feature extraction unit and the features of the image dataset, which is an executed task, acquired by the existing feature acquisition unit; a model selection unit selecting, based on the similarity of the features calculated by the feature comparison unit, one or more trained models corresponding to an image dataset that is an executed task and has a high similarity of features to the image dataset that is an inspection target acquired by the learning image acquisition unit, from among the image datasets that are the executed task; a model training unit generating a new trained model having a non-defective product distribution by inputting the image dataset that is the inspection target into the trained model based on the image dataset that is the inspection target acquired by the learning image acquisition unit and the trained model selected by the model selection unit; a model evaluation unit determining, based on the new trained model generated by the model training unit, whether an evaluation of the new trained model is equal to or greater than a threshold; and a model output unit outputting, based on a result of the determination by the model evaluation unit, information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold.

Images