Learning device and learning method

Abstract

The system includes a learning image acquisition unit (1101) that acquires an image dataset to be inspected, an existing feature acquisition unit (1102) that acquires features of an image dataset that is an executed task, a feature extraction unit (1103) that extracts features of the image dataset to be inspected, a feature comparison unit (1104) that calculates similarities between the features of the image dataset to be inspected and the features of the image dataset that is an executed task, a model selection unit (1105) that selects, based on the similarity of the features, one or more trained models corresponding to an image dataset that is an executed task and has high similarity in features to the image dataset to be inspected, from among the image datasets that are executed tasks, a model training unit (1106) that generates a new trained model that is a non-defective product distribution by inputting the image dataset to be inspected into the trained model selected by the model selection unit (1105), a model evaluation unit (1107) that determines whether an evaluation of the new trained model is equal to or greater than a threshold, and a model output unit (1108) that outputs information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold.

Description

[Technical field]

[0001] The present disclosure relates to a learning device and a learning method for obtaining a trained model. [Background technology]

[0002] When applying AI-based automated visual inspection to a product at a production site, it is necessary to collect an image data set of the product to be inspected as training data to realize the desired function. Therefore, there has been a demand for small-data learning using past models such as transfer learning.

[0003] On the other hand, when multiple past models exist, in order to obtain the optimal model for the image data set to be inspected, it is necessary to train and evaluate all past models and select the most accurate model. Therefore, when the number of past models is huge, a huge training cost is incurred to obtain the optimal model.

[0004] Therefore, a method has been proposed in which a past model is selected based on the characteristics of the image dataset, and transfer learning is performed.

[0005] An example of the above-mentioned transfer learning technology is disclosed in Patent Document 1. This technology combines multiple AI models to improve the accuracy of inspection of good and defective products. In this technology, first, the inspection data is input to multiple past models, and a past model with a correlation of intermediate or final output below a certain level is selected.The selected model is then used to create multiple hybrid model candidates.The hybrid model with the highest accuracy is then adopted. In this way, with this technology, test data is input to multiple hybrid models, and the models are trained to make correct label determinations based on the weighted sum of the outputs of each model. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2022 / 215559 Summary of the Invention [Problem to be solved by the invention]

[0007] As described above, existing transfer learning techniques select multiple pairs of models that have low correlation between intermediate or final outputs when test data (image dataset to be tested) is input to past models. In this case, although the models are independent of each other, there is a possibility that an appropriate model for the test data cannot be selected.

[0008] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a learning device that, unlike conventional devices, makes it possible to obtain a trained model that is appropriate for the image dataset to be inspected. [Means for solving the problem]

[0009] The learning device according to the present disclosure includes an image acquisition unit for learning that acquires an image dataset to be inspected, an existing feature acquisition unit that acquires features of the image dataset that is an executed task, the features being based on outputs at a plurality of intermediate layers in a trained model corresponding to the image dataset that is the executed task, a feature extraction unit that extracts features of the image dataset to be inspected, the features being based on outputs at a plurality of intermediate layers in the trained model, based on the trained model corresponding to the image dataset that is the executed task and the image dataset to be inspected acquired by the image acquisition unit for learning, a feature comparison unit that calculates a similarity of features based on the features of the image dataset to be inspected extracted by the feature extraction unit and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit, and The system is characterized by comprising: a model selection unit that selects, based on the similarity of features, one or more trained models corresponding to an image dataset that is an executed task and has a high similarity of features to the image dataset that is an inspection target acquired by the training image acquisition unit, from among the image datasets that are executed tasks; a model training unit that generates a new trained model having a non-defective product distribution by inputting the image dataset that is the inspection target acquired by the training image acquisition unit and the trained model selected by the model selection unit; a model evaluation unit that determines whether an evaluation of the new trained model generated by the model training unit is equal to or greater than a threshold value, based on the determination result by the model evaluation unit; and a model output unit that outputs information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold value, based on the determination result by the model evaluation unit. Effect of the Invention

[0010] According to the present disclosure, as configured above, it is possible to obtain a trained model that is more appropriate for the image dataset to be inspected than in the past. [Brief description of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of a learning system according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of the configuration of a learning device according to a first embodiment. [Diagram 3] 4 is a flowchart showing an operation example of the learning device according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an overview of the overall operation of the learning device according to the first embodiment. [Diagram 5] FIG. 2 is a diagram showing an outline of the operation of the learning device according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an outline of the operation of the learning device according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an outline of the operation of the learning device according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an outline of the operation of the learning device according to the first embodiment. [Figure 9] FIG. 11 is a block diagram showing an example of the configuration of a learning device according to a second embodiment. [Figure 10] 10 is a flowchart showing an example of the operation of the learning device according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an outline of the operation of a learning device according to the second embodiment. [Figure 12] FIG. 11 is a diagram showing an outline of the operation of a learning device according to the second embodiment. [Figure 13] FIG. 11 is a diagram showing an outline of the operation of a learning device according to the second embodiment. [Figure 14] 14A and 14B are block diagrams showing examples of hardware configurations of the learning devices according to the first and second embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of a learning system 1 according to the first embodiment. As shown in FIG. 1, the learning system 1 includes a learning device 11, an operation input device 12, a storage device 13, and a display output device .

[0013] The learning device 11 outputs information indicating a new trained model based on the image dataset to be inspected input via the operation input device 12, and the features of the image dataset that is an executed task indicated by the information stored in the memory device 13, and the trained model corresponding to the image dataset that is the executed task. An example of the configuration of this learning device 11 will be described later.

[0014] The operation input device 12 is a device that accepts operations by a user. For example, the operation input device 12 outputs to the learning device 11 an image data set that is an inspection target input by the user.

[0015] The storage device 13 stores various data handled by the learning system 1. For example, the storage device 13 stores information indicating features of an image dataset that is an executed task and a learned model corresponding to the image dataset that is the executed task. Here, examples of the storage device 13 include non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), magnetic disk, flexible disk, optical disk, compact disk, mini disk, or DVD (Digital Versatile Disc).

[0016] The display output device 14 displays information indicating the new trained model output by the training device 11.

[0017] Next, an example of the configuration of the learning device 11 will be described with reference to FIG. As shown in FIG. 2, the learning device 11 includes a learning image acquisition unit 1101, an existing feature acquisition unit 1102, a feature extraction unit 1103, a feature comparison unit 1104, a model selection unit 1105, a model learning unit 1106, a model evaluation unit 1107, and a model output unit 1108.

[0018] The learning image acquisition unit 1101 acquires an image data set to be inspected. At this time, the learning image acquisition unit 1101 acquires the image data set to be inspected that is input via the operation input device 12.

[0019] The existing feature acquisition unit 1102 acquires features of the image dataset that is an executed task. At this time, the existing feature acquisition unit 1102 acquires features of the image dataset that is an executed task indicated by the information stored in the storage device 13. Furthermore, the features of the image dataset that is an executed task acquired by the existing feature acquisition unit 1102 are features based on outputs in multiple intermediate layers in a trained model corresponding to the image dataset that is the executed task.

[0020] The feature extraction unit 1103 extracts features of the image dataset that is the inspection target, based on a trained model corresponding to the image dataset that is the executed task, and the image dataset that is the inspection target acquired by the learning image acquisition unit 1101. The features of the image dataset that is the inspection target extracted by this feature extraction unit 1103 are features based on outputs from the multiple intermediate layers in the trained model.

[0021] In this case, for example, the feature extraction unit 1103 may extract, as features, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task. Also, for example, the feature extraction unit 1103 may extract, as features, a group of vectors obtained by averaging, for each channel, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task. Also, for example, the feature extraction unit 1103 may extract, as a feature, a vector obtained by averaging outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task.

[0022] At this time, the feature extraction unit 1103 obtains a trained model corresponding to the image data set, which is the executed task indicated by the information stored in the storage device 13, and then performs the above-mentioned feature extraction.

[0023] The feature comparison unit 1104 calculates the similarity between features based on the features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102. Here, in the learning device 11 according to embodiment 1, the feature comparison unit 1104 calculates the similarity between the overall features of the image dataset to be inspected, which is extracted by the feature extraction unit 1103, and the overall features of the image dataset, which is the executed task, which is acquired by the existing feature acquisition unit 1102.

[0024] At this time, for example, the feature comparison unit 1104 may calculate the similarity between the overall feature of the image dataset to be inspected and the overall feature of the image dataset to be executed, using a distribution difference. Also, for example, the feature comparison unit 1104 may calculate the similarity between the overall feature of the image dataset to be inspected and the overall feature of the image dataset to be executed, using the common area between the distributions.

[0025] Based on the similarity of features calculated by the feature comparison unit 1104, the model selection unit 1105 selects one or more trained models corresponding to the image dataset that is the executed task and has a high similarity of features to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from among the image datasets that are the executed tasks. At this time, for example, the model selection unit 1105 selects one or more trained models corresponding to the image datasets, in the order of image datasets that are executed tasks having high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from among the image datasets that are executed tasks.

[0026] The model learning unit 1106 generates a new trained model by inputting the image data set to be inspected, which is acquired by the learning image acquisition unit 1101, into the trained model based on the trained model selected by the model selection unit 1105. Note that the new trained model generated by the model learning unit 1106 is a non-defective product distribution.

[0027] In this case, for example, the model learning unit 1106 inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and combines all of the outputs from each intermediate layer to generate a new trained model.

[0028] The model evaluation unit 1107 determines whether the accuracy of the new trained model is equal to or higher than a threshold based on the new trained model generated by the model training unit 1106. The threshold can be appropriately set to a value for evaluating the trained model.

[0029] The model output unit 1108 outputs, based on the determination result by the model evaluation unit 1107, information indicating the new trained model whose evaluation is determined to be equal to or higher than the threshold to the outside.

[0030] Next, an example of the operation of learning device 11 according to embodiment 1 shown in FIGS. 1 and 2 will be described with reference to FIGS. In the learning device 11 according to the first embodiment, first, features are extracted from an image dataset to be inspected (a new task) (step ST1), as shown in Fig. 4. In the example of Fig. 4, the image dataset to be inspected is an image dataset X (a bottle mouth), and the features of the image dataset X are features X1 to X3. The learning device 11 also acquires features of the image datasets that are executed tasks (step ST2). In the example of FIG. 4, the image datasets that are executed tasks are three types of image datasets (image dataset A (nut), image dataset B (cable cross section), and image dataset C (skin surface)). In addition, the image datasets that are executed tasks are associated with features and learned models. In the example of FIG. 4, the image dataset A is associated with the feature A and the learned model A, the image dataset B is associated with the feature B and the learned model B, and the image dataset C is associated with the feature C and the learned model C. Note that information indicating the features and learned models that correspond to these image datasets that are executed tasks is stored in advance in the storage device 13. Then, the learning device 11 finds an image dataset having similar features to the image dataset to be inspected from among the image datasets that are executed tasks, and selects a trained model corresponding to the found image dataset (step ST3). The example in Fig. 4 shows a case where feature A is similar to feature X1, feature B is similar to feature X2, and the learning device 11 selects trained model A corresponding to image dataset A and trained model B corresponding to image dataset B. Then, the learning device 11 generates a new trained model (non-defective product distribution) based on the selected trained model (step ST4). The example of Fig. 4 shows a case where the learning device 11 generates a new trained model X based on the selected trained model A and trained model B.

[0031] Also, as shown in Fig. 5, for example, in the learning device 11 according to the first embodiment, it is assumed that feature extraction is performed using multiple intermediate layers among the intermediate layers of the trained model, and a new trained model (non-defective product distribution) is generated. The example of Fig. 5 shows a case where the multiple intermediate layers are three layers (layer a, layer b, and layer c). In this case, the learning device 11 according to the first embodiment extracts features by utilizing a past model (a trained model corresponding to an image data set that is an executed task). In addition, considering the premise of generating a non-defective product distribution, it is desirable to utilize multiple intermediate layers useful for feature extraction from the past model.

[0032] Therefore, as shown in, for example, FIG. 6 and FIG. 7, the learning device 11 according to the first embodiment uses features based on the outputs of the multiple intermediate layers and selects one or more past models having a high similarity of the overall features. The example of FIG. 6 shows a case where the learning device 11 compares the feature X1 of the image dataset X with the feature A of the image dataset A. In this case, the learning device 11 inputs the image dataset X into the trained model A corresponding to the image dataset A to extract the feature X1. Note that the example of FIG. 6 shows a case where the similarity between the feature X1 and the feature A is high. Also, the example of FIG. 7 shows a case where the learning device 11 compares the feature X3 of the image dataset X with the feature C of the image dataset C. In this case, the learning device 11 extracts the feature X3 by inputting the image dataset X into the trained model C corresponding to the image dataset C. Note that the example of FIG. 7 shows a case where the similarity between the feature X3 and the feature C is low. Then, for example, as shown in Fig. 8, the learning device 11 inputs the image data set to be inspected into the selected past model and combines all the outputs from each intermediate layer to generate a new trained model. The example in Fig. 8 shows a case where the learning device 11 selects trained model A and trained model B to generate a new trained model X.

[0033] In an operation example of the learning device 11 according to the first embodiment shown in Fig. 1 and Fig. 2, the learning image acquisition unit 1101 first acquires an image data set to be inspected (step ST101), for example, as shown in Fig. 3. At this time, the learning image acquisition unit 1101 acquires the image data set to be inspected input via the operation input device 12.

[0034] In the examples of FIGS. 4 to 8, the learning image acquisition unit 1101 acquires an image dataset X.

[0035] Furthermore, the existing feature acquisition unit 1102 acquires features of the image dataset which is the executed task (step ST102). At this time, the existing feature acquisition unit 1102 acquires features of the image dataset which is the executed task indicated by the information stored in the storage device 13. Furthermore, the features of the image dataset which is the executed task acquired by this existing feature acquisition unit 1102 are features based on outputs in multiple intermediate layers in the trained model corresponding to the image dataset which is the executed task.

[0036] 4 to 8, the existing feature acquisition unit 1102 acquires a feature A of image dataset A, a feature B of image dataset B, and a feature C of image dataset C. In the examples of Fig. 6 and Fig. 7, the features acquired by the existing feature acquisition unit 1102 are features based on outputs from layers a, b, and c of the past model.

[0037] Next, the feature extraction unit 1103 extracts features of the image dataset to be inspected based on the trained model corresponding to the image dataset that is the executed task and the image dataset to be inspected acquired by the learning image acquisition unit 1101 (step ST103). The features of the image dataset to be inspected extracted by the feature extraction unit 1103 are features based on outputs in the multiple intermediate layers in the trained model.

[0038] In this case, for example, the feature extraction unit 1103 may extract, as features, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task. Also, for example, the feature extraction unit 1103 may extract, as features, a group of vectors obtained by averaging, for each channel, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task. Also, for example, the feature extraction unit 1103 may extract, as a feature, a vector obtained by averaging outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task.

[0039] At this time, the feature extraction unit 1103 obtains a trained model corresponding to the image data set, which is the executed task indicated by the information stored in the storage device 13, and then performs the above-mentioned feature extraction.

[0040] In the examples of FIGS. 4 to 8, the feature extraction unit 1103 extracts features X1 to X3. The feature X1 is a feature extracted by inputting the image dataset X to a trained model A. The feature X2 is a feature extracted by inputting the image dataset X to a trained model B. The feature X3 is a feature extracted by inputting the image dataset X to a trained model C. In the examples of FIGS. 6 and 7, the features extracted by the feature extraction unit 1103 are features based on the outputs of the past models in layers a, b, and c.

[0041] Next, the feature comparison unit 1104 calculates the similarity of the features based on the features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features of the image dataset to be executed, acquired by the existing feature acquisition unit 1102 (step ST104). Here, in the learning device 11 according to embodiment 1, the feature comparison unit 1104 calculates the similarity between the overall features of the image dataset to be inspected, which is extracted by the feature extraction unit 1103, and the overall features of the image dataset, which is the executed task, which is acquired by the existing feature acquisition unit 1102.

[0042] In this case, for example, the feature amount comparison unit 1104 may calculate the similarity between the overall feature amount of the image data set to be inspected and the overall feature amount of the image data set to be executed using a distribution difference. In this case, the feature amount comparison unit 1104 may adopt, for example, the Frechet inception distance, the KL divergence, the JS divergence, or the Mahalanobis distance as the distribution difference. Also, for example, the feature amount comparison unit 1104 may calculate the similarity between the overall feature amount of the image data set to be inspected and the overall feature amount of the image data set to be the executed task by using a common area between the distributions. In this case, the feature amount comparison unit 1104 may adopt, for example, a histogram intersection as the common area between the distributions.

[0043] Next, based on the similarity of the features calculated by the feature comparison unit 1104, the model selection unit 1105 selects one or more trained models corresponding to the image dataset that is the executed task and has a high similarity of features to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101 from among the image datasets that are the executed tasks (step ST105). At this time, for example, the model selection unit 1105 selects one or more trained models corresponding to the image datasets, in the order of image datasets that are executed tasks having high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from among the image datasets that are executed tasks.

[0044] In the examples of Figures 4 to 8, there is a high similarity between feature X1 (feature for layers a, b, and c as a whole) and feature A (feature for layers a, b, and c as a whole), and between feature X2 (feature for layers a, b, and c as a whole) and feature B (feature for layers a, b, and c as a whole), and therefore the feature extraction unit 1103 selects trained model A and trained model B. On the other hand, in the examples of Figures 4 to 8, the similarity between feature X3 (feature for layers a, b, and c as a whole) and feature C (feature for layers a, b, and c as a whole) is low, and the feature extraction unit 1103 does not select the trained model C.

[0045] Next, the model learning unit 1106 generates a new trained model by inputting the image data set to be inspected, which is acquired by the learning image acquisition unit 1101, into the trained model based on the trained model selected by the model selection unit 1105 (step ST106). Note that the new trained model generated by the model learning unit 1106 is a non-defective product distribution.

[0046] At this time, for example, the model learning unit 1106 inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and generates a new trained model by combining all of the outputs from each intermediate layer.

[0047] 4 to 8, the model learning unit 1106 generates a new trained model X from trained model A and trained model B. At this time, the model learning unit 1106 inputs image dataset X to trained model A and trained model B, respectively, and generates a new trained model X (non-defective product distribution) by combining all of the outputs from the intermediate layers (layers a, b, and c) of trained model A and the outputs from the intermediate layers (layers a, b, and c) of trained model B.

[0048] Next, the model evaluation unit 1107 determines whether the accuracy of the new trained model generated by the model training unit 1106 is equal to or higher than a threshold (step ST107). Note that the threshold can be appropriately set to a value for evaluating the trained model.

[0049] Next, the model output unit 1108 outputs, based on the determination result by the model evaluation unit 1107, information indicating the new trained model whose evaluation is determined to be equal to or higher than the threshold to the outside (step ST108).

[0050] Here, in the conventional technology, when selecting from multiple past models, it may not be possible to select a trained model appropriate for the image dataset to be inspected. In contrast, in the learning device 11 according to embodiment 1, by comparing the features of the image dataset to be inspected (features based on the model intermediate output) with the features of the image dataset that is the executed task (features based on the model intermediate output), a trained model can be selected using data similar to the image dataset to be inspected, taking into consideration the generation of a new trained model for anomaly detection, and thus higher accuracy can be expected.

[0051] As described above, according to the first embodiment, the learning device 11 includes the learning image acquisition unit 1101 that acquires an image dataset to be inspected, an existing feature acquisition unit 1102 that acquires features of the image dataset, which is an executed task, that are based on outputs from a plurality of intermediate layers in a trained model corresponding to the image dataset, which is the executed task, a feature extraction unit 1103 that extracts features of the image dataset to be inspected, which are based on outputs from a plurality of intermediate layers in the trained model, based on the trained model corresponding to the image dataset, which is the executed task, and the image dataset to be inspected acquired by the learning image acquisition unit 1101, a feature comparison unit 1104 that calculates a similarity of features based on the features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features of the image dataset, which is the executed task, acquired by the existing feature acquisition unit 1102, and The apparatus further includes a model selection unit 1105 that selects, from among the image datasets that are executed tasks, one or more trained models corresponding to an image dataset that is an executed task and has a high similarity in features to the image dataset that is an inspection target acquired by the training image acquisition unit 1101, based on the similarity of the features calculated by the trainer 1104; a model training unit 1106 that generates a new trained model having a non-defective product distribution, based on the image dataset that is the inspection target acquired by the training image acquisition unit 1101 and the trained model selected by the model selection unit 1105, by inputting the image dataset that is the inspection target into the trained model; a model evaluation unit 1107 that determines, based on the new trained model generated by the model training unit 1106, whether an evaluation of the new trained model is equal to or greater than a threshold; and a model output unit 1108 that outputs information indicating the new trained model whose evaluation is determined to be equal to or greater than the threshold, based on the determination result by the model evaluation unit 1107. In particular, in the learning device 11 according to embodiment 1, the feature comparison unit 1104 calculates the similarity between the overall features of the image dataset to be inspected, which are extracted by the feature extraction unit 1103, and the overall features of the image dataset, which is the executed task, which are acquired by the existing feature acquisition unit 1102, and selects a trained model. As a result, the learning device 11 according to embodiment 1 is able to obtain a trained model that is more appropriate for the image dataset to be inspected than in the past.

[0052] Embodiment 2 In the learning device 11 according to the first embodiment, a case is shown in which the similarity between the overall feature amount of the image dataset to be inspected and the overall feature amount of the image dataset to be executed is calculated, and a trained model is selected. In contrast, in the learning device 11 according to the second embodiment, a case is shown in which the similarity between the feature amount for each intermediate layer of the image dataset to be inspected and the feature amount for each intermediate layer of the image dataset to be executed is calculated, and a trained model (intermediate layer) is selected.

[0053] Fig. 9 is a diagram showing a configuration example of a learning device 11 according to embodiment 2. In learning device 11 according to embodiment 2 shown in Fig. 9, feature amount comparison unit 1104 is changed to feature amount comparison unit 1104b, and model learning unit 1106 is changed to model learning unit 1106b, compared to learning device 11 according to embodiment 1 shown in Fig. 2. The other configuration example of learning device 11 according to embodiment 2 shown in Fig. 9 is similar to the configuration example of learning device 11 according to embodiment 1 shown in Fig. 2, and the same reference numerals are used and only different parts will be described.

[0054] The feature comparison unit 1104b calculates the similarity between features based on the features of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the features of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102. Here, in the learning device 11 according to the second embodiment, the feature comparison unit 1104b calculates the similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the overall feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102.

[0055] At this time, for example, the feature comparison unit 1104b may calculate the similarity between the feature for each intermediate layer of the image dataset to be inspected and the feature for each intermediate layer of the image dataset that is the executed task, using a distribution difference. Also, for example, the feature comparison unit 1104b may calculate the similarity between the feature for each intermediate layer of the image dataset to be inspected and the feature for each intermediate layer of the image dataset that is the executed task, using a common area between the distributions.

[0056] The model learning unit 1106b generates a new trained model by inputting the image data set to be inspected, which is acquired by the learning image acquisition unit 1101, into the trained model based on the image data set to be inspected and the trained model selected by the model selection unit 1105. Note that the new trained model generated by the model learning unit 1106b is a non-defective product distribution.

[0057] In this case, for example, the model learning unit 1106b inputs a non-defective image dataset from the image dataset to be inspected to the trained model selected by the model selection unit 1105, and selectively combines tensors from the outputs of each intermediate layer to generate a new trained model.

[0058] Next, an example of the operation of learning device 11 according to the second embodiment shown in FIGS. 1 and 9 will be described with reference to FIGS. For example, as shown in FIG. 11 and FIG. 12, the learning device 11 according to the second embodiment uses features based on the outputs in the multiple intermediate layers and selects one or more past models having high similarity between the features for each of the intermediate layers. The example of FIG. 11 shows a case where the learning device 11 compares a feature X1 of an image dataset X with a feature A of an image dataset A. In this case, the learning device 11 extracts the feature X1 by inputting the image dataset X into a trained model A corresponding to the image dataset A. Note that the example of FIG. 11 shows a case where the similarity between the feature X1 and the feature A is high for the a layer and the b layer. Also, the example of FIG. 12 shows a case where the learning device 11 compares a feature X2 of the image dataset X with a feature B of the image dataset B. In this case, the learning device 11 extracts the feature X2 by inputting the image dataset X into a trained model B corresponding to the image dataset B. Note that the example of FIG. 12 shows a case where the similarity between the feature X2 and the feature B is high for the c layer. Then, as shown in Fig. 13, for example, the learning device 11 inputs the image data set to be inspected into the selected past model, and selectively combines tensors from the outputs of each of the intermediate layers to generate a new trained model. The example in Fig. 13 shows a case where the learning device 11 selects trained model A (layers a and b) and trained model B (layer c) to generate a new trained model X.

[0059] In an operation example of the learning device 11 according to the second embodiment shown in Fig. 1 and Fig. 9, the learning image acquisition unit 1101 first acquires an image data set to be inspected (step ST201), for example, as shown in Fig. 10. At this time, the learning image acquisition unit 1101 acquires the image data set to be inspected input via the operation input device 12.

[0060] In the examples of FIGS. 4, 5, and 11 to 13, the learning image acquisition unit 1101 acquires an image dataset X.

[0061] Furthermore, the existing feature acquisition unit 1102 acquires features of the image dataset which is the executed task (step ST202). At this time, the existing feature acquisition unit 1102 acquires features of the image dataset which is the executed task indicated by the information stored in the storage device 13. Furthermore, the features of the image dataset which is the executed task acquired by this existing feature acquisition unit 1102 are features based on outputs in multiple intermediate layers in the trained model corresponding to the image dataset which is the executed task.

[0062] 4, 5, and 11 to 13, the existing feature acquisition unit 1102 acquires a feature A of image dataset A, a feature B of image dataset B, and a feature C of image dataset C. Note that in the examples of Figures 11 and 12, the features acquired by the existing feature acquisition unit 1102 are features based on outputs from layers a, b, and c of the past model.

[0063] Next, the feature extraction unit 1103 extracts features of the image dataset to be inspected based on the trained model corresponding to the image dataset that is the executed task and the image dataset to be inspected acquired by the learning image acquisition unit 1101 (step ST203). The features of the image dataset to be inspected extracted by the feature extraction unit 1103 are features based on outputs from the multiple intermediate layers in the trained model.

[0064] In this case, for example, the feature extraction unit 1103 may extract, as features, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task. Also, for example, the feature extraction unit 1103 may extract, as features, a group of vectors obtained by averaging, for each channel, outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task. Also, for example, the feature extraction unit 1103 may extract, as a feature, a vector obtained by averaging outputs from multiple intermediate layers when the image dataset to be inspected or a good image dataset from the image dataset to be inspected is input to a trained model corresponding to the image dataset that is the executed task.

[0065] At this time, the feature extraction unit 1103 obtains a trained model corresponding to the image data set, which is the executed task indicated by the information stored in the storage device 13, and then performs the above-mentioned feature extraction.

[0066] In the examples of FIGS. 4, 5, and 11 to 13, the feature extraction unit 1103 extracts features X1 to X3. The feature X1 is a feature extracted by inputting the image dataset X to the trained model A. The feature X2 is a feature extracted by inputting the image dataset X to the trained model B. The feature X3 is a feature extracted by inputting the image dataset X to the trained model C. In the examples of FIGS. 11 and 12, the features extracted by the feature extraction unit 1103 are features based on the outputs of the past models in layers a, b, and c.

[0067] Next, the feature comparison unit 1104b calculates the similarity of the features based on the features of the image dataset to be inspected extracted by the feature extraction unit 1103 and the features of the image dataset to be executed, acquired by the existing feature acquisition unit 1102 (step ST204). Here, in the learning device 11 according to the second embodiment, the feature comparison unit 1104b calculates the similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102.

[0068] In this case, for example, the feature amount comparison unit 1104b may calculate the similarity between the feature amount for each intermediate layer of the image dataset to be inspected and the feature amount for each intermediate layer of the image dataset to be executed, using a distribution difference. In this case, the feature amount comparison unit 1104b may adopt, for example, Frechet inception distance, KL divergence, JS divergence, or Mahalanobis distance as the distribution difference. Also, for example, the feature amount comparison unit 1104b may calculate the similarity between the feature amount for each intermediate layer of the image dataset to be inspected and the feature amount for each intermediate layer of the image dataset to be executed, using a common area between the distributions. In this case, the feature amount comparison unit 1104b may adopt, for example, a histogram intersection as the common area between the distributions.

[0069] Next, based on the similarity of the features calculated by the feature comparison unit 1104b, the model selection unit 1105 selects one or more trained models corresponding to the image dataset that is the executed task and has a high similarity of features to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101 from among the image datasets that are the executed tasks (step ST205). At this time, for example, the model selection unit 1105 selects one or more trained models corresponding to the image datasets, in the order of image datasets that are executed tasks having high feature similarity to the image dataset that is the inspection target acquired by the learning image acquisition unit 1101, from among the image datasets that are executed tasks.

[0070] In the examples of Figures 4, 5, and 11 to 13, the similarity between feature X1 and feature A relating to layers a and b, and between feature X2 and feature B relating to layer c are high, so the feature extraction unit 1103 selects trained model A and trained model B. On the other hand, in the examples of Figures 4, 5, and 11 to 13, the similarity between feature X3 and feature C for layers a, b, and c is low, and the feature extraction unit 1103 does not select the trained model C.

[0071] Next, the model learning unit 1106b generates a new trained model by inputting the image data set to be inspected, which is acquired by the learning image acquisition unit 1101, into the trained model based on the trained model selected by the model selection unit 1105 (step ST206). Note that the new trained model generated by the model learning unit 1106b is a non-defective product distribution.

[0072] In this case, for example, the model learning unit 1106b inputs a non-defective image dataset from the image dataset to be inspected into the trained model selected by the model selection unit 1105, and selectively combines tensors from the outputs of each intermediate layer to generate a new trained model.

[0073] In the examples of Figures 4, 5, and 11 to 13, the model learning unit 1106b generates a new trained model X from trained model A and trained model B. At this time, the model learning unit 1106 inputs image dataset X to trained model A and trained model B, respectively, and generates a new trained model X (non-defective product distribution) by combining the output of the trained model A in the intermediate layer (layer a and layer b) having the above-mentioned high similarity with the output of the trained model B in the intermediate layer (layer c) having the above-mentioned high similarity.

[0074] Next, the model evaluation unit 1107 determines whether the accuracy of the new trained model generated by the model training unit 1106b is equal to or higher than a threshold (step ST207). Note that the threshold can be appropriately set to a value for evaluating the trained model.

[0075] Next, the model output unit 1108 outputs, based on the determination result by the model evaluation unit 1107, information indicating the new trained model whose evaluation is determined to be equal to or higher than the threshold value to the outside (step ST208).

[0076] Here, in the conventional technology, when selecting from multiple past models, there is a risk that a trained model appropriate for the image dataset to be inspected cannot be selected. In contrast, in the learning device 11 according to the second embodiment, by comparing the features of the image dataset to be inspected (features based on the model intermediate output) with the features of the image dataset that is the executed task (features based on the model intermediate output), a trained model (intermediate layer) can be selected using data similar to the image dataset to be inspected, with the aim of generating a new trained model for anomaly detection, and thus higher accuracy can be expected.

[0077] As described above, in the learning device 11 according to the second embodiment, the feature comparison unit 1104 calculates the similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit 1103 and the feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit 1102, and selects a trained model (intermediate layer). As a result, the learning device 11 according to the second embodiment can obtain a trained model that is more appropriate for new data than in the past.

[0078] Finally, a hardware configuration example of the learning device 11 according to the first and second embodiments will be described with reference to Fig. 14. Here, a hardware configuration example of the learning device 11 according to the first embodiment will be described, but the same applies to the hardware configuration example of the learning device 11 according to the second embodiment. Each function of the learning image acquisition unit 1101, existing feature acquisition unit 1102, feature extraction unit 1103, feature comparison unit 1104, model selection unit 1105, model learning unit 1106, model evaluation unit 1107, and model output unit 1108 in the learning device 11 is realized by a processing circuit 51. The processing circuit 51 may be dedicated hardware as shown in Fig. 14A, or may be a CPU (also called a Central Processing Unit, central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 52 that executes a program stored in a memory 53 as shown in Fig. 14B.

[0079] When the processing circuit 51 is a dedicated hardware, the processing circuit 51 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these. The functions of each of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 may be realized by the processing circuit 51 individually, or the functions of each unit may be realized by the processing circuit 51 collectively.

[0080] When the processing circuit 51 is a CPU 52, the functions of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 are realized by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the memory 53. The processing circuit 51 realizes the functions of each unit by reading and executing the programs stored in the memory 53. That is, the learning device 11 includes a memory 53 for storing a program that, when executed by the processing circuit 51, results in the execution of each step shown in FIG. 3, for example. These programs can also be said to cause a computer to execute procedures and methods of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108. Here, examples of the memory 53 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs (Digital Versatile Discs).

[0081] Note that the functions of the learning image acquisition unit 1101, the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 may be partially realized by dedicated hardware and partially realized by software or firmware. For example, the learning image acquisition unit 1101 may be realized by the processing circuit 51 as dedicated hardware, and the existing feature acquisition unit 1102, the feature extraction unit 1103, the feature comparison unit 1104, the model selection unit 1105, the model learning unit 1106, the model evaluation unit 1107, and the model output unit 1108 may be realized by the processing circuit 51 reading out and executing a program stored in the memory 53.

[0082] Thus, the processing circuitry 51 can realize each of the above-mentioned functions by hardware, software, firmware, or a combination of these.

[0083] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted. [Industrial Applicability]

[0084] Compared to conventional methods, the learning device disclosed herein is capable of obtaining a trained model appropriate for the image dataset to be inspected, and is suitable for use in learning devices that obtain trained models. [Explanation of symbols]

[0085] 1 learning system, 11 learning device, 12 operation input device, 13 storage device, 14 display output device, 51 processing circuit, 52 CPU, 53 memory, 1101 learning image acquisition unit, 1102 existing feature acquisition unit, 1103 feature extraction unit, 1104, 1104b feature comparison unit, 1105 model selection unit, 1106, 1106b model learning unit, 1107 model evaluation unit, 1108 model output unit.

Claims

1. a learning image acquisition unit that acquires an image dataset to be inspected; an existing feature acquisition unit that acquires features of an image dataset that is an executed task, the features being based on outputs from a plurality of intermediate layers in a trained model that corresponds to the image dataset that is the executed task; a feature extraction unit that extracts features of the image dataset that is an inspection target, based on a trained model corresponding to the image dataset that is the executed task and the image dataset that is an inspection target acquired by the learning image acquisition unit, the features being based on outputs from the multiple intermediate layers in the trained model; a feature comparison unit that calculates a similarity of features based on the features of an image dataset that is an inspection target extracted by the feature extraction unit and the features of an image dataset that is an executed task acquired by the existing feature acquisition unit; and a model selection unit that selects, from among the image datasets that are the executed tasks, one or more trained models that have a high similarity in features to an image dataset that is an inspection target acquired by the learning image acquisition unit, based on the similarity in features calculated by the feature comparison unit; and a model learning unit that generates a new trained model that is a non-defective product distribution by inputting the image data set that is the inspection target acquired by the learning image acquisition unit and the trained model selected by the model selection unit, into the trained model; and A model evaluation unit that determines whether the accuracy of the new trained model generated by the model learning unit is equal to or greater than a threshold, based on the new trained model generated by the model learning unit; a model output unit that outputs information indicating a new trained model whose accuracy is determined to be equal to or greater than a threshold based on a determination result by the model evaluation unit; A learning device equipped with

2. The feature extraction unit extracts, as features, outputs from a plurality of intermediate layers when an image dataset that is an inspection target acquired by the learning image acquisition unit or a non-defective image dataset from among the image datasets that are the inspection target is input to a trained model that corresponds to the image dataset that is the executed task. The learning device according to claim 1, wherein the feature is

3. The feature extraction unit extracts, as features, a group of vectors obtained by averaging, for each channel, outputs from a plurality of intermediate layers when the image dataset of the inspection target acquired by the learning image acquisition unit or a non-defective image dataset from among the image dataset of the inspection target is input to a trained model corresponding to the image dataset of the executed task. The learning device according to claim 1, wherein the feature is

4. The feature extraction unit extracts, as a feature, a vector obtained by averaging outputs from a plurality of intermediate layers when the image dataset to be inspected acquired by the learning image acquisition unit or a non-defective image dataset from among the image datasets to be inspected is input to a trained model corresponding to the image dataset to be the executed task. The learning device according to claim 1, wherein the feature is

5. The feature comparison unit calculates a similarity between an overall feature of the image data set that is the inspection target extracted by the feature extraction unit and an overall feature of the image data set that is the executed task acquired by the existing feature acquisition unit, using a distribution difference.

4. The learning device according to claim 2, wherein the feature is

6. The feature comparison unit calculates a similarity between the overall feature of the image data set that is the inspection target extracted by the feature extraction unit and the overall feature of the image data set that is the executed task acquired by the existing feature acquisition unit, using a common area between the distributions.

5. The learning device according to claim 4, wherein the feature is

7. The feature comparison unit calculates a similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit and the feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit, using a distribution difference.

4. The learning device according to claim 2, wherein the feature is

8. The feature comparison unit calculates a similarity between the feature for each intermediate layer of the image dataset that is the inspection target extracted by the feature extraction unit and the feature for each intermediate layer of the image dataset that is the executed task acquired by the existing feature acquisition unit, using a common area between the distributions.

5. The learning device according to claim 4, wherein the feature is

9. The model learning unit inputs a non-defective product image data set from the image data set to be inspected acquired by the learning image acquisition unit to the trained model selected by the model selection unit, and combines all the outputs from each intermediate layer to generate a new trained model that is a non-defective product distribution. The learning device according to claim 1, wherein the feature is

10. The model learning unit inputs a good product image data set from the image data set of the inspection target acquired by the learning image acquisition unit to the trained model selected by the model selection unit, and selectively combines tensors from the output of each intermediate layer to generate a new trained model that is a good product distribution. The learning device according to claim 1, wherein the feature is

11. A learning method using a learning device including a learning image acquisition unit, an existing feature acquisition unit, a feature extraction unit, a feature comparison unit, a model selection unit, a model learning unit, a model evaluation unit, and a model output unit, comprising: The learning image acquisition unit acquires an image data set to be inspected; The existing feature acquisition unit acquires features of an image dataset that is an executed task, the features being based on outputs of a plurality of intermediate layers in a trained model corresponding to the image dataset that is the executed task; The feature extraction unit extracts, based on a trained model corresponding to the image dataset that is the executed task and an image dataset that is an inspection target acquired by the learning image acquisition unit, a feature of the image dataset that is an inspection target, the feature being based on outputs at the multiple intermediate layers in the trained model; a step of the feature comparison unit calculating a similarity of features based on the features of an image dataset that is an inspection target extracted by the feature extraction unit and the features of an image dataset that is an executed task acquired by the existing feature acquisition unit; The model selection unit selects, based on the similarity of the features calculated by the feature comparison unit, one or more trained models corresponding to an image dataset that is an executed task and has a high similarity of features to an image dataset that is an inspection target acquired by the learning image acquisition unit, from among the image datasets that are the executed tasks; The model learning unit generates a new trained model that is a non-defective product distribution by inputting the image data set of the inspection target acquired by the learning image acquisition unit and the trained model selected by the model selection unit into the trained model; A step in which the model evaluation unit determines whether an evaluation of the new trained model generated by the model training unit is equal to or greater than a threshold, based on the new trained model generated by the model training unit; a step of outputting information indicating a new trained model whose evaluation is determined to be equal to or higher than a threshold based on a determination result by the model evaluation unit; A learning method that has

Images