HYBRID MODEL CREATION METHOD, HYBRID MODEL CREATION DEVICE, AND PROGRAM

Abstract

The present invention involves, first, pooling a plurality of models for inferring a category of data being inputted (S1). At least one of the plurality of models is a model obtained by machine learning. The present invention next involves: creating a plurality of hybrid model candidates for determining a category by selecting and combining at least two models from among the plurality of pooled models (S2); making a comparison between the plurality of hybrid model candidates (S4); and selecting, as a hybrid model, one of the plurality of hybrid model candidates (S6).

Description

[Technical field]

[0001] The present disclosure relates to a hybrid model creation method, a hybrid model creation device, and a program. [Background technology]

[0002] Visual inspection systems using AI technology are becoming more common. Since different types of AI models have different advantages and disadvantages, a technology has been proposed to improve accuracy by combining multiple AI models to obtain the complementary advantages of each (see, for example, Patent Document 1). Patent Document 1 discloses that the final judgment result is obtained by integrating the results obtained using all of the multiple models possessed by the device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 079840 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technique disclosed in the above-mentioned Patent Document 1 uses all of the multiple models that the device has, which causes a problem in that redundant models that are not complementary to other models are used in combination.

[0005] The present disclosure has been made in consideration of the above-mentioned circumstances, and aims to provide a hybrid model creation method and the like that can create a hybrid model with higher accuracy. [Means for solving the problem]

[0006] In order to achieve the above object, a hybrid model creation method according to one embodiment of the present disclosure pools multiple models that estimate a category of input data, at least one of the multiple models being a machine-learned model, and by selecting and combining two or more models from the multiple pooled models, creates multiple hybrid model candidates that determine the category, and compares the multiple hybrid model candidates to select one of the multiple hybrid model candidates as a hybrid model.

[0007] This allows a more accurate hybrid model to be created using multiple models.

[0008] These general or specific aspects may be realized by an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, or may be realized by any combination of a system, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0009] The present disclosure can provide a hybrid model creation method that can create a more accurate hybrid model by using multiple models. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a functional configuration of a hybrid model creating device according to an embodiment. [Diagram 2] FIG. 2 is a diagram for conceptually explaining the processing performed when the hybrid model creation method according to the embodiment is executed. [Diagram 3] FIG. 3 is a flowchart showing an outline of the operation of the hybrid model creating device according to the embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of detailed processing of step S1 according to the first embodiment. [Diagram 5]FIG. 5 is a flowchart illustrating an example of detailed processing of step S1 according to the second embodiment. [Figure 6] FIG. 6 is a flowchart illustrating an example of detailed processing of step S3 according to the third embodiment. [Figure 7A] FIG. 7A is a diagram for explaining the accuracy of a hybrid model candidate when three models having a high correlation according to the third embodiment are combined. [Figure 7B] FIG. 7B is a diagram for explaining the accuracy of a hybrid model candidate when three models with weak correlation according to the third embodiment are combined. [Figure 8] FIG. 8 is a diagram for conceptually explaining an example of details of the hybrid model candidate generating process according to the fourth embodiment. [Figure 9] FIG. 9 is a flowchart of an example of a process performed by the hybrid model creating device according to the fourth embodiment. [Figure 10] FIG. 10 is a flowchart illustrating an example of detailed processing of step S3 according to the fifth embodiment. [Figure 11] FIG. 11 is a diagram conceptually illustrating an example of a hybrid model candidate created by combining the model 1 and the model 2 according to the sixth embodiment. [Figure 12] FIG. 12 is a flowchart illustrating an example of detailed processing of step S3 according to the sixth embodiment. [Figure 13] FIG. 13 is a diagram conceptually illustrating the outputs of Model 1 and Model 2 according to the seventh embodiment and the convex hull of the distribution of the outputs corresponding to the defective product images. [Figure 14] FIG. 14 is a diagram conceptually showing an example of a hybrid model candidate created from the outputs of Model 1 and Model 2 from which outputs corresponding to defective product images excluding the vertices of the convex hull shown in FIG. 13 have been removed. [Figure 15] FIG. 15 is a flowchart illustrating an example of detailed processing of step S3 according to the seventh embodiment. [Figure 16] FIG. 16 is a diagram conceptually illustrating the outputs and excluded regions of the model 1 and the model 2 according to the seventh embodiment. [Figure 17]FIG. 17 is a diagram conceptually showing an example of a hybrid model candidate created from the outputs of Model 1 and Model 2 from which the outputs corresponding to the defective product images included in the exclusion area shown in FIG. 16 have been removed. [Figure 18] FIG. 18 is a diagram for explaining a method of calculating the FAR curve for the model 1 according to the eighth embodiment. [Figure 19] FIG. 19 is a diagram illustrating an example of the FAR table of the model 1 according to the eighth embodiment. [Figure 20] FIG. 20 is a diagram conceptually illustrating the first FAR values ​​of the two models according to the eighth embodiment and the second FAR value of a hybrid model candidate created by combining the two models. [Figure 21] FIG. 21 is a diagram illustrating an example of a hybrid model creating method according to another embodiment. [Figure 22] FIG. 22 is a diagram showing another example of a hybrid model creating method according to another embodiment. [Figure 23A] FIG. 23A is a diagram showing an example of a table of a confusion matrix according to another embodiment. [Figure 23B] FIG. 23B is a diagram showing an example of a table of a confusion matrix according to another embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, standards, components, arrangement and connection forms of the components, steps, order of steps, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in the independent claims of the present disclosure will be described as optional components. In addition, each figure is not necessarily a strict illustration. In each figure, substantially the same configuration is given the same symbol, and duplicated explanations may be omitted or simplified.

[0012] (Embodiment) First, an overview of the hybrid model creating device and hybrid model creating method according to the present embodiment will be described.

[0013] [1. Overview of Hybrid Model Creation Device 10] The following provides an overview of the configuration of the hybrid model creation device 10 according to this embodiment.

[0014] Fig. 1 is a block diagram showing a functional configuration of a hybrid model creation device 10 according to the present embodiment. Fig. 2 is a diagram for conceptually explaining processing when a hybrid model creation method according to the present embodiment is executed.

[0015] The hybrid model creation device 10 is realized by a computer or the like, and is a device capable of creating a more accurate hybrid model by using a plurality of models.

[0016] 1, the hybrid model creation device 10 includes a model pool unit 11, a model selection unit 12, a hybrid model candidate creation unit 13, a hybrid model selection unit 14, and a decision threshold determination unit 15. The decision threshold determination unit 15 may be provided in a device separate from the hybrid model creation device 10.

[0017] [1-1. Model Pool Section 11] The model pool unit 11 is composed of a HDD (Hard Disk Drive) or a memory, and pools (stores) a plurality of models that estimate the category of input data. In this embodiment, the model pool unit 11 pools a plurality of models 11a created in advance, such as model 1, model 2, model 3, and model 4, as shown in FIG. 2. Here, at least one of the plurality of models 11a is a machine-learned model. Each of the plurality of models 11a can also be called an AI model. In this embodiment, the input data will be described as an inspection image of a manufactured product. At least one of the plurality of models 11a is an AI model learned by deep learning. The plurality of models 11a may include an AI model whose features are designed manually. For example, each of the plurality of models 11a receives an inspection image of a manufactured product as an input, estimates the probability that the manufactured product shown in the inspection image is defective, and outputs the estimate. Note that each of the plurality of models 11a may output a binary estimation result indicating whether the manufactured product shown in the inspection image is defective or not.

[0018] [1-2. Model Selection Section 12] The model selection unit 12 selects two or more models from the multiple models pooled in the model pool unit 11. In this embodiment, the model selection unit 12 selects two or more models after excluding a predetermined model from the multiple models pooled in the model pool unit 11. In the example shown in FIG. 2, the model selection unit 12 performs a model selection process 12a in which, for example, model 4 is excluded as a predetermined model from among models 1, 2, 3, and 4, and then selects two or more models. The model selection unit 12 may select two or more models after excluding a predetermined model from the multiple models pooled in the model pool unit 11, or may select two or more models from the multiple models pooled in the model pool unit 11 after excluding the predetermined model. In addition, the predetermined model may be, for example, a model with low estimation accuracy, or a model with a strong correlation with other models. Details of such a method of excluding a predetermined model will be described in Example 1 and Example 2 described later, so the description here will be omitted.

[0019] [1-3. Hybrid model candidate creation section 13] The hybrid model candidate creation unit 13 creates multiple hybrid model candidates for determining categories by combining two or more models selected by the model selection unit 12. The hybrid model candidate creation unit 13 may create multiple hybrid model candidates by combining two or more models selected by the model selection unit 12 so as not to include a combination of models having a stronger correlation than a threshold. The hybrid model candidates may be combined by simply linking (cascading) two or more models selected by the model selection unit 12, or may be combined using logistic regression or the like as described below.

[0020] In the example shown in FIG. 2, the hybrid model candidate creation unit 13 performs a hybrid model candidate creation process 13a to create a hybrid model candidate by combining the model 1, model 2, and model 3 selected by the model selection unit 12. More specifically, the hybrid model candidate creation unit 13 creates a hybrid model candidate 1 by combining, for example, the model 1 and the model 2, and a hybrid model candidate 2 by combining, for example, the model 2 and the model 3. In addition, the hybrid model candidate creation unit 13 creates a hybrid model candidate 3 by combining, for example, the model 1 and the model 3, and a hybrid model candidate 4 by combining, for example, the model 1, the model 2, and the model 3. In this embodiment, the category to be determined is whether the manufactured product shown in the inspection image is a good product or a defective product. That is, the hybrid model candidate 3 determines whether the manufactured product shown in the inspection image is a good product or a defective product. The hybrid model candidates 1 to 3 may output a determination result that determines (estimates) whether the manufactured product shown in the inspection image is defective by probability.

[0021] Details of the method of creating hybrid model candidates will be described in Examples 3 to 6 below, and therefore will not be described here.

[0022] Moreover, the hybrid model candidate creating unit 13 compares the created hybrid model candidates.

[0023] 2, the hybrid model candidate creation unit 13 performs a comparison process to compare the created hybrid model candidates 1 to 4. Examples of a method for comparing the hybrid model candidates 1 to 4 include a method for comparing the accuracy of the judgment results of the hybrid model candidates 1 to 4, and a method for comparing the importance (also referred to as the contribution degree) of each of two or more models that can be calculated from the judgment results.

[0024] Details of the method for comparing a plurality of hybrid model candidates will be described later in Example 2 and so on, and therefore will not be described here.

[0025] [1-4. Hybrid model selection section 14] The hybrid model selection unit 14 selects one of the plurality of hybrid model candidates as a hybrid model based on a comparison result of the plurality of hybrid model candidates.

[0026] In the example shown in FIG. 2, the hybrid model selection unit 14 performs a hybrid model selection process 14a for selecting one of the hybrid model candidates 1 to 4 as a hybrid model based on the comparison result of the hybrid model candidates 1 to 4.

[0027] In the hybrid model selection process 14a, based on the comparison results of the hybrid model candidates 1 to 4, the hybrid model candidate having the highest accuracy in the judgment result or the combination of models having the highest importance is selected as the hybrid model.

[0028] The details of the hybrid model selection method will be described later, so a detailed explanation will be omitted here.

[0029] [1-5. Judgment threshold determination unit 15] The judgment threshold determination unit 15 adjusts the sensitivity of the hybrid model selected by the hybrid model selection unit 14 using a validation data set such as an inspection image of a manufactured product, and determines a threshold of an overdetection rate that is acceptable for suppressing erroneous judgment. The judgment threshold determination unit 15 inputs a validation data set such as an inspection image of a manufactured product, and obtains a judgment result of whether the manufactured product is a good product or a defective product. The judgment threshold determination unit 15 generates a confusion matrix from the obtained judgment result, and determines a threshold of an overdetection rate (judgment threshold) that is acceptable for suppressing erroneous judgment. Note that the Cascading Model shown in the judgment threshold determination process 15a shown in FIG. 2 means a hybrid model selected by the hybrid model selection unit 14, and the judgment threshold is optimized.

[0030] [2. Overview of Operation of Hybrid Model Creation Device 10] An outline of the operation of the hybrid model creating device 10 configured as above will now be described.

[0031] FIG. 3 is a flowchart showing an outline of the operation of the hybrid model creation device 10 according to this embodiment.

[0032] First, the hybrid model creation device 10 pools a plurality of models that estimate the category of input data (S1). In this embodiment, at least one of the plurality of models is a machine-learned model. Also, for example, each of the plurality of models receives an inspection image of a manufactured product as input, estimates the probability that the manufactured product shown in the inspection image is defective, and outputs the estimated probability.

[0033] Next, the hybrid model creation device 10 selects two or more models from the plurality of pooled models (S2). In this embodiment, the hybrid model creation device 10 selects two or more models from the plurality of pooled models excluding some models (predetermined models).

[0034] Next, the hybrid model creation device 10 creates a plurality of hybrid model candidates for determining categories by combining the two or more models selected in step S2 (S3). In this embodiment, the hybrid model creation device 10 may combine the two or more models selected in step S2 by sequential cascading, or may combine them using logistic regression.

[0035] Next, the hybrid model creation device 10 compares the multiple hybrid model candidates created in step S3 (S4). In this embodiment, the hybrid model creation device 10 can, for example, compare the accuracy of the judgment results of the hybrid model candidates, or compare the importance of two or more models that can be calculated from the judgment results of the hybrid model candidates.

[0036] Next, the hybrid model creation device 10 judges whether or not all hybrid model candidates have been compared (S5). If all hybrid model candidates have not been compared in step S5 (No in S5), the process returns to step S4.

[0037] On the other hand, in step S5, if comparison has been completed with all hybrid model candidates (Yes in S5), one of the hybrid model candidates is selected as the hybrid model (S6). In this embodiment, the hybrid model creation device 10 can select, as the hybrid model, a hybrid model candidate consisting of a combination of models with the highest accuracy or importance among the respective judgment results of the hybrid model candidates.

[0038] In this way, according to the hybrid model creation method of the present embodiment, multiple hybrid model candidates are created without using all of the pooled multiple models, and the multiple hybrid model candidates are compared using, for example, the judgment accuracy. This makes it possible to select, for example, the hybrid model candidate with the highest accuracy among the judgment results as the hybrid model. In other words, it is possible to create a hybrid model with higher accuracy by using multiple models.

[0039] Example 1 In step S1 shown in FIG. 3, a model with low estimation accuracy may be excluded as a predetermined model from the multiple pooled models. That is, a model with low estimation accuracy may be excluded from the hybrid model candidates from the multiple pooled models. A specific example of this case will be described below as Example 1. Note that the estimation accuracy is not limited to the accuracy rate, and may be at least one combination of the precision rate, the recall rate, the F value calculated by the harmonic mean of the precision rate and the recall rate, the AUC (Area Under Curve) of the ROC (Receiver Operating Characteristic) curve, and the accuracy rate.

[0040] FIG. 4 is a flowchart illustrating an example of detailed processing of step S1 according to the first embodiment.

[0041] In step S1, the hybrid model creating device 10 first pools a plurality of models that estimate the category of input data (S111).

[0042] Next, the hybrid model creation device 10 acquires the estimation accuracy of each of the multiple models using the validation data set (S112). More specifically, before selecting two or more models, the model selection unit 12 inputs multiple validation data sets to each of the multiple models pooled in the model pool unit 11 and causes them to estimate categories, thereby acquiring the estimation accuracy of each of the multiple models.

[0043] The estimation accuracy of each of the pooled models may be calculated using all of the validation datasets prepared in advance, but is not limited to this. Of all the validation datasets, a validation dataset in which the estimation results differ depending on the model may be used. For example, when the pooled models are model 1, model 2, model 3, and model 4, a validation dataset in which the estimation results of model 1 are different from those of model 2, model 3, and model 4 is used.

[0044] Next, the hybrid model creation device 10 excludes models whose estimation accuracy is equal to or less than a threshold (S113). More specifically, the model selection unit 12 excludes models whose estimation accuracy is equal to or less than a threshold from the multiple models pooled in the model pool unit 11. Then, the model selection unit 12 selects two or more models from the multiple models from which the models whose estimation accuracy is equal to or less than the threshold have been excluded. The threshold is set in advance by the user.

[0045] For example, if the multiple pooled models are model 1, model 2, model 3, and model 4, and only the estimation accuracy of model 4 is equal to or lower than the threshold, the model selection unit 12 excludes model 4 from models 1 to 4 pooled in the model pool unit 11. Then, the model selection unit 12 selects two or more models from models 1, 2, and 3 pooled in the model pool unit 11.

[0046] In this way, the hybrid model creation device 10 can exclude models whose estimation accuracy is equal to or lower than a threshold value from among the multiple pooled models, from hybrid model candidates.

[0047] Example 2 In step S1 shown in Fig. 3, a model that is highly correlated with all other models may be excluded as a predetermined model from the multiple pooled models. In other words, a model that is highly correlated with all other models may be excluded from the multiple pooled models as a hybrid model candidate. A specific example of this case will be described below as Example 2.

[0048] FIG. 5 is a flowchart illustrating an example of detailed processing of step S1 according to the second embodiment.

[0049] In step S1, the hybrid model creating device 10 first pools a plurality of models that estimate the category of input data (S121).

[0050] Next, the hybrid model creation device 10 uses the validation data set to acquire the estimation results of each of the multiple models (S122). More specifically, before selecting two or more models, the model selection unit 12 inputs multiple validation data sets to each of the multiple models pooled in the model pool unit 11 and causes the models to estimate categories, thereby acquiring the estimation results of each of the multiple models. Here, the estimation result may be the final output result of the model or an intermediate amount of the model. For example, in a deep learning model, the estimation result is the output result of an intermediate layer or a final layer of the deep learning model.

[0051] Next, the hybrid model creation device 10 calculates the correlation of all the pooled models using the estimation result acquired in step S122 (S123). More specifically, the model selection unit 12 calculates the correlation between two models for all the models pooled in the model pool unit 11.

[0052] Here, a method for calculating the correlation will be described.

[0053] The estimation result of the jth model for the validation data set is c (j is a natural number). j For example, let the estimation result for the i-th validation data (i is a natural number) in the validation data set be c j,i In addition, the estimation result is assumed to be the final output result of the model or a scalar intermediate quantity.

[0054] In this case, the correlation between the jth and kth models (k is a natural number and j≠k) can be calculated using (Formula 1), (Formula 2), (Formula 3), or (Formula 4). Note that (Formula 1) is a formula for calculating the coincidence rate (Jcacard coefficient) of the estimation result, and can be used when the estimation result is a binary value of 0 or 1. In (Formula 1), δ is Kronecker's δ.

[0055]

number

[0056]

number

[0057]

number

[0058]

number

[0059] On the other hand, (Equation 2) to (Equation 4) can be used not only when the estimation result is a binary value, but also when it is a continuous value. (Equation 2) is a formula for calculating covariance, and E[X] indicates the average of X. V[X] in (Equation 3) indicates the variance of X. Furthermore, (Equation 3) is a formula for calculating the correlation coefficient, and (Equation 4) is a formula for calculating the cosine similarity, and c j is j,k This is a vector created by arranging these with respect to i.

[0060] Next, a method for calculating correlation when the estimation result is an intermediate quantity of a vector will be described.

[0061] In this case, the correlation between the jth and kth models is calculated by the intermediate similarity sim for each validation data set using (Equation 5) or (Equation 6). i It is possible to calculate f j,iis the intermediate quantity of a vector consisting of multiple values. Then, calculate a statistical quantity such as the median or the average value shown in (Equation 7). This makes it possible to compare the calculated correlations even if the estimated result is the intermediate quantity of a vector.

[0062]

number

[0063]

number

[0064]

number

[0065] Returning to FIG. 5, the explanation will be continued below.

[0066] Next, the hybrid model creation device 10 excludes models whose correlation with all other models is stronger than a threshold based on the correlation calculated in step S123 (S124). More specifically, the model selection unit 12 excludes models whose average or median correlation coefficient with all other models is stronger than a threshold from the multiple models pooled in the model pool unit 11. Then, the model selection unit 12 selects two or more models from the multiple models from which models below the threshold have been excluded. The threshold is set in advance by the user.

[0067] For example, if the multiple pooled models are model 1, model 2, model 3, and model 4, and the correlation between model 4 and the other models 1, 2, or 3 is stronger than a threshold, the model selection unit 12 excludes model 4 from models 1 to 4 pooled in the model pool unit 11. Then, the model selection unit 12 selects two or more models from model 1, model 2, and model 3 pooled in the model pool unit 11.

[0068] In this way, the hybrid model creation device 10 can exclude from the hybrid model candidates, among the multiple pooled models, models whose correlations with all other models are stronger than a threshold value.

[0069] Example 3 In the second embodiment, a case has been described in which a model that is highly correlated with all other models is excluded as a predetermined model from a plurality of pooled models in step S1 shown in Fig. 3, but this is not limited thereto. In step 3 shown in Fig. 3, a hybrid model candidate may be created without including a combination of models that are highly correlated. A specific example of this case will be described below as a third embodiment.

[0070] FIG. 6 is a flowchart illustrating an example of detailed processing of step S3 according to the third embodiment.

[0071] In step S3, first, the hybrid model creation device 10 acquires the estimation results of each of the multiple models using the validation data set (S311). More specifically, before creating multiple hybrid model candidates, the hybrid model candidate creation unit 13 acquires the estimation results of each of the multiple models by inputting multiple validation data sets to each of the multiple models pooled in the model pool unit 11 and estimating a category. Note that the hybrid model candidate creation unit 13 may acquire the estimation results of each of the multiple models selected by the model selection unit 12 by inputting multiple validation data sets to each of the multiple models and estimating a category. Here, the estimation result may be the final output result of the model or the intermediate amount of the model, as described in the second embodiment. For example, in a deep learning model, the estimation result is the output result of the intermediate layer or the final layer of the deep learning model.

[0072] Next, the hybrid model creation device 10 calculates the correlation of all the pooled or selected models using the estimation result acquired in step S311 (S312). More specifically, the hybrid model candidate creation unit 13 calculates the correlation between two models for all the models pooled in the model pool unit 11 or selected by the model selection unit 12. Note that the method of calculating the correlation has been described in the second embodiment, and therefore will not be described here.

[0073] Next, the hybrid model creation device 10 selects two or more models from the pooled models, so as not to include a combination of two models having a stronger correlation than a threshold (S313). More specifically, the hybrid model candidate creation unit 13 creates a plurality of hybrid model candidates by combining the two or more models selected by the model selection unit 12, so as not to include a combination of two models having a stronger correlation than a threshold.

[0074] In this way, the hybrid model creation device 10 can create hybrid model candidates that combine models with weak correlations from among a plurality of selected models.

[0075] Here, the reason for creating a hybrid model candidate that combines models with weak correlation will be described.

[0076] FIG. 7A is a diagram for explaining the accuracy of a hybrid model candidate when three models with strong correlations according to Example 3 are combined. FIG. 7B is a diagram for explaining the accuracy of a hybrid model candidate when three models with weak correlations according to Example 3 are combined. The hybrid model candidate shown in FIG. 7A and FIG. 7B combines the estimation results of three models using logistic regression or the like. For the sake of simplicity, the hybrid model candidate in FIG. 7A and FIG. 7B will be explained as outputting the majority vote of the estimation results of the three models.

[0077] 7A shows the binary estimation results and judgment results when 10 pieces of validation data from the validation dataset are used for each of strongly correlated models 1, 2, and 3, and a hybrid model candidate, as well as the true values ​​of the 10 pieces of validation data. As shown in FIG. 7A, the accuracies (estimation accuracies) of models 1, 2, and 3 are 80%, 70%, and 80%, and the accuracy (judgment accuracy) of the hybrid model candidate that combines models 1, 2, and 3 is 80%.

[0078] 7B shows the binary estimation results and judgment results when 10 pieces of validation data from the validation dataset are used for each of Model 1, Model 2, and Model 3, which have weak correlation, and the hybrid model candidate, as well as the true values ​​of the 10 pieces of validation data. As shown in FIG. 7B, the accuracies (estimation accuracies) of Model 1, Model 2, and Model 3 are 80%, 60%, and 50%, and the accuracy (judgment accuracy) of the hybrid model candidate, which is a combination of Model 1, Model 2, and Model 3, is 90%.

[0079] In other words, combining three highly correlated models does not improve the accuracy of the hybrid model candidate. On the other hand, even if the accuracy of the three weakly correlated models is not high, it is possible to improve the accuracy of the hybrid model candidate that combines three weakly correlated models.

[0080] As described above, according to the third embodiment, the hybrid model creation device 10 can create hybrid model candidates that combine models with weak correlations. The hybrid model creation device 10 can select one hybrid model from these hybrid model candidates, and can create a hybrid model with higher accuracy.

[0081] Example 4 In the fourth embodiment, a specific example of creating a hybrid model candidate using logistic regression or the like will be described.

[0082] In this embodiment, the hybrid model candidate creation unit 13 creates multiple hybrid model candidates for determining categories by combining two or more models selected by the model selection unit 12 using logistic regression or the like. The maximum number of models to be combined is set in advance, but may be set each time a hybrid model candidate is created. The hybrid model candidate creation unit 13 creates each of the multiple hybrid model candidates as a machine learning model. The machine learning model is a model that inputs a validation dataset into each of the two or more models selected to configure the hybrid model candidate, and two or more output results obtained by estimating a category are input, and outputs a determination result of determining the category of the validation dataset.

[0083] In addition, the hybrid model candidate creation unit 13 compares the judgment results output from the multiple created hybrid model candidates. More specifically, the hybrid model candidate creation unit 13 compares the judgment results output from the multiple created hybrid model candidates after machine learning.

[0084] 8 is a diagram conceptually illustrating an example of details of the hybrid model candidate creation process 13a according to the embodiment 4. The hybrid model candidate creation process 13a illustrated in FIG. 8 is an example of details of the hybrid model candidate creation process 13a illustrated in FIG.

[0085] In the example shown in FIG. 8, the hybrid model candidate creation unit 13 performs a hybrid model candidate creation process 13a to create a hybrid model candidate by combining model 1, model 2, and model 3 selected by the model selection unit 12. More specifically, the hybrid model candidate creation unit 13 creates a machine learning model 1&2 (hybrid model candidate 1) by combining model 1 and model 2 using logistic regression, for example. In addition, the hybrid model candidate creation unit 13 creates a machine learning model 2&3 (hybrid model candidate 2) by combining model 2 and model 3 using logistic regression, for example. In addition, the hybrid model candidate creation unit 13 creates a machine learning model 1&3 (hybrid model candidate 3) by combining model 1 and model 3 using logistic regression, for example. In the example shown in FIG. 8, the maximum number of models to be combined is 2, and machine learning models that are combined in a brute force manner are created.

[0086] In the example shown in FIG. 8, the hybrid model candidate creation unit 13 acquires output results (determination results) obtained after machine learning model 1&2, machine learning model 2&3, and machine learning model 1&3 are trained using a validation dataset. The hybrid model candidate creation unit 13 performs a comparison process to compare the output results of machine learning model 1&2, machine learning model 2&3, and machine learning model 1&3. The hybrid model candidate creation unit 13 ranks the results of the comparison process, for example, in order of high accuracy. In the example shown in FIG. 8, the machine learning models 2&3, 1&3, and 1&2 are ranked in that order.

[0087] Here, we explain how to combine multiple models using logistic regression.

[0088] The machine learning model obtained by combining models using logistic regression can be expressed using a logistic function (sigmoid function) as shown in the following (Equation 8). Note that in (Equation 8), two models are combined, but the same applies when combining three or more models.

[0089]

number

[0090] In (Equation 8), the function S b (β0+β1x1+β2x2) is a sigmoid function with outputs from 0 to 1, where β0 is a constant, and β1 and β2 are the coefficients of x1 and x2. Also, x1 and x2 represent the outputs of the two models.

[0091] In this embodiment, x1 and x2 correspond to the outputs (estimated results) obtained after training two models, respectively, and are expressed as probabilities. b The output of (β0 + β1x1 + β2x2) corresponds to the output (judgment result) obtained after the machine learning model that combines the two models is trained on the coefficients using a validation dataset, and is expressed as a probability between 0 and 1.

[0092] For example, machine learning models 1&2 obtained by combining using logistic regression are hybrid model candidates that take the output of model 1 and the output of model 2 as inputs, apply a logistic function whose coefficients are trained using a validation data set, and output a judgment result. Similarly, machine learning models 2&3 obtained by combining using logistic regression are hybrid model candidates that take the output of model 2 and the output of model 3 as inputs, apply a logistic function whose coefficients are trained using a validation data set, and output a judgment result. Machine learning models 1&3 obtained by combining using logistic regression are hybrid model candidates that take the output of model 1 and the output of model 3 as inputs, apply a logistic function whose coefficients are trained using a validation data set, and output a judgment result.

[0093] The method of combining multiple models is not limited to the method using logistic regression. If machine learning can be performed using the outputs (estimated results) obtained after training each of the multiple models as inputs, machine learning methods such as support vector machines, random forests, gradient boosting, and neural networks can be appropriately selected.

[0094] Next, the processing of the hybrid model creating device 10 according to the fourth embodiment described above will be explained.

[0095] Fig. 9 is a flowchart showing an example of the process of the hybrid model creating device 10 according to the embodiment 4. Note that, since steps S1, S2, S5, and S6 shown in Fig. 9 are similar to steps S1, S2, S5, and S6 described in Fig. 3, the description thereof will be omitted.

[0096] In step S321, the hybrid model creation device 10 uses the validation data set to acquire the estimation results of each of the multiple models. More specifically, the hybrid model candidate creation unit 13 inputs multiple validation data sets to each of the multiple models pooled in the model pool unit 11 or selected by the model selection unit 12, and causes the multiple models to estimate categories, thereby acquiring the estimation results of each of the multiple models. As described above, the estimation results may be the final output results of the models, or may be intermediate amounts of the models. Note that, when acquiring the estimation results of each of the multiple models pooled in the model pool unit 11, step S321 may be executed before step S2.

[0097] Next, the hybrid model creation device 10 creates, as machine learning models, a plurality of hybrid model candidates each of which combines the two or more models selected in step S2 (S322).

[0098] Here, each of the multiple hybrid model candidates is a machine learning model that receives as input the estimation results output from the two or more models selected as the combination, and outputs a determination result that determines the category of the validation dataset. Typically, this machine learning model is a model obtained by combining the two or more models selected as the combination using logistic regression. In addition, the machine learning model is created by the hybrid model candidate creation unit 13 according to a user's instruction.

[0099] Next, the hybrid model creation device 10 compares the judgment results outputted from each of the multiple hybrid model candidates created in step S322 (S41). More specifically, the hybrid model candidate creation unit 13 inputs, for example, a validation data set to each of the hybrid model candidates, and compares the accuracy of the judgment results outputted.

[0100] The hybrid model candidate creation unit 13 may compare the importance of each of the two or more models that can be calculated from the judgment results of each of the hybrid model candidates. More specifically, when comparing a plurality of hybrid model candidates, the hybrid model candidate creation unit 13 may calculate the importance of each of the two or more models selected to configure the hybrid model candidate from the judgment results outputted from each of the plurality of hybrid model candidates. Then, the hybrid model candidate creation unit 13 may perform the comparison process of step S41 by notifying a model whose importance is below a preset threshold value among the calculated importances.

[0101] Furthermore, the hybrid model candidate creation unit 13 may provide the above notification to the hybrid model selection unit 14, or may provide the above notification by displaying the models with importance below the threshold on a display or the like. This allows the hybrid model selection unit 14 to select, in step S6, one of the multiple hybrid model candidates excluding the hybrid model candidates having models with importance below a preset threshold, as a hybrid model.

[0102] A method for comparing the importance (contribution) will be described below.

[0103] In (Equation 8), as described above, the function S b The output of (β0+β1x1+β2x2) is the output (judgment result) obtained after learning the coefficients of a machine learning model that combines two models using a validation dataset. The coefficient β1 indicates the importance of the model that outputs x1 in this machine learning model, and the coefficient β2 indicates the importance of the model that outputs x2 in this machine learning model. In other words, the coefficient β i is the number of models that can be combined to produce a machine learning model. i indicates the importance of model i that outputs

[0104] Here, the coefficient β i is 0 or the coefficient β i are the other coefficients β k When it is smaller than (i ≠ k), it can be analyzed that model i has little influence (contribution) on the judgment result in the machine learning model.

[0105] In addition, the coefficient β i If is a negative value, it can be analyzed that model i may be causing overfitting of the machine learning model. This is because it is considered that the judgment result of the machine learning model and each of the multiple models that make up the machine learning model should have a positive correlation.

[0106] In this way, by training the coefficients of a machine learning model that combines multiple models using a validation dataset and analyzing the coefficients, it is possible to analyze the importance of each of the multiple combined models.

[0107] In addition, the coefficient β i is 0 or the coefficient βi is different from other coefficients β k or coefficient β i If is a negative value, model i should not be used as a model constituting the machine learning model.

[0108] In this way, when combining multiple models using logistic regression, it is easy to calculate the importance of each of the multiple models to be combined. On the other hand, with other machine learning models, the above-mentioned coefficient analysis method may not be applicable. However, by using SHAP (SHapley Additive exPlanation), a tool for interpreting the contribution of features when inference results are obtained, it is possible to calculate the importance of each of the multiple models to be combined.

[0109] Example 5 The multiple models may include models with high computational costs. In such cases, a hybrid model candidate created by combining models with high computational costs may not be able to meet the requirements of the hardware or execution time used. Note that even if the execution time is within the requirements, it is considered better to have a faster processing speed.

[0110] Therefore, in the fifth embodiment, when a hybrid model candidate is created by machine learning using logistic regression, the processing speed is taken into consideration. A specific example will be described below.

[0111] Methods for taking processing speed into account include using the sum of the execution times of multiple models that make up a hybrid model candidate, and adding a regularization term to the loss function when performing machine learning.

[0112] As a first method for taking processing speed into account, we will explain a method that uses the total execution time of each of the multiple models that make up the hybrid model candidate.

[0113] First, the hybrid model candidate creation unit 13 measures (acquires) the processing time required from inputting a validation dataset to estimating the category of the validation dataset for each of the multiple models pooled in the model pool unit 11 or selected by the model selection unit 12. Here, it is assumed that the validation dataset includes X pieces of sample data.

[0114] Next, the hybrid model candidate creation unit 13 calculates an average processing time, which is the processing time per piece of sample data, for each of the multiple models from the measured processing time.

[0115] Next, the hybrid model candidate creation unit 13 creates multiple hybrid model candidates from only combinations of models whose total average processing time satisfies the execution time requirement, among the two or more models selected by the model selection unit 12. Note that the method of creating hybrid model candidates by machine learning using logistic regression is the same as that described in the fourth embodiment, and therefore will not be described here.

[0116] Next, we will explain the second method of taking processing speed into account, which is to add a regularization term to the loss function when performing machine learning.

[0117] First, the hybrid model candidate creation unit 13 acquires the processing time required from inputting a validation dataset to estimating the category of the validation dataset for each of the multiple models pooled in the model pool unit 11 or selected by the model selection unit 12. Here, it is assumed that the validation dataset includes X pieces of sample data.

[0118] Next, the hybrid model candidate creation unit 13 calculates an average processing time, which is the processing time per one sample data, for each of the multiple models from the acquired processing times. The hybrid model candidate creation unit 13 defines the value of the average processing time of each of the multiple models relative to the sum of all the average processing times of the multiple models as the hardware cost.

[0119] That is, the hardware cost C m can be defined as follows (Equation 9): c m =avg(model processing speed) / sum(avg_all_models processing speed) (Equation 9)

[0120] The method of creating hybrid model candidates by machine learning using logistic regression is as described in Example 4, so the description thereof will be omitted here.

[0121] Next, the hybrid model candidate creation unit 13 adds a regularization term that takes into account the hardware costs of each of the two or more models selected to configure the hybrid model candidate to the loss function of the machine learning model of each of the multiple hybrid model candidates.

[0122] A regularization term that takes into account hardware cost is, for example, a regularization term such as Lasso (L1 norm or L1 regularization) with a parameter α and a hardware cost C m α·C multiplied by m It can be expressed as an L1 regularization term. Here, the parameter α is a hyperparameter that can change the weight of the hardware cost. Details will be described later.

[0123] Next, the hybrid model candidate creation unit 13 adds a regularization term that takes into account the hardware cost, and then executes machine learning of logistic regression. This makes it possible to reduce the coefficient (weight) of a model that has a large calculation cost but a small contribution to the hybrid model candidates, and therefore makes it possible to exclude models that have a small contribution to the hybrid model candidates.

[0124] Here, a detailed example of a method for adding the hardware cost as a regularization term will be described.

[0125] Let the number of data in the dataset used for learning be N, and the true value of the nth data in the dataset be tn Let the explanatory variable set for the nth data be φ n Then, the loss function E(w) of logistic regression is expressed as (Equation 10). When performing machine learning, learning is performed to obtain a combination of weights (coefficients) that minimizes the loss function E(w).

[0126]

number

[0127] Here, if the number of dimensions of the explanatory variables is m, for example, the loss function E(w) with an L1 regularization term added is ´ (w) is expressed as in (Equation 11). In (Equation 11), the parameter α is a hyperparameter.

[0128]

number

[0129] In this embodiment, the explanatory variables are the output values ​​of each model. Therefore, the hardware cost C m Using this, the loss function E ´ (w) can be expressed as (Equation 12).

[0130]

number

[0131] Although the method of adding the hardware cost as a regularization term in the case of logistic regression has been described, this is not the only method. For general machine learning, the hardware cost C can be similarly increased by adding the second term on the right side of (Equation 12) to the loss function E(w). m Loss function E ´ (w) can be made.

[0132] In the above, an example in which an L1 regularization term is used as the regularization term has been described, but an L2 regularization term may be used. In this case, the hardware cost C m It is possible to define a loss function that takes into account the above. Note that the L1 regularization term is expected to have the effect of not only reducing the weight (coefficient) value but also setting it to 0. For this reason, it is better to use the L1 regularization term rather than the L2 regularization term when creating hybrid model candidates by combining models that exclude models with long processing times.

[0133] Next, the process of generating a hybrid model candidate according to the fifth embodiment described above will be described.

[0134] 10 is a flowchart illustrating an example of detailed processing of step S3 according to the embodiment 5. Note that, FIG. 10 corresponds to another example of the processing of steps S321 and S322 illustrated in FIG.

[0135] In step S3, the hybrid model creation device 10 acquires the processing time and the estimation result for each of the multiple models using the validation data set (S331). More specifically, the hybrid model candidate creation unit 13 inputs multiple validation data sets to each of the multiple models selected by the model selection unit 12 to estimate a category. The hybrid model candidate creation unit 13 acquires the processing time required from inputting the validation data set to estimating the category of the validation data set and the estimation result for each of the multiple models selected by the model selection unit 12. As described above, the estimation result may be the final output result of the model or an intermediate amount of the model. The processing time and the estimation result may be acquired from each of the multiple models pooled in the model pool unit 11. In this case, step S331 may be executed before step S2.

[0136] Next, the hybrid model creation device 10 defines the value of the time required for each of the multiple models relative to the sum of all the processing times of the multiple models as a hardware cost based on the processing times acquired in step S331 (S332). Here, the processing time used to define the hardware cost is the average processing time.

[0137] Next, the hybrid model creation device 10 creates multiple hybrid model candidates, which are combinations of the two or more models selected in step S2, as machine learning models. Here, the hybrid model creation device 10 adds a regularization term that takes into account the hardware cost to the loss function of each of the multiple hybrid model candidates when performing machine learning (S333). More specifically, a regularization term that takes into account (multiplies) the hardware cost of each of the two or more models selected to configure the hybrid model candidate is added to the loss function of each of the multiple hybrid model candidates.

[0138] Note that, before comparing the multiple hybrid model candidates in the subsequent step S4, the hybrid model candidate creation unit 13 performs coefficient analysis from the output (judgment result) obtained after learning using the validation dataset. This allows the hybrid model candidate creation unit 13 to exclude hybrid model candidates including models with long processing times. Therefore, in the subsequent step S4, the hybrid model candidate creation unit 13 may perform comparison processing of the multiple hybrid model candidates after excluding hybrid model candidates including models with long processing times.

[0139] Example 6 The hybrid model candidate is created and machine-learned as a machine learning model that outputs a judgment result of judging the category of a validation data set using the estimation results output from two or more models selected as a combination as input. In the sixth embodiment, a case will be described in which, when machine learning a machine learning model, outputs that are clearly NG, that is, the estimation results output from two or more models are definitely NG and the true value (label) is also NG, are excluded.

[0140] FIG. 11 is a conceptual diagram illustrating an example of a hybrid model candidate created by combining model 1 and model 2 according to the sixth embodiment. The hybrid model candidate illustrated in FIG. 11 is a logistic regression model (boundary) created by machine learning. The vertical axis is an output value output (estimated) when the validation data set is input to model 2, and is expressed in terms of probability. Similarly, the horizontal axis is an output value output (estimated) when the validation data set is input to model 1, and is expressed in terms of probability. In FIG. 11, if the sample data included in the validation data set is an inspection image of a manufactured product, the black circle is an inspection image in which the true value of the sample data is a good product, and is referred to as a good product image. The white circle is an inspection image in which the true value of the sample data is a defective product, and is referred to as a defective product image.

[0141] When creating a hybrid model candidate, it is necessary to minimize oversight judgments, such as judging an inspection image whose true value is a defective product as a good product, that is, to improve the judgment accuracy. Oversight judgments are erroneous judgments of NG (whose true value is a defective product) as OK (good product). As a method for improving the judgment accuracy, as described above, machine learning may be performed using sample data in which the estimation results differ for each model. In addition, in order to machine learn the logistic regression model (boundary) shown in FIG. 11, it is important that there are outputs from Model 1 and Model 2 corresponding to good product images close to the boundary. On the other hand, it can be seen that the output of a defective product image (referred to as an output indicating a clear NG) in which both the output values ​​(probabilities) of Model 1 and Model 2 are large is relatively less important when machine learning is performed to obtain the logistic regression model (boundary) shown in FIG. 11.

[0142] In the example shown in FIG. 11, the output in the circled area is an output that clearly indicates NG. The number of clearly NGs contained in the circled area is large. For this reason, when creating a hybrid model candidate by machine learning using the outputs of Model 1 and Model 2 as shown in FIG. 11 and the true values ​​(labels) of the validation dataset, it may be strongly influenced by the outputs that clearly indicate NG, such as those in the circled area, and it may not be possible to obtain the boundary shown in FIG. 11.

[0143] Therefore, when creating a candidate hybrid model using machine learning, outputs that are clearly not acceptable, that is, outputs in the circled area, are excluded during machine learning.

[0144] Specifically, the hybrid model candidate creation unit 13 excludes output values ​​that are higher than a threshold and are estimated to be defective from multiple output values ​​obtained by inputting a validation data set into each of two or more models selected to configure the hybrid model candidate and estimating a category. Next, the hybrid model candidate creation unit 13 creates multiple hybrid model candidates by performing machine learning using the multiple output values ​​from which the output values ​​higher than the threshold have been excluded as input and the true values ​​of the validation data set that correspond to the multiple output values.

[0145] Next, a process of generating a hybrid model candidate according to the sixth embodiment described above will be described.

[0146] 12 is a flowchart illustrating an example of detailed processing of step S3 according to the embodiment 6. Note that, FIG. 12 corresponds to another example of the processing of steps S321 and S322 illustrated in FIG.

[0147] In step S3, the hybrid model creation device 10 uses the validation data set to perform estimation by each of two or more models constituting a hybrid model candidate to obtain a plurality of output values ​​(S341).

[0148] Next, the hybrid model creation device 10 excludes output values ​​that are higher than a threshold value and are estimated to be defective from the multiple output values ​​acquired in step S341 (S342). Here, the output values ​​that are higher than a threshold value and are estimated to be defective are output values ​​that clearly indicate NG as described with reference to FIG.

[0149] Next, the hybrid model creation device 10 performs machine learning using the multiple output values, from which the output values ​​higher than the threshold value have been excluded in step S342, as inputs, and using the true values ​​of the validation data set corresponding to the multiple output values, thereby creating multiple hybrid model candidates (S 343 ).

[0150] In this way, the hybrid model creation device 10 performs machine learning while excluding outputs included in areas where clear NGs are concentrated, and can create multiple hybrid model candidates with high determination accuracy.

[0151] Example 7 In the sixth embodiment, a case where machine learning is performed by excluding outputs included in a region where clear NGs are concentrated among the outputs of two or more models constituting a hybrid model candidate is described, but this is not limited to the above. In the seventh embodiment, a method using a convex hull is described as another method for excluding outputs that show clear NGs. Note that the convex hull means the smallest convex polygon (convex polyhedron) that includes all given points.

[0152] FIG. 13 is a diagram conceptually illustrating the output of model 1 and model 2 according to the seventh embodiment and the convex hull of the distribution of the output corresponding to the defective product image. FIG. 14 is a diagram conceptually illustrating an example of a hybrid model candidate created from the output of model 1 and model 2, from which the output corresponding to the defective product image except for the vertices of the convex hull shown in FIG. 13 is removed. FIG. 14(a) conceptually illustrates the output of model 1 and model 2, from which the output indicating NG other than the vertices of the convex hull shown in FIG. 13 is removed. FIG. 14(b) conceptually illustrates an example of a logistic regression model (boundary) as a hybrid model candidate created by machine learning from the output of model 1 and model 2 shown in FIG. 14(a).

[0153] Specifically, the hybrid model candidate creation unit 13 calculates a convex hull when output values ​​estimated to be defective are plotted from among multiple output values ​​obtained by inputting a validation data set into each of two or more models selected to configure the hybrid model candidate and estimating a category. Next, the hybrid model candidate creation unit 13 excludes output values ​​included in the convex hull except for the vertices of the convex hull from the multiple output values. The hybrid model candidate creation unit 13 then inputs and uses the multiple output values ​​from which the output values ​​included in the convex hull except for the vertices of the convex hull have been excluded, and performs machine learning using the true values ​​of the validation data set corresponding to the multiple output values, thereby creating multiple hybrid model candidates.

[0154] This makes it possible to create a hybrid model candidate having such a determination accuracy that the oversight (oversight determination) is zero.

[0155] Next, the process of generating a hybrid model candidate according to the seventh embodiment described above will be described.

[0156] 15 is a flowchart illustrating an example of detailed processing of step S3 according to the embodiment 7. Note that, FIG. 15 corresponds to another example of the processing of steps S321 and S322 illustrated in FIG.

[0157] In step S3, the hybrid model creation device 10 uses the validation data set to perform estimation for each of two or more models constituting a hybrid model candidate, thereby acquiring a plurality of output values ​​(S351).

[0158] Next, the hybrid model creation device 10 calculates a convex hull when the output values ​​estimated to be defective among the multiple output values ​​acquired in step S351 are plotted (S352).

[0159] Next, the hybrid model creation device 10 excludes output values ​​contained in the convex hull excluding the vertices of the convex hull from the multiple output values ​​acquired in step S351 (S353).

[0160] Next, the hybrid model creation device 10 creates multiple hybrid model candidates by performing machine learning using, as input, multiple output values ​​excluding output values ​​contained in the convex hull excluding the vertices of the convex hull, and using true values ​​of the validation dataset corresponding to the multiple output values ​​(S354).

[0161] In this way, by using the convex hull to exclude outputs that clearly indicate NG, the hybrid model creation device 10 can create multiple hybrid model candidates with high judgment accuracy such that the number of oversights (oversight judgments) is zero.

[0162] In addition, when the number of models (number of dimensions) selected to construct the hybrid model candidates is large, for example 10 or more, the number of vertices of the convex hull may become enormous or the calculation cost of the convex hull may become large, so it may not be possible to adopt a method using the convex hull.

[0163] In such a case, as explained in the sixth embodiment, it is sufficient to exclude outputs included in an area (exclusion area) where clear NGs are concentrated as shown in FIG.

[0164] FIG. 16 is a diagram conceptually showing the outputs of model 1 and model 2 and the excluded region according to the seventh embodiment. FIG. 17 is a diagram conceptually showing an example of a hybrid model candidate created from the outputs of model 1 and model 2 from which the outputs corresponding to the defective product images included in the excluded region shown in FIG. 16 have been removed. FIG. 17(a) conceptually shows the outputs of model 1 and model 2 from which the outputs indicating NG included in the excluded region shown in FIG. 16 have been removed. FIG. 17(b) conceptually shows an example of a logistic regression model (boundary) as a hybrid model candidate created by machine learning from the outputs of model 1 and model 2 shown in FIG. 17(a).

[0165] As shown in Figures 16 and 17, the output value (probability) indicating NG is large in both the output of Model 1 and the output of Model 2, and the region where outputs indicating clear NG, where the true value is also NG, are concentrated can be calculated as the excluded region. This calculation method can be used as an approximation method for calculating the convex hull. Then, machine learning can be performed by excluding outputs indicating clear NG that are in the excluded region.

[0166] This approximation method is also effective in cases of low dimensions, because when using a method that uses a convex hull, the output of the model that can be used for machine learning becomes too small when the number of dimensions is low, making the machine learning unstable.

[0167] Example 8 As a method for comparing a plurality of hybrid model candidates, there is a method for comparing the judgment results of each of the hybrid model candidates.

[0168] Here, the judgment result by machine learning is usually output as a probability. However, the probability output as the judgment result does not represent the actual probability of the category indicated as the judgment result. In other words, for example, even if the judgment result of whether or not a manufactured product shown in an inspection image input as sample data is defective is 0.9, the probability that the manufactured product is defective is not necessarily 90%, and it is known that there is a difference between the actual probability and the judgment result.

[0169] There is also a known technology that aligns the probability shown as an AI judgment result with the actual probability, known as Confidence Calibration.

[0170] In this embodiment, the oversight rate of the hybrid model candidates is used as the judgment result outputted from each of the multiple hybrid model candidates. In addition, the FAR table of the multiple models selected to create the hybrid model candidates is calculated and used as a parameter to adjust the oversight rate of the hybrid model candidates.

[0171] Fig. 18 is a diagram for explaining a method for calculating an FAR curve for model 1 according to Example 8. Fig. 19 is a diagram showing an example of an FAR table for model 1 according to Example 8. Model 1 is one of a plurality of models selected for creating a hybrid model candidate.

[0172] Here, FAR is an abbreviation for False Acceptance Rate, which is the probability of erroneously determining that an NG is OK. In this embodiment, the FAR value is referred to as the oversight rate. Also, FA is an abbreviation for False Accept, which is the erroneous determination that an NG is OK. In this embodiment, FA is referred to as oversight or oversight determination. The FAR table is a table of oversight rates (FAR values) when the threshold is changed by a predetermined step size.

[0173] Specifically, first, during machine learning, the hybrid model candidate creation unit 13 uses a validation data set to create an FAR table for each of a plurality of models selected to create a hybrid model candidate.

[0174] For example, in the example shown in FIG. 18, first, the output value and frequency obtained by inputting the validation data set into model 1 and estimating the category are obtained. Next, as shown in FIG. 18(a), the validation data set is stratified into a distribution indicated by the output value (probability) indicating a good product and its frequency, and a distribution indicated by the output value (probability) indicating a defective product and its frequency. Next, in the distribution of the output value (probability) indicating a defective product shown in FIG. 18(a), the area ratio of the output value determined to be a good product when the threshold is gradually increased can be obtained to obtain the FAR curve shown in FIG. 18(b). Note that the area ratio when the area of ​​the entire distribution is set to 1 corresponds to the oversight rate (FAR value). In addition, in the distribution of the output value (probability) indicating a defective product shown in FIG. 18(a), the oversight rate (FAR value) is obtained at a predetermined step size, and the FAR table shown in FIG. 19 can be obtained. In the FAR table shown in FIG. 19, the step size is set to 0.0078125, and FAR values ​​of 0 to 1 are recorded for the indexes assigned to each step size.

[0175] In this way, the hybrid model candidate creation unit 13 can create an FAR table from the distribution of output values ​​obtained by inputting multiple data indicating defective products from the validation data set and estimating a category for each of the multiple selected models. The hybrid model candidate creation unit 13 can obtain the oversight rate by varying the threshold in the distribution of the acquired output values, and can therefore create an FAR table, which is a table of oversight rates.

[0176] Next, the hybrid model candidate creation unit 13 acquires an estimation result by inputting a data sample included in the validation data set into each of the two or more models selected to configure the hybrid model candidate and estimating a category. The hybrid model candidate creation unit 13 acquires a first FAR value, which is the FAR value of each of the two or more models for the data sample, by comparing the acquired estimation result (output value) with a FAR table created in advance.

[0177] Here, suppose that an estimation result of 0.99 is obtained when a sample image, which is an inspection image whose true value indicates a defective product, is input to, for example, model 1 constituting a hybrid model candidate. In this case, the FAR value when the estimation result is 0.99 is obtained from the FAR table of model 1 shown in FIG. 19. Specifically, since it can be calculated as Table[(1-estimation result) / step_size]=Table[(1-0.99) / 0.0078125]=Table[1], the FAR value of Index=1 is obtained in the FAR table shown in FIG. 19. As a result, the hybrid model candidate creation unit 13 can obtain the FAR value=0.000023 as the first FAR value.

[0178] In this manner, in this embodiment, the probability that the inspection image is a defective product can be estimated (adjusted) based on the distribution of output values ​​(probabilities) indicating defective products when the FAR table is created.

[0179] Next, the hybrid model candidate creation unit 13 multiplies the first FAR values ​​of the two or more acquired models, thereby enabling the hybrid model candidate creation unit 13 to acquire a second FAR value, which is the FAR value of the hybrid model candidate combined with the two or more models.

[0180] Fig. 20 is a diagram conceptually illustrating the first FAR value of each of the two models according to the eighth embodiment and the second FAR value of a hybrid model candidate created by combining the two models. As shown in Fig. 20, by multiplying the first FAR values ​​of the two or more models, it is possible to obtain a second FAR value that is improved over the first FAR values ​​of the two or more models.

[0181] Here, we assume that the FAR distributions of the multiple models that make up the hybrid model candidate are independent, so that according to the law of probability for independent events, we can obtain a second FAR value for the hybrid model candidate that is a combination of two or more models by multiplying the first FAR values ​​of the two or more models.

[0182] If the FAR distributions of the multiple models constituting the hybrid model candidates are not independent, the correlation coefficients of all the multiple models can be calculated and the second FAR value can be corrected so that the model with better performance dominates.

[0183] The method of calculating the correlation coefficient is, for example, as follows. That is, first, the hybrid model candidate creation unit 13 inputs a plurality of validation data sets and estimates categories for each of a plurality of models selected to create a plurality of hybrid model candidates, thereby acquiring an estimation result for each of the plurality of models. Next, the hybrid model candidate creation unit 13 may use the acquired estimation result to calculate correlation coefficients for all combinations of two models among the plurality of models. In this way, the hybrid model candidate creation unit 13 can acquire a corrected second FAR value by multiplying the first FAR values ​​of the acquired two or more models, and further multiplying the first FAR value by a coefficient that becomes smaller as the correlation coefficient becomes larger.

[0184] Next, the hybrid model candidate creation unit 13 determines that the data sample is a non-defective product when the second FAR value is smaller than a preset threshold value (FAR threshold value). The hybrid model candidate creation unit 13 can acquire this determination result as a determination result when the data sample is input to the hybrid model candidate. In this way, the hybrid model candidate creation unit 13 can acquire a determination result adjusted using the second FAR value and the preset threshold value as a determination result when the data sample is input to a plurality of hybrid model candidates. Then, the hybrid model candidate creation unit 13 can compare a plurality of hybrid model candidates using the adjusted determination result.

[0185] The FAR threshold may be determined in advance based on the degree of oversight rate that a user who uses the hybrid model can tolerate.

[0186] Here, for example, assume that a user decides to tolerate a 1 ppm oversight rate among inspection images whose true values ​​indicate defective products, and sets a threshold (FAR threshold) in advance to 1 / 1,000,000. Also, the multiple models constituting the hybrid model candidates are model 1 and model 2. In this case, when the first FAR values ​​of model 1 and model 2 for certain sample data are obtained as described above, the second FAR value can be obtained as a value obtained by multiplying these values. Then, if the second FAR value is smaller than the FAR threshold of 1 / 1,000,000, the sample data can be determined as NG (indicating a defective product), and if it is larger, it can be determined as OK (indicating a non-defective product).

[0187] According to the above-mentioned embodiment and examples, the hybrid model creation device 10 and the hybrid model creation method according to the present disclosure can create a hybrid model that does not use all of the multiple models that are prepared and pooled in advance. Also, the hybrid model creation device 10 and the hybrid model creation method according to the present disclosure can create a hybrid model candidate that excludes models that have high calculation costs and little contribution from the viewpoint of processing speed, so that the hybrid model can be created lightly and effectively. Furthermore, the hybrid model creation device 10 and the hybrid model creation method according to the present disclosure can create a hybrid model candidate that excludes models that do not contribute to improving accuracy using importance, so that the hybrid model can be created lightly and effectively.

[0188] The hybrid model creation device 10 and the like according to the present disclosure have been described above based on the embodiment and each example, but the present disclosure is not limited to these embodiments, etc. As long as they do not deviate from the gist of the present disclosure, various modifications conceivable by a person skilled in the art to the embodiment and each example, and other forms constructed by combining some of the components in the embodiment and each example are also included within the scope of the present disclosure.

[0189] (Other embodiments) (1) In the above embodiment, the hybrid model creation device 10 creates hybrid model candidates by combining multiple models selected from multiple pooled models using logistic regression or the like, and selects one hybrid model by comparing the hybrid model candidates, but this is not limited to the above. It is also possible to create hybrid model candidates by combining multiple models selected from multiple pooled models and performing estimation processing using a logical formula in the order of combination, and select one hybrid model by comparing the accuracy.

[0190] FIG. 21 is a diagram illustrating an example of a hybrid model creating method according to another embodiment.

[0191] FIG. 21 shows a hybrid model creation method in which model 1, model 2, and model 3 are selected from a plurality of pooled models. In FIG. 21, three different models connected by arrows are combined in this order to create a hybrid model candidate. The hybrid model candidates are compared in terms of accuracy obtained by taking the logical sum or logical product of the respective accuracies of model 1, model 2, and model 3 in the order in which they are combined. In the example shown in FIG. 21, it is shown that the hybrid model candidate combined in the order of model 3-model 1-model 2 has the highest accuracy of 93%, and is therefore selected as the hybrid model.

[0192] Fig. 22 is a diagram showing another example of a hybrid model creation method according to another embodiment. Fig. 22 shows a method for creating a hybrid model candidate by combining at least two or more of Model 1, Model 2, and Model 3 when Model 1, Model 2, and Model 3 are selected from a plurality of pooled models. In Fig. 22, the hybrid model candidate in which Model 2 and Model 1 are combined in this order has the highest accuracy of 93%, and is therefore selected as the hybrid model.

[0193] (2) In the above embodiment, the decision threshold determination unit 15 constituting the hybrid model creation device 10 has been described as determining the decision threshold by using a confusion matrix. However, the decision threshold may be determined in the following two steps by using the confusion matrix tables shown in FIGS. 23A and 23B.

[0194] 23A and 23B are diagrams showing examples of tables of confusion matrices according to other embodiments.

[0195] First, in step 1, the decision threshold determination unit 15 uses the validation data set to obtain the decision results (binary predicted values ​​of OK or NG) of the hybrid model selected by the hybrid model selection unit 14. The decision threshold determination unit 15 sets the threshold to 0.5 and creates a table that summarizes, for example, a confusion matrix as shown in FIG. 23A from combinations of the decision results and true values ​​(binary values ​​of OK or NG).

[0196] Next, in step 2, a desired accuracy is input, for example, an overdetection rate of 0.86%, and the above judgment results (binary predicted values ​​of OK or NG) are rearranged into a list of true values ​​(binary values ​​of OK or NG) to create a confusion matrix table as shown in Fig. 23B. Here, if an overdetection rate of 0.86% is the desired accuracy, the threshold value of 0.42 shown in Fig. 23B can be selected as the optimal threshold value (judgment threshold value).

[0197] The following forms may also be included within the scope of one or more aspects of the present disclosure.

[0198] (3) Some of the components constituting the hybrid model creation device 10 may be a computer system composed of a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, etc. A computer program is stored in the RAM or the hard disk unit. The microprocessor operates according to the computer program to achieve its functions. Here, the computer program is composed of a combination of multiple instruction codes that indicate commands for a computer to achieve a predetermined function.

[0199] (4) Some of the components constituting the hybrid model creation device 10 may be composed of one system LSI (Large Scale Integration). The system LSI is an ultra-multifunctional LSI manufactured by integrating multiple components on one chip, and specifically, is a computer system including a microprocessor, ROM, RAM, etc. A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating in accordance with the computer program.

[0200] (5) Some of the components constituting the hybrid model creation device 10 may be composed of an IC card or a standalone module that can be attached to each device. The IC card or the module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or the module may include the ultra-multifunction LSI. The microprocessor operates according to a computer program, causing the IC card or the module to achieve its functions. The IC card or the module may be tamper-resistant.

[0201] (6) Furthermore, some of the components constituting the hybrid model creation device 10 may be the computer program or the digital signal recorded on a computer-readable recording medium, such as a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), a semiconductor memory, etc. Also, some of the components may be the digital signals recorded on these recording media.

[0202] In addition, some of the components constituting the above-mentioned hybrid model creation device 10 may transmit the computer program or the digital signal via a telecommunication line, a wireless or wired communication line, a network such as the Internet, data broadcasting, etc.

[0203] (7) The present disclosure may be embodied as any of the methods described above. It may also be embodied as a computer program for implementing these methods by a computer, or as a digital signal comprising the computer program.

[0204] (8) The present disclosure may also be a computer system having a microprocessor and a memory, the memory storing the above-mentioned computer program, and the microprocessor operating in accordance with the computer program.

[0205] (9) The program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network, etc.

[0206] (10) The above-described embodiments and modifications may be combined with each other. [Industrial Applicability]

[0207] The present disclosure can be used in a method for creating a hybrid model that combines machine learning models to perform pass / fail product determination in an inspection process, a hybrid model method, a hybrid model creation device, and a program. [Explanation of symbols]

[0208] 10 Hybrid model creation device 11 Model Pool Section 11a model 12 Model Selection Section 12a Model Selection Process 13 Hybrid Model Candidate Creation Department 13a Hybrid model candidate creation process 14 Hybrid model selection section 14a Hybrid model selection process 15. Judgment threshold determination unit 15a Judgment threshold determination process

Claims

1. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; When generating the plurality of hybrid model candidates, excluding output values ​​that are estimated to be defective because they are higher than a threshold value from a plurality of output values ​​obtained by inputting a validation data set into each of the two or more models selected to configure the hybrid model candidate and causing the model to estimate a category; performing machine learning using the plurality of output values, from which output values ​​higher than the threshold have been excluded, as inputs, and using true values ​​of a validation data set corresponding to the plurality of output values, thereby generating the plurality of hybrid model candidates; Hybrid model creation method.

2. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; When generating the plurality of hybrid model candidates, one of the two or more models selected to constitute the hybrid model candidate Inputting multiple validation data sets to each of them and estimating categories, we obtained multiple output values, and calculated the convex hull when plotting the output values ​​estimated to be defective. excluding output values ​​included in the convex hull except for vertices of the convex hull from the plurality of output values; creating the plurality of hybrid model candidates by inputting and using the plurality of output values, excluding the vertices of the convex hull and the output values ​​contained in the convex hull, and performing machine learning using true values ​​of a validation data set corresponding to the plurality of output values; Hybrid model creation method.

3. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Before selecting the two or more models, a plurality of validation data sets are input to each of the pooled models to estimate categories, thereby obtaining estimation results for each of the pooled models; Using the estimation results, calculate all correlations of the multiple pooled models; removing from the pool of models any model that has a correlation with all other models greater than a threshold; selecting the two or more models from the plurality of models after models stronger than the threshold have been eliminated; Hybrid model creation method.

4. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Before generating the plurality of hybrid model candidates, a plurality of validation data sets are input to each of the pooled or selected models to estimate categories, thereby obtaining estimation results for each of the plurality of models; Using the estimation results, calculate all correlations of the multiple models; When creating the multiple hybrid model candidates, the multiple hybrid model candidates are created by combining the two or more selected models so as not to include a combination of two models having a stronger correlation than a threshold value. Hybrid model creation method.

5. In a deep learning model, the estimation result is an output result of an intermediate layer or a final layer of the deep learning model. The hybrid model creation method according to claim 3 or 4.

6. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Each of the plurality of hybrid model candidates is a machine learning model that receives as input two or more output results obtained by inputting a plurality of validation datasets into each of the two or more models selected to configure the hybrid model candidate and causing them to estimate categories, and outputs a determination result of determining categories of the plurality of validation datasets; calculating and comparing the importance of each of the two or more models selected to configure the hybrid model candidate from the judgment results output to the plurality of hybrid model candidates, and selecting one of the plurality of hybrid model candidates as a hybrid model; Hybrid model creation method.

7. When comparing the plurality of hybrid model candidates, Notify models whose calculated importance falls below a preset threshold value; The hybrid model creation method of claim 6.

8. When selecting the hybrid model, selecting, as a hybrid model, one of the plurality of hybrid model candidates excluding hybrid model candidates having models with the calculated importance levels lower than a preset threshold value; The hybrid model creation method of claim 6.

9. Further, in each of the multiple models selected to generate the multiple hybrid model candidates, a processing time required from inputting a validation dataset to estimating a category of the validation dataset is obtained; Based on the acquired processing times, a value of the processing time of each of the plurality of models relative to a sum of the processing times of all of the plurality of models is defined as a hardware cost; When generating the plurality of hybrid model candidates, adding a regularization term taking into account the hardware cost of each of the two or more models selected to configure the hybrid model candidate to a loss function of a machine learning model of each of the plurality of hybrid model candidates; The hybrid model creation method according to any one of claims 6 to 8.

10. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; Furthermore, in each of the multiple models selected to create the multiple hybrid model candidates, a FAR table is created, which is a table of oversight rates when a threshold is varied, based on a distribution of output values ​​obtained by inputting multiple data indicating defective products from the validation data set as the input data and estimating a category, When comparing the plurality of hybrid model candidates, a data sample included in a validation data set is input to each of the two or more models selected to configure the hybrid model candidate, and an output value obtained by estimating a category is compared with the FAR table to obtain a first FAR value of each of the two or more models for the data sample; multiplying the first FAR values ​​of the two or more models to obtain a second FAR value of the hybrid model candidate; When the second FAR value is smaller than a preset threshold value, a determination result that the data sample is a non-defective product is acquired as a determination result outputted from each of the plurality of hybrid model candidates, and the determination results are compared. Hybrid model creation method.

11. Further, in each of the multiple models selected to generate the multiple hybrid model candidates, a multiple validation data set is input and a category is estimated, thereby obtaining an estimation result for each of the selected multiple models; Using the estimation result, calculate correlation coefficients for all combinations of two models from the selected plurality of models; When obtaining the second FAR value, multiplying the first FAR values ​​of the two or more models obtained, and further multiplying the first FAR value by a coefficient that decreases as the correlation coefficient increases, thereby obtaining the second FAR value. The hybrid modeling method of claim 10.

12. a model pooling unit that pools a plurality of models for estimating categories of input data; At least one of the plurality of models is a machine-learned model, and a model selection unit selects two or more models from the plurality of pooled models; a hybrid model candidate creation unit that creates a plurality of hybrid model candidates for determining the category by combining the two or more selected models, and compares the plurality of hybrid model candidates; a hybrid model selection unit that selects one of the plurality of hybrid model candidates as a hybrid model; The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; When generating the plurality of hybrid model candidates, the hybrid model candidate generation unit excluding output values ​​that are estimated to be defective because they are higher than a threshold value from a plurality of output values ​​obtained by inputting a validation data set into each of the two or more models selected to configure the hybrid model candidate and causing the model to estimate a category; performing machine learning using the plurality of output values, from which output values ​​higher than the threshold have been excluded, as inputs, and using true values ​​of a validation data set corresponding to the plurality of output values, thereby generating the plurality of hybrid model candidates; Hybrid model creation device.

13. a model pooling unit that pools a plurality of models for estimating categories of input data; At least one of the plurality of models is a machine-learned model, and a model selection unit selects two or more models from the plurality of pooled models; a hybrid model candidate creation unit that creates a plurality of hybrid model candidates for determining the category by combining the two or more selected models, and compares the plurality of hybrid model candidates; a hybrid model selection unit that selects one of the plurality of hybrid model candidates as a hybrid model; The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; When generating the plurality of hybrid model candidates, the hybrid model candidate generation unit a convex hull is calculated when an output value estimated to be a defective product is plotted among a plurality of output values ​​obtained by inputting a plurality of validation data sets into each of the two or more models selected to configure the hybrid model candidate and estimating a category; excluding output values ​​included in the convex hull except for vertices of the convex hull from the plurality of output values; creating the plurality of hybrid model candidates by inputting and using the plurality of output values, excluding the vertices of the convex hull and the output values ​​contained in the convex hull, and performing machine learning using true values ​​of a validation data set corresponding to the plurality of output values; Hybrid model creation device.

14. a model pooling unit that pools a plurality of models for estimating categories of input data; At least one of the plurality of models is a machine-learned model, and a model selection unit selects two or more models from the plurality of pooled models; a hybrid model candidate creation unit that creates a plurality of hybrid model candidates for determining the category by combining the two or more selected models, and compares the plurality of hybrid model candidates; a hybrid model selection unit that selects one of the plurality of hybrid model candidates as a hybrid model; The model selection unit is Before selecting the two or more models, a plurality of validation data sets are input to each of the pooled models to estimate categories, thereby obtaining estimation results for each of the pooled models; Using the estimation results, calculate all correlations of the multiple pooled models; removing from the pool of models any model that has a correlation with all other models greater than a threshold; selecting the two or more models from the plurality of models after models stronger than the threshold have been eliminated; Hybrid model creation device.

15. a model pooling unit that pools a plurality of models for estimating categories of input data; At least one of the plurality of models is a machine-learned model, and a model selection unit selects two or more models from the plurality of pooled models; a hybrid model candidate creation unit that creates a plurality of hybrid model candidates for determining the category by combining the two or more selected models, and compares the plurality of hybrid model candidates; a hybrid model selection unit that selects one of the plurality of hybrid model candidates as a hybrid model; The hybrid model candidate creation unit Before generating the plurality of hybrid model candidates, a plurality of validation data sets are input to each of the pooled or selected models to estimate categories, thereby obtaining estimation results for each of the plurality of models; Using the estimation results, calculate all correlations of the multiple models; When creating the multiple hybrid model candidates, the multiple hybrid model candidates are created by combining the two or more selected models so as not to include a combination of two models having a stronger correlation than a threshold value. Hybrid model creation device.

16. a model pooling unit that pools a plurality of models for estimating categories of input data; At least one of the plurality of models is a machine-learned model, and a model selection unit selects two or more models from the plurality of pooled models; a hybrid model candidate creation unit that creates a plurality of hybrid model candidates for determining the category by combining the two or more selected models, and compares the plurality of hybrid model candidates; a hybrid model selection unit that selects one of the plurality of hybrid model candidates as a hybrid model; The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; The hybrid model candidate creation unit Furthermore, in each of the multiple models selected to create the multiple hybrid model candidates, a FAR table is created, which is a table of oversight rates when a threshold is varied, based on a distribution of output values ​​obtained by inputting multiple data indicating defective products from the validation data set as the input data and estimating a category, When comparing the plurality of hybrid model candidates, a data sample included in a validation data set is input to each of the two or more models selected to configure the hybrid model candidate, and an output value obtained by estimating a category is compared with the FAR table to obtain a first FAR value of each of the two or more models for the data sample; multiplying the first FAR values ​​of the two or more models to obtain a second FAR value of the hybrid model candidate; When the second FAR value is smaller than a preset threshold value, a determination result that the data sample is a non-defective product is acquired as a determination result outputted from each of the plurality of hybrid model candidates, and the determination results are compared. Hybrid model creation device.

17. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Let the computer run The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; When generating the plurality of hybrid model candidates, excluding output values ​​that are estimated to be defective because they are higher than a threshold value from a plurality of output values ​​obtained by inputting a validation data set into each of the two or more models selected to configure the hybrid model candidate and causing the model to estimate a category; generating the plurality of hybrid model candidates by performing machine learning using the plurality of output values, from which output values ​​higher than the threshold have been excluded, as inputs, and using true values ​​of a validation data set corresponding to the plurality of output values; A program executed by the computer.

18. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Let the computer run The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; When generating the plurality of hybrid model candidates, a convex hull is calculated when an output value estimated to be a defective product is plotted among a plurality of output values ​​obtained by inputting a plurality of validation data sets into each of the two or more models selected to configure the hybrid model candidate and estimating a category; excluding output values ​​included in the convex hull except for vertices of the convex hull from the plurality of output values; generating the plurality of hybrid model candidates by inputting the plurality of output values, excluding the vertices of the convex hull and the output values ​​contained in the convex hull, and performing machine learning using true values ​​of a validation data set corresponding to the plurality of output values; A program executed by the computer.

19. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Before selecting the two or more models, a plurality of validation data sets are input to each of the pooled models to estimate categories, thereby obtaining estimation results for each of the pooled models; Using the estimation results, calculate all correlations of the multiple pooled models; removing from the pool of models any model that has a correlation with all other models greater than a threshold; selecting the two or more models from the plurality of models after models stronger than the threshold are eliminated; A program that a computer runs.

20. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Before generating the plurality of hybrid model candidates, a plurality of validation data sets are input to each of the pooled or selected models to estimate categories, thereby obtaining estimation results for each of the plurality of models; Using the estimation results, calculate all correlations of the multiple models; When generating the plurality of hybrid model candidates, the plurality of hybrid model candidates are generated by combining the two or more selected models so as not to include a combination of two models having a stronger correlation than a threshold value; A program that a computer runs.

21. Pooling multiple models that estimate categories of input data, At least one of the plurality of models is a machine-learned model; creating a plurality of hybrid model candidates for determining the category by selecting and combining two or more models from the plurality of pooled models; selecting one of the plurality of hybrid model candidates as a hybrid model by comparing the plurality of hybrid model candidates; Let the computer run The input data is an inspection image of a manufactured product, The category to be determined is whether the manufactured product is good or bad; Furthermore, in each of the multiple models selected to create the multiple hybrid model candidates, a FAR table is created, which is a table of oversight rates when a threshold is varied, based on a distribution of output values ​​obtained by inputting multiple data indicating defective products from the validation data set as the input data and estimating a category, When comparing the plurality of hybrid model candidates, a data sample included in a validation data set is input to each of the two or more models selected to configure the hybrid model candidate, and an output value obtained by estimating a category is compared with the FAR table to obtain a first FAR value of each of the two or more models for the data sample; multiplying the first FAR values ​​of the two or more models to obtain a second FAR value of the hybrid model candidate; When the second FAR value is smaller than a preset threshold value, a determination result that the data sample is a non-defective product is acquired as a determination result outputted from each of a plurality of hybrid model candidates, and the determination results are compared. A program executed by the computer.

Images