Trained model generation method, determination device, determination method, and program

Abstract

To provide a learned model generation method capable of keeping one of a false positive rate and a false negative ratio below a predetermined level while suppressing the other.SOLUTION: A learned model generation method which is a generation method of a learned model for binary classification includes: setting at least one of a weighting parameter to weight one error more than another in a loss function which is a sum of an error when learning data is positive and an error when learning data is negative and a determination threshold to determine whether it is positive or negative as a hyper parameter; performing machine learning of a learning model to output a probability in which the learning data is positive or negative, using the loss function; and searching for the hyper parameter so that at least one of a false positive rate and a false negative ratio is below a predetermined value.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a method for generating a trained model, a determination device, a determination method, and a program. [Background technology]

[0002] Patent Document 1 discloses a technique for training a variational autoencoder using multiple object images as training data so as to reduce the error between input and output and to output the mean, variance, and higher-order statistics of a distribution approximated by a specific distribution for each unit pixel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-144314 Summary of the Invention [Problem to be solved by the invention]

[0004] Inspections to determine whether a product is pass or fail may use trained models for binary classification. In binary classification, not only true positives (TP) and true negatives (TN), but also false positives (FP) and false negatives (FN) can occur (see Figure 2).

[0005] A false positive (FP) is when a negative (fail) is classified as positive (pass). The false positive rate (FPR) is expressed as FP / (FP+TN). A false negative (FN) is when a positive (pass) is classified as negative (fail). The false negative rate (FNR) is expressed as FN / (FN+TP).

[0006] In inspections to determine whether a product passes or fails, a low false positive rate (FPR) is required to ensure quality. However, conventional learning models train to increase the accuracy rate (TP + TN) / (TP + TN + FP + FN), and therefore do not guarantee that the required level of false positive rate (FPR) (FPR≦α) will be met.

[0007] In addition, from the viewpoint of test efficiency, it is desirable that the false negative rate (FNR) is as low as possible.

[0008] The present invention has been made in consideration of the above-mentioned problems, and its main purpose is to provide a method for generating a trained model, a judgment device, a judgment method, and a program that can reduce at least one of the false positive rate and the false negative rate to a predetermined level or less. [Means for solving the problem]

[0009] To solve the above problem, one aspect of the present invention provides a trained model generation method for binary classification, which sets at least one of a weighting parameter that weights one error more heavily than the other in a loss function that adds up the error when the training data is positive and the error when the training data is negative, and a determination threshold for determining whether the training data is positive or negative as a hyperparameter, performs machine learning on the training model to output the probability that the training data is positive or negative using the loss function, and searches for the hyperparameters so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value. This makes it possible to keep at least one of the false positive rate and the false negative rate equal to or less than a predetermined value.

[0010] In the above aspect, the loss function may include a corrected probability obtained by correcting the probability output by the trained model using the decision threshold. This makes it possible to obtain a threshold suitable for classification.

[0011] In the above aspect, generating the trained model may include a learning step of provisionally setting the decision threshold and training a learning model using the loss function determined by the decision threshold, and an adjustment step of determining, from the classification results of the training model, an adjustment decision threshold that reduces the false positive rate or false negative rate to a predetermined value or less, and the learning step and the adjustment step may be repeated until the difference between the decision threshold and the adjustment decision threshold becomes a predetermined value or less. This makes it possible to obtain a threshold suitable for classification.

[0012] In the above aspect, the weighting parameter may be set so that the error when the training data is negative is weighted more heavily than the error when the training data is positive in the loss function, the decision threshold may be set to a predetermined fixed value, and the weighting parameter may be searched until the false positive rate is equal to or less than a predetermined value. This makes it possible to generate a trained model that guarantees that the false positive rate is equal to or less than a predetermined value. Furthermore, by setting the decision threshold to a fixed value, a common decision threshold can be used even when multiple trained models are generated, making management easier. Furthermore, by setting the decision threshold to a fixed value, there is no need to search for a decision threshold.

[0013] In the above aspect, in the search for the weighting parameters, the weights may be increased each time the weighting parameters are updated. By increasing the weights each time the weighting parameters are updated, it becomes possible to quickly search for weighting parameters that will result in a false positive rate of a predetermined value or less.

[0014] In the above aspect, a plurality of the weighting parameters may be prepared, and machine learning of the learning model may be performed in parallel for the plurality of the weighting parameters, and a trained model having a false positive rate of a predetermined value or less may be extracted. By performing machine learning of the learning model in parallel for the plurality of weighting parameters, it is possible to reduce the number of searches.

[0015] In the above aspect, when image data is input, the trained model may output a pass / fail determination result for a product included in the image data, thereby making it possible to determine the pass / fail of a product so that at least one of a false positive rate and a false negative rate is equal to or less than a predetermined value.

[0016] Another aspect of the present invention provides a determination device that includes an acquisition unit that acquires determination data, and a determination unit that sets at least one of a weighting parameter that weights one error more heavily than the other error in a loss function that adds up the error when the training data is positive and the error when the training data is negative as hyperparameters, performs machine learning on a learning model to output the probability that the training data is positive or negative using the loss function, and searches for the hyperparameters so that at least one of a false positive rate and a false negative rate is equal to or less than a predetermined value, and determines whether the determination data is positive or negative using a trained model that has been generated. This makes it possible to keep at least one of the false positive rate and the false negative rate equal to or less than a predetermined value.

[0017] In the above aspect, the loss function may include a corrected probability obtained by correcting the probability output by the trained model using the judgment threshold, and the judgment unit may compare the probability that the judgment data output from the trained model is positive or negative with the threshold. This makes it possible to obtain a threshold suitable for judgment.

[0018] In the above aspect, when image data is input as the judgment data, the trained model may output a judgment result of whether the product included in the image data is pass / fail, thereby making it possible to judge whether the product is pass / fail so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value.

[0019] Another aspect of the present invention is a determination method that acquires determination data, sets at least one of a weighting parameter that weights one error more heavily than the other error in a loss function that sums the error when the training data is positive and the error when the training data is negative, and a determination threshold for determining whether the data is positive or negative as a hyperparameter, performs machine learning on a learning model using the loss function to output the probability that the training data is positive or negative, and determines whether the determination data is positive or negative using a trained model generated by searching for the hyperparameters so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value. This makes it possible to keep at least one of the false positive rate and the false negative rate equal to or less than a predetermined value.

[0020] Another aspect of the present invention provides a program that causes a computer to acquire judgment data, set at least one of a weighting parameter that weights one error more heavily than the other error in a loss function that sums the error when the training data is positive and the error when the training data is negative, and a judgment threshold for determining whether the data is positive or negative as hyperparameters, perform machine learning on a learning model using the loss function to output the probability that the training data is positive or negative using the loss function, and determine whether the judgment data is positive or negative using a trained model generated by searching for the hyperparameters so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value. This makes it possible to keep at least one of the false positive rate and the false negative rate equal to or less than a predetermined value. [Effects of the Invention]

[0021] According to the present invention, it is possible to reduce either the false positive rate or the false negative rate to a predetermined value or less. [Brief explanation of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a determination system. [Figure 2] FIG. 1 is a diagram illustrating binary classification. [Figure 3] FIG. 1 is a diagram for explaining an ROC curve. [Figure 4] FIG. 1 is a diagram illustrating an example of a procedure for generating a trained model. [Figure 5] FIG. 10 is a diagram illustrating a loss function. [Figure 6] FIG. 10 is a diagram for explaining a determination method. [Figure 7] FIG. 10 is a diagram illustrating an example of a procedure of a determination method. [Figure 8] FIG. 10 is a diagram for explaining an example of a determination result. [Figure 9] FIG. 1 is a diagram illustrating an example of a procedure for generating a trained model. [Figure 10] FIG. 10 is a diagram for explaining a search for a weighting parameter. [Figure 11] FIG. 10 is a diagram illustrating an example of a procedure of a determination method. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0024] [System Configuration] 1 is a block diagram showing an example of the configuration of a determination system 10. The determination system 10 includes a determination device 1, a storage unit 2, a camera 3, and a display unit 4. The determination system 10 is an appearance inspection system in which the determination device 1 determines whether a product imaged by the camera 2 is pass or fail.

[0025] The determination device 1 is a computer including a CPU, RAM, ROM, nonvolatile memory, an input / output interface, etc. The CPU of the determination device 1 executes information processing in accordance with a program loaded from the ROM or nonvolatile memory to the RAM.

[0026] The program may be supplied via an information storage medium such as an optical disk or a memory card, or may be supplied via a communication network such as the Internet or a LAN.

[0027] The storage unit 2 is a storage device such as an HDD or an SSD. The storage unit 2 stores a trained model, a threshold, and the like used for determination by the determination device 1. The trained model and the threshold are generated in a learning phase, which will be described later.

[0028] The camera 3 is a digital camera that captures an image of a product and generates image data. The camera 3 outputs the generated image data to the determination device 1. The display unit 4 is a display device such as a liquid crystal display. The display unit 4 outputs the determination result by the determination device 1 on a screen.

[0029] The determination device 1 includes an acquisition unit 11 and a determination unit 12. These functional units are realized by the CPU of the determination device 1 executing information processing in accordance with a program loaded from the ROM or nonvolatile memory to the RAM.

[0030] The acquisition unit 11 acquires data for determination. Specifically, the acquisition unit 11 acquires image data generated by the camera 3. The image data is an example of data for determination, and includes the product to be determined.

[0031] The determination unit 12 determines whether the determination data is positive or negative using the trained model. Specifically, the determination unit 12 determines whether the product included in the image data is pass or fail using the trained model and thresholds stored in the storage unit 2. Details of the determination will be described later.

[0032] The trained model is a trained model for binary classification. In this embodiment, the trained model is an image discrimination model such as a convolutional neural network (CNN). A deep neural network in which neurons are combined in multiple stages is suitable for the neural network.

[0033] When image data is input as judgment data, the trained model outputs a judgment result of whether the product contained in the image data passes or fails. For example, a sigmoid function is used as the output element of the trained model, and a value between 0 and 1 is output, which represents the probability of the product passing or failing.

[0034] [Purpose of this embodiment] Before explaining the method for generating a trained model, the purpose of this embodiment will be explained.

[0035] In tests to determine whether a product passes or fails, it is possible that a non-conforming product may be judged as passing (false positive, or FP) or a passing product may be judged as failing (false negative, or FN) (see Figure 2). To ensure quality, it is necessary to reduce the false positive rate (FPR), but lowering the false positive rate (FPR) can lead to a higher false negative rate (FNR).

[0036] Therefore, in this embodiment, from the viewpoint of quality assurance, it is an object to suppress the false positive rate (FPR) to a predetermined value a1 or less, and at the same time, to suppress the false negative rate (FNR) as much as possible.

[0037] To explain this using the ROC curve (Receiver Operating Characteristic curve) in Figure 3, in conventional examples, the recall rate (TPR) may not be sufficient in the range of FPR≦a1. Therefore, in this embodiment, the objective is to improve the recall rate (TPR) as much as possible in the range of FPR≦a1, that is, to reduce the false negative rate (FNR) as much as possible.

[0038] [Learning Phase] A method for generating a trained model using machine learning will be described below. Figure 4 is a flow diagram showing an example of the procedure for generating a trained model. Each step shown in the diagram is realized by information processing by a computer.

[0039] If a model is trained to guarantee FPR≦a1, the TNR is likely to be high. Therefore, in this embodiment, in order to prioritize improving the TPR, that is, to prioritize suppressing the FNR, the model is trained using a loss function in which the error when the training data is pass (positive) is weighted more heavily than the error when the training data is fail (negative).

[0040] In this embodiment, in order to obtain an appropriate threshold value θ that achieves the above-mentioned objective, the loss function includes a corrected probability obtained by correcting the probability of passing (positive) or failing (negative) in accordance with the relationship with a given threshold value θ. The threshold value θ is a threshold value for determining failure (negative) (threshold value 1-θ is a threshold value for determining pass (positive)). Details of the loss function will be described later.

[0041] 4, first, the learning data is divided into model parameter learning data and tuning data (S11). The learning data is a data set in which learning images are associated with pass / fail labels.

[0042] The training data may further include validation data for validating the accuracy of the model. For example, 80% of the training data may be used as model parameter training data, 10% as tuning data, and 10% as validation data (data for evaluating overfitting).

[0043] Next, the threshold θ model is temporarily set to a certain value (S12). model may be determined as a positive constant multiple of a value based on a weighting coefficient in a loss function, as will be described later.

[0044] Next, a learning step is performed using the model parameter learning data (S13). In the learning step, the provisionally set threshold value θ model The model is trained using a loss function determined by

[0045] The model is trained by calculating a loss function from the pass / fail probability obtained when training images are input into the model and the pass / fail labels associated with the training images, and updating the model parameters to minimize the loss function.

[0046] Next, an adjustment step is performed using the tuning data (S14). In the adjustment step, an adjustment threshold θ is set so that FPR≦a1 based on the model determination result. tune is required.

[0047] The model judgment result is the pass / fail probability obtained by inputting the learning image of the tuning data into the model. Based on the obtained pass / fail probability and the pass / fail label associated with the learning image, the boundary of the probability where FPR≦a1 is determined by the adjustment threshold θ tune It is said that.

[0048] Next, the threshold θ model and the adjustment threshold θ tune It is determined whether the difference between the threshold values ​​θ is equal to or less than a predetermined value c (S15). model and the adjustment threshold θ tune If the difference exceeds the predetermined value c (S15: NO), the threshold θ model is updated (S16), and the learning step (S13) and the adjustment step (S14) are executed again.

[0049] Threshold θ model is a hyperparameter and cannot be explored by the loss function, so the threshold θ model The update is performed by, for example, bisection. model +θ tune ) / 2 is the new threshold θ model For example, as will be described later, the lower limit value at which the FNR monotonically decreases with respect to θ is estimated, and θ that is equal to or greater than this lower limit is searched for by a bisection method based on the monotonically decreasing nature of the FNR.

[0050] The learning step (S13) and the adjustment step (S14) are carried out by adjusting the threshold θ model and the adjustment threshold θ tune This is repeated until the difference between the threshold θmodel is within an appropriate range, the adjustment threshold θ tune This is repeated until it approaches .

[0051] Threshold θ model and the adjustment threshold θ tune When the difference between these two is equal to or less than the predetermined value c (S15: YES), the model's judgment accuracy, FPR, FNR, etc. are confirmed using the validation data, and all steps are completed. This makes it possible to obtain a trained model that can achieve FPR≦a1 while suppressing FNR, and an appropriate threshold θ for pass / fail judgment.

[0052] [Loss function] The loss function used in the learning step (S13) will be explained below.

[0053] As described above, this embodiment uses a loss function that can weight errors when the training data is acceptable (positive) more than errors when the training data is unacceptable (negative). For example, BCE (Binary Cross Entropy) with Logistic Loss, which allows such weighting, is used as the loss function (see Equation 1).

[0054]

number

[0055] p is a weighting coefficient, and by selecting a value greater than 1, the first term can be weighted. Note that selecting a value for p less than 1 will weight the second term. p is a hyperparameter and cannot be searched for using this loss function, so you can set an appropriate value that meets the conditions described below. x is the model output value, and σ(x) is the probability of predicting a pass (positive). The first term in the square brackets in Equation 1 represents the error when the training data is pass (positive), and the second term represents the error when the training data is fail (negative). The weighting coefficient p is included in the first term. This loss function is configured so that the loss increases the more the model's prediction class differs from the correct answer (i.e., FP or FN); when p is greater than 1, the loss increases the more the predicted class is FN.

[0056] Let σ0 be the probability that y=1 (pass) and 1-σ0 be the probability that y=0 (fail), and determine the threshold value θ as shown in Equation 2 (see Figure 5). Note that the threshold value θ is a threshold value for determining a fail (negative), and the threshold value 1-θ is a threshold value for determining a pass (positive).

[0057]

number

[0058] σ n (θ) is the corrected probability obtained by correcting σ0 according to the relationship with the threshold θ. n (θ) is discontinuous before and after the threshold value 1-θ.

[0059] If the weighting coefficient p is such that the difference in the loss function is l(θ+Δθ)-l(θ)<0 at a certain θ, Δθ>0, the FNR will decrease monotonically. When y=1, it is expressed by Equation 3, and when y=0, it is expressed by Equation 4.

[0060]

number

[0061]

number

[0062] Therefore, the difference in the loss function due to changing θ occurs only in the vicinity of σ0~1-θ. y=1 >N y=0 Therefore, we can assume that the number of pass (positive) and fail (negative) cases where σ0~1-θ is the same. Therefore, the weighting coefficient p c The condition is expressed by Equation 5.

[0063]

number

[0064] Based on this formula, it is possible to confirm the numerical range that the threshold value θ should satisfy.

[0065] [Inference phase] Hereinafter, a determination method using a trained model generated in the learning phase and a threshold value θ, which is realized in the determination system 10 (see FIG. 1), will be described. FIG. 6 is a diagram for explaining the determination method. FIG. 7 is a flow chart showing an example of the procedure of the determination method. FIG. 8 is a diagram for explaining an example of a determination result.

[0066] The determination device 1 functions as an acquisition unit 11 and a determination unit 12 by executing the information processing shown in FIG.

[0067] First, the determination device 1 acquires image data captured by the camera 3 (S21, function as the acquisition unit 11).

[0068] Next, the judgment device 1 uses the learned model and threshold value θ generated in the learning phase and stored in the memory unit 2 (see Figure 1) to judge whether the product included in the image data is OK (pass) or NG (fail) (S22-S26, function as the judgment unit 12).

[0069] Specifically, the determination device 1 inputs the image data into the trained model and calculates the OK probability p2 that the product is OK (accepted) (S22). The output element of the trained model is configured as a sigmoid function, and the OK probability p2 is output as a value between 0 and 1.

[0070] Next, the determination device 1 calculates the NG probability p1 that the product is NG (rejected) from the OK probability p2 (S23). The NG probability p1 is expressed as 1-p2. The OK probability p2 or the NG probability p1 is an example of the pass / fail determination result of the product contained in the image data.

[0071] Next, the determination device 1 compares the NG probability p1 with the threshold value θ, and makes a determination based on the magnitude relationship between the NG probability p1 and the threshold value θ (S24).

[0072] If the NG probability p1 is equal to or greater than the threshold value θ (S24: YES), the determination device 1 determines that the product is NG (rejected) (S25).

[0073] On the other hand, if the NG probability p1 is less than the threshold value θ (S24: NO), the determination device 1 determines that the product is OK (passed) (S26).

[0074] As shown in the example of Figure 8, when the threshold θ is 5%, images A and D, whose NG probability p1 is 5% or more, are judged as NG (fail), and images B and C, whose NG probability p1 is less than 5%, are judged as OK (pass).

[0075] According to the embodiment described above, in an inspection to determine whether a product included in image data is OK (pass) or NG (fail), it is possible to achieve FPR≦a1 while also suppressing FNR.

[0076] [Variations] In the above embodiment, the objective was to suppress the false positive rate (FPR) below a predetermined value a1 while also suppressing the false negative rate (FNR), but the opposite may also be true: to suppress the false negative rate (FNR) below a predetermined value b1 while also suppressing the false positive rate (FPR).

[0077] If a model is trained to guarantee FNR≦b1, the TPR is likely to be high. Therefore, in this modification, in order to prioritize improving the TNR, that is, to prioritize suppressing the FPR, the model is trained using a loss function in which the error when the training data is unacceptable (negative) is weighted more heavily than the error when the training data is acceptable (positive).

[0078] Specifically, in the loss function expressed by the above formula 1, the weighting coefficient p c is included in the second term in square brackets, which represents the error when the training data is negative (negative), rather than in the first term.

[0079] In the learning step (S13), the model is trained using such a loss function, and in the adjustment step (S14), an adjustment threshold θ , where FNR≦b1, is determined based on the model's judgment results. tune is required.

[0080] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made by those skilled in the art.

[0081] In the above embodiment, image data is used as the determination data, but the present invention is not limited to this, and various types of data can be used as the determination data.

[0082] In the above embodiment, the NG probability p1 is calculated and compared with a threshold value θ for determining NG (failure), but this is not limited to this; the OK probability p2 may be calculated and compared with a threshold value 1-θ for determining OK (pass).

[0083] Alternatively, a loss function may be used in which the error when the training data is acceptable (positive) is weighted more heavily than the error when the training data is unacceptable (negative), and the model may be trained so that the FNR is below a predetermined value.

[0084] Furthermore, a loss function may be used in which the error when the training data is unacceptable (negative) is weighted more heavily than the error when the training data is acceptable (positive), and the model may be trained so that the FPR is below a predetermined value.

[0085] [Second embodiment] The second embodiment will be described below. Fig. 9 is a flowchart showing an example of the procedure of a trained model generation method according to the second embodiment. Each step shown in the diagram is realized by information processing by a computer.

[0086] In the second embodiment, a weighting parameter r is set as a hyperparameter in the loss function, which weights the error when the training data is unsuccessful (negative) more than the error when the training data is successful (positive). The decision threshold θ is set as a predetermined fixed value, and machine learning of the learning model is performed using the loss function. The weighting parameter r is searched for until the FPR becomes equal to or less than a predetermined value α.

[0087] As shown in Fig. 9, first, the training data is divided into model training data, tuning data, and test data (S31). The training data is a data set in which training images are associated with pass / fail labels. For example, 80% of the training data is model training data, 10% is tuning data, and 10% is test data (data for overfitting evaluation).

[0088] Next, the weighting parameter r is set to a value r0 (S32), where r0 is a value greater than 1.

[0089] Next, model parameters are learned using the model learning data (S33). Specifically, learning is performed based on a loss function including a weighting parameter r that weights the error when the learning data is unsuccessful (negative) more than the error when the learning data is successful (positive), and the learned parameters k r (Hats are omitted in the text) to obtain the trained parameter k r is expressed by the following Equation 6.

[0090]

number

[0091] where D T represents the total training data. r is a weighting parameter and is a value greater than 1. Y l is the pass / fail label for the training data (1: pass, 0: fail). l (k) is the predicted value (probability) that data l will be judged as Y=1 (pass) or Y=0 (fail).

[0092] l0 is the number of failing (negative) learning data, and l1 is the number of passing (positive) learning data. Because it is difficult to prepare equal amounts of passing and failing learning data, the weights l0 and l1 of the number of data are used to suppress the effects of this bias.

[0093] Equation 6 is configured so that the weighting parameter r is set to a value greater than 1, so that if the training data is unacceptable and the prediction deviates, the loss becomes relatively large. Therefore, under this loss function, learning is performed to minimize the FPR.

[0094] In S32 and S33, the weighting parameter r=r0 is set, and then the model parameters are learned using the model learning data, and the learned parameters k ro is obtained.

[0095] Next, the learned parameter k ro The value of y for checking the FPR is calculated from the determination result using tuning data based on a model including (S34). y is expressed by the following Equation 7.

[0096]

number

[0097] where FPR is the number of trained parameters kro is an FPR calculated from the determination results of a model including the above. α is a preset value that is appropriately selected based on the level of FPR required for the model.

[0098] Next, it is determined whether the value of y can be approximately regarded as 0 (S35). Specifically, it is determined whether the value of y is within a predetermined range including 0.

[0099] If the value of y cannot be regarded as approximately 0 (S35: NO), the weighting parameter r is updated (S36), and the learning of the model parameters (S33) and the calculation of the value of y (S34) are performed again.

[0100] In this way, the weighting parameter r is searched for by repeating the processes of S33 to S36. The weighting parameter r is searched for until the value of y can be regarded as approximately 0 (S35: YES), that is, until the FPR becomes equal to or less than a predetermined value.

[0101] A method such as a linear search method is used to search for the weighting parameter r. As shown by the solid line in Fig. 10, it is expected that the value of y will generally decrease as the weighting parameter r increases, so it is preferable to increase the weight each time the weighting parameter r is updated.

[0102] However, as shown by the dashed line in Figure 10, in actual learning, the value of y often fluctuates due to fluctuations in data selection, etc. Therefore, it is also possible to prepare multiple weighting parameters r, perform learning on them in parallel, and then compare the values ​​of y to determine the weighting parameter r. In this case, it is expected that the number of search loops can be reduced.

[0103] Note that if the weighting parameter r is set too large, the influence of the first term in the loss function becomes relatively small, which tends to increase the FNR, so it is desirable to keep the weighting parameter r at the minimum value within the range that satisfies FPR≦α and suppress the increase in FNR. This makes it possible to suppress the FNR while ensuring FPR≦α.

[0104] If the value of y can be considered to be approximately 0 (S35: YES), the learned parameter k obtained by the search r That is, the value of y is calculated again from the determination result using the test data (S37), and it is determined whether the value of y can be approximately regarded as 0 (S38).

[0105] If the value of y can be considered to be approximately 0 (S38: YES), the search ends. Here, it is preferable to compare with a value smaller than that in S35 above. For example, if it is determined in S35 above that the value of y is 0 or less, it is preferable to determine in S38 whether the value of y is equal to or less than a value a (e.g., a=-0.01) that is slightly smaller than 0.

[0106] If the value of y cannot be regarded as approximately 0 (S38: NO), the processes of S32 to S36 are repeated.

[0107] By following the above steps, it is possible to obtain a trained model that has an excellent balance between FPR and FNR, ensuring that FPR≦α while also suppressing FNR.

[0108] 11 is a diagram showing an example of the procedure of a determination method according to the second embodiment, which uses a trained model generated by the trained model generation method according to the second embodiment. The determination device 1 executes the information processing shown in the diagram according to a program.

[0109] First, the determination device 1 acquires image data captured by the camera 3 (S41, function as the acquisition unit 11).

[0110] Next, the determination device 1 uses the trained model to determine whether the product included in the image data is OK (pass) or NG (fail) (S42-S45, function as the determination unit 12).

[0111] Specifically, the determination device 1 inputs the image data into the trained model and calculates the OK probability p2 that the product is OK (accepted) (S42). The output element of the trained model is configured as a sigmoid function, and the OK probability p2 is output as a value between 0 and 1.

[0112] Next, the determination device 1 calculates the OK probability p2 by the determination threshold θ f Compared with the OK probability p2, the judgment threshold θ f In the second embodiment, the determination threshold θ f is a predetermined fixed value.

[0113] In S43, the OK probability p2 is determined based on the judgment threshold θ f If the above is true, the determination device 1 determines that the product is OK (passes) (S44).

[0114] On the other hand, in S43, the OK probability p2 is determined by the judgment threshold θ f If it is less than this, the determination device 1 determines that the product is NG (rejected) (S45).

[0115] Here, the OK probability p2 and the judgment threshold θ f However, the present invention is not limited to this. As in the first embodiment, the NG probability p1 and the determination threshold value 1-θ f The determination may be made by comparing [Explanation of symbols]

[0116] 1 Determination device, 2 Memory unit, 3 Camera, 4 Display unit, 10 Determination system, 11 Acquisition Section, 12 Judgment Section

Claims

1. A method for generating a trained model for binary classification, comprising: At least one of a weighting parameter that weights one error in a loss function obtained by adding together an error when the learning data is positive and an error when the learning data is negative more than the other error, and a determination threshold for determining whether the data is positive or negative is set as a hyperparameter; performing machine learning of a learning model using the loss function so as to output the probability that the learning data is positive or negative; The hyperparameters are searched for so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value. How to generate a trained model.

2. The loss function includes a corrected probability obtained by correcting the probability output by the learning model by the decision threshold. The method for generating a trained model according to claim 1.

3. The generation of the trained model is a learning step of provisionally setting the decision threshold and learning a learning model using the loss function determined by the decision threshold; an adjustment step of calculating an adjustment decision threshold value from the classification result of the learning model such that the false positive rate or false negative rate is equal to or less than a predetermined value; Including, The learning step and the adjustment step are repeated until the difference between the determination threshold value and the adjustment determination threshold value becomes equal to or less than a predetermined value. The method for generating a trained model according to claim 2.

4. setting the weighting parameter such that an error when the training data is negative in the loss function is weighted more heavily than an error when the training data is positive; The determination threshold is set as a predetermined fixed value, The weighting parameters are searched until the false positive rate is equal to or less than a predetermined value. The method for generating a trained model according to claim 1.

5. In the search for the weighting parameters, the weighting is increased every time the weighting parameters are updated. The method for generating a trained model according to claim 4.

6. preparing a plurality of the weighting parameters; Machine learning of a learning model is performed in parallel for a plurality of the weighting parameters; extracting a trained model in which the false positive rate is equal to or less than a predetermined value; The method for generating a trained model according to claim 4.

7. When image data is input, the trained model outputs a pass / fail judgment result for the product included in the image data. The method for generating a trained model according to claim 1.

8. an acquisition unit that acquires data for determination; a determination unit that sets at least one of a weighting parameter that weights one error in a loss function obtained by adding together an error when the training data is positive and an error when the training data is negative more heavily than the other error, and a determination threshold for determining whether the data is positive or negative, as hyperparameters, performs machine learning on a learning model so as to output the probability that the training data is positive or negative using the loss function, and determines whether the data for determination is positive or negative using a trained model that is generated by searching for the hyperparameters so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value; A determination device comprising:

9. the loss function includes a corrected probability obtained by correcting the probability output by the trained model by the decision threshold; The determination unit compares the probability that the determination data output from the trained model is positive or negative with the determination threshold. The determination device according to claim 8 .

10. When image data as the judgment data is input, the trained model outputs a judgment result of whether the product included in the image data is pass or fail. The determination device according to claim 8 .

11. Obtaining data for judgment, At least one of a weighting parameter that weights one error in a loss function that adds together the error when the training data is positive and the error when the training data is negative more than the other error, and a judgment threshold for determining whether the data is positive or negative, is set as a hyperparameter, and machine learning is performed on a learning model so that the probability that the training data is positive or negative is output using the loss function, and a trained model generated by searching for the hyperparameters so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value is used to determine whether the judgment data is positive or negative. Judgment method.

12. Obtaining data for judgment, and a weighting parameter that weights one error in a loss function obtained by adding together an error when the training data is positive and an error when the training data is negative, more than the other error, and a judgment threshold for determining whether the data is positive or negative are set as hyperparameters; machine learning is performed on a learning model so that the probability that the training data is positive or negative is output using the loss function; and a trained model is generated by searching for the hyperparameters so that at least one of the false positive rate and the false negative rate is equal to or less than a predetermined value, and the trained model is used to determine whether the data for judgment is positive or negative; A program that causes a computer to execute the following.

Images