Method for generating learned model, determination device, determination method, and program

Abstract

The method for generating a learned model is a method for generating a learned model for binary classification, and sets, as a hyper-parameter, at least one of: a weighting parameter for weighting an error of one of loss functions compared to an error of the other of the loss functions; the loss function is a function obtained by adding an error when the learning data is positive and an error when the learning data is negative, and the determination threshold value is used for determining whether the learning data is positive or negative, and machine learning of the learning model is performed using the loss function so that the probability that the output learning data is positive or negative is equal to the probability that the output learning data is negative or positive. The search of the hyper-parameter is performed such that at least one of a false positive rate and a false negative rate is equal to or less than a predetermined value.

Description

Technical Field

[0001] The present disclosure relates to a method for generating a learned model, a determination device, a determination method, and a program. Background Art

[0002] Patent document 1 discloses the following technology: a variational autoencoder is trained using multiple object images as learning data so that the error between input and output becomes smaller, and the mean, variance, and higher-order statistics of the distribution approximated by a specific distribution are output for each unit pixel.

[0003] Prior Art Literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-144314 Summary of the invention

[0006] -Problems to be solved by the invention-

[0007] In the inspection to determine whether a product is qualified or not, a learned model for binary classification is sometimes used. In binary classification, not only true positives (TP: True Positive) and true negatives (TN: True Negative) may occur, but also false positives (FP: False Positive) and false negatives (FN: False Negative) may occur (see Figure 2 ).

[0008] False positive (FP) is the classification of negative (unqualified) as positive (qualified). The false positive rate (FPR: False Positive Rate) is expressed by FP / (FP+TN). False negative (FN) is the classification of positive (qualified) as negative (unqualified). The false negative rate (FNR: False Negative Rate) is expressed by FN / (FN+TP).

[0009] In the inspection to determine whether a product is qualified or not, in order to ensure quality, the false positive rate (FPR) is reduced. However, the previous learning model learns to improve the accuracy rate (TP+TN) / (TP+TN+FP+FN), so it cannot guarantee that the required level of the false positive rate (FPR) is met (FPR≤α).

[0010] Furthermore, from the viewpoint of the work efficiency of the examination, the false negative rate (FNR) is preferably as low as possible.

[0011] The present disclosure has been made in view of the above-mentioned problems. A main object of the present disclosure is to provide a generation method, a determination device, a determination method, and a program for a learned model that can set at least one of a false positive rate and a false negative rate to a predetermined value or less.

[0012] -Methods for solving problems-

[0013] In order to solve the above-mentioned problems, a method for generating a learned model in one mode of the present disclosure is a method for generating a learned model for binary classification, at least one of a weighting parameter and a judgment threshold is set as a hyperparameter, the weighting parameter is a parameter for weighting the error of one side of the loss function compared with the error of the other side of the loss function, the loss function is a function obtained by adding the error when the learning data is positive and the error when the learning data is negative, the judgment threshold is used to judge whether it is positive or negative, the loss function is used to perform machine learning of the learning model so as to output the probability of the learning data being positive or negative, and the hyperparameter is searched so that at least one of the false positive rate and the false negative rate is below a given value. Thus, at least one of the false positive rate and the false negative rate can be below a given value.

[0014] In the above aspect, the loss function may include a corrected probability obtained by correcting the probability output by the learning model by the determination threshold, thereby obtaining a threshold suitable for classification.

[0015] In the above method, the generation of the learned model may include: a learning step, temporarily setting the judgment threshold, using the loss function determined by the judgment threshold to learn the learning model; and an adjustment step, according to the classification result of the learning model, finding the false positive rate or the false negative rate to be below a given adjustment judgment threshold, repeating the learning step and the adjustment step until the difference between the judgment threshold and the adjustment judgment threshold is below a given value. In this way, a threshold suitable for classification can be obtained.

[0016] In the above method, the weighting parameter that weights the error when the learning data is negative compared to the error when the learning data is positive is set in the loss function, the judgment threshold is set to a given fixed value, and the weighting parameter is searched until the false positive rate becomes below the given value. In this way, a learning model that guarantees a false positive rate below the given value can be generated. In addition, by setting the judgment threshold to a fixed value, even if multiple learning models are generated, a common judgment threshold can be used, which can simplify management. Furthermore, by setting the judgment threshold to a fixed value, there is no need to search for the judgment threshold.

[0017] In the above-mentioned manner, in the search of the weighted parameter, the weighting may be increased each time the weighted parameter is updated. Thus, by increasing the weighting each time the weighting is updated, a weighted parameter with a false positive rate below a given value can be quickly searched.

[0018] In the above method, it is also possible to prepare a plurality of weighting parameters, perform machine learning of the learning model in parallel for the plurality of weighting parameters, and extract a learning model with a false positive rate below a given value. Thus, by performing machine learning of the learning model in parallel for the plurality of weighting parameters, the number of searches can be reduced.

[0019] In the above method, when the learned model is input with image data, it may output a determination result of whether the product included in the image data is qualified or not. Thus, the qualification of the product can be determined so that at least one of the false positive rate and the false negative rate is below a given value.

[0020] In addition, another mode of the determination device disclosed in the present invention comprises: an acquisition unit, which acquires determination data; and a determination unit, which sets at least one of a weighting parameter and a determination threshold as a hyperparameter, wherein the weighting parameter is a parameter for weighting the error of one side of the loss function compared with the error of the other side of the loss function, wherein the loss function is a function obtained by adding the error when the learning data is positive to the error when the learning data is negative, wherein the determination threshold is used to determine whether it is positive or negative, and the machine learning of the learning model is performed using the loss function to make the probability of outputting the learning data to be positive or negative, and the hyperparameter is searched to make at least one of the false positive rate and the false negative rate become a given value or less, and the learning completed model is generated to determine whether the determination data is positive or negative. Thus, at least one of the false positive rate and the false negative rate can be made below a given value.

[0021] In the above method, the loss function may include a corrected probability after the probability output by the learning model is corrected by the judgment threshold, and the judgment unit compares the probability that the judgment data output from the learned model is positive or negative with the threshold. Thus, a threshold suitable for judgment can be obtained.

[0022] In the above method, the learned model may output a determination result of whether a product included in the image data is qualified when the image data is input as the determination data. Thus, the qualification of the product can be determined so that at least one of the false positive rate and the false negative rate is below a given value.

[0023] In addition, another mode of the determination method disclosed in the present invention includes the following steps: obtaining determination data; and setting at least one of a weighting parameter and a determination threshold as a hyperparameter, wherein the weighting parameter is a parameter for weighting the error of one side of the loss function compared to the error of the other side of the loss function, wherein the loss function is a function obtained by adding the error when the learning data is positive to the error when the learning data is negative, wherein the determination threshold is used to determine whether it is positive or negative, and the machine learning of the learning model is performed using the loss function to output the probability that the learning data is positive or negative, and the learning model generated by searching the hyperparameter so that at least one of the false positive rate and the false negative rate becomes below a given value, and determining whether the determination data is positive or negative. Thus, at least one of the false positive rate and the false negative rate can be below a given value.

[0024] In addition, a program of another mode of the present disclosure causes a computer to perform the following processing: obtaining judgment data; and setting at least one of a weighting parameter and a judgment threshold as a hyperparameter, wherein the weighting parameter is a parameter for weighting the error of one side of the loss function compared to the error of the other side of the loss function, wherein the loss function is a function obtained by adding the error when the learning data is positive to the error when the learning data is negative, wherein the judgment threshold is used to judge whether it is positive or negative, and the machine learning of the learning model is performed using the loss function to make the probability of outputting the learning data to be positive or negative, and the hyperparameter is searched so that at least one of the false positive rate and the false negative rate becomes a learning model generated below a given value, and it is determined whether the judgment data is positive or negative. Thus, at least one of the false positive rate and the false negative rate can be made below a given value.

[0025] -Effects of the Invention-

[0026] According to the present disclosure, one of the false positive rate and the false negative rate can be set to a predetermined value or less. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a diagram showing a configuration example of a determination system.

[0028] Figure 2 It is a diagram used to illustrate binary classification.

[0029] Figure 3 It is a graph for explaining the ROC curve.

[0030] Figure 4 This is a diagram showing an example of the steps of a method for generating a learned model.

[0031] Figure 5 is a graph used to illustrate the loss function.

[0032] Figure 6 It is a diagram for explaining the determination method.

[0033] Figure 7 It is a diagram showing an example of the procedure of the determination method.

[0034] Figure 8 It is a diagram for explaining an example of the determination result.

[0035] Fig. 9 This is a diagram showing an example of the steps of a method for generating a learned model.

[0036] Fig.10 It is a diagram for explaining the search of weighting parameters.

[0037] Fig.11 It is a diagram showing an example of the procedure of the determination method. DETAILED DESCRIPTION

[0038] [First embodiment]

[0039] Hereinafter, a first embodiment of the present disclosure will be described with reference to the drawings.

[0040] [System Structure]

[0041] Figure 1 1 is a block diagram showing a configuration example of a determination system 10. The determination system 10 includes a determination device 1, a storage unit 2, a camera 3, and a display unit 4. The determination system 10 is an appearance inspection system that uses the determination device 1 to determine whether a product captured by the camera 3 is acceptable.

[0042] The determination device 1 is a computer including a CPU, a RAM, a ROM, a nonvolatile memory, an input / output interface, etc. The CPU of the determination device 1 executes information processing according to a program loaded from the ROM or the nonvolatile memory to the RAM.

[0043] The program may be provided via an information storage medium such as an optical disk or a memory card, or may be provided via a communication network such as the Internet or a LAN.

[0044] The storage unit 2 is a storage device such as a HDD or an SDD. The storage unit 2 stores a learned model and a threshold value and the like used in the determination of the determination device 1. The learned model and the threshold value are generated in a learning phase described later.

[0045] The camera 3 is a digital camera that captures a product and generates image data. The camera 3 outputs the generated image data to the determination device 1. The display unit 4 is a display device such as a liquid crystal display. The display unit 4 outputs the determination result of the determination device 1 to a screen.

[0046] The determination device 1 includes an acquisition unit 11 and a determination unit 12. These functional units are realized by the CPU of the determination device 1 executing information processing according to a program loaded from a ROM or a nonvolatile memory to a RAM.

[0047] The acquisition unit 11 acquires determination data. Specifically, the acquisition unit 11 acquires image data generated by the camera 3. The image data is an example of determination data and includes a product to be determined.

[0048] The determination unit 12 determines whether the determination data is positive or negative using the learned model. Specifically, the determination unit 12 determines whether the product included in the image data is good or bad using the learned model and the threshold value stored in the storage unit 2. The details of the determination will be described later.

[0049] The learned model is a learned model for binary classification. In the present embodiment, the learned model is, for example, an image discrimination model such as a convolutional neural network (CNN). The neural network is preferably a deep neural network that combines neurons in multiple stages.

[0050] When the learned model receives image data as determination data, it outputs a determination result of whether the product contained in the image data is acceptable or not. The output element of the learned model outputs a value greater than 0 and less than 1 indicating the probability of whether the product is acceptable or not, using, for example, a sigmoid function.

[0051] [Purpose of this embodiment]

[0052] Before describing a method for generating a learned model, the purpose of this embodiment will be described.

[0053] In the process of determining whether a product is qualified or not, it is possible that a non-conforming product is judged as qualified (false positive, i.e. FP), and a qualified product is judged as unqualified (false negative, i.e. FN) (refer to Figure 2 ). In order to ensure quality, the false positive rate (FPR) is reduced, but if the false positive rate (FPR) is reduced, the false negative rate (FNR) may increase.

[0054] Therefore, in the present embodiment, from the viewpoint of quality assurance, an object is to suppress the false positive rate (FPR) to a given value a1 or less, while also suppressing the false negative rate (FNR) as much as possible.

[0055] If using Figure 3As illustrated by the ROC curve (Receiver Operating Characteristic curve), in the conventional example, the reproducibility rate (TPR) is sometimes insufficient within the range of FPR≤a1. Therefore, in the present embodiment, the purpose is to increase the reproducibility rate (TPR) as much as possible within the range of FPR≤a1, that is, to suppress the false negative rate (FNR) as much as possible.

[0056] [Learning phase]

[0057] The following describes a method for generating a learned model based on machine learning. Figure 4 1 is a flowchart showing an example of a method for generating a learned model. Each step shown in the figure is implemented by computer-based information processing.

[0058] When the model is learned so as to ensure that FPR ≤ a1, TNR tends to become higher. Therefore, in this embodiment, in order to give priority to improving TPR, that is, to give priority to suppressing FNR, a loss function is used to weight the error when the learning data is qualified (positive) compared to the error when the learning data is unqualified (negative), and the model is learned.

[0059] In addition, in this embodiment, in order to obtain an appropriate threshold value θ to achieve the above-mentioned purpose, the loss function includes a corrected probability that corrects the probability of passing (positive) or failing (negative) according to the relationship with a given threshold value θ. The threshold value θ is a threshold value for determining failure (negative) (the threshold value 1-θ becomes the threshold value for determining passing (positive)). The details of the loss function will be described later.

[0060] like Figure 4 As shown, first, the learning data is divided into model parameter learning data and tuning data (S11). The learning data is a data set in which learning images are associated with labels of pass or fail.

[0061] The learning data may also include verification data for verifying the accuracy of the model. For example, 80% of the learning data may be model parameter learning data, 10% may be tuning data, and 10% may be verification data (overlearning evaluation data).

[0062] Next, the threshold θ model Temporarily set to a certain value (S12). Threshold θ model For example, as described later, it may be determined as a positive constant multiple of the value based on the weighting coefficient in the loss function.

[0063] Next, a learning step is performed using the model parameter learning data (S13). In the learning step, a threshold value θ set temporarily is used. modelThe loss function is determined to learn the model.

[0064] The model is learned by calculating a loss function based on the pass / fail probability obtained by inputting a learning image into the model and the pass / fail label associated with the learning image, and updating the model parameters to minimize the loss function.

[0065] Next, the tuning data is used to perform an adjustment step (S14). In the adjustment step, an adjustment threshold value θ that satisfies FPR ≤ a1 is obtained based on the judgment result of the model. tune .

[0066] The judgment result of the model is the probability of passing or failing obtained by inputting the learning image of the tuning data into the model. Based on the obtained probability of passing or failing and the pass or failing label associated with the learning image, the probability boundary of FPR≤a1 can be obtained as the adjustment threshold θ tune .

[0067] Next, we determine the threshold θ model and the adjustment threshold θ tune Is the difference between the given value c or less (S15)? model and the adjustment threshold θ tune If the difference exceeds the given value c (S15: No), the threshold θ is updated. model ( S16 ), the learning step ( S13 ) and the adjustment step ( S14 ) are performed again.

[0068] Threshold θ model is a hyperparameter and cannot be searched through the loss function, so the threshold θ model The update of is performed, for example, by binary search. Specifically, (θ model +θ tune ) / 2 is used as the new threshold θ model For example, as described later, a lower limit value of the FNR monotonically decreasing with respect to θ is estimated, and a binary search is performed based on the monotonically decreasing property of the FNR to obtain θ that is equal to or greater than the lower limit value.

[0069] The learning step (S13) and the adjustment step (S14) are repeated until the threshold θ model and the adjustment threshold θ tune Until the difference between θ and c is less than the given value c. That is, the process is repeated until the threshold value θ model Approaching the adjustment threshold value θ within an appropriate range tune until.

[0070] If the threshold θ model and the adjustment threshold θ tuneIf the difference is less than the given value c (S15: Yes), all steps are completed after confirming the judgment accuracy, FPR, FNR, etc. of the model using the verification data. In this way, a learning model that can achieve FPR≤a1 and suppress FNR and a threshold value θ suitable for pass / fail judgment can be obtained.

[0071] [Loss function]

[0072] The loss function used in the learning step ( S13 ) is described below.

[0073] As described above, in this embodiment, a loss function is used that can weight the error when the learning data is qualified (positive) compared to the error when the learning data is unqualified (negative). For example, BCE (Binary Cross Entropy) with Logistic Loss that can perform such weighting is used as a loss function (see Formula 1).

[0074] [Number 1]

[0075]

[0076] p is a weighting coefficient, and the first term can be weighted by selecting a value greater than 1. In addition, when a value less than 1 is selected for p, the second term is weighted. p is a hyperparameter and cannot be searched through this loss function, so an appropriate value that meets the conditions described later can be set. x is the output value of the model, and σ(x) is the probability of being predicted as qualified (positive). The first term in the square brackets of Formula 1 represents the error when the learning data is qualified (positive), and the second term represents the error when the learning data is unqualified (negative). The weighting coefficient p is included in the first term. In this loss function, the more the prediction of the model is different from the correct answer (i.e., FP or FN), the greater the loss. When p is a value greater than 1, the loss increases the more the predicted category is FN.

[0077] Let σ0 be the probability of y=1 (pass), and 1-σ0 be the probability of y=0 (fail), and the judgment for the threshold value θ is determined as shown in Formula 2 (refer to Figure 5 ). In addition, the threshold value θ is a threshold value for judging failure (negative), and the threshold value 1-θ is a threshold value for judging success (positive).

[0078] [Number 2]

[0079]

[0080] σ n (θ) is the corrected probability obtained by correcting σ0 based on the relationship with the threshold θ. n (θ) is discontinuous before and after the threshold 1-θ.

[0081] If the difference in the loss function is a weighting coefficient p of l(θ+Δθ)-l(θ)<0 when θ and Δθ>0, the FNR decreases monotonically. The case of y=1 is expressed by equation 3, and the case of y=0 is expressed by equation 4.

[0082] [Number 3]

[0083]

[0084] [Number 4]

[0085]

[0086] Therefore, the difference in the loss function caused by changing θ only occurs in the vicinity of σ0 to 1-θ. y＝1 >N y＝0 , so the case of σ0~1-θ can be assumed that the number of qualified (positive) and unqualified (negative) is the same. Therefore, the condition of the weighting coefficient p of the loss function is monotonically decreasing, which is expressed by Equation 5.

[0087] [Number 5]

[0088]

[0089] Based on this formula, it is possible to confirm the numerical range that the threshold value θ should satisfy.

[0090] [Inference Phase]

[0091] Next, in the determination system 10 (refer to Figure 1 ) is implemented in the following, using the learned model generated in the learning phase and the determination method of the threshold θ. Figure 6 It is a diagram for explaining the determination method. Figure 7 This is a flowchart showing an example of the procedure of the determination method. Figure 8 It is a diagram for explaining an example of the determination result.

[0092] The determination device 1 executes the Figure 6 The information processing shown in FIG. 1 is performed to function as the acquisition unit 11 and the determination unit 12 .

[0093] First, the determination device 1 acquires image data captured by the camera 3 ( S21 , as a function of the acquisition unit 11 ).

[0094] Next, the determination device 1 uses the data generated in the learning phase and stored in the storage unit 2 (see Figure 1 ) and the threshold value θ, it is determined whether the product included in the image data is OK (qualified) or NG (unqualified) (S22-S26, as the function of the determination unit 12).

[0095] Specifically, the determination device 1 inputs the image data into the learned model and calculates the probability p2 of the product being OK (qualified) ( S22 ). The output element of the learned model is composed of a Sigmoid function, and the OK probability p2 is output as a value greater than or equal to 0 and less than or equal to 1.

[0096] Next, the determination device 1 calculates the NG probability p1 of the product being NG (failed) based on the OK probability p2 (S23). The NG probability p1 is represented by 1-p2. The OK probability p2 or the NG probability p1 is an example of the determination result of whether the product is good or not included in the image data.

[0097] Next, the determination device 1 compares the NG probability p1 with the threshold value θ, and makes a determination based on the magnitude relationship between the NG probability p1 and the threshold value θ ( S24 ).

[0098] When the NG probability p1 is equal to or greater than the threshold value θ ( S24 : YES), the determination device 1 determines that the product is NG (defective) ( S25 ).

[0099] On the other hand, when the NG probability p1 is smaller than the threshold value θ ( S24 : No), the determination device 1 determines that the product is OK (passive) ( S26 ).

[0100] like Figure 8 As shown in the example, when the threshold θ is 5%, images A and D with NG probability p1 of 5% or more are judged as NG (failed), and images B and C with NG probability p1 of less than 5% are judged as OK (passed).

[0101] According to the first embodiment described above, in the inspection for determining whether the product included in the image data is OK (qualified) or NG (unqualified), FPR≦a1 can be achieved while suppressing FNR.

[0102] [Modifications]

[0103] In the first embodiment described above, the purpose is to suppress the false positive rate (FPR) below a given value a1 while also suppressing the false negative rate (FNR), but conversely, the false negative rate (FNR) may be suppressed below a given value b1 while also suppressing the false positive rate (FPR).

[0104] When the model is learned to ensure that FNR ≤ b1, TPR tends to become higher. For this reason, in this variation, in order to give priority to improving TNR, that is, to give priority to suppressing FPR, a loss function is used to weight the error when the learning data is unqualified (negative) compared to the error when the learning data is qualified (positive), and the model is learned.

[0105] Specifically, in the loss function represented by the above-mentioned equation 1, the weight coefficient p is included in the second term in the square brackets representing the error when the learning data is unqualified (negative), not in the first term.

[0106] In the learning step (S13), the model is learned using such a loss function. In the adjustment step (S14), an adjustment threshold θ is obtained to satisfy FNR ≤ b1 based on the judgment result of the model. tune .

[0107] As mentioned above, although the 1st embodiment of this disclosure was described, this disclosure is not limited to the embodiment described above, and various modifications can be made by those skilled in the art.

[0108] In the above-mentioned embodiment, image data is used as determination data, but the present invention is not limited to this, and various data can be used as determination data.

[0109] In the above embodiment, the NG probability p1 is calculated and compared with the threshold value θ for determining NG (failure), but the present invention is not limited thereto. The OK probability p2 may be calculated and compared with the threshold value 1-θ for determining OK (pass).

[0110] Alternatively, a loss function that weights the error when the learning data is qualified (positive) compared to the error when the learning data is unqualified (negative) may be used to perform model learning so that the FNR becomes equal to or less than a given value.

[0111] Alternatively, a loss function that weights the error when the learning data is unsatisfactory (negative) compared to the error when the learning data is acceptable (positive) may be used to learn the model so that the FNR becomes equal to or less than a given value.

[0112] [Second Embodiment]

[0113] Hereinafter, a second embodiment will be described. Fig. 9 1 is a flowchart showing an example of the procedure of the method for generating a learned model according to the second embodiment. Each step shown in the figure is implemented by information processing using a computer.

[0114] In the second embodiment, a weighting parameter r that weights the error when the learning data is unqualified (negative) in the loss function compared to the error when the learning data is qualified (positive) is set as a hyperparameter, and the judgment threshold θ is set to a given fixed value. Then, the machine learning of the learning model is performed using the loss function, and the weighting parameter r is searched until the FPR becomes less than a given value α.

[0115] like Fig. 9As shown, first, the learning data is divided into model learning data, tuning data, and test data (S31). The learning data is a data set that associates learning images with pass / fail labels. For example, 80% of the learning data can be model learning data, 10% can be tuning data, and 10% can be test data (overlearning evaluation data).

[0116] Next, the weight parameter r is set to a certain value r0 ( S32 ). r0 is a value greater than 1.

[0117] Next, the model parameters are learned using the model learning data (S33). Specifically, the learning is performed based on a loss function including a weighted parameter r that weights the error when the learning data is unqualified (negative) compared to the error when the learning data is qualified (positive), and the learned parameter k is obtained. r (The hat is omitted in this paper). The learning is completed parameter k r It is expressed by the following formula 6.

[0118] [Number 6]

[0119]

[0120] Here, D T represents all the learning data. r is a weighting parameter, which is a value greater than 1. l is the pass or fail label of the learning data (1: pass, 0: fail). l (k) is the predicted value (probability) of whether data l is judged as Y=1 (pass) or Y=0 (fail).

[0121] l0 is the number of unqualified (negative) learning data, and l1 is the number of qualified (positive) learning data. Since it is difficult to prepare qualified and unqualified learning data equally, the weights l0 and l1 of the number of data are used to suppress the influence of the deviation.

[0122] This equation 6 is constructed such that when the prediction is deviated when the learning data is defective, the loss becomes relatively large by setting the weight parameter r to be greater than 1. Therefore, based on this loss function, learning is performed so as to minimize the FPR.

[0123] In S32 and S33, the weighted parameter r=r0 is set, and on this basis, the model parameters are learned using the model learning data to obtain the learned parameter k ro .

[0124] Next, according to the learning completion parameter k roThe value of y for FPR confirmation is calculated based on the judgment result of the tuning data using the model (S34). y is represented by the following equation 7.

[0125] [Number 7]

[0126]

[0127] Here, FPR is calculated based on the learned parameter k ro The FPR is calculated based on the judgment result of the model. α is a preset value, which is appropriately selected from the level of FPR required by the model.

[0128] Next, it is determined whether the value of y can be regarded as approximately 0 ( S35 ). Specifically, it is determined whether the value of y is within a given range including 0.

[0129] When the value of y cannot be regarded as approximately 0 ( S35 : No), the weight parameter r is updated ( S36 ), and the model parameter learning ( S33 ) and the value of y are calculated ( S34 ) again.

[0130] By repeating the processes of S33 to S36 in this way, the weight parameter r is searched. The weight parameter r is searched until the value of y is approximately regarded as 0 (S35: Yes), that is, until the FPR becomes equal to or less than a given value.

[0131] In searching for the weighted parameter r, for example, a straight line search method or the like is used. Fig.10 As shown by the solid line in the middle, as a general tendency, it is expected that the value of y will decrease as the weight parameter r increases, and therefore it is preferable to increase the weight each time the weight parameter r is updated.

[0132] However, if Fig.10 As shown by the dotted line in the middle, in actual learning, the value of y often oscillates due to fluctuations in data selection, so multiple weighting parameters r can be prepared, and the values ​​of y can be compared after learning them in parallel to determine the weighting parameter r. In this case, it is expected that the number of search cycles can be reduced.

[0133] In addition, if the weighting parameter r is too large, the influence of the first term of the loss function becomes relatively small, and the FNR tends to increase. Therefore, it is preferred that the weighting parameter r stays at the minimum value within the range of satisfying FPR≤α to suppress the increase of FNR. In this way, the FNR can be suppressed while ensuring FPR≤α.

[0134] When the value of y is approximately regarded as 0 (S35: Yes), the learned parameter k obtained by searching is used. r The same verification is performed. That is, based on the determination result using the test data, the value of y is calculated again ( S37 ), and it is determined whether the value of y is approximately regarded as 0 ( S38 ).

[0135] When the value of y is approximately regarded as 0 (S38: Yes), the search ends. Here, it is preferable to compare with a value smaller than that in the above S35. For example, when it is determined in the above S35 whether the value of y is less than 0, it is preferable to determine in S38 whether the value of y is less than a value a (for example, a=-0.01) slightly smaller than 0.

[0136] When the value of y cannot be regarded as approximately 0 ( S38 : No), the processes of S32 to S36 are performed again.

[0137] Through the above steps, a learned model with an excellent balance between FPR and FNR can be obtained, which ensures FPR≤α while suppressing FNR.

[0138] Fig.11 This is a diagram showing an example of the steps of the determination method according to the second embodiment using the learned model generated by the method for generating a learned model according to the second embodiment. The determination device 1 executes the information processing shown in this diagram according to a program.

[0139] First, the determination device 1 acquires image data captured by the camera 3 ( S41 , as a function of the acquisition unit 11 ).

[0140] Next, the determination device 1 determines whether the product included in the image data is OK (qualified) or NG (unqualified) using the learned model ( S42 - S45 , as the function of the determination unit 12 ).

[0141] Specifically, the determination device 1 inputs the image data to the learned model and calculates the OK probability p2 of the product being OK (S42). The output element of the learned model is composed of a Sigmoid function, and the OK probability p2 is output as a value greater than or equal to 0 and less than or equal to 1.

[0142] Next, the determination device 1 performs the following processing: the OK probability p2 is compared with the determination threshold θ f Compare and determine the threshold value θ f In the second embodiment, the determination threshold θ f is a given fixed value.

[0143] In S43, the OK probability p2 is the decision threshold θ f In the above case, the determination device 1 determines that the product is OK (qualified) ( S44 ).

[0144] On the other hand, in S43, the OK probability p2 is less than the determination threshold θ f In the case of the above, the determination device 1 determines that the product is NG (failed) (S45).

[0145] In addition, here, the OK probability p2 is compared with the decision threshold θ f The determination is performed by comparing the NG probability p1 and the determination threshold 1-θ as in the first embodiment. f To make a judgment.

[0146] [Summary of methods]

[0147] As can be seen from the above description, the present disclosure includes the following aspects. In the following, only for the purpose of clarifying the correspondence with the embodiment, the reference numerals are given in parentheses.

[0148] (Mode 1) The method for generating a learned model according to the present disclosure is a method for generating a learned model for binary classification.

[0149] At least one of a weighting parameter and a judgment threshold value is set as a hyperparameter, wherein the weighting parameter is a parameter for weighting an error of one side of the loss function compared with an error of the other side of the loss function, wherein the loss function is a function obtained by adding an error when the learning data is positive and an error when the learning data is negative, and the judgment threshold value is used to judge whether it is positive or negative,

[0150] The loss function is used to perform machine learning of the learning model so that the output learning data has a probability of being positive or negative.

[0151] The hyperparameter search is performed so that at least one of the false positive rate and the false negative rate becomes equal to or less than a given value.

[0152] (Aspect 2) In the method for generating a learned model according to aspect 1, the loss function may include a corrected probability obtained by correcting the probability output by the learning model using the determination threshold.

[0153] (Method 3) The method for generating the learning model of Method 2 may also include:

[0154] A learning step of temporarily setting the judgment threshold and using the loss function determined by the judgment threshold to learn a learning model; and

[0155] The adjustment step is to obtain the false positive rate or the false negative rate according to the classification result of the learning model so as to obtain a given adjustment judgment threshold value as follows:

[0156] The learning step and the adjusting step are repeatedly performed until the difference between the determination threshold and the adjustment determination threshold becomes equal to or less than a predetermined value.

[0157] (Method 4) The method for generating a learned model according to any one of Methods 1 to 3 may be:

[0158] The weighting parameter for weighting the error when the learning data is negative compared to the error when the learning data is positive is set in the loss function,

[0159] The determination threshold is set to a given fixed value,

[0160] The weighting parameter is searched until the false positive rate becomes below a given value.

[0161] (Mode 5) In the method for generating a learned model according to Mode 4, in searching for the weight parameter, the weight may be increased each time the weight parameter is updated.

[0162] (Method 6) The method for generating a learning model of method 4 or method 5 may also be used.

[0163] preparing a plurality of said weighting parameters,

[0164] Performing machine learning of the learning model in parallel for the plurality of weighted parameters,

[0165] The false positive rate is extracted given the following learned model.

[0166] (Mode 7) In the method for generating a learned model according to any one of Modes 1 to 6, when image data is input, the learned model may output a result of determining whether a product included in the image data is acceptable or not.

[0167] (Mode 8) The determination device (1) according to the present disclosure includes:

[0168] an acquisition unit (11) for acquiring determination data; and

[0169] A determination unit (12) sets at least one of a weighting parameter and a determination threshold as a hyperparameter, wherein the weighting parameter is a parameter for weighting the error of one side of the loss function compared to the error of the other side of the loss function, wherein the loss function is a function obtained by adding the error when the learning data is positive and the error when the learning data is negative, and the determination threshold is used to determine whether it is positive or negative, and the loss function is used to perform machine learning of the learning model so as to output the probability that the learning data is positive or negative, and the hyperparameter is searched so that at least one of the false positive rate and the false negative rate becomes a given rate or less to generate a learning completion model, and determine whether the determination data is positive or negative.

[0170] (Mode 9) In the determination device (1) of Mode 8, it is also possible that:

[0171] The loss function includes a modified probability obtained by modifying the probability output by the learning model by the determination threshold,

[0172] The determination unit compares the probability that the determination data output from the learned model is positive or negative with the determination threshold.

[0173] (Mode 10) In the determination device (1) of Mode 8 or Mode 9, when the learned model is input with image data as the determination data, it may output a determination result of whether a product included in the image data is acceptable or not.

[0174] (Method 11) The determination method involved in the present disclosure includes the following steps:

[0175] Obtaining determination data; and

[0176] At least one of a weighting parameter and a judgment threshold is set as a hyperparameter, the weighting parameter being a parameter for weighting the error of one side of the loss function compared with the error of the other side of the loss function, the loss function being a function obtained by adding the error when the learning data is positive and the error when the learning data is negative, the judgment threshold is used to judge whether it is positive or negative, the loss function is used to perform machine learning of the learning model to output the probability that the learning data is positive or negative, the hyperparameter is searched so that at least one of the false positive rate and the false negative rate becomes a learning completed model generated below a given value, and whether the judgment data is positive or negative is judged.

[0177] (Mode 12) The program according to the present disclosure causes a computer to execute the following processing:

[0178] Obtaining determination data; and

[0179] At least one of a weighting parameter and a judgment threshold is set as a hyperparameter, the weighting parameter being a parameter for weighting the error of one side of the loss function compared with the error of the other side of the loss function, the loss function being a function obtained by adding the error when the learning data is positive and the error when the learning data is negative, the judgment threshold is used to judge whether it is positive or negative, the loss function is used to perform machine learning of the learning model to output the probability that the learning data is positive or negative, the hyperparameter is searched so that at least one of the false positive rate and the false negative rate becomes a learning completed model generated below a given value, and whether the judgment data is positive or negative is judged.

[0180] [CROSS-REFERENCE TO RELATED APPLICATIONS]

[0181] This application claims priority to Japanese Patent Application No. 2022-126088 filed with the Japan Patent Office on August 8, 2022, the contents of which are incorporated herein by reference in their entirety. This application claims priority to Japanese Patent Application No. 2022-182474 filed with the Japan Patent Office on November 15, 2022, the contents of which are incorporated herein by reference in their entirety.

[0182] -Description of Reference Numerals-

[0183] 1 determination device, 2 storage unit, 3 camera, 4 display unit, 10 determination system, 11 acquisition unit, 12 determination unit.

Claims

1. A method for generating a learned model, which is a method for generating a learned model for binary classification, At least one of a weighting parameter and a judgment threshold value is set as a hyperparameter, wherein the weighting parameter is a parameter for weighting an error of one side of the loss function compared with an error of the other side of the loss function, wherein the loss function is a function obtained by adding an error when the learning data is positive and an error when the learning data is negative, and the judgment threshold value is used to judge whether it is positive or negative, The loss function is used to perform machine learning of the learning model so that the output learning data has a probability of being positive or negative. The hyperparameter search is performed so that at least one of the false positive rate and the false negative rate becomes equal to or less than a given value.

2. The method for generating a learning model according to claim 1, wherein: The loss function includes a corrected probability obtained by correcting the probability output by the learning model by the decision threshold.

3. The method for generating a learning model according to claim 2, wherein: The generation of the learning completed model includes: A learning step of temporarily setting the judgment threshold and using the loss function determined by the judgment threshold to learn a learning model; and The adjustment step is to obtain the false positive rate or the false negative rate according to the classification result of the learning model so as to obtain a given adjustment judgment threshold value as follows: The learning step and the adjusting step are repeatedly performed until the difference between the determination threshold and the adjustment determination threshold becomes equal to or less than a predetermined value.

4. The method for generating a learning model according to claim 1, wherein: The weighting parameter for weighting the error when the learning data is negative compared to the error when the learning data is positive is set in the loss function, The determination threshold is set to a given fixed value, The weighting parameter is searched until the false positive rate becomes below a given value.

5. The method for generating a learning model according to claim 4, wherein: In the search of the weighting parameter, the weight is increased each time the weighting parameter is updated.

6. The method for generating a learning model according to claim 4, wherein: preparing a plurality of said weighting parameters, Performing machine learning of the learning model in parallel for the plurality of weighted parameters, The false positive rate is extracted given the following learned model.

7. The method for generating a learned model according to claim 1, wherein: When image data is input to the learned model, the learned model outputs a determination result of whether the product included in the image data is acceptable or not.

8. A determination device comprising: an acquisition unit that acquires determination data; and A determination unit sets at least one of a weighting parameter and a determination threshold as a hyperparameter, wherein the weighting parameter is a parameter for weighting an error of one side of a loss function compared to an error of the other side of the loss function, wherein the loss function is a function obtained by adding an error when the learning data is positive to an error when the learning data is negative, and the determination threshold is used to determine whether it is positive or negative, and the loss function is used to perform machine learning of a learning model so as to output a probability that the learning data is positive or negative, and a learning model is generated by searching the hyperparameter so that at least one of a false positive rate and a false negative rate becomes below a given value, and determines whether the determination data is positive or negative.

9. The determination device according to claim 8, wherein: The loss function includes a modified probability obtained by modifying the probability output by the learning model by the determination threshold, The determination unit compares the probability that the determination data output from the learned model is positive or negative with the determination threshold.

10. The determination device according to claim 8, wherein: When the image data as the determination data is input, the learned model outputs a determination result of whether a product included in the image data is acceptable or not.

11. A determination method comprising the following steps: Obtaining determination data; and At least one of a weighting parameter and a judgment threshold is set as a hyperparameter, the weighting parameter being a parameter for weighting the error of one side of the loss function compared with the error of the other side of the loss function, the loss function being a function obtained by adding the error when the learning data is positive and the error when the learning data is negative, the judgment threshold is used to judge whether it is positive or negative, the loss function is used to perform machine learning of the learning model to output the probability that the learning data is positive or negative, the hyperparameter is searched so that at least one of the false positive rate and the false negative rate becomes a learning completed model generated below a given value, and whether the judgment data is positive or negative is judged.

12. A program that causes a computer to execute the following processing: Obtaining determination data; and At least one of a weighting parameter and a judgment threshold is set as a hyperparameter, the weighting parameter being a parameter for weighting the error of one side of the loss function compared with the error of the other side of the loss function, the loss function being a function obtained by adding the error when the learning data is positive and the error when the learning data is negative, the judgment threshold is used to judge whether it is positive or negative, the loss function is used to perform machine learning of the learning model to output the probability that the learning data is positive or negative, the hyperparameter is searched so that at least one of the false positive rate and the false negative rate becomes a learning completed model generated below a given value, and whether the judgment data is positive or negative is judged.

Images