Determination device, determination method, and determination program

Abstract

This determination device can access a database storing one or more sets of combinations of first generated data relating to a first generated result generated under a first generation condition, and the determination success or failure of the first generated result, and performs: an acquisition process for acquiring the first generated data from the database; a determination process for outputting a determination uncertainty relating to the determination success or failure of the first generated result by entering the first generated data acquired by the acquisition process into a determination model that has been trained on the basis of the one or more sets and that outputs a determination uncertainty relating to the determination success or failure of the first generated result upon receiving the first generated data; and a decision process for deciding whether or not a second generated result different from the first generated result needs to be generated, on the basis of the determination uncertainty output by the determination process.

Description

Determination device, determination method, and determination program Incorporation by Reference

[0001] This application claims priority from Japanese Patent Application No. 2023-144321, filed on September 6, 2023, the contents of which are incorporated herein by reference.

[0002] The present invention relates to a determination device, a determination method, and a determination program for performing a necessity determination.

[0003] In fields such as medicine and materials science, there is little training data available for machine learning. Furthermore, the cost of creating labels for training data is high, and creating the labels can be difficult. If information that has not been learned from the training data is input into a machine learning model, the output of the machine learning model cannot be trusted. For example, if an image of a dog is input into a machine learning model that reads numbers, the machine learning model will output a number. Therefore, a technique called uncertainty is used as an indicator of whether a machine learning model knows the input information (see, for example, Non-Patent Document 1 below).

[0004] Furthermore, the data creation support device disclosed in Patent Document 1 below includes a storage device that stores a neural network model and test data, and a computing device that identifies the uncertainty of the inference results from the neural network model, obtains gradient information of the test data by backpropagation processing that treats the uncertainty as loss, calculates the deviation between each of a plurality of small-change test data and the test data, and identifies small changes that increase or decrease the uncertainty based on the uncertainty information, gradient information, and deviation.

[0005] Japanese Patent Application Laid-Open No. 2022-108816

[0006] Eyke Hullermeier, “QUANTIFYING ALEATORIC AND EPISTEMIC UNCERTAINTY IN MACHINE LEARNING: ARE CONDITIONAL ENTROPY AND MUTUAL INFORMATION APPROPRIATE MEASURES?”, arXiv:2209.03302v1 [cs.LG] 7 Sep 2022

[0007] However, the above-mentioned conventional technology does not take into consideration the fact that the necessity of generating teacher data is determined before the teacher data is generated.Furthermore, since the input to the machine learning model that calculates uncertainty is the generated teacher data, the above-mentioned conventional technology does not take into consideration the fact that the machine learning model calculates uncertainty with high accuracy based only on the generation conditions before the teacher data is generated.

[0008] The present invention aims to provide in advance a determination result as to whether or not a result will contribute to improving the accuracy of a machine learning model.

[0009] A determination device according to one aspect of the invention disclosed in the present application is a determination device having a processor that executes a program and a storage device that stores the program, and is capable of accessing a database that stores one or more sets of combinations of first generation data regarding a first generation result generated under first generation conditions and the success or failure of the determination of the first generation result, and the processor executes the following steps: an acquisition process that acquires the first generation data from the database; a determination process that outputs a determination uncertainty regarding the success or failure of the determination of the first generation result by inputting the first generation data acquired by the acquisition process into a determination model that learns based on the set and outputs a determination uncertainty regarding the success or failure of the determination of the first generation result when the first generation data is input; and a decision process that determines whether or not to generate a second generation result that is different from the first generation result, based on the determination uncertainty output by the determination process.

[0010] According to a representative embodiment of the present invention, it is possible to provide in advance a determination result as to whether or not a method contributes to improving the accuracy of a machine learning model. Problems, configurations, and effects other than those described above will become clear from the following description of the examples.

[0011] FIG. 1 is an explanatory diagram showing uncertainty. FIG. 2 is an explanatory diagram showing an example of a prior determination of the need for generation of teacher data before generation. FIG. 3 is a block diagram showing an example of the hardware configuration of a determination device. FIG. 4 is a block diagram showing an example of the functional configuration of the determination device. FIG. 5 is an explanatory diagram showing an example of experiment condition information. FIG. 6 is an explanatory diagram showing an example of a past experiment DB. FIG. 7 is an explanatory diagram showing an example of past data. FIG. 8 is an explanatory diagram showing an example of a determination model. FIG. 9 is an explanatory diagram showing an example of determination uncertainty information. FIG. 10 is an explanatory diagram showing an example of a prediction model. FIG. 11 is an explanatory diagram showing an example of prediction uncertainty information. FIG. 12 is an explanatory diagram showing an example of a determination result by a determination unit. FIG. 13 is an explanatory diagram showing an example of a display screen. FIG. 14 is a flowchart showing an example of a processing procedure executed by the determination device.

[0012] <Figure 1: Uncertainty> Figure 1 is an explanatory diagram showing uncertainty. Uncertainty is an index that indicates whether or not a machine learning model knows its input information, i.e., the low reliability of the output result of a machine learning model. An output result with high uncertainty has low reliability, while an output result with low uncertainty has high reliability. Uncertainty can be expressed, for example, by the variance of the output results from multiple machine learning models in which some of the multiple nodes that make up the machine learning models are intentionally dropped out using the Monte Carlo method (hereinafter referred to as Monte Carlo dropout models).

[0013] There are two types of uncertainty: epistemic and aleatric. Epistemic uncertainty is uncertainty caused by a lack of training data. Aleatric uncertainty is uncertainty caused by the peculiarities of the training data, and the more training data there is, the more difficult it becomes to distinguish.

[0014] Graph 100 shows the uncertainty associated with an increase or decrease in the number of training data. The horizontal axis represents the number of training data, and the vertical axis represents uncertainty. Uncertainty ranges from 0 to 1, with larger values ​​indicating higher uncertainty.

[0015] Waveform 101, which indicates epistemicity, decreases in uncertainty, i.e., becomes more reliable, as the number of teacher data increases. Waveform 102, which indicates aleatricity, increases in uncertainty, i.e., becomes less reliable, as the number of teacher data increases. Waveform 103 is a composite waveform of waveforms 101 and 102. Because composite waveform 103 asymptotically approaches a certain uncertainty value T as the amount of teacher data increases, there is a limit to how much accuracy can be improved by adding teacher data.

[0016] <Figure 2: Example of Preliminary Determination of Whether or Not to Generate Teacher Data Before Generating It> Figure 2 is an explanatory diagram showing an example of a preliminary determination of whether or not to generate teacher data before generating it. Figure 2 shows an example of an experiment on material composition, etc. In this case, the teacher data is image data of the experimental product generated by the experiment.

[0017] When an experiment is conducted, experimental condition information 201 is prepared. If the amount of training data in the past experiment DB 210 is small, it is difficult to generate image data using AI (Artificial Intelligence). Therefore, image data 202 of the experimental product is obtained by actually conducting the experiment.

[0018] The judgment AI 203 is a machine learning model trained to judge the success or failure of a target product based on the measurement results of the experimental product. The judgment AI 203 is trained using a training data set in which the measurement results of the experimental product are used as training data and the judgment success or failure 213 of the target product is used as correct answer data.

[0019] When image data 202 of the experimental product is input into the judgment AI 203, judgment uncertainty 204 is output, which is an indicator of whether the desired product has been produced or not. However, since judgment uncertainty 204 depends on the experimental product in the experimental condition information 201, it incurs experimental costs.

[0020] The judgment device of this embodiment performs the following steps: (1) acquiring image data 212 of past experimental products that are similar cases; (2) calculating judgment uncertainty 214 using judgment AI 203; (3) calculating prediction uncertainty 224 using prediction AI 220; and (4) deciding whether or not an experiment is necessary.

[0021] Regarding (1), the determination device acquires image data 212 of past experiment products that are similar cases from the past experiment DB 210. The past experiment DB 210 stores past experiment conditions 211 and image data 212 of past experiment products. The determination device may acquire image data 212 of any experiment product of a similar case from the past experiment DB 210, or may search for and acquire image data 212 of past experiment products that were experimented under past experiment conditions 211 that are similar to the experiment condition information 201.

[0022] Regarding (2), the judgment device calculates the judgment uncertainty 214 by inputting the image data 212 of the experimental product of the similar case acquired in (1) into the judgment AI 203.

[0023] Regarding (3), the determination device calculates the prediction uncertainty 224 by inputting the experimental condition information 201 into the prediction AI 220. The prediction AI 220 is a machine learning model trained to calculate the prediction uncertainty 224 when the experimental condition information 201 is input. The prediction AI 220 is trained using a teacher dataset in which the experimental condition information 201 is used as training data and a prediction success / failure of the target product 223 indicating whether or not a past experimental product was generated as the target product based on the past experiment conditions 211 in the past experiment DB 210 is used as correct answer data.

[0024] The prediction uncertainty 224 is an index value indicating whether or not the desired product has been produced, similar to the determination uncertainty 214. In other words, the prediction uncertainty 224 is an index value indicating how many experimental conditions surrounding the input experimental condition information 201 exist as teacher data in the past experiment DB 210.

[0025] Regarding (4), the determination device determines whether or not to conduct an experiment 230 based on the determination uncertainty 214 of (2) and the prediction uncertainty 224 of (3).

[0026] <FIG. 3: Example of Hardware Configuration of Determination Device> FIG. 3 is a block diagram showing an example of the hardware configuration of the determination device. The determination device 300 includes a processor 301, a storage device 302, an input device 303, an output device 304, and a communication interface (communication IF) 305. The processor 301, the storage device 302, the input device 303, the output device 304, and the communication IF 305 are connected via a bus 306. The processor 301 controls the determination device 300. The storage device 302 serves as a working area for the processor 301. The storage device 302 is a non-transitory or temporary recording medium that stores various programs and data. Examples of the storage device 302 include a read-only memory (ROM), a random access memory (RAM), a hard disk drive (HDD), and a flash memory. The input device 303 inputs data. Examples of the input device 303 include a keyboard, mouse, touch panel, numeric keypad, scanner, microphone, and sensor. The output device 304 outputs data. Examples of the output device 304 include a display, printer, and speaker. The communication IF 305 connects to a network and transmits and receives data.

[0027] <Fig. 4: Example of functional configuration of determination device 300> Fig. 4 is a block diagram showing an example of the functional configuration of the determination device 300. The determination device 300 has an acquisition unit 401, a determination unit 402, a prediction unit 403, a determination unit 404, and an output unit 405. Specifically, the acquisition unit 401, the determination unit 402, the prediction unit 403, the determination unit 404, and the output unit 405 are realized, for example, by having the processor 301 execute a program stored in the storage device 302 shown in Fig. 3.

[0028] The determination device 300 can also access a past experiment DB 211. The past experiment DB 211 is stored in the storage device 302 shown in FIG.

[0029] 2 (1), the acquisition unit 401 acquires image data 212 of past experiment products from the past experiment DB 211. Specifically, for example, the acquisition unit 401 acquires image data 212 of an experiment product selected by a user by operating the input device 303. The acquisition unit 401 also acquires the experiment condition information 201, searches the past experiment DB 211, acquires image data 212 of past experiment products similar to the experiment condition information 201 from the past experiment DB 211, and outputs the image data 212 as past data 410.

[0030] [Figure 5 Experimental Condition Information 201] Figure 5 is an explanatory diagram showing an example of the experimental condition information 201. The experimental condition information 201 has an input experimental condition ID 501, material A 502, material B 503, and coordinate values ​​504. One row represents one input experimental condition. The acquisition unit 401 searches the past experiment DB 211 for each input experimental condition.

[0031] The input experimental condition ID 501 is identification information that uniquely identifies the input experimental condition to the determination device 300. Material A 502, material B 503, and coordinate value 504 are examples of experimental condition information 201. Material A 502 and material B 503 are materials used to generate the experimental product, and their values ​​may be, for example, their quantities or ratios. The coordinate value 504 is the coordinate position indicating the measurement point of the experimental product. For example, if the experimental product is generated on a semiconductor wafer, the coordinate value 504 is the measurement position where the experimental product exists when the center position of the semiconductor wafer is the coordinate origin.

[0032] 6 is an explanatory diagram showing an example of the past experiment DB 211. The past experiment DB 211 has fields of past experiment conditions 211, image data 212 of past experiment products, judgment success / failure 213, and prediction success / failure 223. The past experiment conditions 211 have a past experiment condition ID 611, material A 612, material B 613, and coordinate values ​​614.

[0033] The past experiment condition ID 611 is identification information that uniquely identifies the past experiment condition. Material A 612, material B 613, and coordinate value 614 are material A 502, material B 503, and coordinate value 504 in the past experiment condition. One row indicates one experimental condition that was previously tested (hereinafter referred to as the past experiment condition).

[0034] The image data 212 of the past experiment product is image data of a past experiment product generated in a past experiment under the past experiment conditions. In Fig. 6, the image data 212 of the past experiment product has R, G, and B values ​​for each pixel. After the experiment, the input experiment conditions are registered in the past experiment DB 211 together with the image data of the experiment product.

[0035] The judgment success / failure 213 is information indicating whether the past experimental product photographed as the image data 212 of the past experimental product is the target product, with "1" indicating success, i.e., the past experimental product is the target product, and "0" indicating failure, i.e., the past experimental product is not the target product. The judgment model 420 is trained using a teacher dataset in which the image data 212 of the past experimental product is used as training data and the judgment success / failure 213 is used as correct answer data.

[0036] The prediction success / failure 223 is information indicating whether or not a past experimental product was generated as a target product based on the past experimental conditions 211, with "1" indicating success, i.e., the past experimental product was the target product, and "0" indicating failure, i.e., the past experimental product was not the target product. The prediction model 430 is trained using a training data set in which the past experimental conditions 211 are training data and the prediction success / failure 223 is correct answer data.

[0037] 7 is an explanatory diagram showing an example of the past data 410. The past data 410 is information that associates input experimental conditions with past experimental conditions. The past data 410 includes an input experimental condition ID 501, past experimental conditions 211 (material A 612, material B 613, and coordinate values ​​614), image data 212 of past experimental products, and a difference 700.

[0038] The difference 700 is information indicating the difference between the input experimental conditions and the past experimental conditions. Specifically, the difference 700 is, for example, the Euclidean distance or cosine similarity between the material B 503 and the coordinate value 504 specified by the input experimental condition ID 501 and the material A 612, the material B 613, and the coordinate value 614 specified by the past experimental condition ID 611.

[0039] The difference 700 may also be a distance in the latent space in the prediction AI 220. Specifically, for example, it is the Euclidean distance or cosine similarity between intermediate data that has been dimensionally compressed when the input experimental conditions are input to the prediction AI 220 and intermediate data that has been dimensionally compressed when the past experimental conditions are input to the prediction AI 220.

[0040] 4, the judgment unit 402 inputs image data 212 of past experimental products in past data 410 into a judgment model 420 and calculates the judgment uncertainty 214. That is, the judgment unit 402 and the judgment model 420 are the judgment AI 203 shown in FIG. 2, and the judgment unit 402 executes (2) calculation of the judgment uncertainty 214 by the judgment AI 203 in FIG. 2.

[0041] [Fig. 8: Determination Model 420] Fig. 8 is an explanatory diagram showing an example of the determination model 420. The determination model 420 has a layer 801, a layer type 802, and a layer parameter 803. The layer 801 is information for identifying a layer of the determination model 420. The layer type 802 is information indicating the type of the layer 801. Examples of layer types include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The layer parameter 803 is information indicating the parameters of the layer 801 (for example, the weight of each neuron).

[0042] 9 is an explanatory diagram showing an example of the determination uncertainty information 421. The determination uncertainty information 421 includes an input experiment condition ID 501, a past experiment condition ID 611, a determination uncertainty 214, and a difference 700. In other words, the determination uncertainty information 421 is information in which the determination uncertainty 214 calculated by the determination unit 402 is added to the input experiment condition ID 501, the past experiment condition ID 611, and the difference 700 in the past data 410.

[0043] 4 , the prediction unit 403 inputs each input experimental condition of the experimental condition information 201 into the prediction model 430 and calculates the prediction uncertainty 224. That is, the prediction unit 403 and the prediction model 430 are the prediction AI 220 shown in FIG. 2 , and the prediction unit 403 executes (3) calculation of the prediction uncertainty 224 by the prediction AI 220 in FIG. 2 .

[0044] [Fig. 10 Prediction Model 430] Fig. 10 is an explanatory diagram showing an example of the prediction model 430. The prediction model 430 has a layer 1001, a layer type 1002, and a layer parameter 1003. The layer 1001 is information for identifying a layer of the prediction model 430. The layer type 1002 is information indicating the type of the layer 1001. Examples of layer types include an input layer, a convolutional layer, a pooling layer, a fully connected layer, and an output layer. The layer parameter 1003 is information indicating the parameters of the layer 1001 (for example, the weight of each neuron).

[0045] 11 is an explanatory diagram showing an example of the prediction uncertainty information 431. The prediction uncertainty information 431 includes an input experiment condition ID 501, a past experiment condition ID 611, and a prediction uncertainty 224. In other words, the prediction uncertainty information 431 is information that associates the input experiment condition ID 501 with the prediction uncertainty 224 calculated by the prediction unit 403.

[0046] 4 , the determination unit 404 determines whether or not the input experimental conditions contribute to improving the accuracy of the determination model 420, specifically, whether or not an experiment based on the input experimental conditions is necessary, based on the determination uncertainty 214 and prediction uncertainty 224 for each input experimental condition. That is, the determination unit 404 executes (4) Determining Whether or Not an Experiment is Necessary in FIG. 2 and outputs a determination result 440.

[0047] Specifically, for example, if the judgment uncertainty 214 is equal to or greater than a predetermined judgment threshold and the prediction uncertainty 224 is equal to or greater than a predetermined prediction threshold, the decision unit 404 determines that an experiment based on the input experimental conditions is necessary. That is, since the judgment AI 203 has not been able to learn sufficiently due to an insufficient amount of training data and the prediction AI 220 also fluctuates significantly, an experiment based on the input experimental conditions is necessary.

[0048] Furthermore, if the judgment uncertainty 214 is not equal to or greater than a predetermined judgment threshold but the prediction uncertainty 224 is equal to or greater than a predetermined prediction threshold, the decision unit 404 determines that an experiment based on the input experimental conditions is necessary. That is, although the judgment AI 203 has learned sufficiently because the number of training data is sufficient, the prediction AI 220 has large fluctuations, and therefore an experiment based on the input experimental conditions is necessary.

[0049] Furthermore, when the judgment uncertainty 214 is equal to or greater than the predetermined judgment threshold but the prediction uncertainty 224 is not equal to or greater than the predetermined prediction threshold, the decision unit 404 also determines that an experiment based on the input experimental conditions is necessary. That is, although fluctuations in the prediction AI 220 are suppressed, the judgment AI 203 has not been able to learn sufficiently due to an insufficient amount of training data, so an experiment based on the input experimental conditions is necessary.

[0050] Furthermore, if the judgment uncertainty 214 is not equal to or greater than a predetermined judgment threshold and the prediction uncertainty 224 is not equal to or greater than a predetermined prediction threshold, the decision unit 404 determines that an experiment based on the input experimental conditions is unnecessary. In other words, since the number of training data is sufficient, the judgment AI 203 has been sufficiently trained, and fluctuations in the prediction AI 220 are also suppressed, so an experiment based on the input experimental conditions is unnecessary.

[0051] In addition, when there is a determination uncertainty 214 of one or more past experimental conditions for one input experimental condition, if there are a predetermined number or more past experimental conditions for which the determination uncertainty 214 is equal to or greater than the determination threshold, the determination unit 404 determines that an experiment based on the input experimental condition is necessary. The predetermined number is an integer equal to or greater than 1 and can be set in advance by a user operation. The smaller the predetermined number, the more accurately it is possible to determine whether an experiment is necessary.

[0052] Instead of a predetermined number, the decision unit 404 may determine that an experiment based on the input experimental conditions is necessary when the proportion of past experimental conditions for which the determination uncertainty 214 is equal to or greater than the determination threshold is equal to or greater than a predetermined proportion. The proportion of past experimental conditions for which the determination uncertainty 214 is equal to or greater than the determination threshold is defined as a numerator of the number of past experimental conditions for which the determination uncertainty 214 is equal to or greater than the determination threshold, and a denominator of the number of past experimental condition IDs 611 associated with the input experimental condition ID 501 of the input experimental condition in the past data 410. The predetermined proportion is, for example, a value greater than 0 and equal to or less than 1, and can be set in advance by a user operation. The smaller the predetermined proportion, the more accurately the necessity of an experiment can be determined.

[0053] 12 is an explanatory diagram showing an example of the determination result 440 by the determination unit 404. The determination result 440 includes an input experimental condition ID 501 and implementation necessity 230. In other words, the determination result 440 is information that associates the input experimental condition ID 501 with the implementation necessity 230 determined by the determination unit 404. If the implementation necessity 230 is "1", it indicates that an experiment based on the input experimental conditions needs to be conducted, and if the implementation necessity 230 is "0", it indicates that an experiment based on the input experimental conditions does not need to be conducted.

[0054] 4 , the output unit 405 outputs the experimental condition information 201, past data 410, judgment uncertainty information 421, prediction uncertainty information 431, and decision result 440. Specifically, for example, the output unit 405 displays this data on a display, which is an example of the output device 304, transmits it to another computer with which it can communicate via the communication IF 305, or outputs it to the storage device 302 for storage.

[0055] 13 is an explanatory diagram showing an example of a display screen. A display screen 1300 is a screen displayed on the display of the determination device 300 or another computer that can communicate with the determination device 300 by the output unit 405.

[0056] The display screen 1300 has a setting area 1301, an execution button 1302, and a result area 1303. The setting area 1301 has selection buttons 1311 to 1314 that allow the user to select the determination model 420, the prediction model 430, the past experiment DB 401, and input experiment conditions from the storage device 302.

[0057] The execute button 1302 is a user interface that, when pressed, reads the judgment model 420, prediction model 430, past experiment DB 401, and input experiment conditions set in the setting area 1301, and executes processing by the acquisition unit 401, judgment unit 402, prediction unit 403, and decision unit 404.

[0058] The result area 1303 has an input experiment condition ID selection pull-down 1331, an experiment necessity result display area 1322, and a detailed data area 1323. The input experiment condition ID selection pull-down 1331 is a user interface that allows the user to select the input experiment condition ID 501 by operation.

[0059] The experiment necessity result display area 1322 is an area for displaying the result of necessity of an experiment based on the input experiment conditions specified by the input experiment condition ID 501 .

[0060] The detailed data area 1323 displays the experimental condition ID (input experimental condition ID 501 and past experimental condition ID 611), material A (502, 612), material B (503, 613), coordinate values ​​(504, 614), an image of the past experimental product based on the image data 212 of the past experimental product, and the difference 700.

[0061] 14 is a flowchart showing an example of the execution procedure by the determination device 300. Note that in FIG. 14, an example will be described in which the predetermined number in the determination unit 404 is set to "1." The determination device 300 executes a process to start a prediction uncertainty loop (step S1401). That is, the determination device 300 sequentially selects unselected rows of the prediction uncertainty information 431 as targets for reading.

[0062] The determination device 300 reads the unselected rows of the prediction uncertainty information 431 selected as the reading target (step S1402).

[0063] The determination device 300 determines whether the prediction uncertainty 224 in the selected row of the prediction uncertainty information 431 read in step S1402 is equal to or greater than the prediction threshold value Tp (step S1403).

[0064] If the prediction uncertainty 224 is equal to or greater than the prediction threshold Tp (step S1403: Yes), the process proceeds to step S1410. On the other hand, if the prediction uncertainty 224 is not equal to or greater than the prediction threshold Tp (step S1403: No), the determination device 300 executes a determination uncertainty loop start process for the selected row of the prediction uncertainty information 431 (step S1404). That is, the determination device 300 sequentially selects unselected rows of the determination uncertainty information 421 as targets for reading.

[0065] The determination device 300 reads the unselected rows of the determination uncertainty information 421 selected as the read target (step S1405).

[0066] The determination device 300 determines whether the input experiment condition ID 501 in the selected row of the prediction uncertainty information 431 matches the input experiment condition ID 501 in the selected row of the prediction uncertainty information 431 read in step S1405 (step S1406).

[0067] If they do not match (step S1406: No), the process proceeds to step S1408. On the other hand, if they do match (step S1406: Yes), the determination device 300 determines whether the determination uncertainty 214 in the selected row of the determination uncertainty information 421 read in step S1405 is equal to or greater than the determination threshold Tj (step S1407).

[0068] If the determination uncertainty 214 is equal to or greater than the determination threshold Tj (step S1407: Yes), the process proceeds to step S1410. On the other hand, if the determination uncertainty 214 is not equal to or greater than the determination threshold Tj (step S1407: No), the determination device 300 executes the determination uncertainty loop termination (step S1408). Specifically, for example, if there is an unselected row in the determination uncertainty information 421 for the selected row in the prediction uncertainty information 431, the determination device 300 returns to step S1404; if there is not an unselected row, the determination device 300 proceeds to step S1409.

[0069] In step S1409, the determination device 300 sets the value of the necessity 230 of conducting an experiment under the input experimental conditions identified by the input experimental condition ID 501 in the selected row of the prediction uncertainty information 431 to "0", i.e., no need to conduct the experiment (step S1409), and proceeds to step S1411.

[0070] Also, in step S1410, the determination device 300 sets the value of the necessity 230 of conducting an experiment under the input experimental conditions identified by the input experimental condition ID 501 in the selected row of the prediction uncertainty information 431 to "1", i.e., necessity of conducting the experiment (step S1410), and proceeds to step S1411.

[0071] In step S1411, the determination device 300 adds a pair of the input experiment condition ID 501 and the implementation necessity 230 as a row to the determination result 440 (step S1411). Then, the determination device 300 executes the prediction uncertainty loop termination (step S1412). Specifically, for example, if there is an unselected row in the determination uncertainty information 421 for the selected row in the prediction uncertainty information 431, the determination device 300 returns to step S1401. If there is not, the determination device 300 executes the output process of displaying the display screen 1300 (step S1413), and ends the series of processes.

[0072] As described above, according to this embodiment, the determination device 300 can determine whether an experiment based on the input experimental conditions is necessary before an experiment based on the input experimental conditions is actually performed. This allows the determination result as to whether or not the experiment will contribute to improving the accuracy of the determination model 420 to be provided before adding teacher data (image data of the experimental product based on the input experimental conditions) used for training the determination model 420. Therefore, the user can determine whether it is worth the effort before adding more teacher data. Furthermore, if it is determined that an experiment is unnecessary, it is possible to reduce the cost and time waste associated with unnecessary experiments, as well as the waste of adding more teacher data without improving accuracy.

[0073] In the above-described embodiment, the determination device 300 performs a preliminary determination of whether an experiment is necessary based on the input experimental conditions using the determination unit 402 and the prediction unit 403, but the preliminary determination may be performed by only the determination unit 402. In this case, the decision unit 404 determines that an experiment based on the input experimental conditions is necessary if the determination uncertainty 214 is equal to or greater than the determination threshold Tj, and determines that an experiment based on the input experimental conditions is unnecessary if the determination uncertainty 214 is not equal to or greater than the determination threshold Tj. In this way, since processing by the prediction unit 403 is not performed, the preliminary determination can be performed faster.

[0074] In addition, in the above-mentioned examples, experimental conditions and experimental products in an experiment were used as examples, but the present invention is not limited to experiments, and can also be applied to various production conditions and production results for achieving a desired result, such as production conditions and products in a production process, cooking conditions in a cooking procedure and cooked dishes.

[0075] In the above-described embodiment, the image data 212 of the past experimental product was used as the training data for the determination model 420, but the training data is not limited to image data. For example, if the generated result is the sound of cutting wood with a machine tool in a manufacturing process or the sound of processing ingredients with cooking utensils in a cooking procedure, the training data may be such audio data.

[0076] The present invention is not limited to the above-described embodiments, and includes various modifications and equivalent configurations within the spirit and scope of the appended claims. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to configurations including all of the described configurations. Furthermore, part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Furthermore, the configuration of another embodiment may be added to the configuration of one embodiment. Furthermore, part of the configuration of each embodiment may be added to, deleted from, or replaced with other configurations.

[0077] Furthermore, the aforementioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits, or may be realized in software by having a processor interpret and execute a program that realizes each function.

[0078] Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, hard disk, or SSD (Solid State Drive), or in a recording medium such as an IC (Integrated Circuit) card, SD card, or DVD (Digital Versatile Disc).

[0079] In addition, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily represent all the control lines and information lines that are necessary for implementation. In reality, it can be assumed that almost all components are interconnected.

Claims

1. A judgment device having a processor that executes a program and a storage device that stores the program, the judgment device being capable of accessing a database that stores one or more sets of combinations of first generation data regarding a first generation result generated under first generation conditions and the success or failure of the judgment of the first generation result, the processor executing an acquisition process that acquires the first generation data from the database; a judgment process that outputs judgment uncertainty regarding the success or failure of the judgment of the first generation result by inputting the first generation data acquired by the acquisition process into a judgment model that learns based on the set and outputs judgment uncertainty regarding the success or failure of the judgment of the first generation result when the first generation data is input; and a decision process that determines whether or not to generate a second generation result different from the first generation result based on the judgment uncertainty output by the judgment process.

2. A determination device as described in claim 1, characterized in that, in the acquisition process, the processor searches the database for the first generation condition similar to the second generation condition of the second generation result, and acquires the first generation data related to the first generation result generated under the searched first generation condition, and, in the decision process, the processor decides whether or not to generate the second generation result generated under the second generation condition based on the decision uncertainty.

3. A determination device as described in claim 1, wherein the database stores the predicted success / failure of the first generation result indicating whether the first generation result has been obtained based on the first generation condition, and the processor executes a prediction process to output a prediction uncertainty regarding the predicted success / failure of the second generation result by inputting a second generation condition of the second generation result into a prediction model that learns based on the first generation condition in the set and the predicted success / failure of the first generation result, and outputs a prediction uncertainty regarding the predicted success / failure of the first generation result when the first generation condition is input, and in the decision process, the processor decides whether or not to generate the second generation result based on the judgment uncertainty and the prediction uncertainty output by the prediction process.

4. A determination device as described in claim 3, characterized in that in the acquisition process, the processor acquires the first generation data regarding the first generation result generated under the first generation condition similar to the second generation condition of the second generation result.

5. A determination device as described in claim 4, characterized in that in the acquisition process, the processor acquires the first generation data regarding the first generation result generated under the first generation condition similar to the second generation condition based on first dimensional compressed data obtained by dimensionally compressing the first generation condition and second dimensional compressed data obtained by dimensionally compressing the second generation condition.

6. A determination device as described in claim 3, characterized in that, in the decision process, the processor determines that generation of the second generation result is unnecessary if the decision uncertainty is not greater than or equal to a decision threshold and the prediction uncertainty is not greater than or equal to a prediction threshold.

7. A judgment method using a judgment device having a processor that executes a program and a storage device that stores the program, the judgment method being characterized in that the processor is capable of accessing a database that stores one or more sets of combinations of first generation data regarding a first generation result generated under first generation conditions and the success or failure of the judgment of the first generation result, and the processor executes the following judgment methods: an acquisition process that acquires the first generation data from the database; a judgment process that outputs judgment uncertainty regarding the success or failure of the judgment of the first generation result by inputting the first generation data acquired by the acquisition process into a judgment model that learns based on the set and outputs judgment uncertainty regarding the success or failure of the judgment of the first generation result when the first generation data is input; and a decision process that determines whether or not to generate a second generation result different from the first generation result based on the judgment uncertainty output by the judgment process.

8. A judgment program that causes a processor that can access a database that stores one or more sets of combinations of first generation data regarding a first generation result generated under first generation conditions and the success or failure of the judgment of the first generation result to execute the following: an acquisition process that acquires the first generation data from the database; a judgment process that outputs judgment uncertainty regarding the success or failure of the judgment of the first generation result by inputting the first generation data acquired by the acquisition process into a judgment model that learns based on the sets and outputs judgment uncertainty regarding the success or failure of the judgment of the first generation result when the first generation data is input; and a decision process that determines whether or not to generate a second generation result different from the first generation result based on the judgment uncertainty output by the judgment process.

Images