Active learning program, method, and apparatus

Abstract

To suppress training costs required for active learning for use in improvement of accuracy of a machine learning model, with using less labeled data.SOLUTION: An active learning apparatus according to the present invention predicts (E), based on an error prediction model 26 trained with using an error prediction training data group 34, prediction error of an energy prediction model 24 with respect to each of a plurality of non-labeled data included in a non-labeled data group 36, selects non-labeled data having a large prediction error from the plurality of non-labeled data, generates (F) a labeled data by giving energy information as a correct label to it, and trains (A) the energy prediction model 24 with using the generated labeled data as energy prediction training data.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The disclosed technology relates to an active learning program, an active learning method, and an active learning device. [Background technology]

[0002] Conventionally, neural networks have been proposed that predict the energy of a molecule based on its structural data. These neural networks are trained by supervised learning using labeled data consisting of "structure" and "energy." This "energy" is calculated, for example, by a numerical calculation method called density functional theory (DFT). The calculation time for molecular energy using DFT is very long, and it can take half a day to three days to calculate the energy of one structure. Therefore, it is difficult to collect a large amount of labeled data for supervised learning.

[0003] Therefore, active learning methods have been proposed to improve the accuracy of neural networks with as little labeled data as possible. For example, a technology has been proposed that predicts the uncertainty of energy prediction values ​​for unlabeled data using a machine learning model that predicts the uncertainty of energy prediction values ​​by neural networks. This technology labels unlabeled data with large predicted uncertainty, and uses the labeled data to train a neural network that predicts energy.

[0004] Also, as a technology related to active learning, an information processing device has been proposed that accepts training examples consisting of features, assigns labels to the training examples, generates one or more student models using the labeled training examples, and calculates the error between the prediction by the student model and the label. This device generates an error prediction model that is a model that predicts the error, and outputs examples that are predicted to have a large error based on the error prediction model. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2022 / 113338 [Non-Patent Document 1] Kristof T. Schutt, Oliver T. Unke, and Michael Gastegger, "Equivariant Message Passing for the Prediction of Tensorial Properties and Molecular Spectra," PMLR, 2021. [Non-Patent Document 2] Johannes Gasteiger, Muhammed Shuaibi, Anuroop Sriram, Stephan Gunnemann, Zachary Ulissi, C. Lawrence Zitnick, and Abhishek Das, "GemNet-OC: Developing Graph Neural Networks for Large and Diverse Molecular Simulation Datasets," Transactions on Machine Learning Research, 2022. [Non-Patent Document 3] Joseph Musielewicz, Xiaoxiao Wang, Tian Tian, ​​and Zachary Ulissi, "FINETUNA: Fine-tuning Accelerated Molecular Simulations," arXiv:2205.01223v2 [physics.comp-ph] 1 Jul 2022. Summary of the Invention [Problem to be solved by the invention]

[0006] However, the conventional technique using a machine learning model to predict the uncertainty of the energy predicted by a neural network requires multiple neural networks to predict the uncertainty, which poses a problem of high training costs due to the active learning.

[0007] In one aspect, the disclosed technology aims to reduce training costs through active learning to improve the accuracy of machine learning models with as little labeled data as possible. [Means for solving the problem]

[0008] In one aspect, the disclosed technology predicts a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model. The second machine learning model is trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data. The disclosed technology then generates labeled data by assigning a correct answer label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error, and retrains the first machine learning model using the generated labeled data. Effect of the Invention

[0009] One aspect of this is that it has the effect of reducing training costs through active learning to improve the accuracy of machine learning models with as little labeled data as possible. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining a NN for predicting molecular energy. [Diagram 2] FIG. 1 is a diagram for explaining the background art. [Diagram 3] FIG. 1 is a diagram for explaining an uncertainty prediction NN in the background art. [Figure 4] FIG. 2 is a functional block diagram of the active learning device according to the present embodiment. [Diagram 5] FIG. 13 is a diagram for explaining an energy prediction model. [Figure 6] FIG. 13 is a diagram for explaining an error prediction model. [Figure 7] FIG. 13 is a diagram showing an example of the relationship between the calculated energy value by DFT and the predicted energy value by the energy prediction model. [Figure 8] FIG. 1 is a block diagram showing a schematic configuration of a computer that functions as an active learning device. [Figure 9] 13 is a flowchart illustrating an example of an active learning process. [Figure 10] FIG. 2 is a diagram illustrating an outline of processing in an active learning device. [Figure 11] FIG. 13 is a diagram showing the results of verification of the effect of the present method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings.

[0012] Before describing the details of the embodiment, we will explain the challenges in active learning for supervised learning of machine learning models for molecular energy prediction.

[0013] As shown in Figure 1, a neural network (hereafter also referred to as "NN") that predicts the energy of a molecule based on its structural data is trained by supervised learning using labeled data consisting of "structure" and "energy." That is, the structural data of the molecule is input to the NN, and the parameters of the NN are updated by backpropagating the difference between the predicted molecular energy, which is the output from the NN, and the correct energy. This correct energy is calculated, for example, by a numerical calculation method called DFT. Because it takes a very long time to calculate the molecular energy using DFT, it is difficult to collect a large amount of labeled data for supervised learning.

[0014] Therefore, an active learning method has been proposed to improve the accuracy of NN with as little labeled data as possible. Specifically, as shown in A of Fig. 2, an information processing device that performs active learning predicts the uncertainty of the predicted value of energy for each unlabeled data included in an unlabeled data group using an uncertainty prediction NN that predicts the uncertainty of the predicted value of energy. Then, the information processing device selects unlabeled data with large uncertainty predicted by the uncertainty prediction NN from among the multiple unlabeled data. As shown in B of Fig. 2, the information processing device generates labeled data by assigning energy calculated by, for example, DFT, as a correct answer label to the selected unlabeled data.

[0015] Furthermore, the information processing device retrains the energy prediction NN that predicts energy using the training data to which the generated labeled data has been added, as shown in C of Fig. 2. The information processing device evaluates the retrained energy prediction NN using the validation data, as shown in D of Fig. 2, and terminates the training if the accuracy of the energy prediction NN meets the standard. On the other hand, if the accuracy of the energy prediction NN does not meet the standard, the information processing device repeats the process from predicting uncertainty, excluding the data labeled in B above, from the unlabeled data group, as shown in E of Fig. 2.

[0016] In this way, the uncertainty of the predicted energy value is predicted using an uncertainty prediction NN, and unlabeled data with large uncertainty is then selected as the data to be labeled, thereby selecting data that can efficiently train the energy prediction NN.

[0017] However, for example, in the technology described in Non-Patent Document 3, this uncertainty prediction NN is composed of multiple neural networks as shown in Fig. 3. Specifically, structural data of the same molecule is input to multiple (N in the example of Fig. 3) energy prediction NNs each having different weights, etc., and the energy of the molecule is predicted. Then, the variation in the predicted value of each energy prediction NN is used as the uncertainty of the structural data. Due to this configuration, the technology described in Non-Patent Document 3 requires the preparation of multiple neural networks to predict uncertainty, which increases the training cost.

[0018] Therefore, in this embodiment, a method for predicting the uncertainty of a predicted value of energy using a single neural network is proposed. The active learning device according to this embodiment will be described in detail below.

[0019] As shown in Fig. 4, active learning device 10 functionally includes a first training unit 12, a verification unit 14, a second training unit 16, an error prediction unit 18, a first generation unit 20, and a second generation unit 22. In addition, an energy prediction model 24 and an error prediction model 26 are stored in a predetermined storage area of ​​active learning device 10. Energy prediction model 24 is an example of a "first machine learning model" of the disclosed technology, and error prediction model 26 is an example of a "second machine learning model" of the disclosed technology.

[0020] The first training unit 12 trains the energy prediction model 24 using an energy prediction training data group 30. The energy prediction training data group 30 includes a plurality of energy prediction training data, which are pairs of molecular structure data and correct labels of the energy of the molecule. The energy prediction model 24 is a machine learning model configured with a neural network or the like. As shown in FIG. 5, when molecular structure data is input, the energy prediction model 24 outputs a predicted value of the energy of the molecule.

[0021] Specifically, the first training unit 12 inputs molecular structure data of the energy prediction training data into the energy prediction model 24 and updates the parameters of the energy prediction model 24 so that the predicted energy value outputted matches the energy indicated by the correct label. Furthermore, when labeled data generated by the first generating unit 20 described below is added to the energy prediction training data group 30, the first training unit 12 retrains the energy prediction model 24 using the energy prediction training data group 30 after the addition.

[0022] The verification unit 14 verifies the accuracy of the energy prediction model 24 by using the verification data group 32. The verification data group 32 includes, as verification data, a plurality of labeled data in which a correct label for the energy of a molecule is assigned to structural data of the molecule and which has not been used in training the energy prediction model 24.

[0023] The verification unit 14 calculates the prediction accuracy of the energy prediction model 24 based on the predicted value predicted by the energy prediction model 24 for the molecular structure data of the verification data and the correct label of the verification data. The verification unit 14 verifies whether the prediction accuracy satisfies a predetermined criterion. If the prediction accuracy satisfies the predetermined criterion, the verification unit 14 ends the training of the energy prediction model 24. On the other hand, if the prediction accuracy does not satisfy the predetermined criterion, the verification unit 14 repeatedly executes the processing of each functional unit to retrain the energy prediction model 24 and the error prediction model 26.

[0024] The second training unit 16 trains the error prediction model 26 using an error prediction training data group 34. The error prediction training data group 34 includes a plurality of error prediction training data, which are pairs of molecular structure data and correct labels of errors in predicted values ​​(hereinafter also referred to as "prediction errors") when the energy of the molecule is predicted by the energy prediction model 24. The error prediction model 26 is a machine learning model composed of a single neural network or the like. When molecular structure data is input, the error prediction model 26 outputs a predicted value of a prediction error of the energy of the molecule, as shown in FIG. 6.

[0025] Specifically, the second training unit 16 updates the parameters of the error prediction model 26 so that the predicted value of the energy prediction error outputted by inputting the molecular structure data of the error prediction training data to the error prediction model 26 matches the prediction error indicated by the correct answer label. Furthermore, when error prediction training data generated by the second generation unit 22 described later is added to the error prediction training data group 34, the second training unit 16 retrains the error prediction model 26 using the error prediction training data group 34 after the addition.

[0026] The error prediction unit 18 predicts the prediction error of the energy prediction model 24 for each of the plurality of unlabeled data included in the unlabeled data group 36 based on the error prediction model 26 .

[0027] The first generating unit 20 selects unlabeled data to be labeled from the plurality of unlabeled data, based on the prediction error predicted by the error predicting unit 18. That is, in this embodiment, the prediction error predicted by the error predicting unit 18 is used as the uncertainty of the prediction result of the unlabeled data for the energy prediction model 24.

[0028] Specifically, the first generation unit 20 selects unlabeled data whose prediction error predicted by the error prediction model 26 is equal to or greater than a predetermined value, or unlabeled data whose prediction error is in the top predetermined number of positions, from among the multiple unlabeled data included in the unlabeled data group 36. In other words, the first generation unit 20 selects unlabeled data whose prediction error is large. The first generation unit 20 acquires information on the energy that is the correct label for the selected unlabeled data.

[0029] For example, the first generating unit 20 acquires energy information calculated by DFT for the molecular structure data indicated by the unlabeled data. The calculation of energy by DFT may be executed inside the active learning device 10 or may be executed by an external device. When executed by an external device, the first generating unit 20 outputs the molecular structure data indicated by the unlabeled data to the external device and acquires the calculation result from the external device. In addition, the first generating unit 20 may present the molecular structure data indicated by the unlabeled data to a user by displaying it on a display device, etc., and acquire the calculation result input by the user.

[0030] The first generating unit 20 generates labeled data by assigning the acquired energy information as a correct label to the molecular structure data of the selected unlabeled data. The first generating unit 20 adds the generated labeled data to the energy prediction training data group 30 as energy prediction training data. The first generating unit 20 also passes the generated labeled data to the second generating unit 22.

[0031] The second generation unit 22 inputs the molecular structure data of the labeled data passed from the first generation unit 20 to the energy prediction model 24 and obtains a predicted value of the energy. The second generation unit 22 also calculates a prediction error from the energy indicated by the correct label of the labeled data and the obtained predicted value of the energy.

[0032] Fig. 7 shows an example of the relationship between the calculated energy value by DFT and the predicted energy value by the energy prediction model 24. Each plot in the graph shown in Fig. 7 corresponds to the calculated value and the predicted value for one piece of structure data. Line A shown in Fig. 7 represents the case where the calculated value and the predicted value match, that is, the case where the prediction error is 0. That is, the second generation unit 22 calculates the distance between line A and each plot as the prediction error for the structure data corresponding to that plot.

[0033] The second generating unit 22 generates, as error prediction training data, a pair of the molecular structure data of the labeled data passed from the first generating unit 20 and the calculated prediction error. The second generating unit 22 adds the generated error prediction training data to an error prediction training data group 34.

[0034] The active learning device 10 may be realized by, for example, a computer 40 shown in FIG. 8. The computer 40 includes a CPU (Central Processing Unit) 41, a GPU (Graphics Processing Unit) 42, a memory 43 as a temporary storage area, and a non-volatile storage device 44. The computer 40 also includes an input / output device 45 such as an input device and a display device, and an R / W (Read / Write) device 46 that controls reading and writing of data to a storage medium 49. The computer 40 also includes a communication I / F (Interface) 47 that is connected to a network such as the Internet. The CPU 41, the GPU 42, the memory 43, the storage device 44, the input / output device 45, the R / W device 46, and the communication I / F 47 are connected to one another via a bus 48.

[0035] The storage device 44 is, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, etc. An active learning program 50 for causing the computer 40 to function as the active learning device 10 is stored in the storage device 44 as a storage medium. The active learning program 50 has a first training process control instruction 52, a verification process control instruction 54, a second training process control instruction 56, an error prediction process control instruction 58, a first generation process control instruction 60, and a second generation process control instruction 62. The storage device 44 also has an information storage area 70 in which information constituting each of the energy prediction model 24 and the error prediction model 26 is stored.

[0036] The CPU 41 reads out the active learning program 50 from the storage device 44, expands it in the memory 43, and sequentially executes the control instructions of the active learning program 50. The CPU 41 operates as the first training unit 12 shown in FIG. 4 by executing the first training process control instruction 52. The CPU 41 also operates as the verification unit 14 shown in FIG. 4 by executing the verification process control instruction 54. The CPU 41 also operates as the second training unit 16 shown in FIG. 4 by executing the second training process control instruction 56. The CPU 41 also operates as the error prediction unit 18 shown in FIG. 4 by executing the error prediction process control instruction 58. The CPU 41 also operates as the first generation unit 20 shown in FIG. 4 by executing the first generation process control instruction 60. The CPU 41 also operates as the second generation unit 22 shown in FIG. 4 by executing the second generation process control instruction 62. Furthermore, CPU 41 reads information from information storage area 70, and loads each of energy prediction model 24 and error prediction model 26 in memory 43. In this way, computer 40 that executes active learning program 50 functions as active learning device 10. Note that CPU 41 that executes the program is hardware. Furthermore, part of the program may be executed by GPU 42.

[0037] The functions realized by active learning program 50 may be realized, for example, by a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

[0038] Next, the operation of the active learning device 10 according to this embodiment will be described. When training of the energy prediction model 24 is instructed, the active learning process shown in FIG. 9 is executed in the active learning device 10. The active learning process is an example of an active learning method of the disclosed technology. The active learning process shown in FIG. 9 will be described below together with an overview of the process in the active learning device 10 according to this embodiment shown in FIG. 10.

[0039] In step S10, the first training unit 12 trains the energy prediction model 24 using the energy prediction training data group 30 (A in FIG. 10). Next, in step S12, the verification unit 14 calculates the prediction accuracy of the energy prediction model 24 based on the predicted value predicted by the energy prediction model 24 for the molecular structure data of the verification data and the correct answer label of the verification data. Next, in step S14, the verification unit 14 determines whether the calculated prediction accuracy satisfies a predetermined criterion (B in FIG. 10). If the criterion is not satisfied, the process proceeds to step S16.

[0040] In step S16, the second generation unit 22 inputs the molecular structure data of the labeled data to the energy prediction model 24 and obtains a predicted value of the energy. Next, in step S18, the second generation unit 22 calculates a prediction error from the energy indicated by the correct label of the labeled data and the obtained predicted value of the energy. Then, the second generation unit 22 generates a pair of the molecular structure data of the labeled data and the calculated prediction error as error prediction training data, and adds it to the error prediction training data group 34 (C in FIG. 10).

[0041] Next, in step S20, the second training unit 16 trains the error prediction model 26 using the error prediction training data group 34 (D in FIG. 10). Next, in step S22, the error prediction unit 18 predicts the prediction error of the energy prediction model 24 for each of the multiple unlabeled data included in the unlabeled data group 36 based on the error prediction model 26 (E in FIG. 10).

[0042] Next, in step S24, the first generation unit 20 selects unlabeled data with a large predicted prediction error from the multiple unlabeled data. Then, the first generation unit 20 acquires energy information that is a correct answer label for the selected unlabeled data, and generates labeled data by assigning the acquired energy information as a correct answer label to the molecular structure data of the selected unlabeled data (F in FIG. 10). Then, the first generation unit 20 adds the generated labeled data to the energy prediction training data group 30 as energy prediction training data (G in FIG. 10), and returns to step S10.

[0043] If it is determined in step S14 that the calculated prediction accuracy satisfies a predetermined standard, the active learning process ends.

[0044] As described above, the active learning device according to the present embodiment trains an error prediction model using a set of a plurality of data and a prediction error of the energy prediction model for each of the plurality of data as training data. The active learning device predicts a prediction error of the energy prediction model for each of the plurality of unlabeled data based on the error prediction model, and generates labeled data by assigning a correct answer label to unlabeled data selected from the plurality of unlabeled data based on the prediction error. Then, the active learning device retrains the energy prediction model using the generated labeled data. In this way, in the present embodiment, labeling is performed on unlabeled data with a large prediction error predicted by an error prediction model composed of a single neural network or the like. This makes it possible to reduce the training cost by active learning compared to the case where the uncertainty of unlabeled data is predicted by a machine learning model composed of multiple neural networks.

[0045] Here, we will explain the experimental results that verify that even when active learning is performed using the method related to this embodiment (hereinafter referred to as "this method"), the accuracy of the machine learning model can be improved with as little labeled data as possible.

[0046] In this verification, to compare with the present method, a method was used in which training data for supervised learning of the energy prediction model was selected by random sampling from the prepared training data. In this verification, 3770 pairs of molecular structure data and the energy of the molecule were prepared, of which 2870 were used as training data and the remaining 900 were used as validation data that were not used for training. In addition, all of the structural data in the 2870 training data were used as unlabeled data to be labeled, and all of the structural data and energy pairs in the 2870 training data were used for supervised learning.

[0047] Figure 11 shows the experimental results for each method, showing the relationship between the number of labeled data used in supervised learning and the energy prediction error of the energy prediction model in which supervised learning was performed with that number of labeled data. As shown in the figure, our method is able to improve the accuracy of the energy prediction model with a smaller amount of labeled data than the comparison method. Note that when all labeled data is used for training, the accuracy of our method and the comparison method is naturally equivalent.

[0048] In the above embodiment, the case of predicting energy from molecular structure data has been described, but the data to which the disclosed technology can be applied is not limited to this example. In particular, when it is difficult to prepare a large amount of labeled data, the effect of applying the disclosed technology is high.

[0049] In the above embodiment, the active learning program is stored (installed) in advance in the storage device, but this is not limiting. The program according to the disclosed technology may be provided in a form stored in a storage medium such as a CD-ROM, a DVD-ROM, or a USB memory.

[0050] The following supplementary notes are further disclosed regarding the above embodiment.

[0051] (Appendix 1) predicting a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data; generating labeled data by assigning a correct label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error; retraining the first machine learning model using the generated labeled data; An active learning program for causing a computer to execute a process including the above.

[0052] (Appendix 2) 2. The active learning program according to claim 1, further comprising: selecting, from the plurality of unlabeled data, unlabeled data whose predicted prediction error is equal to or greater than a predetermined value, or unlabeled data whose predicted prediction error is in the top predetermined range.

[0053] (Appendix 3) Calculating each of the prediction errors from each of prediction results obtained by inputting each of a plurality of labeled data into the first machine learning model and each of correct labels of the plurality of labeled data, and generating a plurality of the training data from each of the calculated prediction errors and each of the plurality of labeled data; training the second machine learning model using the training data; 3. The active learning program according to claim 1 or 2, further comprising:

[0054] (Appendix 4) The active learning program according to any one of claims 1 to 3, further comprising repeating the prediction of the prediction error and the generation of the labeled data until the accuracy of the retrained first machine learning model meets a predetermined standard.

[0055] (Appendix 5) 4. The active learning program of claim 3, further comprising: retraining the second machine learning model using training data generated based on the labeled data generated from the selected unlabeled data.

[0056] (Appendix 6) The labeled data is data in which molecular energy information is added as a correct label to molecular structure data, The first machine learning model is a machine learning model that outputs the energy of a molecule as a prediction result when structural data of the molecule is input. An active learning program according to any one of claims 1 to 5.

[0057] (Appendix 7) The active learning program of claim 6, wherein generating the labeled data includes calculating the energy of the molecule from the structural data of the molecule, which is the selected unlabeled data, using density functional theory.

[0058] (Appendix 8) predicting a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data; generating labeled data by assigning a correct label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error; retraining the first machine learning model using the generated labeled data; The active learning method includes a computer executing a process including:

[0059] (Appendix 9) 9. The active learning method according to claim 8, further comprising selecting, from the plurality of unlabeled data, unlabeled data whose predicted prediction error is equal to or greater than a predetermined value, or unlabeled data whose predicted prediction error is in the top predetermined range.

[0060] (Appendix 10) Calculating each of the prediction errors from each of prediction results obtained by inputting each of a plurality of labeled data into the first machine learning model and each of correct labels of the plurality of labeled data, and generating a plurality of the training data from each of the calculated prediction errors and each of the plurality of labeled data; training the second machine learning model using the training data; 10. The active learning method according to claim 8 or 9, further comprising:

[0061] (Appendix 11) 11. The active learning method according to any one of claims 8 to 10, further comprising repeating the prediction of the prediction error and the generation of the labeled data until the accuracy of the retrained first machine learning model meets a predetermined criterion.

[0062] (Appendix 12) 11. The active learning method of claim 10, further comprising retraining the second machine learning model using training data generated based on the labeled data generated from the selected unlabeled data.

[0063] (Appendix 13) The labeled data is data in which molecular energy information is added as a correct label to molecular structure data, The first machine learning model is a machine learning model that outputs the energy of a molecule as a prediction result when structural data of the molecule is input. An active learning method according to any one of claims 8 to 12.

[0064] (Appendix 14) 14. The active learning method of claim 13, wherein generating the labeled data includes calculating the energy of the molecule from the structural data of the molecule, which is the selected unlabeled data, using density functional theory.

[0065] (Appendix 15) an error prediction unit that predicts a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data; a first generation unit that generates labeled data by assigning a correct label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error; A first training unit that retrains the first machine learning model using the generated labeled data; 1. An active learning device comprising:

[0066] (Appendix 16) 16. The active learning device of claim 15, wherein the first generation unit selects, from the plurality of unlabeled data, unlabeled data whose predicted prediction error is equal to or greater than a predetermined value, or unlabeled data whose predicted prediction error is in the top predetermined range.

[0067] (Appendix 17) a second generation unit that calculates each of the prediction errors from each of prediction results obtained by inputting each of a plurality of labeled data into the first machine learning model and each of correct labels of the plurality of labeled data, and generates a plurality of the training data from each of the calculated prediction errors and each of the plurality of labeled data; a second training unit that trains the second machine learning model using the training data; 17. The active learning device of claim 15 or 16, further comprising:

[0068] (Appendix 18) 18. The active learning device according to any one of claims 15 to 17, wherein the prediction of the prediction error and the generation of the labeled data are repeated until the accuracy of the retrained first machine learning model meets a predetermined criterion.

[0069] (Appendix 19) The second generation unit generates the training data based on the labeled data generated by the first generation unit; The second training unit retrains the second machine learning model using the training data generated based on the labeled data generated by the first generation unit. 18. The active learning device of claim 17.

[0070] (Appendix 20) The labeled data is data in which molecular energy information is added as a correct label to molecular structure data, The first machine learning model is a machine learning model that outputs the energy of a molecule as a prediction result when structural data of the molecule is input. 20. The active learning device according to claim 15, wherein the active learning device is a [Explanation of symbols]

[0071] 10 Active learning device 12 1st Training Department 14 Verification Department 16 2nd Training Department 18 Error prediction section 20 1st generation part 22 Second generation part 24 Energy forecast model 26 Error prediction model 30 Energy prediction training data set 32 Validation data set 34 Error prediction training data set 36 Unlabeled Data Set 40 Computer 41 CPU 42 GPU 43 Memory 44 Storage device 45 I / O Devices 46 R / W device 47 Communication I / F 48 Bus 49 Storage medium 50 Active Learning Programs 52 First Training Process Control Instructions 54 Verification Process Control Instructions 56 Second Training Process Control Instructions 58 Error Prediction Process Control Instructions 60 First Generation Process Control Instruction 62 Second generation process control instruction 70 Information storage area

Claims

1. predicting a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data; generating labeled data by assigning a correct label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error; retraining the first machine learning model using the generated labeled data; An active learning program for causing a computer to execute a process including the above.

2. 2. The active learning program according to claim 1, further comprising: selecting, from the plurality of unlabeled data, unlabeled data whose predicted prediction error is equal to or greater than a predetermined value, or unlabeled data whose predicted prediction error is in the top predetermined range.

3. Calculating each of the prediction errors from each of prediction results obtained by inputting each of a plurality of labeled data into the first machine learning model and each of correct labels of the plurality of labeled data, and generating a plurality of the training data from each of the calculated prediction errors and each of the plurality of labeled data; training the second machine learning model using the training data; 3. The active learning program according to claim 1, further comprising:

4. The active learning program according to claim 1 or 2, wherein the prediction of the prediction error and the generation of the labeled data are repeated until the accuracy of the retrained first machine learning model meets a predetermined criterion.

5. The active learning program of claim 4 , further comprising: retraining the second machine learning model using the training data generated based on the labeled data generated from the selected unlabeled data.

6. The labeled data is data in which molecular energy information is added as a correct label to molecular structure data, The first machine learning model is a machine learning model that outputs the energy of a molecule as a prediction result when structural data of the molecule is input.

3. The active learning program according to claim 1 or 2.

7. The active learning program of claim 6 , wherein generating the labeled data includes calculating the energy of the molecule from the structural data of the molecule, which is the selected unlabeled data, using density functional theory.

8. predicting a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data; generating labeled data by assigning a correct label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error; retraining the first machine learning model using the generated labeled data; The active learning method includes a computer executing a process including:

9. an error prediction unit that predicts a prediction error of the first machine learning model for each of a plurality of unlabeled data based on a second machine learning model trained using a set of a plurality of data and a prediction error of the first machine learning model for each of the plurality of data as training data; a first generation unit that generates labeled data by assigning a correct label to unlabeled data selected from the plurality of unlabeled data based on the predicted prediction error; A first training unit that retrains the first machine learning model using the generated labeled data; 1. An active learning device comprising:

Images