Machine learning program, machine learning method, and information processing apparatus

Abstract

To improve prediction accuracy of a machine learning model.SOLUTION: An information processing apparatus 10 inputs data samples 14a, 14b, 14c to a machine learning model 13 and extracts features 15a, 15b, 15c from the machine learning model 13. The information processing apparatus 10 normalizes the features 15a, 15b, 15c to normalized features 16a, 16b, 16c. The information processing apparatus 10 selects, based on the normalized features 16a, 16b, 16c, at least one data sample, which is part of the data samples 14a, 14b, 14c from the data samples 14a, 14b, 14c. The information processing apparatus 10 trains the machine learning model 13 using the at least one data sample.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a machine learning program, a machine learning method, and an information processing device. [Background technology]

[0002] The computer may use the trained machine learning model to generate prediction data corresponding to the objective variable from input data corresponding to the explanatory variables. The computer may train the machine learning model by machine learning using a plurality of data samples in which the input data corresponding to the explanatory variables are associated with the correct labels corresponding to the objective variables.

[0003] A machine learning model may calculate feature quantities representing characteristics of input data from the input data, and may generate prediction data using the calculated feature quantities. A neural network may be used to calculate the feature quantities.

[0004] In addition, machine learning models are sometimes used to predict the results of computationally intensive computer simulations in a short time. Such machine learning models are sometimes called surrogate models. For example, a structural analysis simulation has been proposed that predicts a displacement vector from a stiffness matrix and a force vector using a graph convolutional neural network. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-9904 A Summary of the Invention [Problem to be solved by the invention]

[0006] When a dataset contains many data samples, training a machine learning model using all the data samples may increase the load on the computer and lengthen the training time. Therefore, it is conceivable that the computer selects some of the data samples from the dataset and trains the machine learning model using the selected data samples.

[0007] At this time, how to select some data samples is important. Even if the number of data samples is the same, the prediction accuracy of the trained machine learning model may differ depending on which data samples are selected. Therefore, in one aspect, the present invention aims to improve the prediction accuracy of the machine learning model. [Means for solving the problem]

[0008] In one embodiment, a machine learning program is provided that causes a computer to execute the following processes: A plurality of data samples are input to a machine learning model, and a plurality of feature quantities corresponding to each of the plurality of data samples are extracted from the machine learning model; The plurality of feature quantities are normalized into a plurality of normalized feature quantities that fall within a certain numerical range; One or more data samples that are a part of the plurality of data samples are selected from the plurality of data samples based on the plurality of normalized feature quantities; The one or more data samples are used to train the machine learning model.

[0009] In one aspect, a machine learning method executed by a computer is provided. In another aspect, an information processing device having a storage unit and a processing unit is provided. Effect of the Invention

[0010] On the one hand, it improves the predictive accuracy of machine learning models. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an information processing device according to a first embodiment. [Diagram 2] FIG. 11 illustrates an example of hardware of an information processing apparatus according to a second embodiment. [Diagram 3] FIG. 1 illustrates an example of the use of a machine learning model to predict the results of a quantum chemical calculation. [Figure 4] FIG. 13 is a diagram illustrating an example of updating a feature vector based on a graph structure. [Diagram 5] FIG. 1 is a diagram illustrating an example of the structure of a machine learning model. [Figure 6] FIG. 1 is a diagram showing an example structure of a molecular dataset. [Figure 7] FIG. 1 is a diagram illustrating an example of sample selection in machine learning. [Figure 8] FIG. 13 is a diagram illustrating an example of normalization of a feature vector. [Figure 9] 1 is a graph showing an example of the average error of a machine learning model. [Figure 10] FIG. 2 is a block diagram showing an example of functions of the information processing device; [Figure 11] 1 is a flowchart showing an example of a machine learning procedure. [Figure 12] 13 is a flowchart illustrating an example of a procedure for selecting samples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The present embodiment will be described below with reference to the drawings. [First embodiment] A first embodiment will be described.

[0013] FIG. 1 is a diagram illustrating an information processing apparatus according to a first embodiment. The information processing device 10 of the first embodiment performs machine learning to train a machine learning model using training data. The machine learning model may be a substitute model (surrogate model) that predicts the result of a quantum chemical calculation. The information processing device 10 may be a client device or a server device. The information processing device 10 may be called a computer or a machine learning device.

[0014] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory such as a random access memory (RAM). The storage unit 11 may also be a non-volatile storage such as a hard disk drive (HDD) or a flash memory.

[0015] The processing unit 12 is, for example, a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). However, the processing unit 12 may include an electronic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor executes a program stored in a memory such as a RAM (which may be the storage unit 11). A collection of processors may be called a multiprocessor or simply a "processor."

[0016] The storage unit 11 stores the machine learning model 13. The machine learning model 13 calculates features from input data and generates prediction data from the calculated features. The machine learning model 13 may be a neural network or a graph neural network (GNN). The graph neural network updates the features of each of multiple nodes based on the connection relationships between nodes indicated by graph data including multiple nodes and multiple edges. The feature of a node is updated using the feature of an adjacent node adjacent to the node. Such a graph neural network is sometimes called an MPNN (Message Passing Neural Network).

[0017] The machine learning model 13 includes parameters whose parameter values ​​are optimized through machine learning. The parameters are, for example, weights included in a neural network. When the machine learning model 13 includes a graph neural network, a function that updates the feature of a node using the feature of an adjacent node may include such parameters.

[0018] The machine learning model 13 may be an alternative model that predicts molecular energy calculated by quantum chemical calculation such as density functional theory (DFT) from molecular data indicating a molecular structure. The density functional theory calculates the ground state energy of a molecule from electron density. A quantum chemical simulation that purely executes the density functional theory requires a large amount of calculation. The alternative model reduces the amount of calculation in the quantum chemical simulation. The molecular structure is defined by graph data including, for example, nodes representing atoms and edges indicating connection relationships between atoms. The alternative model, for example, gives the nuclear charge amount of an atom as the initial value of the feature amount of the node, updates the feature amount of the node according to the graph structure, and calculates the energy from the feature amount.

[0019] The storage unit 11 also stores a plurality of data samples, such as data samples 14a, 14b, and 14c. Each of the plurality of data samples includes input data corresponding to an input of the machine learning model 13. Each of the plurality of data samples may include a correct answer label indicating a correct answer of the output of the machine learning model 13. For example, each of the data samples 14a, 14b, and 14c includes a molecular structure and a molecular energy. The molecular structure is defined, for example, by the coordinates and elements of each atom included in the molecule.

[0020] The processing unit 12 trains the machine learning model 13 using some of the multiple data samples. The processing unit 12 selects the data samples to be used for machine learning in a manner described below. The selection of the data samples may be based on the idea of ​​so-called active learning. When selecting the data samples, the machine learning model 13 is tentatively trained. For example, parameter values ​​included in the machine learning model 13 are updated in advance based on a small number of data samples.

[0021] The processing unit 12 inputs each of the multiple data samples to the machine learning model 13, thereby extracting multiple feature amounts corresponding to the multiple data samples from the machine learning model 13. For example, the processing unit 12 extracts feature amounts 15a, 15b, and 15c corresponding to data samples 14a, 14b, and 14c from the machine learning model 13.

[0022] The extracted feature is, for example, a feature close to the output side of the machine learning model 13, and is not the prediction data itself output by the machine learning model 13. When the machine learning model 13 includes a graph neural network, the extracted feature is, for example, a feature of each node updated by the graph neural network, and is a feature immediately before aggregating these features to generate prediction data. The extracted feature may be a numerical vector including multi-dimensional numerical values.

[0023] The processing unit 12 normalizes the extracted multiple feature amounts to multiple normalized feature amounts within a certain numerical range. For example, the processing unit 12 normalizes the feature amounts 15a, 15b, and 15c to normalized feature amounts 16a, 16b, and 16c. The certain numerical range is, for example, a range from 0 to 1. The processing unit 12 may perform scale adjustment by associating the minimum value of the multiple feature amounts with 0 and the maximum value of the multiple feature amounts with 1. When the feature amounts are vectors, the processing unit 12 may perform normalization on a dimensional basis. In this case, a correspondence relationship between a numerical value before normalization and a numerical value after normalization is determined for each dimension.

[0024] The processing unit 12 selects some data samples from the plurality of data samples based on the plurality of normalized features. At this time, it is preferable that the processing unit 12 selects data samples having normalized features that are not similar to other data samples. The number of data samples to be selected may be specified by the user or may be determined from the number of data samples in the population.

[0025] For example, the processing unit 12 converts the normalized features into a plurality of principal component features by performing principal component analysis (PCA) on the normalized features. The principal component features have a smaller number of dimensions than the normalized features. The processing unit 12 selects a certain number of data samples whose principal component features are as dissimilar as possible based on the distribution of the plurality of principal component features in the principal component space.

[0026] The processing unit 12 may calculate the distance between a plurality of normalized features or a plurality of principal component features, and may preferentially select a data sample that is far from other data samples. The processing unit 12 may also classify the plurality of data samples into clusters by a clustering algorithm such as the k-means method or hierarchical clustering, and may select a representative data sample from each of the plurality of clusters.

[0027] The processing unit 12 uses the selected data samples to train the machine learning model 13. The data samples that are not selected are not used in this machine learning. For example, the processing unit 12 inputs the selected data samples to the machine learning model 13 and calculates an error between the output of the machine learning model 13 and a correct label. The processing unit 12 updates parameter values ​​included in the machine learning model 13 so that the error is reduced. The processing unit 12 may update weights included in the neural network by an error backpropagation method.

[0028] If a correct label has not been assigned to the selected data sample, the processing unit 12 may request a correct label from the user. The processing unit 12 may repeat the training of the machine learning model 13 and the selection of the data sample multiple times. The processing unit 12 outputs the trained machine learning model 13. The processing unit 12 may store the machine learning model 13 in a non-volatile storage, may display it on a display device, or may transmit it to another information processing device. The processing unit 12 may perform a prediction process using the trained machine learning model 13, or may measure the prediction accuracy of the machine learning model 13 using test data.

[0029] As described above, the information processing device 10 of the first embodiment inputs each of a plurality of data samples to the machine learning model 13, thereby extracting a plurality of feature amounts corresponding to the plurality of data samples from the machine learning model 13. The information processing device 10 normalizes the plurality of feature amounts to a plurality of normalized feature amounts that fall within a certain numerical range. The information processing device 10 selects one or more data samples that are a part of the plurality of data samples from among the plurality of data samples, based on the plurality of normalized feature amounts. The information processing device 10 trains the machine learning model 13 using the one or more data samples.

[0030] This narrows down the data samples used for machine learning, reducing the load on the machine learning and shortening the training time, and improving the prediction accuracy of the machine learning model13 compared to using all of the miscellaneous data samples.

[0031] In addition, the data samples are narrowed down using normalized features obtained by normalizing the features calculated in the machine learning model 13. Thus, data samples that are likely to contribute to improving the prediction accuracy are selected, improving the prediction accuracy of the machine learning model 13. In particular, the features extracted from the machine learning model 13 are closer to the prediction result than the data sample itself, and are therefore highly relevant to the prediction accuracy of the machine learning model 13. In addition, normalizing the features unifies the scale of the features, improving the selection accuracy of the data samples.

[0032] The machine learning model 13 may include a graph neural network that updates the feature values ​​of each of the multiple nodes based on the connection relationships between the multiple nodes indicated by the graph data. The extracted feature values ​​may be feature values ​​after passing through the graph neural network. The graph neural network specifies a nonlinear transformation between input data and output data. Therefore, the feature values ​​after passing through the graph neural network are more relevant to the final prediction result than the feature values ​​before passing through the graph neural network, and are suitable for selecting data samples. As a result, the accuracy of selecting data samples is improved, and the prediction accuracy of the machine learning model 13 is improved.

[0033] Furthermore, the information processing device 10 may convert a plurality of normalized features into a plurality of principal component features by principal component analysis, and may select a data sample based on the distance between the plurality of principal component features. This improves the accuracy of selecting data samples. In particular, normalizing the features before the principal component analysis reduces the risk that some dimensions included in the features become dominant and other dimensions are underestimated.

[0034] Furthermore, the machine learning model 13 may predict molecular energy from molecular data representing a molecule including multiple atoms, and the multiple feature amounts may be feature amounts calculated for the multiple atoms. This allows the machine learning model 13 to predict the results of quantum chemistry simulations such as density functional theory with high accuracy without executing the original algorithm. As a result, the load of the quantum chemistry simulation is reduced, and the simulation time is shortened. In addition, the accuracy of selecting data samples corresponding to molecular data is improved.

[0035] [Second embodiment] Next, a second embodiment will be described. The information processing device 100 of the second embodiment performs machine learning to train a machine learning model using training data. The machine learning model of the second embodiment is a substitution model that predicts the result of a quantum chemical calculation using density functional theory. The information processing device 100 may be a client device or a server device. The information processing device 100 may be called a computer, a machine learning device, or a simulation device. The information processing device 100 corresponds to the information processing device 10 of the first embodiment.

[0036] FIG. 2 illustrates an example of hardware of an information processing apparatus according to the second embodiment. The information processing device 100 has a CPU 101, a RAM 102, a HDD 103, a GPU 104, an input interface 105, a medium reader 106, and a communication interface 107, all connected via a bus. The CPU 101 corresponds to the processing unit 12 in the first embodiment. The RAM 102 or the HDD 103 corresponds to the storage unit 11 in the first embodiment.

[0037] The CPU 101 is a processor that executes instructions of a program. The CPU 101 loads the program and data stored in the HDD 103 into the RAM 102 and executes the program. The information processing device 100 may have multiple processors.

[0038] The RAM 102 is a volatile semiconductor memory that temporarily stores programs executed by the CPU 101 and data used in calculations by the CPU 101. The information processing device 100 may have a type of volatile memory other than a RAM.

[0039] The HDD 103 is a non-volatile storage that stores software programs such as an operating system (OS), middleware, and application software, and data. The information processing device 100 may have other types of non-volatile storage, such as a flash memory or a solid state drive (SSD).

[0040] The GPU 104 performs image processing in cooperation with the CPU 101, and outputs an image to a display device 111 connected to the information processing device 100. The display device 111 is, for example, a CRT (Cathode Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, or a projector. Other types of output devices, such as a printer, may be connected to the information processing device 100.

[0041] The GPU 104 may be used as a general purpose computing on graphics processing unit (GPGPU). The GPU 104 may execute a program in response to an instruction from the CPU 101. The information processing device 100 may include a volatile semiconductor memory other than the RAM 102 as a GPU memory.

[0042] The input interface 105 receives an input signal from an input device 112 connected to the information processing device 100. The input device 112 is, for example, a mouse, a touch panel, or a keyboard. A plurality of input devices may be connected to the information processing device 100.

[0043] The medium reader 106 is a reading device that reads the program and data recorded on the recording medium 113. The recording medium 113 is, for example, a magnetic disk, an optical disk, or a semiconductor memory. The magnetic disk includes a flexible disk (FD) and a HDD. The optical disk includes a compact disc (CD) and a digital versatile disc (DVD). The medium reader 106 copies the program and data read from the recording medium 113 to another recording medium such as the RAM 102 or the HDD 103. The read program may be executed by the CPU 101.

[0044] The recording medium 113 may be a portable recording medium. The recording medium 113 may be used for distributing programs and data. The recording medium 113 and the HDD 103 may be called computer-readable recording media.

[0045] The communication interface 107 communicates with other information processing devices via the network 114. The communication interface 107 may be a wired communication interface connected to a wired communication device such as a switch or a router, or a wireless communication interface connected to a wireless communication device such as a base station or an access point.

[0046] The machine learning and prediction processing of the second embodiment may be implemented in a dedicated server in a data center or as an application service. The machine learning and prediction processing of the second embodiment may be applied to various industrial fields such as material development and catalyst selection.

[0047] Next, a machine learning model according to the second embodiment will be described. FIG. 3 is a diagram showing an example of the use of a machine learning model to predict the results of quantum chemical calculations. To predict the result of the quantum chemical calculation, graph data 161 showing a molecular structure is provided. The graph data 161 includes a plurality of nodes and a plurality of edges. The nodes included in the graph data 161 represent atoms. The edges included in the graph data 161 indicate a connection between two atoms. The coordinates of the position where the atom exists and the element which is the type of the atom are specified for each node. In addition, a physical quantity for determining an initial value of a feature vector is assigned to each node. For example, the nuclear charge of the atom is assigned as the physical quantity.

[0048] The information processing device 100 inputs graph data 161 to a machine learning model 162. The machine learning model 162 is a graph neural network. The graph neural network is sometimes called an MPNN. The machine learning model 162 updates a feature vector of a node based on the feature vector of an adjacent node adjacent to the node and an edge vector assigned to an edge between the two nodes. The edge vector reflects the distance and direction between the two nodes. The feature vector is sometimes called a hidden state.

[0049] The function for updating the feature vector includes parameter values ​​determined through machine learning. The machine learning model 162 repeats updating the feature vector of each node based on the feature vector of the adjacent node multiple times. The machine learning model 162 aggregates the feature vectors of all the nodes to generate and output a prediction result 163 as a scalar value. The prediction result 163 indicates the molecular energy in the ground state of the molecule indicated by the graph data 161. The prediction result 163 corresponds to a predicted value of the result of a quantum chemical calculation using density functional theory. By using the machine learning model 162, the result is predicted without performing a high-load quantum chemical calculation.

[0050] FIG. 4 is a diagram illustrating an example of updating a feature vector based on a graph structure. Nodes 171, 172, 173, and 174 are nodes included in graph data 161, and each represents an atom. Node 171 has a feature vector 181. Node 172 has a feature vector 182. Node 173 has a feature vector 183. Node 174 has a feature vector 184. The adjacent nodes of node 171 are nodes 172, 173, and 174. However, in the calculation described below, node 171 itself may be included in the adjacent nodes of node 171. Here, we consider updating feature vector 181.

[0051] The machine learning model 162 calculates a message 185 from the feature vector 182 and the edge vector of the edge between the nodes 171 and 172 using a message function. The machine learning model 162 also calculates a message 186 from the feature vector 183 and the edge vector of the edge between the nodes 171 and 173 using a message function. The machine learning model 162 also calculates a message 187 from the feature vector 184 and the edge vector of the edge between the nodes 171 and 174 using a message function. The message function includes parameter values ​​determined through machine learning.

[0052] The machine learning model 162 calculates a message by adding up messages 185, 186, and 187. The machine learning model 162 updates the feature vector 181 to a feature vector 188 using an update function based on the feature vector 181 and the added up message. The update function includes parameter values ​​determined through machine learning. The machine learning model 162 executes message transmission and feature vector updating for all nodes. Furthermore, the machine learning model 162 repeats message transmission and feature vector updating for all nodes multiple times.

[0053] The machine learning model 162 is, for example, a Polarizable Atom Interaction Neural Network (PaiNN). PaiNN is described in, for example, the following literature: Kristof Schutt, Oliver Unke and Michael Gastegger, "Equivariant Message Passing for the Prediction of Tensorial Properties and Molecular Spectra", Proc. of the 38th International Conference on Machine Learning (ICML 2021), Volume 139, pp. 9377-9388, July 2021.

[0054] The machine learning model 162 assigns feature vectors s and v to each node. The feature vectors s and v are, for example, 128-dimensional numeric vectors. i represents the residual scalar of the i-th node. i represents the equivariant representation of the i-th node. i The initial value of is the nuclear charge Z of the i-th atom. i Typically, nuclear charges close to each other are transformed into similar embedded representations. The embedding part may be trained in advance or may be trained together with other parts of the machine learning model 162. i The initial value of is a zero vector.

[0055] The machine learning model 162 calculates the feature vector s of node i according to formula (1). i In formula (1), M s is the message function for the feature vector s, and U s is the update function for feature vector s. j is the feature vector s of node j adjacent to node i. ij is an edge vector indicating the distance and direction from the coordinates of node i to the coordinates of node j. s,i is the message that node i receives from its neighbors regarding feature vector s.

[0056]

number

[0057] Message function M s is a parameter, and the feature vector s j may include a coefficient matrix acting on the edge vector r ij Also, the update function U s is a parameter, and the feature vector s i may include a coefficient matrix acting on the message m s,iThe edge vector r may include a coefficient matrix acting on ij or message m s,i is, for example, a 128-dimensional numeric vector.

[0058] In addition, the machine learning model 162 calculates the feature vector v of the node i according to the formula (2). i In formula (2), M v is the message function for the feature vector v, and U v is the update function for the feature vector v. j is the feature vector v of node j adjacent to node i. v,i is the message that node i receives from its neighbors regarding feature vector v.

[0059]

number

[0060] Message function M v is a parameter, and the feature vector v j may include a coefficient matrix acting on the feature vector s j may include a coefficient matrix acting on the edge vector r ij Also, the update function U v is a parameter, and the feature vector v i may include a coefficient matrix acting on the message m v,i The message m v,i is, for example, a 128-dimensional numeric vector.

[0061] The machine learning model 162 updates the feature vectors s and v of each node multiple times according to formulas (1) and (2). After that, the machine learning model 162 linearly transforms the feature vector s of each of the multiple nodes. For the linear transformation, parameter values ​​determined through machine learning are used. For example, for the linear transformation, a coefficient matrix acting on the feature vector s is used. Then, the machine learning model 162 sums up the linearly transformed feature vectors s of all nodes and converts them into a prediction result 163 indicating energy.

[0062] FIG. 5 is a diagram illustrating an example of the structure of a machine learning model. The machine learning model 162 includes an embedding layer 140, message layers 141, 143, and 145, update layers 142, 144, and 146, linear layers 147 and 149, and a SiLU (Sigmoid Linear Unit) layer 148. The embedding layer 140 is a layer that represents a nuclear charge Z i This distributed representation vector is a feature vector s i The initial value of the feature vector v i The initial value of is a zero vector.

[0063] The message layer 141 receives the feature vector s i The initial value of and the feature vector v i and the edge vector r indicated by the graph data 161 ij The message layer 141 receives the message function M s ,M v Then, the feature vector s i ,v i Here, we transform the feature vector s i ,v i The feature vectors of the adjacent nodes are used to transform the feature vector s i The output of the message layer 141 for i In this way, the message layer 141 adds the feature vector s i The method is trained to calculate the appropriate difference between

[0064] The update layer 142 is located after the message layer 141 and updates the feature vector s i ,v i The update layer 142 receives the update function U s ,U v Then, the feature vector s i ,v i Here, we transform the feature vector s i ,v i The transformation of is done node by node, and the feature vectors of adjacent nodes are not used. i The output of the update layer 142 for i is added. Also, the feature vector v i The output of the update layer 142 for i In this way, the update layer 142 adds the feature vector s i ,v i The method is trained to calculate the appropriate difference between

[0065] The message layer 143 is located after the update layer 142 and receives the feature vector s i ,v i and the edge vector r ij The message layer 143 receives the message function M s ,M v Then, the feature vector s i ,v i Transform the feature vector s i ,v i The output of the message layer 143 for i ,v i are added together, respectively.

[0066] The update layer 144 is located after the message layer 143 and updates the feature vector s i ,v i The update layer 144 receives an update function Us ,U v Then, the feature vector s i ,v i Transform the feature vector s i ,v i The output of the update layer 144 for i ,v i are added together, respectively.

[0067] The message layer 145 is located after the update layer 144 and receives the updated feature vector s i ,v i and the edge vector r ij The message layer 145 accepts a message function M s ,M v Then, the feature vector s i ,v i Transform the feature vector s i ,v i The output of the message layer 145 for i ,v i are added together, respectively.

[0068] The update layer 146 is located after the message layer 145 and updates the feature vector s i ,v i The update layer 146 receives an update function U s ,U v Then, the feature vector s i ,v i Transform the feature vector s i The output of the update layer 146 for i is added.

[0069] Note that the message function M s ,M v The message layers 141, 143, and 145 using the update function U may contain the same types of parameters, but the parameter values ​​may be trained differently. s,U v The update layers 142, 144, 146 using the same parameter types may be trained differently, although the parameter values ​​may be different.

[0070] The linear layer 147 is located after the update layer 146 and is used to calculate the feature vector s i The linear layer 147 receives the feature vector s i is linearly transformed. The linear transformation here is performed on a node-by-node basis.

[0071] The SiLU layer 148 is located after the linear layer 147 and receives the feature vector from the linear layer 147. The SiLU layer 148 converts the numerical values ​​of each dimension included in the feature vector using SiLU shown in Equation (3). The SiLU is a type of activation function and is sometimes called a swish function.

[0072]

number

[0073] The linear layer 149 is located after the SiLU layer 148 and receives feature vectors from the SiLU layer 148. The linear layer 149 linearly transforms the feature vectors using a coefficient matrix. The linear transformation here is performed on a node-by-node basis. The feature vectors of multiple nodes output by the linear layer 149 are summed up and finally converted into a prediction result 163.

[0074] Next, machine learning for training the machine learning model 162 will be described. FIG. 6 is a diagram showing an example of the structure of a molecular data set. The information processing device 100 stores a molecular dataset 131. The information processing device 100 includes a plurality of samples that can be used for machine learning. Each sample indicates a molecular structure and energy. The molecular structure indicates the position coordinates and elements of each of a plurality of atoms included in the molecule. The nuclear charge of the atom is calculated from the elements. The molecular structure corresponds to the explanatory variables and input data of the machine learning model 162. The energy is the energy in the ground state of the molecule. The energy corresponds to the objective variable and correct label of the machine learning model 162.

[0075] As described below, the information processing device 100 selects some samples from the molecular dataset 131 that are useful for improving the prediction accuracy of the machine learning model 162, and trains the machine learning model 162 using the selected samples. Samples that are not selected may not be used for training the machine learning model 162. At the time of sample selection, each sample may already have a correct answer label assigned thereto, or may not have a correct answer label assigned thereto yet. In the latter case, the information processing device 100 may request the user to assign a correct answer label to the selected sample.

[0076] FIG. 7 is a diagram illustrating an example of sample selection in machine learning. The information processing device 100 acquires a molecular dataset 151 including a plurality of samples. The molecular dataset 151 corresponds to the above-mentioned molecular dataset 131. The information processing device 100 predicts the energy of each of a plurality of samples included in the molecular dataset 151 using a machine learning model 152. The machine learning model 152 corresponds to the above-mentioned machine learning model 162. The machine learning model 152 is provisionally trained in advance. For example, the machine learning model 152 is trained in advance using a small number of samples. At this point, the prediction accuracy of the machine learning model 152 does not need to be very high.

[0077] The information processing device 100 extracts, for each sample, a feature vector calculated in the machine learning model 152 in the process of predicting energy from the machine learning model 152. The extracted feature vector is a feature vector after passing through a graph neural network, and is a feature vector of a plurality of atoms before being added together. In the second embodiment, the information processing device 100 extracts a feature vector to be input to the linear layer 147 shown in FIG. 5.

[0078] The information processing device 100 concatenates multiple feature vectors corresponding to multiple atoms for each sample to generate a feature vector 153. When the molecules targeted by the molecular dataset 131 are large-scale, the number of dimensions of the feature vector 153 becomes large, such as tens of thousands of dimensions.

[0079] The information processing device 100 normalizes the feature vector 153 among a plurality of samples into a normalized feature vector 154. In normalization, the information processing device 100 identifies the maximum and minimum values ​​of each dimension included in the feature vector 153, and scales the numerical range from the minimum value to the maximum value into a numerical range of 0 to 1. This normalization is performed according to Equation (4). x is a variable representing a certain dimension. max(x) and min(x) are the maximum and minimum values ​​of the dimension x among a plurality of feature vectors corresponding to a plurality of samples.

[0080]

number

[0081] The information processing device 100 performs principal component analysis on a plurality of normalized feature vectors corresponding to a plurality of samples, and converts the normalized feature vector 154 into a principal component vector 155. This generates a plurality of principal component vectors corresponding to a plurality of samples. The principal component vector 155 is a numerical vector having fewer dimensions than the normalized feature vector 154.

[0082] In the principal component analysis, the information processing device 100 generates a variance-covariance matrix from a plurality of normalized feature vectors corresponding to a plurality of samples, and performs eigenvalue decomposition on the variance-covariance matrix. The eigenvalue decomposition obtains an eigenvalue λ and an eigenvector v that satisfy Av=λv for a certain matrix A. A new coordinate axis of the vector space is determined by this eigenvector v. This coordinate axis is a coordinate axis with a large variance in the vector space, and represents the distribution characteristics of the samples.

[0083] The information processing device 100 calculates the Euclidean distance 156 between the principal component vectors 155 for each pair of two different samples. However, other distance indices such as cosine similarity may be used as the distance indices. The information processing device 100 extracts some samples from the molecular dataset 151 based on the Euclidean distance 156, and generates a molecular data subset 157 formed from the extracted samples.

[0084] The number of samples to be extracted may be specified by the user. Alternatively, the number of samples to be extracted may be determined from the number of samples in the molecular dataset 151, such as 1 / 10 of the number of samples in the molecular dataset 151. Here, the information processing device 100 selects samples such that the samples included in the molecular data subset 157 have principal component vectors 155 that are as dissimilar as possible. Thus, samples with similar principal component vectors 155 are not included in the molecular data subset 157 in a duplicated manner as much as possible.

[0085] For example, the information processing device 100 calculates the minimum value of the Euclidean distance 156 between each of the multiple samples and other samples (the Euclidean distance 156 between each of the multiple samples and the closest sample). The information processing device 100 preferentially removes samples having the smallest minimum value from among the multiple samples. The information processing device 100 repeats removing samples until the number of remaining samples becomes equal to or less than the target number of samples.

[0086] However, the information processing device 100 may detect a pair of samples having a small Euclidean distance 156, and exclude one or both samples of the detected pair. Furthermore, the information processing device 100 may classify a plurality of principal component vectors corresponding to a plurality of samples into clusters based on the Euclidean distance 156, and may select a representative sample from each cluster. The number of clusters may be the same as the number of target samples. A clustering algorithm such as the k-means method or hierarchical clustering may be used.

[0087] The information processing device 100 further trains the machine learning model 152 using samples included in the molecular data subset 157. For example, the information processing device 100 generates input data from the molecular structure indicated by the sample, and inputs the input data to the machine learning model 152. The information processing device 100 calculates the error between the energy predicted by the machine learning model 152 and the correct label indicated by the sample, and updates the parameter values ​​included in the machine learning model 152 by the error backpropagation method. This improves the prediction accuracy of the machine learning model 152 compared to that at the time of temporary training.

[0088] 7 may be repeated multiple times. In this case, the information processing device 100 inputs multiple samples included in the molecular dataset 151 to the machine learning model 152 trained using the molecular data subset 157. The information processing device 100 calculates a feature vector 153, a normalized feature vector 154, a principal component vector 155, and a Euclidean distance 156. The information processing device 100 extracts a new molecular data subset 157 from the molecular dataset 151.

[0089] FIG. 8 is a diagram illustrating an example of normalization of a feature vector. The information processing device 100 generates a feature vector table 132. The feature vector table 132 is a table that lists a plurality of feature vectors corresponding to the above-mentioned feature vector 153 for a plurality of samples. One row of the feature vector table 132 represents one feature vector corresponding to one sample. A plurality of columns of the feature vector table 132 correspond to a plurality of dimensions included in the feature vector. One feature vector is a concatenation of a plurality of feature vectors calculated for a plurality of atoms included in the sample. The concatenated feature vector is extracted from immediately before the linear layer 147 in FIG. 5.

[0090] The information processing device 100 generates a normalized feature vector table 133 by normalizing the feature vector table 132. The information processing device 100 identifies a maximum value and a minimum value for each column of the feature vector table 132, and determines a scale conversion function for converting a numerical range from the minimum value to the maximum value into a numerical range of 0 or more and 1 or less. The information processing device 100 performs normalization for each column using the determined scale conversion function.

[0091] As described above, the information processing device 100 extracts the feature vector of each atom after passing through the graph neural network from the machine learning model. The data conversion of the graph neural network is a nonlinear conversion. Therefore, by determining the similarity between samples from the feature vector after the nonlinear conversion, it is easier to identify redundant samples that contribute less to improving prediction accuracy than by determining the similarity from the input data of the samples.

[0092] In addition, the information processing device 100 normalizes the feature vector extracted from the machine learning model before performing the principal component analysis. By normalizing the feature vector, a specific dimension included in the original feature vector is prevented from dominating in the principal component analysis, and all dimensions included in the original feature vector are fully taken into consideration. This improves the accuracy of determining the similarity. As a result, a sample useful for improving the prediction accuracy is selected.

[0093] FIG. 9 is a graph showing an example of the average error of a machine learning model. Curves 191 and 192 show the relationship between the number of samples and the average error. The horizontal axis of FIG. 9 represents the number of samples, and the vertical axis of FIG. 9 represents the average error. The number of samples is the number of samples selected for machine learning from the population, and corresponds to the number of samples in the molecular data subset 157 in FIG. 7. Here, the number of samples is specified by the user. The average error is the average of the prediction errors between the energy predicted by the machine learning model and the correct label when the machine learning model is trained using the selected samples.

[0094] Curve 191 shows the average error when the feature vector is not normalized when selecting a sample from the population. Curve 192 shows the average error when the feature vector is normalized when selecting a sample from the population. As shown by curves 191 and 192, if the number of samples is the same, the average error is smaller when the feature vector is normalized than when the feature vector is not normalized. Therefore, by normalizing the feature vector, appropriate samples that contribute to improving the prediction accuracy of the machine learning model are selected, and the prediction accuracy is improved.

[0095] In addition, normalizing the feature vectors achieves a desired prediction accuracy with fewer samples than not normalizing the feature vectors, thereby reducing the load of machine learning using selected samples and shortening the machine learning time.

[0096] Next, the functions and processing procedures of the information processing device 100 will be described. FIG. 10 is a block diagram illustrating an example of functions of the information processing device. The information processing device 100 has a dataset storage unit 121, a model storage unit 122, a prediction unit 123, a feature extraction unit 124, a sample selection unit 125, and a machine learning unit 126. The dataset storage unit 121 and the model storage unit 122 are implemented using, for example, a RAM 102, a GPU memory, or a HDD 103. The prediction unit 123, the feature extraction unit 124, the sample selection unit 125, and the machine learning unit 126 are implemented using, for example, a CPU 101, a GPU 104, and a program.

[0097] The dataset storage unit 121 stores a plurality of samples as a population. For example, the dataset storage unit 121 stores the above-mentioned molecular dataset 131. Each sample includes a molecular structure and an energy. The energy is a correct answer label corresponding to the correct answer of the output of the machine learning model. However, the correct answer label may be assigned to the sample after it is determined to use the sample for machine learning, as in active learning.

[0098] The model storage unit 122 stores a machine learning model. For example, the model storage unit 122 stores the machine learning model 162 shown in FIG. 5. The machine learning model is provisionally trained in advance. This machine learning model is further trained using some samples among the multiple samples stored in the dataset storage unit 121.

[0099] The prediction unit 123 generates input data that matches the input format of the machine learning model from the molecular structure defined for each of the multiple samples stored in the dataset storage unit 121. The prediction unit 123 inputs the generated input data for each of the multiple samples to the machine learning model to predict the energy of the molecule.

[0100] The feature extraction unit 124 extracts, for each of the multiple samples, a feature vector calculated while the prediction unit 123 predicts the energy from the machine learning model. The extracted feature vector is a feature vector calculated for each atom included in the molecule, and is a feature vector after sufficient message propagation between atoms has been performed. If the machine learning model includes alternating message layers and update layers, the extracted feature vector is, for example, a feature vector updated by the last update layer. The feature extraction unit 124 concatenates multiple feature vectors corresponding to multiple atoms to obtain one feature vector for each sample.

[0101] The sample selection unit 125 selects some samples from the dataset storage unit 121 based on the feature vectors acquired by the feature extraction unit 124. First, the sample selection unit 125 normalizes a plurality of feature vectors corresponding to a plurality of samples to generate a plurality of normalized feature vectors corresponding to the plurality of samples. The sample selection unit 125 performs principal component analysis on the plurality of normalized feature vectors to generate a plurality of principal component vectors corresponding to the plurality of samples. The sample selection unit 125 calculates a Euclidean distance indicating the similarity of the principal component vectors between two different samples.

[0102] The sample selection unit 125 selects some samples from the data set storage unit 121 based on the Euclidean distance. The some samples are selected so that the mutual Euclidean distance is as large as possible. For example, the sample selection unit 125 excludes one or both of two samples with small Euclidean distance from the population of samples. Thus, redundant samples with similar principal component vectors are excluded.

[0103] The machine learning unit 126 trains the machine learning model using the samples selected by the sample selection unit 125. Samples not selected by the sample selection unit 125 are not used in the machine learning here. For example, the machine learning unit 126 generates input data that matches the input format of the machine learning model from the molecular structure defined in the sample, and inputs the input data to the machine learning model. The machine learning unit 126 calculates the error between the predicted data output by the machine learning model and the correct label, and updates the parameter values ​​so as to reduce the error.

[0104] The machine learning unit 126 may update parameter values ​​included in the machine learning model by backpropagation. Alternatively, the machine learning unit 126 may divide the selected samples into a plurality of mini-batches and repeat a forward process for calculating an error and a backward process for updating parameter values ​​using the mini-batches.

[0105] The information processing device 100 may output information about the selected sample. The information processing device 100 may store the information about the selected sample in a non-volatile storage, may display it on the display device 111, or may transmit it to another information processing device. The information processing device 100 may store the trained machine learning model in a non-volatile storage, may display it on the display device 111, or may transmit it to another information processing device.

[0106] FIG. 11 is a flowchart showing an example of a machine learning procedure. (S10) The prediction unit 123 generates input data from the molecular structure for each sample. (S11) The prediction unit 123 inputs input data to a machine learning model for each sample and executes forward processing to generate predicted data. The feature extraction unit 124 extracts a feature vector from the machine learning model for each sample. The extracted feature vector is a feature vector generated during the process in which the input data is converted into predicted data.

[0107] (S12) The sample selection unit 125 normalizes the feature vectors of the multiple samples. In normalization, the feature vectors are scaled for each dimension so that the values ​​fall within a certain range. (S13) The sample selection unit 125 performs principal component analysis on the normalized feature vectors of the multiple samples, and converts the normalized feature vectors into principal component vectors.

[0108] (S14) The sample selection unit 125 calculates the Euclidean distance between the principal component vectors for each pair of two different samples from among the multiple samples. (S15) The sample selection unit 125 selects a certain number of samples from the data set, which is the population, based on the Euclidean distance calculated in step S14. The number of samples to be selected is specified by, for example, a user. Samples with a large Euclidean distance to other samples are preferentially selected. Sample selection will be described later.

[0109] (S16) The machine learning unit 126 trains the machine learning model using the samples selected in step S15. For example, the machine learning unit 126 inputs input data for each sample to the machine learning model, and calculates the error between the predicted data output by the machine learning model and the correct label. The machine learning unit 126 updates the parameter values ​​included in the machine learning model by the error backpropagation method so as to reduce the error.

[0110] FIG. 12 is a flowchart showing an example of a procedure for selecting samples. This sample selection is performed in step S15 described above. (S20) The sample selection unit 125 initializes a threshold to the minimum value of the Euclidean distances comprehensively calculated in step S14.

[0111] (S21) The sample selection unit 125 initializes a selection dataset indicating the selected samples to the original molecular dataset as a population. Thus, the selection dataset in the initial state indicates that all samples have been selected.

[0112] (S22) The sample selection unit 125 sets a target sample number. The target sample number is an upper limit for the number of samples to be selected. The target sample number may be specified by a user, or may be determined from the number of samples in the population.

[0113] (S23) The sample selection unit 125 determines whether the number of samples included in the selected data set exceeds the target sample number. If the number of samples in the selected data set exceeds the target sample number, the process proceeds to step S24. If the number of samples in the selected data set is equal to or smaller than the target sample number, the process proceeds to step S27.

[0114] (S24) The sample selection unit 125 searches the selected data set for samples whose Euclidean distance is equal to or greater than a threshold and less than a threshold+α. For example, α=0.01. For example, the sample selection unit 125 searches for a sample whose Euclidean distance to the closest other sample (the minimum value of the Euclidean distance to other samples) satisfies the above range. In this case, the corresponding sample becomes a target for deletion in step S25. Also, for example, the sample selection unit 125 searches for a pair of samples whose Euclidean distance satisfies the above range. In this case, one or both of the corresponding pair becomes a target for deletion in step S25.

[0115] (S25) If there is a sample that satisfies the condition of step S24, the sample selection unit 125 deletes the corresponding sample from the selected data set. (S26) The sample selection unit 125 updates the threshold to threshold+α. For example, the sample selection unit 125 adds 0.01 to the threshold. Then, the process returns to step S23.

[0116] (S27) The sample selection unit 125 outputs the samples included in the selected data set as samples to be used in machine learning. As described above, the information processing device 100 of the second embodiment predicts the results of quantum chemical calculations using a machine learning model instead of quantum chemical calculations using density functional theory. This reduces the load of quantum chemical simulations and shortens the simulation time. In addition, the machine learning model includes a graph neural network. This allows the feature vector of each atom to be calculated taking into account the molecular structure, improving prediction accuracy.

[0117] In addition, a portion of the samples in the population is used for machine learning instead of all of them. This reduces the load on the machine learning process and shortens the machine learning time. In addition, by selecting samples, the prediction accuracy of the trained machine learning model is improved compared to using all of the miscellaneous samples that contain similar redundant samples.

[0118] Furthermore, the information processing device 100 determines the similarity between samples using a feature vector calculated internally when a sample is input to the machine learning model. Then, the information processing device 100 selects samples that are as dissimilar as possible. This eliminates redundant samples, and selects samples that contribute greatly to improving the prediction accuracy of the machine learning model. Therefore, if the number of selected samples is kept constant, the prediction accuracy of the machine learning model improves. Furthermore, a target prediction accuracy is achieved with a small number of samples, and the machine learning time is shortened.

[0119] Furthermore, the information processing device 100 determines the similarity of samples using the feature vectors of each atom before addition, which are feature vectors after nonlinear transformation by the graph neural network. This allows the similarity to be determined from feature vectors that are strongly related to the prediction results of the machine learning model, improving the accuracy of sample selection.

[0120] Furthermore, the information processing device 100 normalizes the feature vector extracted from the machine learning model to convert it into a normalized feature vector, and further converts it into a principal component vector by principal component analysis. Normalization unifies the scale among multiple dimensions included in the feature vector. This reduces the risk that some dimensions will become dominant and other dimensions will be underestimated, and a principal component vector that appropriately expresses the features of each sample is generated. Furthermore, principal component analysis generates a principal component vector suitable for determining the similarity between samples. [Explanation of symbols]

[0121] 10. Information processing device 11 Storage section 12 Processing section 13 Machine Learning Models 14a, 14b, 14c Data samples 15a, 15b, 15c Features 16a, 16b, 16c Normalized features

Claims

1. inputting each of a plurality of data samples into a machine learning model, thereby extracting a plurality of feature quantities corresponding to each of the plurality of data samples from the machine learning model; normalizing each of the plurality of feature quantities into a plurality of normalized feature quantities each falling within a certain numerical range; selecting, from the plurality of data samples, one or more data samples that are a part of the plurality of data samples based on the plurality of normalized features; training the machine learning model using the one or more data samples; A machine learning program that causes a computer to carry out processing.

2. The machine learning model includes a graph neural network that updates feature amounts of the plurality of nodes based on a connection relationship between the plurality of nodes indicated by graph data, and the plurality of feature amounts are feature amounts after passing through the graph neural network. The machine learning program according to claim 1.

3. and selecting the one or more data samples includes converting each of the normalized features into a plurality of principal component features by principal component analysis, and selecting the one or more data samples based on distances between the plurality of principal component features. The machine learning program according to claim 1.

4. The machine learning model predicts molecular energy from molecular data representing a molecule including a plurality of atoms, and the plurality of features are feature amounts calculated for the plurality of atoms. The machine learning program according to any one of claims 1 to 3.

5. inputting each of a plurality of data samples into a machine learning model, thereby extracting a plurality of feature quantities corresponding to each of the plurality of data samples from the machine learning model; normalizing each of the plurality of feature quantities into a plurality of normalized feature quantities each falling within a certain numerical range; selecting, from the plurality of data samples, one or more data samples that are a part of the plurality of data samples based on the plurality of normalized features; training the machine learning model using the one or more data samples; A machine learning method in which processing is performed by a computer.

6. A storage unit that stores a plurality of data samples and a machine learning model; a processing unit that inputs each of the plurality of data samples into the machine learning model, thereby extracting a plurality of feature amounts corresponding to each of the plurality of data samples from the machine learning model, normalizing each of the plurality of feature amounts into a plurality of normalized feature amounts falling within a certain numerical range, selecting one or more data samples that are a part of the plurality of data samples from among the plurality of data samples based on the plurality of normalized feature amounts, and training the machine learning model using the one or more data samples; An information processing device having the above configuration.

Images