Learning device

Abstract

A learning device 1 comprises a learning unit 23 that learns a plurality of training data sets 11 and outputs model data 13 that specifies a model that infers a target variable y from an explanatory variable X. Each learning data set 11 includes data S that specifies a simulator, an explanatory variable X that is input for the simulator, a setting value C for the simulator, and a target variable y obtained by running the simulator using the explanatory variable X and the setting value C. The learning unit 23 learns a first model f1 that specifies the relationship between the explanatory variable X and the target variable y, a second model f2 that specifies the relationship between the explanatory variable X and the data S that specifies the simulator, a third model f3 that specifies the relationship between the explanatory variable X and the setting value C for the simulator, and a fourth model f4 that specifies the relationship between the explanatory variable X, the data S that specifies the simulator, and the setting value C for the simulator.

Description

Learning Device

[0001] The present disclosure relates to a learning device.

[0002] Numerical models consisting of complex systems of equations are used in climate simulations to predict future climates. Simulations using these numerical models may produce different results depending on the model used or the simulation settings, even if they are based on the same physical phenomenon.

[0003] To achieve higher speeds, there is a surrogate modeling technique, which uses a machine learning model to learn a numerical model and then uses the high-speed inference of the machine learning model to imitate the simulation results of the numerical model.

[0004] There is a surrogate modeling technology that takes into account differences in simulation settings (see Non-Patent Document 1). Non-Patent Document 1 separates the parameters of a machine learning model into parameters that learn parts that are common between numerical models regardless of the simulation settings, and parameters that learn parts that change depending on the simulation settings.

[0005] In Non-Patent Document 1, first, for each simulation setting, machine learning model parameters that vary depending on the simulation setting are learned. Then, Non-Patent Document 1 fixes the machine learning model parameters that vary depending on the simulation setting, and learns parameters that are common to the simulation settings.

[0006] Chen, S., Kalanat, N., Xie, Y. et al. Physics-guided machine learningfrom simulated data with different physical parameters. Knowl. Inf. Syst. 65, 3223-3250 (2023).

[0007] However, the machine learning model employed in Non-Patent Document 1 does not assume inputs related to simulators such as numerical models or simulator setting values. Non-Patent Document 1 has a problem in that it is unable to predict output results that reflect differences in simulators or differences in simulator setting values.

[0008] The present disclosure has been made in consideration of the above circumstances, and an object of the present disclosure is to provide a technology that can predict output results by reflecting differences in simulators or differences in simulator setting values.

[0009] A learning device according to one aspect of the present disclosure includes a storage device that stores multiple learning datasets, each including an explanatory variable and a target variable, and a learning unit that learns the learning datasets and outputs model data that identifies a model that infers the target variable from the explanatory variables. One learning dataset includes data that identifies one of multiple simulators, explanatory variables that serve as input to the simulator, setting values ​​of the simulator, and a target variable obtained by running the simulator using the explanatory variables and the setting values. The learning unit learns a first model that identifies the relationship between the explanatory variables and the target variable, a second model that identifies the relationship between the explanatory variables and data that identifies the simulator, a third model that identifies the relationship between the explanatory variables and the setting values ​​of the simulator, and a fourth model that identifies the relationship between the explanatory variables, the data that identifies the simulator, and the setting values ​​of the simulator, and outputs the model data that identifies the learned first, second, third, and fourth models.

[0010] According to the present disclosure, it is possible to provide a technique that can predict output results by reflecting differences in simulators or differences in simulator setting values.

[0011] Fig. 1 is a diagram illustrating functional blocks of a learning device according to the present disclosure. Fig. 2 is a diagram illustrating input and output of a model learned by the learning device. Fig. 3 is a diagram illustrating a model learned by the learning device. Fig. 4 is a flowchart illustrating a learning process performed by the learning device. Fig. 5 is a diagram illustrating the hardware configuration of a computer used in the learning device.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0013] (Learning Device) A learning device 1 according to the present disclosure constructs a model that predicts a target variable y from a certain explanatory variable X. The learning device 1 according to the present disclosure differs from a general machine learning model in that the target variable y in the present disclosure is obtained by setting a setting value C of a simulator specified by data S in the simulator and inputting an explanatory variable X to the simulator. The learning device 1 constructs a model that predicts the target variable y from a certain explanatory variable X, taking into consideration the simulator and the setting value C of the simulator. In the present disclosure, the simulator is, for example, a numerical model that targets physical phenomena such as climate prediction.

[0014] This allows the learning device 1 to predict output results by reflecting differences in simulators such as numerical models or different setting values ​​of the simulators.

[0015] 1, the learning device 1 includes a training data set 11, a converted training data set 12, and model data 13, and functions of a simulator conversion unit 21, a setting conversion unit 22, and a learning unit 23. Each piece of data is stored in a storage device such as a memory 902 or a storage 903. Each function is implemented in a CPU 901.

[0016] The training data set 11 is input data used for training by the learning device 1. The learning device 1 stores N training data sets 11. As shown in FIG. 2A, in the present disclosure, one training data set 11 includes an explanatory variable X, a target variable y, data S for identifying a simulator, and a setting value C of the simulator.

[0017] The explanatory variable X is the input of the simulator. The data S for specifying the simulator specifies one of the multiple simulators targeted by the learning device 1 according to the present disclosure. The setting value C is the setting value of the simulator. The target variable y is a value obtained by running the simulator specified by the data S using the explanatory variable X and the setting value C.

[0018] As shown in Figure 2(b) , a typical training dataset contains only an explanatory variable X and a target variable y. A training dataset 11 according to the present disclosure includes, in addition to the explanatory variable X and the target variable y, data S for identifying a simulator and a simulator setting value C. In each training dataset 11, the target variable y is the result of processing the explanatory variables identified in that training dataset 11 with a simulator for which a setting value is set.

[0019] The N sets of training data sets 11 included in the training device 1 may include training data sets 11 for each of a plurality of simulators. The N sets of training data sets 11 may include a plurality of training data sets 11 with different setting values ​​for one simulator. The N sets of training data sets 11 may also include a plurality of training data sets 11 that associate a plurality of explanatory variables with a combination of one simulator and setting value.

[0020] By training a model using such a training data set 11, it is possible to predict output results by reflecting differences in simulators such as numerical models or different setting values ​​of the simulator.

[0021] The explanatory variable X and the target variable y of the training dataset 11 are embedding vectors having a predetermined number of elements. For example, the explanatory variable X has n real number elements, and the target variable y has m real number elements.

[0022] The explanatory variable X and the target variable y input and output by the simulator may be data in any format, such as image data, graph data, document data, etc. In the present disclosure, data in any format is converted into an embedding vector having a predetermined number of elements in advance.

[0023] One method for converting data in any format into embedding vectors is to use machine learning models, such as convolutional neural networks (CNNs), graph neural networks (GNNs), or large language models (LLMs).

[0024] Other common methods can be used to convert data of any format into embedding vectors. Vision Transformer [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, Dosovitskiy+2021] is a method for converting image data into embedding vectors. Graph Convolutional Networks [Semi-Supervised Classification with Graph Convolutional Networks, Kipf and Welling 2017] is a method for converting graph data into embedding vectors. BERT [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, Devlin+2019] is a method for converting document data into embedding vectors.

[0025] In the present disclosure, the data S specifying the simulator of the training dataset 11 and the simulator setting value C may be in any data format. The data S specifying the simulator and the simulator setting value C are converted into a fixed-size embedding vector by a simulator conversion unit 21 and a setting conversion unit 22, which will be described later.

[0026] The transformed training dataset 12 is data obtained by converting the data S that identifies the simulator of the training dataset 11 and the simulator setting value C into fixed-size embedding vectors. The transformed training dataset 12 is generated by a simulator conversion unit 21 and a setting conversion unit 22, which will be described later. Each of the four elements of the transformed training dataset 12 has a fixed-size embedding vector format.

[0027] The model data 13 is data that specifies the model that the learning device 1 learns.

[0028] Here, three examples of the training dataset 11 according to the present disclosure will be described.

[0029] The first example relates to a simulator for predicting the global annual mean temperature. The explanatory variables X are basic climate variables, such as the global surface temperature and the east-west and north-south wind speeds as of January 1, 2020. The simulator specified by the data S is a predetermined Earth system model (ESM), and the setting values ​​C are the settings in that simulator. The simulator uses CO ２ The amount of greenhouse gas emissions from 2001 to 2100 is calculated, and the trend in the global annual average temperature from 2001 to 2100 is output as the target variable y.

[0030] In the training dataset 11 of the present disclosure, a plurality of different Earth system models may be set in the simulator. For a certain Earth system model, different modeling methods, such as a plurality of precipitation process modeling methods or a plurality of carbon cycle modeling methods, may be incorporated into the simulator, thereby setting a plurality of different simulators for a plurality of training datasets 11. Furthermore, a plurality of different setting values ​​may be set for a plurality of training datasets 11 by incorporating values ​​such as trends in greenhouse gas (GHG) emissions or global land use patterns into the setting value C.

[0031] The second example relates to a simulator that predicts the average annual crop yield of a certain crop. The explanatory variables X are the monthly average temperature and precipitation trends. The target variable y is the annual average crop yield of a certain crop. If there are multiple different modeling methods for existing crop growth models, each of these modeling methods is treated as a simulator specified by the data S. The setting value C is a parameter set for each growth model or the type of crop.

[0032] The third example relates to a simulator that predicts the trends in variables related to economic activity. The explanatory variable X is a variable related to economic activity. Examples of variables related to economic activity include each country's gross domestic product as of 2024, and the percentage of the population engaged in the primary, secondary, and tertiary industries within the country. The target variable y is the trend of a similar variable related to economic activity until 2100. Here, since results vary depending on the numerical model, each of these numerical models is considered to be a simulator specified by data S. The setting value C is a graph showing the amount and volume of trade between countries.

[0033] In each learning dataset 11, when the data S for identifying a simulator has an arbitrary size, the simulator conversion unit 21 converts the data S for identifying a simulator into an embedding vector of a fixed size. In the present disclosure, the fixed-size embedding vector converted from the data S for identifying a simulator is converted into the data S for identifying a simulator. ｖ It is sometimes referred to as.

[0034] The data S for specifying the simulator is given in a format that indicates which model the output data comes from, using one-hot vector encoding according to the number of models handled by the simulator. For example, there may be expressions that are common among different models within the simulator, such as several ways of expressing the precipitation process. The data S for specifying the simulator can express the different models within the simulator as discrete information using one-hot vector encoding.

[0035] If the size of the vector of the simulator-specific data S obtained by one-hot vector encoding is large, the simulator conversion unit 21 may generate an embedding vector having a predetermined vector size using a common dimension reduction technique, such as principal component analysis (PCA), singular value decomposition (SVD), or autoencoder.

[0036] Alternatively, the simulator conversion unit 21 may convert the data S in any format into an embedding vector of a fixed size by machine learning.

[0037] In each learning data set 11, when the setting value C has an arbitrary size, the setting conversion unit 22 converts the setting value C into an embedding vector of a fixed size. In the present disclosure, the fixed-size embedding vector converted from the setting value C is referred to as the setting value C. ｖ It is sometimes referred to as.

[0038] The setting value C is generally given in the form of a real value, a discrete value, a one-dimensional vector, a two-dimensional or three-dimensional tensor, or the like. If the setting value C does not include a discrete value, the setting conversion unit 22 can obtain a one-dimensional embedding vector by arranging given constants, vectors, and tensors in any order. If the setting value C includes a discrete value, the setting conversion unit 22 converts the setting value C into a real value vector using one-hot encoding or the like.

[0039] The setting value C may be given as natural language data such as a climate change scenario, or may be represented by a graph such as trade volume between countries. In this case, the setting conversion unit 22 converts the setting value C into a one-dimensional embedding vector using BERT, Graph2vec, or the WL kernel. If the vector size of the setting value C obtained by these conversions is large, the setting conversion unit 22 may generate an embedding vector having a predetermined vector size using a general dimensionality reduction technique.

[0040] Alternatively, the setting conversion unit 22 may convert the setting value C in any format into an embedding vector of a fixed size by machine learning.

[0041] The learning unit 23 learns the transformed training dataset 12 and outputs model data 13 that identifies a model that estimates a target variable y from an explanatory variable X. The learning unit 23 references the transformed training dataset 12, each of which has a fixed size, as data S that identifies a simulator and simulator setting values ​​C. When the data S that identifies a simulator of the training dataset 11 before conversion and the simulator setting values ​​C each have a fixed size, the learning unit 23 may reference the training dataset 11 before conversion.

[0042] In the present disclosure, as shown in equation (1), the learning unit 23 learns a first model f1 that specifies the relationship between the explanatory variable X and the target variable y, a second model f2 that specifies the relationship between the explanatory variable X and the data S that specifies the simulator, a third model f3 that specifies the relationship between the explanatory variable X and the setting value C of the simulator, and a fourth model f4 that specifies the relationship between the explanatory variable X, the data S that specifies the simulator, and the setting value C of the simulator. The learning unit 23 outputs model data 13 that specifies the learned first model f1, second model f2, third model f3, and fourth model f4.

[0043]

[0044] As shown in Figure 3, in equation (1), the term related to the first model f1 is referred to as the f1 term. The term related to the second model f2 is referred to as the f2 term. The term related to the third model f3 is referred to as the f3 term. The term related to the fourth model f4 is referred to as the f4 term. The learning model output by the learning unit 23 predicts the target variable y as the sum of the prediction results of the f1 term, f2 term, f3 term, and f4 term.

[0045] The first model f1 models the portion that can be predicted from the explanatory variable X, regardless of differences in simulators or simulator setting values. As shown in the f1 section, the input to the first model f1 is only the explanatory variable X, and does not include simulator-specific data S or setting values ​​C. The first model f1 is a learning model that reflects differences only in the explanatory variable X, without depending on the simulator-specific data S or setting values ​​C. Therefore, the first model f1 learns portions related to universal physical laws, regardless of simulators or setting values.

[0046] The second model f2 models the portion that can be predicted from the explanatory variable X and the simulator specified by the data S, regardless of differences in the setting value C. The second model f2 receives only the explanatory variable X and the data S that specifies the simulator as input, and does not receive the setting value C. The second model f2 is a learning model that reflects differences in the explanatory variable X and the data S that specifies the simulator, without depending on the setting value C. The sum of the f1 and f2 terms is considered to represent the bias of the simulator.

[0047] The third model f3 models the portion that can be predicted from the simulator specified by the explanatory variable X and the setting value C, regardless of the difference in the simulator specified by the data S. The third model f3 takes only the explanatory variable X and the setting value C as input, and does not take the setting value C as input. The third model f3 is a learning model that reflects differences in the explanatory variable X and the setting value C, without depending on the simulator. The sum of the f1 and f3 terms is considered to represent a plausible prediction result for the setting values ​​of a certain simulator.

[0048] The fourth model f4 models the predicted portion using all of the explanatory variables X, the simulator, and the setting value C.

[0049] Here, the first model f1, the second model f2, the third model f3, and the fourth model f4 are functions that output, for each input, a vector of the same shape, specifically, the same size, as the target variable y. The first model f1, the second model f2, the third model f3, and the fourth model f4 are machine learning models such as linear regression models or deep learning models, but may also be existing trained models or functions expressed as mathematical formulas.

[0050] For example, the first model f1 may be an existing, highly accurate trained model trained using a large amount of data, or a function derived from existing physics knowledge. The first model f1 allows differences in simulators or setting values ​​to be incorporated into the structure (architecture) of the machine learning model. This allows the machine learning model identified by the model data 13 output by the learning unit 23 to appropriately reflect differences in numerical models, simulators, or setting values ​​in predictions.

[0051] Next, the learning process performed by the learning unit 23 will be described.

[0052] The loss function or optimization method in training the first model f1, the second model f2, the third model f3, and the fourth model f4 can take any form, such as a general mean squared error or stochastic gradient descent. When the loss function or optimization method takes the form of a function, or when training is performed with parameters of a trained model fixed, the parameter training process may be skipped.

[0053] After learning the first model f1, the learning unit 23 uses the first model f1 to learn the second model f2 and the third model f3, and then uses the first model f1, the second model f2, and the third model f3 to learn the fourth model f4. Specifically, the learning unit 23 learns each model in the order of Equation (2), Equation (3), and Equation (4).

[0054]

[0055] The learning unit 23 first learns the first model f1. The learning unit 23 learns the first model f1 so that the output of the f1 term to which the explanatory variable Xi is set approaches the target variable yi.

[0056] The learning unit 23 then uses the learned first model f1 to learn a second model f2 and a third model f3. Specifically, the learning unit 23 learns the second model f2 and the third model f3 so that the output of the f2 term in which the explanatory variable Xi and the data Si specifying the simulator are set, and the output of the f3 term in which the explanatory variable Xi and the setting value Ci are set, approach the difference between the output of the f1 term in which yi and the explanatory variable Xi are set.

[0057] In equation (2), the part that depends on the explanatory variable Xi has already been trained as the first model f1, so the second model f2 and the third model f3 are considered to be independent models. The training unit 23 can train the second model f2 and the third model f3 simultaneously.

[0058] The learning unit 23 uses the trained first model f1, second model f2, and third model f3 to learn, as a fourth model f4, a portion of the target variable yi that cannot be resolved by the first model f1, the second model f2, and the third model f3. Specifically, the learning unit 23 learns the fourth model f4 so that the output of the f4 term in which the explanatory variable Xi, the data Si specifying the simulator, and the setting value Ci are set approaches a value obtained by subtracting from yi the output of the f1 term in which the explanatory variable Xi is set, the output of the f3 term in which the explanatory variable Xi and the data Si specifying the simulator are set, and the output of the f3 term in which the explanatory variable Xi and the setting value Ci are set.

[0059] The model learned by the learning unit 23 is used to estimate the target variable y. When the explanatory variables Xi, data Si specifying the simulator, and setting values ​​Ci are input to the model, a predicted value of the target variable yi is obtained.

[0060] (Learning Method) A learning method in the learning device 1 according to the present disclosure will be described with reference to FIG.

[0061] In step S11, the learning device 1 converts data S that identifies the simulator of each learning data set 11 into data S of fixed-size embedding vectors. ｖIn step S12, the learning device 1 converts the setting value C of each learning data set 11 into the setting value C of the fixed-size embedding vector ｖ Convert to.

[0062] In step S13, the learning device 1 learns the first model f1 so that the output value of the f1 term to which the explanatory variable Xi is input approaches yi. The learning device 1 learns the first model f1 using equation (2).

[0063] In step S14, the learning device 1 learns a second model f2 using the first model f1 learned in step S13. In step S15, the learning device 1 learns a third model f3 using the first model f1 learned in step S13. The learning device 1 learns the second model f2 and the third model f3 using equation (3).

[0064] In step S16, the learning device 1 learns a fourth model f4 using the first model f1 learned in step S13, the second model f2 learned in step S14, and the third model f3 learned in step S15. The learning device 1 learns the fourth model f4 using equation (4).

[0065] In step S17, the learning device 1 outputs model data 13 that identifies each model learned in steps S13 to S16.

[0066] The learning device 1 according to the present disclosure converts differences between simulators, such as numerical models, and differences between setting values ​​into embedding vectors, and models these in a prediction model. The prediction model is an additive model formed from multiple models that use these embedding vectors as arguments. The prediction model formed in this way can predict output results that reflect the differences between different simulators and different setting values.

[0067] The learning device 1 inputs an embedding vector, which is a vector of the differences between simulators and the differences between setting values, into the learning model. The model learned by the learning device 1 can predict output results for different simulators and setting values. Such a model does not require the introduction of parameters into the machine learning model for different simulators and setting values. Unlike Non-Patent Document 1, the learning device 1 does not need to learn as many combinations of simulators and setting values ​​as there are to be targeted, thereby reducing learning costs.

[0068] The learning device 1 formulates the structure (architecture) of the machine learning model by individually considering differences in the numerical models adopted by the simulators or differences in the setting values ​​of the simulators. The learning device 1 can appropriately reflect differences in the simulators or differences in the setting values ​​of the simulators in the prediction results. By analyzing each of the first model f1, the second model f2, the third model f3, and the fourth model f4, the learning device 1 can analyze biases due to differences in the numerical models adopted by the simulators or trends commonly observed in the setting values ​​of the simulators.

[0069] Each model may have a function format other than a machine learning model. For example, the learning device 1 may use an existing model for each model to improve the accuracy of the existing model.

[0070] The learning device 1 learns differences between simulators and settings, and can learn a machine learning model without explicitly separating common parts, etc., regardless of the simulator or setting values.

[0071] The learning device 1 according to the present disclosure can predict output results by reflecting differences in simulators or differences in simulator setting values.

[0072] The learning device 1 according to the present disclosure described above uses, for example, a general-purpose computer system including a CPU (Central Processing Unit, processor) 901, memory 902, storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), communication device 904, input device 905, and output device 906. In this computer system, the CPU 901 executes a program loaded on the memory 902, thereby realizing each function of the learning device 1.

[0073] The learning device 1 may be implemented on one computer or on multiple computers, or may be a virtual machine implemented on a computer.

[0074] The program of the learning device 1 can be stored on a computer-readable recording medium such as a HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), or DVD (Digital Versatile Disc), or can be distributed via a network. The computer-readable recording medium is, for example, a non-transitory recording medium.

[0075] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the present disclosure.

[0076] REFERENCE SIGNS LIST 1 Learning device 11 Learning data set 12 Converted learning data set 13 Model data 21 Simulator conversion unit 22 Setting conversion unit 23 Learning unit 901 CPU 902 Memory 903 Storage 904 Communication device 905 Input device 906 Output device

Claims

1. A learning device comprising: a storage device that stores multiple learning datasets each including an explanatory variable and a target variable; and a learning unit that learns the learning datasets and outputs model data that identifies a model that infers the target variable from the explanatory variables, wherein one learning dataset includes: data that identifies one of multiple simulators; explanatory variables that serve as input to the simulator; setting values ​​of the simulator; and a target variable obtained by running the simulator using the explanatory variables and the setting values, and the learning unit learns a first model that identifies the relationship between the explanatory variables and the target variable, a second model that identifies the relationship between the explanatory variables and data that identifies the simulator, a third model that identifies the relationship between the explanatory variables and the setting values ​​of the simulator, and a fourth model that identifies the relationship between the explanatory variables, the data that identifies the simulator, and the setting values ​​of the simulator, and outputs the model data that identifies the learned first, second, third, and fourth models.

2. The learning device according to claim 1, wherein the learning unit learns the first model, then uses the first model to learn the second model and the third model, and then uses the first model, the second model, and the third model to learn the fourth model.

3. The learning device according to claim 1, wherein in each learning data set, the data specifying the simulator and the setting values ​​of the simulator each have a fixed size.

4. The learning device of claim 1, wherein in each learning dataset, the data specifying the simulator and the setting values ​​of the simulator each have an arbitrary size, and further comprising a simulator conversion unit that converts the data specifying the simulator into an embedding vector of a fixed size, and a setting conversion unit that converts the setting values ​​of the simulator into an embedding vector of a fixed size, and the learning unit refers to the converted embedding vector as the data specifying the simulator and the converted embedding vector as the setting values ​​of the simulator.

Images