Automatic generation of machine learning models from computational simulation data

Abstract

Systems and methods for automatically training a machine learning model are described herein. An exemplary method includes running a set of computational simulations and assembling a dataset associated with the set of computational simulations. The dataset includes data associated with at least one simulation result for at least one computational simulation in the set of computational simulations. The method also includes training a machine learning model using the dataset. At least one feature and at least one target for the machine learning model are part of the dataset.

Description

[Background technology]

[0001] Computer-aided engineering (CAE) is the practice of simulating representations of physical objects using computational methods, including but not limited to the finite element method (FEM) and the finite difference method (FDM). To perform a simulation using FEM and / or FDM, the domain must be discretized into a finite number of elements, called a mesh. FEM and FDM are techniques for converting differential equations (e.g., partial differential equations (PDEs)) into systems of equations that can be solved numerically.

[0002] Machine learning, a branch of artificial intelligence, fosters the ability of systems to automatically "learn" from experience. In some cases, the systems are able to learn without human intervention. Machine learning requires large amounts of data to train the algorithms. Summary of the Invention

[0003] SUMMARY OF THE INVENTION Systems and methods for automatically generating machine learning models from computational simulation data are described herein.

[0004] An exemplary method for automatically training a machine learning model is described herein. The method includes: running a set of computational simulations; and assembling a dataset associated with the set of computational simulations. The dataset includes data associated with at least one simulation result for at least one computational simulation in the set of computational simulations. The method also includes training a machine learning model using the dataset. At least one feature and at least one target for the machine learning model are part of the dataset.

[0005] Additionally, the data set further includes data associated with at least one simulation input for at least one computational simulation in the set of computational simulations.

[0006] In some implementations, the method further includes receiving, from a user, an input specification associated with the at least one simulation input.

[0007] In some implementations, the method further includes receiving, from a user, a result specification associated with the at least one simulation result.

[0008] Alternatively or additionally, at least one simulation input for at least one computational simulation in the set of computational simulations includes a parameterized variable. Optionally, the parameterized variable is a parameterized condition at a location on the simulation model. Alternatively or additionally, the parameterized variable is optionally a parameterized attribute of the simulation model.

[0009] Alternatively or additionally, at least one simulation result for at least one computational simulation in the set of computational simulations includes a calculated property at a location on the simulation model. Optionally, the calculated property represents an output of a measurement device.

[0010] In some implementations, the method further receives, from a user, a target specification relating to at least one target for the machine learning model. Optionally, the at least one target is a simulation input or a simulation result.

[0011] In some implementations, the method further includes receiving, from a user, a feature specification relating to at least one feature for the machine learning model. Optionally, the at least one feature is a simulation input or a simulation result.

[0012] In some implementations, training a machine learning model using the dataset includes training a plurality of machine learning models using the dataset. Optionally, the machine learning models are trained in parallel. Alternatively or additionally, the method further includes evaluating performance of each of the trained machine learning models.

[0013] Alternatively or additionally, the trained machine learning model is configured to predict a state, characteristic, or behavior of a physical system based on at least one measurement obtained by a measurement device of the physical system.

[0014] Alternatively or additionally, the machine learning model is a supervised learning model, a semi-supervised learning model, or an unsupervised learning model. Optionally, the machine learning model is a deep learning model. Optionally, the machine learning model is an artificial neural network.

[0015] In some implementations, the set of computational simulations and machine learning model training is run until a stopping criterion is reached. Optionally, the stopping criterion is a predetermined performance level of the trained machine learning model, a predetermined number of simulations, a predetermined number of computational core hours, or a predetermined cost.

[0016] In some implementations, the method further includes providing the trained machine learning model to a user. For example, the method includes optionally transmitting to the user an architecture, hyperparameter values, and / or parameter values ​​of the trained machine learning model.

[0017] Alternatively or additionally, the data set further comprises real data associated with the physical system. Optionally, the real data is measured by at least one measurement device of the physical system.

[0018] An exemplary system is described herein. The system includes a device including a network of measurement devices, a machine learning module, and a controller. The machine learning module is trained using a dataset, the dataset including data associated with at least one simulation result for at least one computational simulation in a set of computational simulations, and at least one feature and at least one target for the machine learning model are part of the dataset. The controller includes a processor and a memory having computer-executable instructions stored in the memory. The controller is configured to receive each measurement from the network of measurement devices, input each measurement to the machine learning module, and receive an output predicted by the machine learning module, the output being a state, characteristic, or behavior of the device.

[0019] Additionally, the data set further includes data associated with at least one simulation input for at least one computational simulation in the set of computational simulations.

[0020] Alternatively or additionally, at least one simulation input for at least one computational simulation in the set of computational simulations includes a parameterized variable. Optionally, the parameterized variable is a parameterized condition at a location on the simulation model. Alternatively or additionally, the parameterized variable is optionally a parameterized attribute of the simulation model.

[0021] Alternatively or additionally, at least one simulation result for at least one computational simulation in the set of computational simulations includes a calculated property at a location on the simulation model, optionally the calculated property representing an output of at least one measurement device of the network of measurement devices.

[0022] Alternatively or additionally, the trained machine learning model is configured to predict a state, characteristic, or behavior of a physical system based on at least one measurement obtained by a network of measurement devices.

[0023] Alternatively or additionally, the machine learning model is a supervised learning model, a semi-supervised learning model, or an unsupervised learning model. Optionally, the machine learning model is a deep learning model. Optionally, the machine learning model is an artificial neural network.

[0024] Alternatively or additionally, the data set further comprises actual data associated with the device. Optionally, the actual data is measured by at least a network of measuring devices.

[0025] It will be appreciated that the above subject matter may be implemented as an article of manufacture such as a computer controller, a computer process, a computing system, or a computer-readable storage medium.

[0026] Other systems, methods, features, and / or advantages will be or become apparent to one with skill in the art upon examination of the following figures and detailed description. It is intended that all such additional systems, methods, features, and / or advantages be included within this description and protected by the accompanying claims. [Brief explanation of the drawings]

[0027] The elements in the drawings are not necessarily to scale relative to each other, and like reference numbers designate corresponding parts throughout the views.

[0028] [Figure 1] 1 is a flowchart illustrating example operations for automatically training a machine learning model according to implementations described herein. [Figure 2] 1 is an exemplary computing device. [Figure 3]1 is a table illustrating a synthetic dataset according to implementations described herein. [Figure 4] FIG. 1 is a block diagram illustrating an example environment for automatically training machine learning models according to implementations described herein. DETAILED DESCRIPTION OF THE INVENTION

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. Methods and materials similar or equivalent to those described herein may be used in the practice or testing of the present disclosure. As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. As used herein, the term "comprising" and variations thereof are used synonymously with the term "including" and variations thereof and are open, non-limiting terms. As used herein, the term "optional" or "optionally" refers to the subsequently described feature, event, or circumstance that may or may not occur, and the description includes instances where the feature, event, or circumstance occurs or does not occur. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, the aspect includes from the one particular value and / or to the other particular value. Similarly, when values ​​are expressed as approximations, by use of the antecedent "about," it will be understood that the particular value forms another aspect. It will be further understood that the endpoints of each of the ranges are significant both in relation to the other endpoint, and independently of the other endpoint.

[0030] Described herein are systems and methods for automatically generating machine learning models from computational simulation data. Traditionally, computational simulation performance has not been integrated with the machine learning training process. As described above, machine learning requires large datasets (sometimes referred to herein as "datasets") for training. Running a set of computational simulations, manually assembling the resulting computational simulation datasets (often referred to as "synthetic data"), and training a machine learning model is time-consuming and error-prone. Raw computational simulation data can be unreliable and therefore unsuitable for model training. For example, raw computational simulation data can be biased, inaccurate, unreliable, and / or ambiguous. Therefore, training datasets (and optionally validation and test datasets) must be purposefully assembled from computational simulation data to serve a specific purpose of the model. This can avoid the garbage-in, garbage-out problem. In conventional systems, dataset assembly is a manual process. In contrast, the automated systems and methods described herein can efficiently process computational simulation data, assemble training datasets, and then use the same to train a machine learning model without user intervention. Such processing may include, but is not limited to, merging data from multiple sources, organizing data, formatting data, cleaning data (e.g., removing unnecessary and / or problematic data), extracting data and / or features from the data, and labeling data (e.g., tagging features and targets for a surveillance model). As discussed above, it should be understood that computational simulations generate large amounts of data. The automated systems and methods described herein are configured to, among other features, identify subsets of simulation inputs and results, assemble the subsets into a training dataset, and then train a machine learning model without user intervention.Thus, the systems and methods described herein solve problems present in conventional systems and processes by integrating computational simulation, synthetic data preparation, and machine learning model training into a fully automated process.

[0031] The synthetic data generated by the set of simulations can be used to train a machine learning model, e.g., a trained machine learning model configured to predict one or more conditions acting on a device including a sensor network. For example, consider a network of sensors embedded in a device. The device can be of any type, and the sensors can be of various types; for example, a sensor network can include a mixture of various types of measurements (temperature, pressure, strain, etc.). One goal for the machine learning model can be to predict, in real time, one or more conditions acting on the device based on the current measured output of the sensor network. For example, in the case of a touchscreen, the screen is the device, and the device includes a set of distributed strain sensors positioned below the screen. The goal for the machine learning model can be to predict force load locations (due to touches) on the screen based on the actual measured output of the distributed set of strain sensors. As described below, the machine learning model is trained using computational simulation data (e.g., a synthetic data set). The trained machine learning model can then be used in inference mode to predict force load locations. It should be understood that real-time prediction of conditions acting on the screen is merely an exemplary goal of the machine learning model. This disclosure contemplates other goals for machine learning models. For example, a goal for a machine learning model may be to predict a particular outcome (such as structural failure or overheating) based on the measured output of a sensor network.

[0032] With a goal of predicting one or more conditions acting on a device, a set of simulations of the device can be configured in a conventional manner, for example, by specifying the geometry, materials, loads / boundary conditions, outputs, etc. Outputs can include measured properties and locations of each sensor in the sensor network. For each condition the machine learning model is configured to predict (e.g., pressure or temperature on the surface of the device), a distribution of values ​​for that condition is specified. A set of simulations is then run to explore the parameter space for the set of conditions. Once the simulations are completed, the simulation results are included in a dataset. Optionally, the dataset can be divided into a training dataset and a validation dataset. A machine learning model is then trained (and optionally validated and / or tuned) using the dataset. Optionally, analysis of the machine learning model can inform the process of which regions of the parameter space to emphasize for further simulations and training. Alternatively or additionally, the set of simulations can optionally continue until a stopping criterion is reached. Exemplary stopping criteria can include, but are not limited to, machine learning model performance reaching an acceptable threshold, the number of simulations exceeding a threshold, monetary cost exceeding a threshold, and / or the number of compute core hours used exceeding a threshold.

[0033]

[0023] Referring now to Figure 1, a flowchart illustrating example operations for automatically training a machine learning model is shown. The present disclosure contemplates that one or more of the operations shown in Figure 1 may be performed automatically, e.g., without user input and / or intervention. For example, a set of simulations is initiated (e.g., step 102 of Figure 1), a computational simulation data set is assembled (e.g., step 104 of Figure 1), and a machine learning model is trained without requiring user input or intervention (e.g., step 106 of Figure 1). Optionally, in some implementations, all of the operations shown in Figure 1 may be performed automatically, e.g., without user input and / or user intervention between any steps.

[0034] In step 102, a set of computational simulations is performed. The set of computational simulations includes those necessary to generate reliable synthetic data that can be used to train a machine learning model. It should be understood that the number, type, and / or size of the set of simulations depends on the nature and purpose of the trained machine learning model. The present disclosure contemplates performing the computational simulations using one or more computing devices, such as the computing device of FIG. 2. As used herein, a set of computational simulations includes one or more computational simulations. In some implementations, a set of computational simulations is a plurality of computational simulations. The present disclosure contemplates that the set of computational simulations relates to a physical system, for example, a device having an embedded sensor network. In the simulation, the physical system is represented by a simulation model (e.g., a two-dimensional or three-dimensional virtual representation of the physical system). Exemplary devices with embedded sensor networks are touch-sensitive electronic devices using a distributed set of strain sensors (see, e.g., Example 2) or a device using a distributed set of thermocouples (see, e.g., Example 1). It should be understood that these physical systems are provided by way of example only. The present disclosure contemplates that the physical system may be any kind of structure and may include various types of sensors (e.g., temperature, pressure, strain, current, optical, flow, chemical, acoustic, etc. sensors).

[0035] In some implementations, the set of computational simulations may optionally be performed by a cloud-based computing cluster. Exemplary systems and methods for running simulations using a cloud-based computing cluster are described in U.S. Patent Application Publication No. 2020 / 0342148, entitled "SYSTEMS AND METHODS FOR RUNNING A SIMULATION," published October 29, 2020, by applicant OnScale, Inc. It should be understood that the systems and methods described herein are not intended to be limited to cloud-based computing implementations. Computational simulations such as FEM and FDM are known in the art and therefore will not be described in further detail herein.

[0036] Prior to running the set of simulations, an input specification related to at least one simulation input (described in detail below) is received from a user, e.g., at a computing device that integrates computational simulation and machine learning model training. For example, the user may specify which simulation inputs to use to train the machine learning model and provide such information to the computing device. The specified simulation inputs may optionally be a subset (i.e., not all) of the simulation inputs used to run the set of computational simulations. The present disclosure contemplates that the input specification (including the specified simulation inputs) may optionally be provided to the computing device via a message, a web-based portal, a mobile application, or the like. Additionally, a result specification related to at least one simulation result (described in detail below) is received from a user, e.g., at a computing device that integrates computational simulation and machine learning model training. For example, the user may specify which simulation results to use to train the machine learning model and provide such information to the computing device. The specified simulation results may optionally be a subset (i.e., not all) of the simulation results generated by running the set of computational simulations. The present disclosure contemplates that the result specification (including the specified simulation results) may optionally be provided to a computing device via a message, a web-based portal, a mobile application, etc. In other words, when a user requests a set of computational simulations related to a physical system, the user specifies which of the simulation inputs and / or simulation results are of interest to the user for purposes of training a machine learning model (e.g., see the table in FIG. 3 ). These simulation inputs and / or simulation results (and associated data) form the dataset used to train the machine learning module.As described below, this dataset contains at least one target and at least one feature for the machine learning model.

[0037] In step 104, a dataset associated with the set of computational simulations is assembled. The dataset assembled in step 104 can serve as a training dataset, i.e., a dataset used to train a machine learning model (see step 106). Optionally, the dataset assembled in step 104 can be split into a training dataset and a validation dataset. It should be understood that the validation dataset includes data returned from model training and subsequently used solely to measure model performance during training. The present disclosure contemplates assembling the dataset using one or more computing devices, such as the computing device of FIG. 2. For example, the dataset may optionally be stored in memory or on a hard drive of a computing device that integrates the computational simulations and the machine learning model training. The present disclosure contemplates that the dataset may be stored locally or remotely (e.g., accessible via a network) with respect to the computing device. Assembling the dataset may include collecting, merging, and / or combining respective computational simulation data from multiple computational simulations. Additionally, assembling a dataset may include data processing steps to, but are not limited to, select, rearrange, modify, reduce, normalize, encode, classify, augment, label, and / or store data in the dataset to prepare the dataset for use in training a machine learning model. The goal is to assemble a reliable dataset for machine learning model training (see step 106). It should be understood that assembling a dataset can include one or more of the above data processing steps, which are provided by way of example only. The present disclosure contemplates using techniques known in the art to assemble a dataset.

[0038] As described above, step 104 can be performed without user input or intervention. That is, the workflow between steps 102 and 104 can be automated. The dataset includes data associated with at least one simulation result for each computational simulation in the set of computational simulations. Additionally, the dataset optionally includes data associated with at least one simulation input for each computational simulation in the set of computational simulations. It should be understood that the input to the simulation can include, but is not limited to, geometry, materials, and initial / load / boundary conditions acting on the simulation model (e.g., forces, heat, current and / or voltage, magnetic fields, light, etc.). Methods such as FEM and / or FDM are used to discretize the domain of the simulation model into a finite number of elements and convert differential equations into a system of equations that can be numerically solved to generate one or more outputs. Thus, the data associated with at least one simulation result can be generated by running a set of computational simulations based on data associated with at least one simulation input. Because the data associated with the simulation input and the simulation result are generated by computer simulation rather than by physical experimentation, they are sometimes referred to herein as "synthetic" data. As described below, one or more machine learning models may be trained using the synthetic data.

[0039] As used herein, a simulation input for which associated data is included in a synthetic dataset includes a parameterized variable. A parameterized variable is a parameter whose value varies from simulation to simulation. Optionally, a parameterized variable is a parameterized condition at a location of a simulation model (e.g., a point, edge, surface, or volume of the model). Conditions may include, but are not limited to, force, displacement, pressure, traction, velocity, mass flux, momentum, dynamic pressure, temperature, heat flux, power, current, voltage, and magnetic field strength. It should be understood that the above conditions are provided by way of example only. For example, the present disclosure generally contemplates that other conditions are possible from the examples. Additionally, a condition may be one of three types: a load condition, a boundary condition, or an initial condition. Alternatively, a parameterized variable is optionally a parameterized characteristic of a simulation model. A characteristic may include a characteristic of a simulation model that is not a condition, such as a material or geometric property. It should be understood that the above characteristics are provided by way of example only. In some implementations, at least one simulation input for each computational simulation includes a single parameterized variable. In other implementations, at least one simulation input for each computational simulation includes multiple parameterized variables.

[0040] As used herein, a simulation result whose associated data is included in a synthetic dataset includes a calculated property at a location of a simulation model (e.g., a point, edge, surface, or volume of the model). The property may include, but is not limited to, stress, strain, displacement, temperature, heat flux, velocity, static pressure, dynamic pressure, current, voltage, and power. It should be understood that the above properties are provided by way of example only. The present disclosure contemplates that other properties may be output by a set of simulations. In some implementations, the calculated property represents the output of a measurement device. Sensors (e.g., strain sensors, temperature sensors) are exemplary measurement devices. In some implementations, at least one simulation result includes a single calculated property. In other implementations, at least one simulation result includes multiple calculated properties.

[0041] Referring now to FIG. 3 , a table illustrating an exemplary synthetic data set is shown. Simulation inputs include a force magnitude applied to surface A (condition 1) and a heat flux applied to surface B (condition 2). As discussed above, these simulation inputs are specified by a user to train a machine learning model. It should be understood that the number and / or type of simulation inputs in the table are provided by way of example only. For example, more than two or less than two simulation inputs may be specified by a user. Conditions 1 and 2 are parameterized variables, e.g., their values ​​may change from simulation to simulation. It should be understood that the number of simulations in the table are provided by way of example only. Simulation results include strain measured at point X (sensor 1), temperature measured at point Y (sensor 2), and temperature measured at point Z (sensor 3). As discussed above, these simulation results are specified by a user to train a machine learning model. It should be understood that the number and / or type of simulation results in the table are provided by way of example only. For example, more than three or less than three simulation results may be specified by a user. A simulation result is a calculated property at a location in a simulation model.

[0042] Referring again to FIG. 1 , in step 106, a machine learning model is trained on a dataset associated with the set of computational simulations. The present disclosure contemplates that the machine learning model may be implemented using one or more computing devices, such as the computing device of FIG. 2 . As described above, step 106 may be performed without user input or intervention. That is, the workflow between steps 104 and 106 may be automated. The machine learning model is trained on a dataset, such as the dataset assembled in step 104. During training, weights, biases, parameters, rewards (e.g., Q-values), etc. associated with the machine learning model are adjusted to minimize a cost function. Upon completion, the trained machine learning model is configured for inference mode, e.g., the trained machine learning model can make predictions based on new data. For example, the trained machine learning model may be configured to predict a state, characteristic, or behavior of a physical system (e.g., an outcome, such as a structural failure or overheating) based on at least one measurement obtained by a measurement device of the physical system. In some implementations, the machine learning model is a classifier model configured to classify model inputs into one of “n” categories, for example. In other implementations, the machine learning model is, for example, a regression model configured to estimate a value based on model inputs.

[0043] In some implementations, the machine learning model may optionally be a neural network. An artificial neural network (ANN) is a computing system including multiple interconnected neurons (e.g., also referred to as "nodes"). The present disclosure contemplates that the nodes may be implemented using a computing device (e.g., a processing unit and memory described herein). The nodes may optionally be arranged in multiple layers, such as an input layer, an output layer, and one or more hidden layers. Each node is connected to one or more other nodes in the ANN. For example, each layer is composed of multiple nodes, and each node is connected to all nodes in the previous layer. The nodes in a given layer are not interconnected with each other, i.e., the nodes in a given layer function independently of each other. As used herein, the nodes in the input layer receive data from outside the ANN, the nodes in the hidden layer modify the data between the input layer and the output layer, and the nodes in the output layer provide results. Each node is configured to receive an input, implement an n activation function (e.g., binary step, linear, sigmoid, tanH, or rectified linear unit (ReLU) function), and provide an output according to the activation function. Additionally, each node is associated with a respective weight. The ANN is trained on a dataset (e.g., a computational simulation dataset described herein) to minimize a cost function, which is a measure of the performance of the ANN. Training algorithms include, but are not limited to, backpropagation. The training algorithm adjusts the weights and / or biases of the nodes to minimize the cost function. It should be understood that any algorithm that minimizes the cost function can be used to train the ANN. It should be understood that a neural network is provided only as an exemplary machine learning model. The present disclosure contemplates that the machine learning model may be any supervised, semi-supervised, or unsupervised learning model. Optionally, the machine learning model is a deep learning model. Machine learning models are known in the art and therefore will not be described in further detail herein.

[0044] In some implementations, the machine learning model is trained using synthetic data (e.g., data associated with simulation inputs and / or results as described above). Optionally, in other implementations, the dataset further includes real data associated with a physical system. Optionally, such real data is measured by at least one measurement device of the physical system. In other words, a dataset including synthetic data can be supplemented with real data. The present disclosure contemplates that supplementing a dataset with real data (e.g., actual real-world measurements) can improve the performance of the machine learning model.

[0045] Prior to training, a target specification associated with at least one target and a feature specification associated with at least one feature for the machine learning model are received from a user, e.g., by a computing device that integrates computational simulation and machine learning model training. As described below, the at least one target and the at least one feature are part of the dataset assembled in step 104. For example, the user may specify which of the simulation inputs and / or simulation results in the dataset assembled in step 104 are the machine learning model target (or targets) and provide such information to the computing device. Additionally, the user may specify which of the simulation inputs and / or simulation results in the dataset assembled in step 104 are the machine learning model feature (or features) and provide such information to the computing device. The present disclosure contemplates that the target and feature specifications may optionally be provided to the computing device via a message, a web-based portal, a mobile application, or the like. Optionally, the target and feature specifications are provided to the computing device when the user requests a set of computational simulations related to a physical system, e.g., simultaneously with the user providing the simulation input and result specifications to the computing device. The at least one target and the at least one feature are part of a dataset associated with a set of computational simulations. In particular, each of the at least one target and the at least one feature is a simulation input or a simulation result in the dataset (e.g., see the table in FIG. 3). In other words, when a user requests a set of computational simulations related to a physical system, the user specifies each one of the simulation inputs and results in the dataset (e.g., the dataset assembled in step 104) as a target or a feature for purposes of training a machine learning model (e.g., see the table in FIG. 3).It should be understood that the at least one target is part of the dataset that the machine learning model is trained to predict in step 106. For example, the machine learning model may be trained to learn patterns and uncover relationships between at least one target and at least one feature in the dataset. After training, i.e., when operating in inference mode to analyze new data, the "target" is the output of the trained model, and the "feature" is the input to the trained model.

[0046] Referring again to FIG. 3 , a table illustrating an exemplary synthetic dataset is shown. Each of the simulation inputs (Condition 1, Condition 2) and simulation results (Sensor 1, Sensor 2, and Sensor 3) is specified as either a target or a feature for the machine learning model. In ML Model 1, the user specified Condition 2 as the target and Condition 1, Sensor 1, Sensor 2, and Sensor 3 as features. ML Model 1 is thus trained to learn patterns and uncover relationships to predict Condition 2, given values ​​(e.g., of a function) for Condition 1, Sensor 1, Sensor 2, and Sensor 3. In ML Model 2, the user specified Condition 1 as the target and Condition 2, Sensor 1, Sensor 2, and Sensor 3 as features. ML Model 2 is thus trained to learn patterns and uncover relationships to predict Condition 1, given values ​​(e.g., of a function) for Condition 2, Sensor 1, Sensor 2, and Sensor 3. As described herein, each of the at least one target and at least one feature is a simulation input or a simulation result from the dataset. It should be understood that the at least one target and the at least one feature are not limited to those in the table. When a user wants to predict a condition based on actual measurements using a trained machine learning model, it should be understood that the features are the output of the measurement device (e.g., calculated characteristics from a simulation of sensor 1, sensor 2, sensor 3, etc. in the table), since the actual measurements are input into the trained machine learning model. It should be understood that the number and / or types of targets, features, and / or ML models in FIG. 3 are provided by way of example only. For example, more or less than two machine learning models can be trained. Alternatively or additionally, more or less than four features for the machine learning model can be specified. Alternatively or additionally, one or more targets for the machine learning model can be specified (indeed, it should be understood that specifying multiple targets for a single machine learning model can be beneficial if they are highly correlated).

[0047] In some implementations, a single machine learning model is trained. In other implementations, multiple machine learning models are trained. Optionally, the machine learning models are trained in parallel. Alternatively or additionally, the performance of each of the trained machine learning models can be evaluated.

[0048] In some implementations, the set of computational simulations and the machine learning model training are performed until a stopping criterion is reached. Optionally, the stopping criterion is a predetermined performance level of the trained machine learning model. In some implementations, the performance level of the trained machine learning model is measured using a validation dataset, e.g., a dataset other than the training dataset. In other implementations, the performance level of the trained machine learning model is measured using a test dataset, e.g., a dataset independent of the training dataset. The present disclosure contemplates comparing the measured performance level with a threshold (i.e., a predetermined performance level).

[0049] The present disclosure contemplates measuring the performance of a machine learning model using techniques known in the art, including, but not limited to, mean absolute error, mean squared error, classification accuracy, logarithmic loss, and area under the curve. As described herein, a machine learning model may be a classifier model or a regression model. The performance of a classifier model may be measured using one or more metrics known to those skilled in the art, including, but not limited to, accuracy rate, precision, sensitivity / recall, and specificity metrics. For example, the performance of a classifier model may be evaluated by measuring the number of true positives (TP), true negatives (TN), false negatives (FN), and false positives (FP). In some implementations of the present disclosure, the accuracy rate may be measured as (TN + TP) / (TP + FP + TN + FN). Precision measures may also be used as stopping criteria in some implementations. A non-limiting example of a precision measure that may be used in some implementations is TP / (TP + FP). Similarly, the true positive rate (which may also be referred to as recall or sensitivity) may also be used as a stopping criterion. In some implementations, the true positive rate can be defined as TP / (TP+FN). Specificity can also be used as a stopping criterion in some implementations. An exemplary measure of specificity is TN / (TN+FP). The present disclosure also contemplates that the stopping criterion can be visualized or graphed and displayed to a user, for example, as a confusion matrix, ROC (Receiver Operating Characteristic), or PR (Precision-Recall) curve. It should be understood that the above exemplary metrics for classifier models are intended as non-limiting examples only, and that additional performance metrics will be apparent to one skilled in the art.

[0050] The performance of a regression model can be measured using one or more metrics known to those skilled in the art, including, but not limited to, mean squared error (MSE), root mean squared error (RSME), and mean absolute error (MAE). The present disclosure also contemplates that the stopping criteria can be visualized or graphed and displayed to a user, for example, as an error curve. It should be understood that the above exemplary metrics for regression models are intended as non-limiting examples only, and that additional performance metrics will be apparent to those skilled in the art.

[0051] Alternatively or additionally, the stopping criteria may be independent of the performance level of the trained machine learning algorithm. For example, the stopping criteria may be a predetermined number of simulations, a predetermined monetary cost, a predetermined number of computing core hours, or other predetermined computational cost. It should be understood that the above exemplary stopping criteria are intended as non-limiting examples only, and that additional stopping criteria will be apparent to those skilled in the art.

[0052] In some implementations, the trained machine learning model can be provided to a user, for example, the architecture, hyperparameter values, and / or parameter values ​​of the trained machine learning model can be sent to the user.

[0053] It should be understood that the logical operations described herein with respect to various aspects may be implemented (1) as a series of computer-implemented operations or program modules (i.e., software) executing on a computing device (e.g., the computing device described in FIG. 2 ), (2) as interconnected machine logic circuits or circuit modules (i.e., hardware) within a computing device, and / or (3) as a combination of software and hardware in a computing device. Thus, the logical operations discussed herein are not limited to any particular combination of hardware and software. The implementation is a matter of choice dependent on the performance and other requirements of the computing device. Accordingly, the logical operations described herein are referred to variously as operations, structural devices, operations, or modules. These operations, structural devices, operations, and modules may be implemented in software, firmware, special-purpose digital logic, and any combination thereof. It should also be understood that more or fewer operations may be performed than illustrated in the figures and described herein. These operations may also be performed in a different order than described herein.

[0054] Referring to FIG. 2, an exemplary computing device 200 is illustrated in which the methods described herein may be implemented. It should be understood that the exemplary computing device 200 is only one example of a suitable computing environment in which the methods described herein may be implemented. Optionally, the computing device 200 may be any well-known computing system, including, but not limited to, a personal computer, a server, a handheld or laptop device, a multiprocessor system, a microprocessor-based system, a network personal computer (PC), a minicomputer, a mainframe computer, an embedded system, and / or a distributed computing environment including any of the above systems or devices. A distributed computing environment enables remote computing devices connected to a communications network or other data transmission medium to perform various tasks. In a distributed computing environment, program modules, applications, and other data may be stored in local and / or remote computer storage media.

[0055] In its most basic configuration, computing device 200 typically includes at least one processing unit 206 and system memory 204. Depending on the exact configuration and type of computing device, system memory 204 may be volatile (such as random access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory), or some combination of the two. This most basic configuration is illustrated in FIG. 2 by dashed line 202. Processing unit 206 may be a standard programmable processor that performs arithmetic and logical operations necessary for the operation of computing device 200. Computing device 200 may also include a bus or other communication mechanism for communicating information between various components of computing device 200.

[0056] Computing device 200 may have additional features / functionality. For example, computing device 200 may include additional storage, such as removable storage 208 and non-removable storage 210, including, but not limited to, magnetic or optical disks or tape. Computing device 200 may also include network connections 216 that allow the device to communicate with other devices. Computing device 200 may also have input devices 214, such as a keyboard, mouse, touchscreen, etc. Output devices 212, such as a display, speakers, printer, etc., may also be included. Additional devices may be connected to the bus to facilitate communication of data between components of computing device 200. All of these devices are well known in the art and need not be discussed at length here.

[0057] The processing unit 206 may be configured to execute program code encoded on tangible computer-readable media. Tangible computer-readable media refers to any medium that can provide data that causes the computing device 200 (i.e., machine) to operate in a specific fashion. A variety of computer-readable media may be utilized to provide instructions to the processing unit 206 for execution. Exemplary tangible computer-readable media may include, but are not limited to, volatile, non-volatile, removable, and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. The system memory 204, removable storage 208, and non-removable storage 210 are all examples of tangible computer storage media. Exemplary tangible computer-readable recording media include, but are not limited to, integrated circuits (e.g., field programmable gate arrays or application specific ICs), hard disks, optical disks, magneto-optical disks, floppy disks, magnetic tape, holographic storage media, solid state devices, RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROMs, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices.

[0058] In an example implementation, processing unit 206 may execute program code stored in system memory 204. For example, a bus may carry data to system memory 204, from which processing unit 206 receives and executes instructions. Data received by system memory 204 may optionally be stored on removable storage 208 or non-removable storage 210 either before or after execution by processing unit 206.

[0059] It should be understood that the various techniques described herein may be implemented in connection with hardware or software, or, where appropriate, with any combination thereof. Accordingly, the methods and apparatus of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in a tangible medium, such as a floppy diskette, a CD-ROM, a hard drive, or any other machine-readable storage medium; when the program code is loaded and executed on a machine, such as a computing device, the machine becomes an apparatus for practicing the presently disclosed subject matter. When program code runs on a programmable computer, the computing device generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. One or more programs may implement or utilize the processes described in connection with the presently disclosed subject matter, for example, through the use of application programming interfaces (APIs), reusable controls, or the like. Such programs may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the programs can be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.

[0060] 4, a block diagram illustrating an example environment 400 for automatically training machine learning models is shown. As described herein, one or more computational simulations 410 are executed within the environment 400. The computational simulations 410 simulate one or more types of physics (e.g., mechanical, electromechanical, electromagnetic, and thermal physics) associated with and acting on a physical system.

[0061] Simulation input 402 and simulation results 404 are shown in FIG. 4. Simulation inputs include, but are not limited to, the geometry, materials, initial conditions, loading conditions, and boundary conditions of a physical system. Exemplary simulation inputs are also shown in FIG. 3. Simulation results include properties calculated at a location in the simulation model (e.g., a point, edge, surface, or volume of the model). Properties can include, but are not limited to, stress, strain, displacement, temperature, heat flux, velocity, static pressure, dynamic pressure, current, voltage, and power. Exemplary simulation results are also shown in FIG. 3. A computational simulation 410 is performed by specifying simulation inputs 402 and calculating simulation results 404.

[0062] Additionally, as described herein, one or more machine learning models 420 are trained in the environment 400. The machine learning models 420 are trained using a dataset 430 associated with one or more computational simulations 410 (e.g., synthetic data). For example, the machine learning models may be any supervised, semi-supervised, or unsupervised learning model, optionally a supervised learning model such as a deep learning model. After training on the synthetic data, the machine learning models are configured to predict states, properties, or behaviors of a physical system based on actual measurements obtained by the physical system.

[0063] As described herein, the input specification 403 specifies or directs which data from the simulation inputs 402 are included in the dataset 430 used to train the machine learning model 420. Additionally, the result specification 405 specifies which data from the simulation results 404 are included in the dataset 430 used to train the machine learning model 420. The directed data forms the dataset 430. It should be understood that the directed simulation inputs and directed simulation results may, optionally, be a subset (e.g., not all) of the simulation inputs and results from one or more computational simulations 410. Additionally, it should be understood that the dataset may include only directed simulation results in some implementations, or both directed simulation inputs, and results in other implementations.

[0064] As described herein, feature and target specifications may be provided by a user. The feature and target specifications specify or indicate which data included in the dataset 430 are the machine learning model features 406 and which data included in the dataset 430 are the machine learning model targets 408. Exemplary feature and target specifications for two machine learning models are shown in FIG. 3. [Example]

[0065] The following examples are presented so as to provide those of ordinary skill in the art with a complete disclosure and description of how the compounds, compositions, articles, devices, and / or methods claimed herein may be made and evaluated, and are intended to be purely illustrative and not intended to limit the disclosure. Efforts have been made to ensure accuracy with respect to numbers (e.g., amounts, temperatures, etc.), but some errors and deviations should be accounted for.

[0066] Example 1

[0067] An exemplary implementation of the present disclosure is described below. In this example, a device (e.g., a physical system) includes multiple thermocouples (e.g., measurement devices) distributed around the exterior of the device. Each thermocouple measures a respective temperature at its exterior location on the device at a sampling rate, e.g., 10 measurements per second. The device also includes internal components necessary to maintain below a certain temperature. However, due to the device's design, it is not possible to measure the temperature inside the device. Thus, for example, it is not possible to use thermocouples to monitor the temperature of the internal components and avoid overheating. Instead, a machine learning model can be trained to predict the temperature of the internal components based on the external thermocouple measurements. Once trained, the machine learning model can be used in inference mode as part of a control system that adjusts the operation of the device when the predicted temperature of the internal components exceeds an overheating threshold. As described herein, the machine learning model can be trained using synthetic data obtained through a set of computational simulations.

[0068] In this example, an automated method may be used to run a set of computational simulations (e.g., step 102 of FIG. 1 , computational simulation 410 of FIG. 4 ). In this example, the set of computational simulations is referred to as a “study,” and running the set of computational simulations that make up a study may optionally be performed as a batch (e.g., one after the other or simultaneously).

[0069] The set of computational simulations is based on simulation inputs specified by a user (e.g., simulation inputs 402 of FIG. 4). In this example, the simulation inputs include a computer-aided design (CAD) model of the device, information about material properties of the device's components, and information about the nature and location of applied conditions, such as initial conditions, loading conditions, and / or boundary conditions. The set of computational simulations can be run based on the simulation inputs to calculate simulation results (e.g., simulation results 404 of FIG. 4).

[0070] In this example, the applied conditions may represent physical effects applied to the CAD model, including information about the size, shape, and placement of the device's components. Each applied condition may be defined by a location (or location) where the applied condition acts and one or more characteristics of the applied condition. For example, an applied condition may include an ambient temperature and a power output of a device's component. An ambient temperature applied condition may be defined to have a location outside the device (i.e., "ambient" relative to the device), and the characteristic of the ambient temperature may be a temperature value. An applied condition for a power output of a device's component may have a location associated with the device's component, and the power output may have a characteristic representing an energy conversion rate (i.e., its power output) related to heat dissipation.

[0071] Additionally, in this example, characteristics of the applied conditions can be parameterized, for example, the power output of the device's components can be parameterized with 50 specified values, and the ambient temperature can also be parameterized with 20 specified values.

[0072] As discussed above, a set of computational simulations can calculate simulation results (e.g., simulation results 404 in FIG. 4 ). In this example, the simulation results can include the temperature of each thermocouple location and the temperature of the device's internal components. Thus, in this example, a simulation can be performed for each combination of parameterized variable values, and a simulation result can be calculated for each combination of parameterized variable values. Thus, in this example, a computational simulation can be performed for each of 1000 combinations of parameterized values ​​representing the component output and parameterized values ​​representing the ambient temperature (i.e., 50×20=1000). For each computational simulation, the temperature of each thermocouple located on the device and the temperature of the device's internal components can be simulated. The thermocouple temperatures correspond to the “calculated properties” described above with reference to FIG. 1 . Thus, the 1000 computational simulations performed result in the calculation of the temperature of each thermocouple and the temperature of the device's internal components for each combination of parameterized values. The temperature of each thermocouple location and the temperature of the internal components can be described as a simulation result of the simulation.

[0073] In this example, the simulation results of the 1000 simulations can be assembled (e.g., step 104 of FIG. 1) into a dataset (e.g., dataset 430 of FIG. 4). As discussed herein, the dataset may include only a subset of the simulation inputs and / or simulation results. For example, a user can specify which data from the simulation inputs and / or simulation results (e.g., input specification 403 and result specification 405 of FIG. 4) to include in the dataset, and the dataset is then used to train a machine learning model. In this example, assembling the dataset can include combining the simulation results of each simulation into the dataset. Thus, the dataset can include thermocouple location temperatures and internal component temperatures from each of the 1000 simulations. In this example, the dataset includes only the simulation results and not the simulation inputs. In this example, the dataset can then be used to train a machine learning model (e.g., step 106 of FIG. 1; machine learning model 420 of FIG. 4). In this example, the simulation results representing the temperature at each thermocouple location may be features of the machine learning model (e.g., model features 406 in FIG. 4 ), and the simulation results representing the internal component temperatures may be targets of the machine learning model (e.g., model targets 408 in FIG. 4 ).

[0074] This embodiment contemplates that the performance of a trained machine learning model may be determined during training. As described herein, performance may be measured using known metrics such as accuracy, precision, sensitivity / recall, and specificity for classifier models or error for regression models. The performance of a machine learning model may be iteratively improved. For example, as described below, steps may be taken to improve the model's performance in response to user input / commands (e.g., by tuning the model) or in response to determining that the model's performance is below a predetermined threshold. Once a machine learning model has been trained using a dataset, the machine learning model is considered a trained machine learning model that may be used in inference mode. The trained model may be stored in a file (e.g., a computer file) and / or transmitted to a user.

[0075] To improve the performance of the model, any one or more of steps 102, 104, and 106 illustrated in FIG. 1 can be repeated. To repeat step 102 of FIG. 1, additional combinations of parameterized variable values ​​can be selected, and additional simulations corresponding to the additional combinations of parameterized variable values ​​can be performed. The additional simulations can be based on the same CAD model as the initial simulation. Similarly, the additional simulations calculate the temperatures of each of the thermocouples and the temperatures of the internal components of the device for each additional combination of parameterized values.

[0076] To repeat step 104 of FIG. 1, additional simulation results (“additional simulation results”) can be added to the dataset to form a new or expanded dataset assembled from the existing dataset and the new simulation results (“augmented dataset”).

[0077] To repeat step 106 of Figure 1, the machine learning model can be retrained using the increased data set. The performance of the retrained machine learning model can be re-evaluated. It should be understood that alternatively or additionally, the machine learning model can be optionally tuned or refined in step 106, for example, by modifying hyperparameters of the model.

[0078] These steps of selecting new parameter values, running new simulations based on the new parameter values, retraining the machine learning model using a larger dataset, measuring the performance of the retrained model, and determining whether the model's performance is sufficient can be performed multiple times. For example, these steps can be performed repeatedly until a performance value reaches a threshold. Alternatively, or additionally, these steps can be repeated until the cost of the step (e.g., monetary, computing, such as number of simulations or core hours) exceeds a cost threshold.

[0079] Example 2

[0080] Another exemplary implementation of the present disclosure is described below. In this example, a device (e.g., a physical system) is an electronic device including a panel (e.g., a measurement device) with multiple strain sensors attached to the underside of and distributed around the panel. Each strain sensor measures a respective strain at its location due to an object (e.g., an instrument or a user's finger) contacting the user-facing surface of the panel. A machine learning model can be trained to predict the location and pressure of the object based on the strain sensor measurements. Once trained, the machine learning model can be used in inference mode, for example, as part of a control system, to accept user input to the device and react accordingly. As described herein, the machine learning model can be trained using synthetic data obtained through a set of computational simulations.

[0081] Similar to the above example, an automated method can be used to run a set of computational simulations (e.g., step 102 of FIG. 1 ; computational simulation 410 of FIG. 4 ). A set of computational simulations is referred to as a “study,” and running the set of computational simulations that make up a study can optionally be performed as a batch (e.g., one after the other or simultaneously). The set of computational simulations is based on simulation input (e.g., simulation input 402 of FIG. 4 ) specified by a user. In this example, the simulation input includes a computer-aided design (CAD) model of the device, information about material properties of the device's components, and information about the nature and location of applied conditions, such as initial conditions, loading conditions, and / or boundary conditions. The set of computational simulations can be run based on the simulation input to calculate simulation results (e.g., simulation results 404 of FIG. 4 ).

[0082] Similar to the above example, the applied conditions can represent physical effects applied to the CAD model, including information about the size, shape, and placement of device components. Each applied condition can be defined by a location where the applied condition acts and one or more characteristics of the applied condition. For example, an applied condition can include a first touch location on a panel of the device, a second touch location on the panel of the device, a first touch pressure, and a second touch pressure. The first touch location can be a parameterized variable with 100 possible values, and the second touch location can be a parameterized variable with 100 possible values. The first touch pressure can be a parameterized variable with a minimum and a maximum value, and the second touch pressure can be a parameterized variable with a minimum and a maximum value.

[0083] As discussed above, a set of computational simulations can calculate simulation results (e.g., simulation results 404 in FIG. 4 ). In this example, the simulation results can include strain at each strain sensor of the device. Thus, in this example, simulations can be performed for combinations of application conditions (e.g., first and second touch locations and first and second touch pressures), and simulation results can be calculated for each combination. For example, parameterized variable values ​​can be selected for combinations of application conditions to generate simulation results. Optionally, in this example, computational simulations can be performed for each of N random combinations of application conditions (e.g., N=50,000). It should be understood that N=50,000 is provided by way of example only. For each computational simulation, strain at each strain sensor of the device can be simulated. The strain at each strain sensor corresponds to the “calculated characteristic” described above with reference to FIG. 1 .

[0084] Similar to the above example, the simulation results of the 50,000 simulations can be assembled (e.g., step 104 of FIG. 1 ) into a dataset (e.g., dataset 430 of FIG. 4 ). As discussed herein, the dataset may include only a subset of the simulation inputs and / or simulation results. For example, a user can specify which data from the simulation inputs and / or simulation results (e.g., input specification 403 and result specification 405 of FIG. 4 ) to include in the dataset, and the dataset is then used to train a machine learning model. In this example, assembling the dataset can include combining both the simulation inputs and the results of each simulation into the dataset. Thus, the dataset can include four application conditions (e.g., parameterized variable values ​​for randomly selected combinations of touch location and touch pressure) and calculated strain values ​​from each of the 50,000 simulations. Optionally, the dataset can be analyzed and cleaned, for example, to detect and remove (or correct to realistic values) calculated strain values ​​that exceed the operating range of the strain sensor.

[0085] In this example, the data set can then be used to train a machine learning model (e.g., step 106 of FIG. 1 , machine learning model 420 of FIG. 4 ). In this example, the simulation results representing the stain in each of the strain sensors can be features of the machine learning model (e.g., model features 406 of FIG. 4 ), and the simulation inputs representing the four application conditions can be targets of the machine learning model (e.g., model targets 408 of FIG. 4 ). This differs from Example 1 above because in Example 2, the targets of the machine learning model (i.e., what the model predicts) are the simulation inputs to the simulation (i.e., the applied conditions).

[0086] Similar to the above embodiment, this embodiment contemplates that the performance of a trained machine learning model may be determined during training using metrics such as accuracy, precision, sensitivity / recall, and specificity of a classifier model or error of a regression model. Also, once a machine learning model has been trained using a dataset, the machine learning model is considered a trained machine learning model that may be used in inference mode. The trained learning model may be stored in a file (e.g., a computer file) and / or transmitted to a user.

[0087] Similar to the above examples, to improve the performance of the model, any one or more of steps 102, 104, and 106 illustrated in FIG. 1 can be repeated in the same manner as described in Example 1. Additionally, these steps of selecting new parameter values, running new simulations based on the new parameter values, retraining the machine learning model using a larger dataset, measuring the performance of the retrained model, and determining whether the model's performance is sufficient can be performed more than once. For example, these steps can be performed repeatedly until a performance value reaches a threshold. Alternatively or additionally, these steps can be repeated until the cost of the step (e.g., monetary, computing, such as number of simulations or core hours) exceeds a cost threshold.

[0088] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the particular features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. It should be noted that the present specification discloses the following aspects. [Aspect 1] 1. A method for automatically training a machine learning model, comprising: running a set of computational simulations; assembling a dataset associated with the set of computational simulations, the dataset including data associated with at least one simulation result for at least one computational simulation in the set of computational simulations; training a machine learning model using the dataset, wherein at least one feature and at least one target for the machine learning model are part of the dataset. [Aspect 2] 2. The method of embodiment 1, wherein the data set further includes data associated with at least one simulation input for the at least one computational simulation in the set of computational simulations. [Aspect 3] 3. The method of embodiment 2, further comprising receiving, from a user, an input specification associated with the at least one simulation input. [Aspect 4] 4. The method of any one of aspects 2 to 3, further comprising receiving, from a user, a result specification associated with the at least one simulation result. [Aspect 5] 3. The method of embodiment 2, wherein the at least one simulation input for the at least one computational simulation in the set of computational simulations comprises a parameterized variable. [Aspect 6] 6. The method of embodiment 5, wherein the parameterized variables are parameterized conditions at a location on a simulation model. [Aspect 7] 6. The method of embodiment 5, wherein the parameterized variables are parameterized characteristics of a simulation model. [Aspect 8] 6. The method of any one of aspects 1 to 5, wherein the at least one simulation result for the at least one computational simulation in the set of computational simulations includes a calculated property at a location on a simulation model. [Aspect 9] 9. The method of embodiment 8, wherein the calculated characteristic represents an output of a measurement device. [Aspect 10] 10. The method of any one of aspects 1-9, further comprising receiving, from a user, a target specification relating to the at least one target for the machine learning model. [Aspect 11] 11. The method of embodiment 10, wherein the at least one target is a simulation input. [Aspect 12] 11. The method of embodiment 10, wherein the at least one target is a simulation result. [Aspect 13] 13. The method of any one of aspects 1-12, further comprising receiving, from a user, a feature specification relating to the at least one feature for the machine learning model. [Aspect 14] 14. The method of embodiment 13, wherein the at least one feature is a simulation input. [Aspect 15] 14. The method of claim 13, wherein the at least one feature is a simulation result. [Aspect 16] 16. The method of any one of aspects 1-15, wherein training the machine learning model using the dataset comprises training a plurality of machine learning models using the dataset. [Aspect 17] 17. The method of embodiment 16, wherein the machine learning models are trained in parallel. [Aspect 18] 18. The method of claim 16 or 17, further comprising evaluating a performance of each of the trained machine learning models. [Aspect 19]

[0033] Aspect 19. The method of any one of aspects 1-18, wherein the trained machine learning model is configured to predict a state, property, or behavior of a physical system based on at least one measurement value obtained by a measurement device of the physical system. [Aspect 20] 20. The method of any one of aspects 1 to 19, wherein the machine learning model is a supervised learning model, a semi-supervised learning model, or an unsupervised learning model. [Aspect 21] 20. The method of any one of aspects 1 to 19, wherein the machine learning model is a deep learning model. [Aspect 22] 20. The method of any one of aspects 1 to 19, wherein the machine learning model is an artificial neural network. [Aspect 23] 23. The method of any one of aspects 1-22, wherein the set of computational simulations and training of the machine learning model are performed until a stopping criterion is reached. [Aspect 24] 24. The method of claim 23, wherein the stopping criterion is a predetermined performance level of the trained machine learning model, a predetermined number of simulations, a predetermined number of computing core hours, or a predetermined cost. [Aspect 25] 25. The method of any one of aspects 1-24, further comprising providing the trained machine learning model to a user. [Aspect 26] 26. The method of claim 25, wherein providing the trained machine learning model to the user includes sending the user an architecture, hyperparameter values, and / or parameter values ​​of the trained machine learning model. [Aspect 27] 27. The method of any one of aspects 1-26, wherein the dataset further comprises real data associated with a physical system. [Aspect 28] 28. The method of embodiment 27, wherein the real data is measured by at least one measurement device of the physical system. [Aspect 29] 1. A system comprising: a device comprising a network of measurement devices; a machine learning module, the machine learning module being trained using a dataset, the dataset including data associated with at least one simulation result for at least one computational simulation in a set of computational simulations, and at least one feature and at least one target of the machine learning module being part of the dataset; 1. A controller comprising a processor and a memory, the memory having computer-executable instructions stored therein, the instructions, when executed by the processor, causing the processor to: receiving respective measurements from said network of measuring devices; inputting each of said measurements into said machine learning module; a controller that receives an output predicted by the machine learning module, the output being a state, characteristic, or behavior of the device. [Aspect 30] 30. The system of embodiment 29, wherein the data set further includes data associated with at least one simulation input for the at least one computational simulation in the set of computational simulations. [Aspect 31] 31. The system of aspect 30, wherein the at least one simulation input for the at least one computational simulation in the set of computational simulations includes a parameterized variable. [Aspect 32] 32. The system of claim 31, wherein the parameterized variable is a parameterized state at a location on a simulation model. [Aspect 33] 32. The system of embodiment 31, wherein the parameterized variables are parameterized characteristics of a simulation model. [Aspect 34] 34. The system of any one of aspects 29 to 33, wherein the at least one simulation result for the at least one computational simulation in the set of computational simulations includes a calculated property at a location on a simulation model. [Aspect 35] 35. The system of embodiment 34, wherein the calculated characteristic represents an output of at least one measurement device of the network of measurement devices. [Aspect 36] 36. The system of any one of aspects 29 to 35, wherein the trained machine learning model is configured to predict a state, characteristic, or behavior of the device based on at least one measurement value acquired by the network of measurement devices. [Aspect 37] 37. The system of any one of aspects 29 to 36, wherein the machine learning model is a supervised learning model, a semi-supervised learning model, or an unsupervised learning model. [Aspect 38] 37. The system of any one of aspects 29 to 36, wherein the machine learning model is a deep learning model. [Aspect 39] 37. The system of any one of aspects 29 to 36, wherein the machine learning model is an artificial neural network. [Aspect 40]

[0039] Aspects 40. The system of any one of aspects 29-39, wherein the dataset further includes actual data associated with the device. [Aspect 41] 41. The system of embodiment 40, wherein the actual data is measured by the network of measurement devices.

Claims

1. 1. A method for automatically training a machine learning model, comprising: running a set of computational simulations; assembling a dataset associated with the set of computational simulations, the dataset including data associated with at least one simulation result for at least one computational simulation in the set of computational simulations; training a machine learning model using the dataset, wherein at least one feature and at least one target for the machine learning model are part of the dataset; receiving, from a user, a target specification relating to the at least one target for the machine learning model.

2. The method of claim 1 , wherein the data set further comprises data associated with at least one simulation input for the at least one computational simulation in the set of computational simulations.

3. The method of claim 2 , further comprising receiving, from a user, an input specification associated with the at least one simulation input.

4. The method of claim 2 or 3, further comprising receiving, from a user, a result specification associated with the at least one simulation result.

5. The method of claim 2 , wherein the at least one simulation input for the at least one computational simulation in the set of computational simulations comprises a parameterized variable.

6. The method of claim 5 , wherein the parameterized variables are parameterized conditions at locations on a simulation model.

7. The method of claim 5 , wherein the parameterized variables are parameterized attributes of a simulation model.

8. The method of any one of claims 1 to 5, wherein the at least one simulation result for the at least one computational simulation in the set of computational simulations comprises a calculated property at a location on a simulation model.

9. The method of claim 8 , wherein the calculated characteristic represents an output of a measurement device.

10. The method of any one of claims 1 to 9, wherein said at least one target is a simulation input.

11. The method according to any one of claims 1 to 9, wherein the at least one target is a simulation result.

12. The method of any one of claims 1 to 11, further comprising receiving, from a user, a feature specification relating to the at least one feature for the machine learning model.

13. The method of claim 12 , wherein the at least one feature is a simulation input.

14. The method of claim 12 , wherein the at least one feature is a simulation result.

15. 15. The method of any one of claims 1 to 14, wherein training the machine learning model with the dataset comprises training a plurality of machine learning models with the dataset.

16. The method of claim 15 , wherein the machine learning models are trained in parallel.

17. The method of claim 15 or 16, further comprising evaluating the performance of each of the trained machine learning models.

18. 18. The method of any one of claims 1 to 17, wherein the trained machine learning model is configured to predict a state, property, or behavior of a physical system based on at least one measurement obtained by a measurement device of the physical system.

19. 19. The method of any one of claims 1 to 18, wherein the machine learning model is a supervised learning model, a semi-supervised learning model, or an unsupervised learning model.

20. 19. The method of any one of claims 1 to 18, wherein the machine learning model is a deep learning model.

21. 19. The method of any one of claims 1 to 18, wherein the machine learning model is an artificial neural network.

22. The method of any one of claims 1 to 21, wherein the set of computational simulations and the training of the machine learning model are performed until a stopping criterion is reached.

23. 23. The method of claim 22, wherein the stopping criterion is a predetermined performance level of the trained machine learning model, a predetermined number of simulations, a predetermined number of computing core hours, or a predetermined cost.

24. The method of any one of claims 1 to 23, further comprising providing the trained machine learning model to a user.

25. 25. The method of claim 24, wherein providing the trained machine learning model to a user comprises transmitting to the user an architecture, hyperparameter values, and / or parameter values ​​of the trained machine learning model.

26. The method of any one of claims 1 to 25, wherein the dataset further comprises real data associated with a physical system.

27. 27. The method of claim 26, wherein the real data is measured by at least one measurement device of the physical system.

28. 1. A system comprising: a device comprising a network of measurement devices; a machine learning module, the machine learning module including a machine learning model trained with a dataset, the dataset including data associated with at least one simulation result for at least one computational simulation in a set of computational simulations, at least one feature and at least one target of the machine learning model being part of the dataset, and training the machine learning model including receiving, from a user, a target specification associated with the at least one target for the machine learning model; 1. A controller comprising a processor and a memory, the memory having computer-executable instructions stored therein, the instructions, when executed by the processor, causing the processor to: receiving respective measurements from said network of measuring devices; inputting each of said measurements into said machine learning module; a controller that receives an output predicted by the machine learning module, the output being a state, characteristic, or behavior of the device.

29. 30. The system of claim 28, wherein the data set further comprises data associated with at least one simulation input for the at least one computational simulation in the set of computational simulations.

30. 30. The system of claim 29, wherein the at least one simulation input for the at least one computational simulation in the set of computational simulations comprises a parameterized variable.

31. 31. The system of claim 30, wherein the parameterized variables are parameterized states at locations on a simulation model.

32. 31. The system of claim 30, wherein the parameterized variables are parameterized attributes of a simulation model.

33. 33. The system of claim 28, wherein the at least one simulation result for the at least one computational simulation in the set of computational simulations includes a calculated property at a location on a simulation model.

34. 34. The system of claim 33, wherein the calculated characteristic represents an output of at least one measurement device of the network of measurement devices.

35. 35. The system of any one of claims 28 to 34, wherein the trained machine learning model is configured to predict a state, characteristic, or behavior of the device based on at least one measurement obtained by the network of measurement devices.

36. 36. The system of any one of claims 28 to 35, wherein the machine learning model is a supervised learning model, a semi-supervised learning model, or an unsupervised learning model.

37. The system of any one of claims 28 to 35, wherein the machine learning model is a deep learning model.

38. The system of any one of claims 28 to 35, wherein the machine learning model is an artificial neural network.

39. The system of any one of claims 28 to 38, wherein the dataset further comprises actual data associated with the device.

40. 40. The system of claim 39, wherein the actual data is measured by the network of measuring devices.

Images