Apparatus and method for training reinforcement learning system

Abstract

A computer-implemented method of training an agent for generating a diverse data set of ordinary differential equations, where the agent comprises a neural network, where the agent selects an action from an action space based on an output of the neural network to sequentially generate a set of ordinary differential equations, where the agent performs the selected action, and where the selected action is based on an output of the neural network to sequentially generate the set of ordinary differential equations. The common differential equations are continuously constructed by concatenating mathematical operators and / or variables to form equations, and wherein after a complete set of common differential equations is generated, the agent receives a reward based on the selected action.

Description

Technical Field

[0001] The present invention relates to methods, corresponding systems, computer programs and machine-readable storage media for training a reinforcement learning system for generating a diverse data set of ordinary differential equations. Background Art

[0002] Large datasets of physical multidimensional signals obeying ordinary differential equations (ODEs) are particularly valuable in training machine learning systems for reliably recovering the symbolic form of the dynamical laws that govern or underlie a series of measurements of one or more physical quantities of a computer-controlled system and / or its environment. For example, the symbolic form of the dynamical laws allows further propagation of the inferred dynamics and meaningful modifications for prediction under the invention. Large datasets of this nature are difficult to obtain, and generating them is also a challenge due to the difficulty in determining a priori which ODEs (in combination with initial conditions) produce valid solutions. Prior art

[0003] In https: / / arxiv.org / abs / 1912.01412 and \url{https: / / openreview.net / pdf?id=JDuEddUsSb}, a scheme for solving the problem of large-scale data generation in the context of differential equations is introduced. The scheme consists of a tree-based scheme from which ODEs are derived, combined with a large set of hand-crafted explicit rules. Formulating the rules mentioned above requires a lot of expertise and effort. In addition, it is challenging to ensure balanced and diverse exploration in this setting. The datasets obtained by the tree-based scheme ultimately contain a limited variety of equations. Summary of the invention

[0004] The present invention solves the problem of generating large amounts of data for training large language models to focus on physical multidimensional signals that obey ODEs. Instead of extracting equation components from a distribution and applying predefined rules to streamline the results, a reinforcement learning (RL) agent is trained to generate a set of ODEs. Thus, after the training process of the agent is complete, obtaining new equations boils down to running its policy. Compared to previous schemes, this scheme produces larger and more diverse data sets of ODEs with valid solutions.

[0005] According to a first aspect, the present invention relates to a computer-implemented method for training an agent for generating a diverse data set of ordinary differential equations. The agent may be given by a machine learning system. A neural network may be composed of the agent. The neural network may be given by a fully connected neural network. In particular, the fully connected neural network may be given by a multi-layer perceptron. Preferably, the fully connected neural network may include several hidden layers. For example, the fully connected neural network may include two hidden layers each having 64 units and a hyperbolic tangent nonlinearity. The agent selects an action from an action space based on the output of the neural network, wherein the action space may include mathematical operators, constants, variables and brackets. Subsequently, the agent performs the selected action and receives a reward for it based on the selected action. The method comprises a step of initializing the agent, wherein the initialization comprises setting parameters of the neural network. Preferably, the parameters may be initialized by a random distribution. Alternatively, some or all parameters of the neural network may be received from a database or a data storage. In a further step, environmental parameters are received by the agent, wherein the environmental parameters include the number of variables N1 and N1 numbers N 2,i , i = 1, ... N1, where each number N 2,i Determine the number of operators. The environment parameters N1 and N 2,i , i=1, ... N1, may be obtained from a random distribution. In a further method step, a full set of N1 ODEs in symbolic form is sequentially generated. To this end, actions from the action space comprising mathematical operators, variables and brackets are sequentially selected by the agent. The selected actions are concatenated to form an ODE in symbolic form. The number N 2,i The number of mathematical operators that can appear in the i-th ODE in the full set of ODEs is thereby determined. In a further step, the generated full set of ODEs is passed to a solver for verifying the solvability of the full set. When attempting to solve the set of ODEs handed over to the solver, the solver can extract initial conditions and constants from a random distribution. In a further step, the agent receives a reward based on the verification result of the solver. Thus, a positive reward is assigned for a set of ODEs with valid solutions, and a negative reward is assigned for a set of ODEs without valid solutions. The previous steps are repeated for multiple episodes. Subsequently, the current values ​​of the parameters of the agent's network are adjusted by a policy optimization algorithm using the received rewards collected in the multiple episodes.

[0006] The value for the reward may be a positive or negative value, e.g., a positive or negative real number. Preferably, the reward is a binary reward, where a positive real number may be assigned to a set of solvable ODEs, and a corresponding negative real number (i.e., a negative real number whose absolute value is equal to the absolute value of the positive real number) may be assigned to a set of ODEs that are found to be "unsolvable" by the solver.

[0007] Preferably, the ODE set may characterize an electrical or chemical energy storage, in particular a fuel cell or a battery. More preferably, the ODE set may characterize the performance, aging, charging and discharging of an electrical or chemical energy storage, in particular a fuel cell or a battery. In a further embodiment, the ODE set may characterize the performance or aging of a manufacturing machine.

[0008] The proposed method enables the agent to autonomously learn rules for "legal" (i.e., valid) ODE generation based on the rewards obtained, thereby making the task of formulating rules for ODE generation obsolete. In addition, due to the intrinsic exploration employed by the reinforcement learning agent, effective exploration of the state space is more easily encouraged in the proposed setting, resulting in a considerably richer and more diverse set of ODEs than, for example, tree-based attempts.

[0009] As supported by the arguments mentioned above, the proposed method solves the problem of obtaining large-scale datasets with many kinds of valid ODEs.

[0010] Preferably, the action space includes the following set of mathematical entities: cos, sin, exp, C, x i , (,), +, -, *, / , ** and END (end), where the index i is assigned the value i=1, ..., N1. In the context of the present disclosure, mathematical entities may also be referred to as / may be understood as mathematical expressions or symbols. The entities cos, sin and exp symbolically represent the mathematical operators cosine, sine and exponential functions. Further, +, -, *, / , ** represent mathematical operators for addition, subtraction, multiplication, division and exponentiation. The entities "(" and ")" represent brackets. C represents a constant value and x represents a constant value. iRepresents equation variables. Depending on the environment parameter N1 received at the beginning of the current segment, there may be more or less such variables. Preferably, after the environment parameter has been received by the agent, the number of variables in the action space in the segment is adjusted to N1. For example, if the value of N1 is 3 in the segment, the number of variables is adjusted to 3 accordingly, and the action space in this case includes three variables, such as x, y and z (or equivalently, x1, x2, x3). END is similar to the sentence end token in natural language processing (NLP), that is, when the action END is selected, this indicates that the construction of the ODE is complete. In other words, when the action END is selected, the generation of the current ODE is completed. Accordingly, then, the complete ODE is given by the ordered concatenation of the actions previously selected in the process of the construction of the ODE. In the case where the set of ODEs is not yet complete, after selecting the action END, the agent continues to select actions sequentially from the action space in order to construct the next ODE in the set of N1 ODEs.

[0011] Preferably, the agent randomly selects the action from the action space by drawing from an action distribution, wherein the parameters of the action distribution are determined by the neural network. In particular, the parameters of the action distribution may include the mean of a Gaussian distribution with a variable standard deviation.

[0012] Preferably, a state space representing progress in the sequential generation of ODEs is modeled, wherein the first state in the state space is defined by an empty set of ordinary differential equations. The current state may then contain all preceding actions previously selected by the agent and concatenated in the order of selection. In the current state, the agent may select an action from the action space, and the state immediately following the current state may be obtained by concatenating the current state with the action selected by the agent. For example, the current state may be given by "sin(exp(", and the agent may select the action "sin" from the action space. In this example, the next state following the current state is then obtained as the concatenation of "sin(exp(" with "sin", i.e., "sin(exp(sin". If the value of N2 in a specific fragment is, for example, 3, the complete ODE generated in the above-mentioned manner may be given by "sin(exp(sin(x)))".

[0013] An intuitive way to understand the operation of the agent is through an NLP analogy, where a complete equation corresponds to a sentence. Similar to how a sentence is generated one word at a time, the agent generates one symbol at a time.

[0014] It should be noted that throughout the present invention, depending on N i dependent variable x iand an independent variable t, the set of N1 ODEs generated by the agent can be understood as a function f i An ordered set of N1 symbolic expressions, i = 1, ..., N1, where the i-th function f i Contains N included in the action space 2,i The independent variable t can be identified by a time variable. That is, in each segment, the agent can generate N1 functions f i Then, the generated set of ordered functions can be substituted into the set of ordinary differential equations to be solved by the solver by identifying the i-th function in the generated set using the time derivative of the i-th variable included in the action space.

[0015] Preferably, the solver tries multiple initial conditions for a given ODE set before determining it as an ODE set without a valid solution. Specifically, the solver can determine initial conditions and constants from a random distribution, and can attempt to solve the ODE set ignored by the agent. If the solver finds a valid solution with the determined initial conditions and constants, a positive reward can be assigned to the corresponding ODE set. However, in the case where the solver may not determine a valid solution to the ODE set, the solver can retry solving the ODE set with different initial conditions and constants for a predefined number of times. This process can be done so as not to cause the agent to move away from the ODE set simply because a certain initial condition or constant for the ODE set results in an invalid solution. Preferably, the process of trying to find a valid solution to the ODE set can be repeated a predefined number of times, for example up to 25 times, wherein each time, the solver can determine new values ​​for the initial conditions and constants from the corresponding random distribution. Only after the predefined number of unsuccessful attempts to solve the set of ODEs in this manner may the set of ODEs be declared "unsolvable", ie, declared as an ODE with no valid solutions.

[0016] Preferably, the solver may be RK45. However, the component may be any off-the-shelf differential equation solver.

[0017] Preferably, the policy optimization algorithm is an actor-critic policy optimization algorithm. For example, the actor-critic policy optimization algorithm can be given by a proximal policy optimization (PPO) algorithm.

[0018] Preferably, a fixed size buffer is defined for each ODE in the set of ODEs to prevent mode collapse scenarios. Mode collapse scenarios may include equations consisting of many nested and / or meaningless brackets and / or mutually cancelling operations. During the training process, the agent learns to plan its actions so that the buffer is filled with meaningful operators and operands.

[0019] The state space of all possible combinations of operators and operands is huge, making it difficult to explore efficiently. Returning to the NLP analogy: like words in a sentence, symbols must obey certain rules for equations to have mathematical meaning. For example, the cos operator cannot be followed by a + operator. Therefore, preferably, constraints can be enforced on the action selection to ensure that the generated ODE follows mathematical rules. In particular, the constraints mentioned above can be enforced by using PPO with invalid action masking. These constraints allow focusing on the relevant parts of the state space.

[0020] Preferably, in an embodiment, a set of ODEs with positive rewards are stored in a data set together with the initial conditions and constants used by the solver when determining the ODE to be valid and the corresponding solutions obtained by the solver. In other words, this embodiment may refer to the case when the training is complete and the agent can act as a data generator. The agent can then output a set of ODEs associated with randomly drawn constants and initial conditions to be solved by the solver. If the solver outputs a valid signal, the entire data point (i.e., ODE, constant, IC and corresponding solution) can be attached to the data set, otherwise, a fixed number of retries with different constants and ICs can be made to eventually end with a valid solution to be attached to the data set after a predetermined number of attempts or to determine the set of ODEs as "unsolvable". In the latter case, the ODEs may obviously not be attached to the data set.

[0021] Preferably, the data set may be used to train a second machine learning system for predicting a set of ODEs describing a time series of performance, aging, charge cycles or discharge cycles of a real-world system. In this case, the second machine learning system for predicting the set of ODEs may receive the time series as input in the reasoning and may then provide the corresponding set of ODEs as output. In particular, the real-world system may be given by an electrical or chemical energy storage (in particular, a battery or a fuel cell) or by a manufacturing machine. The time series may include measured physical observables describing the state of the system over time.

[0022] Preferably, the performance, aging, charging cycle or discharging cycle of the energy storage or manufacturing machine is predicted depending on the predicted set of ODEs of a trained second machine learning system, wherein the prediction of the second machine learning system is based on a given time series of measured physical observables describing the state of the energy storage or manufacturing machine.

[0023] In case the predicted ODE set determines the performance, aging or charge (discharge) cycle of an energy storage or manufacturing machine, the dynamics of the energy storage or manufacturing machine can be better understood / explored and meaningful modifications to the real-world system may be performed to improve its performance and / or life cycle. Preferably, depending on the predicted ODE set determining the performance, aging or charge (discharge) cycle of the energy storage or manufacturing machine, such storage or manufacturing machine is controlled to improve its performance, aging or charge (discharge).

[0024] According to a further aspect, the invention relates to a system configured to implement the training method according to the steps and / or features described above.

[0025] According to a further aspect, the present invention relates to a computer program having machine-readable instructions which, when executed on one or several computers, cause (one or more) computers to perform one of the computer-implemented methods described above and below. Furthermore, according to another aspect, the present invention relates to a machine-readable storage medium on which the above computer program is stored. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Embodiments of the present invention will be discussed in more detail with reference to the following drawings. The drawings show: Figure 1 Reinforcement learning systems; Figure 2 Flowchart of a method for training a reinforcement learning system. DETAILED DESCRIPTION

[0027] Figure 1 An embodiment of a reinforcement learning system 100 is shown. The system 100 includes an environment 101 and an agent 102 and a solver 103. In addition, the system 100 includes a dataset storage 104. In other embodiments, the dataset storage 104 may be an external component that is not included by the system 100 itself. The dataset storage 104 may be a database or cloud storage. The agent 102 may be trained to generate a variety of datasets for ODEs and includes a neural network 112. Actions from the action space are selected by the agent 102 based on the output of the neural network 112. The environment 101 is responsible for selecting the action space by determining the parameters N1, N2, and N3. 2,i To set up the task, where i=1, ... N1, where the parameter N1 determines the number of variables (ie, the number of ODEs in the set) and the number N 2,i Determine the number of mathematical operators in the ith ODE in the set of ODEs. All parameters N1, N 2,i The agent 102 receives the environment parameters N1, N 2,i, and starts each episode with an empty set of differential equations. At each time step, the agent 102 selects an action from the following set: cos, sin, exp, C, (,), x i , +, -, *, / , **, END, where i = 1, ... N1. For example, if N1 = 3, the set may be given by cos, sin, exp, C, (,), x, y, z, +, -, *, / , **, END, where x, y, z represent equation variables. C represents a constant value and END is similar to the end-of-sentence token in natural language processing. Depending on the number of variables received from the environment 101 in each segment, there may be more or fewer variables in the set. For example, for the number of variables N1 = 4, the set may include variables w, x, y, and z. An intuitive way to understand the operation of the agent 102 is through an NLP analogy, where a complete equation corresponds to a sentence. Similar to the way a sentence is generated one word at a time, the agent generates one symbol at a time. Once the full set of ODEs has been created by the agent 101, the full set is passed on to the solver 103, which may be, for example, RK45 or any "off-the-shelf" differential equation solver. In order to solve the ODE set, a value P including an initial condition (IC) and values ​​for constants in the ODE is determined or received by the solver 103, alternatively. The solver can generate the value P by itself, or the value P can be generated by another component and can be provided to the solver 103. The purpose of the solver 103 is to determine whether the generated ODE set has a valid solution. If the answer is affirmative ("yes"), the agent 102 receives a positive reward (R>0). Otherwise, the agent 102 receives a negative value as a reward (R<0). For a given ODE set, some initial conditions produce valid solutions, while other initial conditions do not produce valid solutions. In order not to make the agent stay away from the ODE set just because a certain initial condition for the ODE set leads to an invalid solution, the solver 103 tries several values ​​for its initial conditions and constants before declaring the given ODE set as "unsolvable". To this end, the solver 103 searches whether there is a valid solution. If the answer is affirmative, a positive reward (R>0) is assigned. Otherwise, if the answer is no (e.g., due to divergent solutions, timeouts in solving, etc.), the solver may retry to find a valid solution. Preferably, the solver 103 may retry to find a valid solution a predefined number of times (e.g., n end times), where n end is a natural number. For example, n end Might be chosen to be 25. However, for the maximum number of attempts / retries n endOther choices of may be possible. In this context, it is advantageous that the solver, or alternatively another component of the RL system, tracks the actual number n of attempts of the solver 103 to find a valid solution. If the solver may not have found a valid solution, the actual number n of retries performed so far may be compared to the maximum number n. end Compare. If n <n end Or n = n end , the solver may retry to find a solution. In this case, the newly extracted parameters P' are substituted for the parameters P in the last attempt, and the solver 103 repeatedly attempts to solve the given set of ODEs with the new parameters P'. However, if n>n end , then the set of ODEs is declared “unsolvable” and a negative reward (R<0) is assigned to the set of ODEs.

[0028] The above-mentioned steps can be repeated for multiple episodes, thereby collecting multiple ODE sets together with their corresponding rewards. Based on these data, the agent 102 is trained using a policy optimization RL algorithm that supports discrete actions. Preferably, the algorithm is a PPO algorithm.

[0029] After training is complete, the agent 102 can act as a data generator. The agent 102 can then output a set of ODEs to be sent to the solver 103, which is then associated with randomly drawn constants and initial conditions P. In other embodiments, the solver 103 itself determines the values ​​for the IC and constants. If the solver 103 outputs a valid signal ("yes", i.e., the set of ODEs is found to be solvable), the entire data point (i.e., the ODEs, constants, ICs, and solutions determined by the solver 103) can be appended to the data set in the data set storage 104. Otherwise, a fixed number of retries with different constants and ICs can be performed before a valid solution is found and the data point can be appended to the data set in the data storage 104.

[0030] Furthermore, the reinforcement learning system 100 may include at least one processor 145 and at least one machine-readable storage medium 146 containing instructions that, when executed by the processor 145 , cause the system 100 to perform a training method according to one of the aspects of the present invention.

[0031] Figure 2 1 shows a flow chart of a method for training a reinforcement learning system 100. In method step S1, the agent 102 is initialized. The agent 102 then receives a number of variables N1 and a number of operators N in step S2. 2,iThe environment parameters of . Thus, the index i operates on i=1, ... N1. In step S3, the agent 102 sequentially generates a full set of N1 ordinary differential equations in symbolic form. This can be achieved by the agent 102 by sequentially selecting actions from an action space including mathematical operators and concatenating the selected actions to form an ODE, where N 2,i The number of mathematical operators in the i-th ODE in the set is determined. Subsequently, in step S4, the generated full set of ODEs is sent to the solver 103 for verification of solvability. The solver 103 may use initial conditions and constants P drawn from a random distribution when attempting to find a valid solution to the set of ODEs. In step S5, a reward R is received by the agent, wherein the reward is based on the verification result of the solver. Positive rewards may be assigned for sets of ODEs with valid solutions, and negative rewards may be assigned for sets of ODEs without valid solutions. The reward R may be a binary reward. In method step S6, the previous steps are repeated for multiple segments, and the received rewards collected in the multiple segments are used by the policy optimization algorithm to adjust the current values ​​of the parameters of the neural network 112 of the agent 102.

[0032] The term "computer" may be understood to cover any device for processing predefined computing rules. These computing rules may exist in the form of software, hardware or a mixture of software and hardware.

[0033] Generally, a plurality may be understood to be indexed, that is, each element in the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if the plurality comprises N elements, where N is the number of elements in the plurality, the elements are assigned integers from 1 to N. It may also be understood that the elements in the plurality may be accessed by their indexes.

Claims

1. A computer-implemented method for training an agent (102) for generating a diverse data set of ordinary differential equations, wherein the agent comprises a neural network (112), wherein the agent selects an action from an action space based on an output of the neural network, executes the selected action and receives a reward based on the selected action, The method comprises: - initializing (S1) the agent (102), - Receives a number N1 including variables and N1 numbers N 2,i Environmental parameters (N1, N 2,i ), where i = 1, ..., N1, where each N 2,i Determine the number of operators (S2), - sequentially generate a set of N1 ordinary differential equations in symbolic form by sequentially selecting actions from an action space including mathematical operators and concatenating the selected actions to form ordinary differential equations, where N 2,i Determine the number of mathematical operators in the ith ordinary differential equation in the set (S3), - passing the generated full set of ordinary differential equations to a solver (103) for verification of solvability, wherein the solver uses initial conditions and constants (P) drawn from a random distribution (S4), - receiving a reward (R) based on the verification result of the solver, wherein a positive reward is assigned to a set of ordinary differential equations having a valid solution and a negative reward is assigned to a set of ordinary differential equations without a valid solution (S5), - repeating the preceding steps for a plurality of episodes and using the received rewards collected in the plurality of episodes to adjust the current values ​​(S5) of the parameters of the neural network (112) of the agent (102) by a policy optimization algorithm.

2. The method according to claim 1, wherein the action space includes mathematical entities cos, sin, exp, C, x i , (,), +, -, *, / , ** and END, where cos, sin, exp, +, -, *, / , ** represent mathematical operators, and where x i Represents a variable, where i=1,...N1.

3. The method according to any of the preceding claims, wherein the agent (102) randomly selects the action from the action space by drawing from an action distribution, wherein parameters of the action distribution are determined by the neural network (112).

4. A method according to any of the preceding claims, wherein a state space representing progress in the sequential generation of ordinary differential equations is modeled, wherein a first state in the state space is defined by an empty set of ordinary differential equations, wherein a current state contains all preceding actions selected by the agent (102) concatenated in a selected order, wherein in the current state the agent selects an action from a predefined action space, and wherein the state immediately following the current state is obtained by concatenating the current state with the action selected by the agent (102).

5. A method according to any one of the preceding claims, wherein the solver (103) tries multiple initial conditions for the given set of ordinary differential equations before determining the given set of ordinary differential equations as a set of ordinary differential equations without valid solutions.

6. The method according to any one of the preceding claims, wherein the policy optimization algorithm is an actor-critic policy optimization algorithm.

7. A method according to any one of the preceding claims, wherein a fixed size buffer is defined for each ordinary differential equation in the set of ordinary differential equations to prevent mode collapse scenarios.

8. A method according to any one of the preceding claims, wherein constraints are enforced on the selection of actions to ensure that the generated equations follow mathematical rules.

9. A method according to any of the preceding claims, wherein a set of ordinary differential equations with positive rewards (R>0) are stored in a data set together with initial conditions and constants (P, P') used by the solver (103) and the corresponding solutions obtained by the solver (103).

10. The method of claim 9, wherein the data set is used to train a second machine learning system for predicting a set of ordinary differential equations describing the time series of performance, aging, charge cycles or discharge cycles of an energy storage or manufacturing machine.

11. The method according to claim 10, further comprising the steps of: - predicting the performance, aging, charging cycle or discharging cycle of the energy storage or manufacturing machine depending on the predicted set of ordinary differential equations of the second machine learning system.

12. A system (100) configured to implement the method according to any one of claims 1 to 11.

13. Computer program, configured to cause a computer to carry out the method according to any one of claims 1 to 11 with all its steps, if the computer program is executed by a processor (145).

14. A machine-readable storage medium (146) on which a computer program according to claim 13 is stored.

Images