Reinforced learning for controlling industrial processes

Abstract

The present disclosure relates to a method of training a machine learning agent (2) for controlling an industrial process in an industrial plant. The method includes inputting to an agent a simulation value of a process variable (PV) from a simulation (20) of the industrial process using a model (29) of the industrial process, and an example value of a perturbation variable (DV). An adjustment (22) is input into the simulation, wherein the PV value of the simulation depends on the adjustment. The agent outputs a value of a manipulated variable (MV) in response to the simulated value and the example value. The MV value is used in the simulation, and the simulation updates the PV value of the simulation. The cost of the simulated industrial process is estimated when the MV values are used. As a function of the estimated cost, a reward is paid and fed to the agent.

Description

Technical Field

[0001] This disclosure relates to a method for training a machine learning agent for controlling industrial processes in an industrial plant using reinforcement learning (RL) algorithms. Background Technology

[0002] Reinforcement learning (RL) is a type of machine learning in which an intelligent agent is trained to maximize cumulative rewards. The agent takes actions in an environment, such as an industrial process, which is interpreted as representations of rewards and environmental states. These representations are fed back to the agent. Reinforcement learning agents typically interact with their environment in discrete time steps. At each step, the agent receives the current state and reward. It then selects an action from the set of available actions, which is subsequently sent to the environment.

[0003] In addition to Real-Time Optimization (RTO) and Advanced Process Control (APC) (or in addition to RTO and APC), RL machine learning can be used for the control of industrial processes. Summary of the Invention

[0004] The purpose of this invention is to provide training for more robust RL agents for controlling industrial processes.

[0005] According to one aspect of the invention, a method is provided for training a machine learning agent for controlling an industrial process in an industrial plant using a reinforcement learning algorithm. The method includes, for each of a plurality of training epochs: inputting simulated values ​​of process variables and example values ​​of perturbation variables to the agent, the simulated values ​​being derived from a simulation of the industrial process using a model of the industrial process, the process variables being sensed or estimated in the plant, and the perturbation variables being sensed or unsensitized in the plant; the agent outputting values ​​of manipulation variables in response to the input simulated values ​​and example values, the manipulation variables being actuated in the plant; using the output values ​​of the manipulation variables in the simulation of the industrial process, the simulation updating the simulated values ​​of the process variables used in the simulation; estimating the cost of the simulated industrial process when using the values ​​of the manipulation variables; and feeding a reward to the agent as a function of the estimated cost. Thus, a trained agent is obtained after the epochs. The method further includes, in each of at least some of the rounds, inputting at least one adjustment of the model of the industrial process into the simulation, wherein the simulated value of the process variable input depends on the adjustment.

[0006] According to another aspect of the invention, a system is provided configured to train a machine learning agent using a reinforcement learning algorithm for controlling industrial processes in an industrial plant. The system includes a processing circuit system and a storage device storing instructions executable by the processing circuit system, wherein the system is operable to perform embodiments of the methods disclosed herein.

[0007] According to another aspect of the present invention, a computer program product is provided, the computer program product including a computer executable component for causing the system to perform an embodiment of the method of the present disclosure when the computer executable component is run on a processing circuit system included in the system.

[0008] By training the agent on adjustments to a model of the industrial process, the agent can also control the industrial process under non-ideal conditions, such as when uncontrolled disturbances affect the process (e.g., chemical reactions in a reactor). In some embodiments, the adjustments may be to parameters (e.g., constants) in the model's equations. For example, the parameter may be a reaction rate constant, which is known and constant under normal operating conditions, but may vary slightly, for example, in response to disturbances to the process (such as potentially uncontrolled high or low ambient temperatures). In some other embodiments, the adjustments may be to add at least one equation to the model and / or remove at least one equation from the model, and / or to add at least one term to the model's equations and / or remove at least one term from the model's equations. For example, under non-ideal conditions, in addition to the (desired) main reaction, side (generally undesirable) reactions may occur, in which case the adjustments may include equation(s) also targeting the side reaction.

[0009] The decision of which adjustments to incorporate into the model can be based on the probability of different adjustments occurring in an industrial process (e.g., calculated from historical process data). If an adjustment has historically had a higher probability of occurrence, the model can be tuned during training for more episodes (iterations) than adjustments that have historically had a lower probability of occurrence.

[0010] It should be noted that any feature of any aspect may be applied to any other aspect, where appropriate. Similarly, any advantage of any aspect may be applied to any other aspect. Further objects, features, and advantages of the appended embodiments will become apparent from the following detailed disclosure, the appended dependent claims, and the accompanying drawings.

[0011] Generally, all terms used in the claims will be interpreted according to their ordinary meaning in the art, unless otherwise expressly defined herein. All references to “an / a / the element, device, component, apparatus, step, etc.” are openly interpreted as relating to at least one instance of an element, apparatus, component, device, step, etc., unless otherwise expressly stated. Unless expressly stated, the steps of any method disclosed herein need not be performed in the exact order disclosed. The use of “first,” “second,” etc., for different features / components of this disclosure is intended only to distinguish features / components from other similar features / components, and not to assign any order or hierarchy to the features / components. Attached Figure Description

[0012] Embodiments will be described by way of example with reference to the accompanying drawings, wherein:

[0013] Figure 1 This is a schematic block diagram of an industrial plant according to some embodiments of the present invention.

[0014] Figure 2 This is a schematic diagram of a system configured according to some embodiments of the present invention for training a machine learning agent using a reinforcement learning algorithm to control industrial processes in an industrial plant.

[0015] Figure 3 This is a schematic block diagram of a system configured for training a machine learning agent according to some embodiments of the present invention.

[0016] Figure 4 This is a schematic flowchart illustrating some embodiments of the method of the present invention. Detailed Implementation

[0017] Embodiments will now be described more fully below with reference to the accompanying drawings, in which certain embodiments are illustrated. However, many other embodiments of different forms are possible within the scope of this disclosure. Rather, the following embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art. Throughout the specification, the same reference numerals denote the same elements.

[0018] As used herein, a process variable (PV) is a variable that depends on the process and changes over time. Process variables are typically monitorable (i.e., sensed or measured over time), so when used to control an industrial process in a plant, the sensed value of PV can be fed into a trained agent. A disturbance variable (DV) is an uncontrolled external input variable, as used herein. A manipulated variable (MV) is a variable that is changed by an agent (e.g., via a PID controller, etc.), as used herein. As a simple illustrative example, PV could be the temperature in a room, DV could be the outdoor temperature, and MV could be a control signal for a heater in the room. In this example, the parameters could relate to heat transfer from the heater to the air in the room.

[0019] Figure 1 The diagram illustrates an industrial plant 1 in which industrial process 3 operates. Process 3 is controlled by a trained agent 2. The agent is fed with sensed process variable (PV) values ​​within process 3, and in response, outputs manipulated variable (MV) values ​​to the process. Typically, MV is controlled to achieve the output values ​​via a PID controller or similar device.

[0020] Figure 2 A system 10 configured to train a machine learning agent 2 using a Reinforcement Learning (RL) algorithm for controlling an industrial process 3 in an industrial plant 1 is illustrated. The RL includes receiving a reward 21 in response to feeding a value of MV selected by agent 2 to a simulation 20, wherein the MV value is selected with the objective of achieving the highest possible cumulative reward. During each training epoch, in one or more sampling iterations, values ​​of PV and DV are fed to agent 2, wherein the value of PV is generated based on the value of MV. The values ​​of PV and DV can be determined based on observations 27, for example, by human and / or computer observers, or by sensors.

[0021] The PV value can be obtained from simulation 20. The DV value can be obtained from, for example, an example perturbation 28 defined by a human operator or a computer operator. The PV value is fed to agent 2, and the DV value is fed to agent 2 and simulation 20, wherein the perturbation is associated with the DV value simulated in simulation 20.

[0022] Simulation 20 is at least partially based on model 29 of the industrial process. Furthermore, the simulation may be based, for example, on structural information about plant 1. In at least some training epochs, model 29 is adjusted by inputting model adjustments 22(s) into model 29. In some embodiments of the invention, at least one adjustment 22 includes adjusting parameters in model 29. In some embodiments, the adjustment 22 of a parameter is based on an estimated probability that the parameter takes different values. For example, the probability of the parameter taking different values ​​may be estimated based on historical data of industrial process 3. Additionally or alternatively, in some embodiments of the invention, the at least one adjustment 22 includes: adding at least one equation to model 29 and / or removing at least one equation from model 29, and / or adding at least one term to the equations of model 29 and / or removing at least one term from the equations of model 29. Adjustments 22 made by adding and / or removing terms and / or equations may also be based on the probability of certain non-ideal situations occurring in process 3, for example, based on historical data of industrial process 3. For example, if it is known from historical data from factory 1 that a side reaction may occur in industrial process 3, model 29 can be adjusted accordingly to reflect this situation, so that agent 2 can also be trained to handle this situation.

[0023] In each round (e.g., for each sample within that round), reward 21 is a function of the estimated cost of the simulated industrial process 3 during that round. In some embodiments of the invention, this reward is calculated based on an economic objective function 23 relating to the estimated cost and an economic objective for the industrial process 3. In some embodiments, the reward is also calculated based on a barrier function 24 relating to at least one of the following: a predefined constraint 25 for at least one process variable PV, and / or a predefined constraint 26 for at least one manipulated variable MV. For example, if the reactor temperature (which may be PV sensed in the industrial process 3) should not exceed a predefined threshold (i.e., PV constraint 25), then when the reactor temperature exceeds this threshold in simulation 20, the barrier function 24 can penalize agent 2 by significantly reducing reward 21.

[0024] Figure 3An embodiment of system 10 of this disclosure is illustrated schematically. System 10 includes a processing circuitry system 31, such as a central processing unit (CPU). The processing circuitry 31 may include one or more processing units in the form of microprocessors. However, other suitable devices with computing capabilities, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or complex programmable logic devices (CPLDs), may be included in the processing circuitry 31. The processing circuitry 31 is configured to run one or more computer programs or software (SW) 33 stored in a storage device 32 of one or more storage units (e.g., storage devices). The storage unit is considered to be a computer-readable device 32, which, together with the SW 33 stored thereon as a computer-executable component, forms a computer program product and may be, for example, in the form of random access memory (RAM), flash memory or other solid-state storage devices, or a hard disk, or a combination thereof. The processing circuitry system 31 may also be configured to store data in the storage device 32 as needed. When at least a portion of the SW 33 is executed, the processing circuitry system 31 may provide the intelligent agent 2 and / or emulation 20 discussed herein. System 10 may also include a communication interface 34 for communicating with other parts of system 10 or the outside of the system, such as a human-machine interface (HMI).

[0025] Figure 4 This is a schematic flowchart illustrating some embodiments of the method disclosed herein. The method is used to train a machine learning agent 2 for controlling an industrial process 3 in an industrial plant 1 via RL. The method includes performing multiple training iterations relative to a simulation 20 of process 3, referred to herein as round 40, before obtaining (S7) the trained agent 2. The trained agent 2 can then be used (S8) to control the industrial process 3 in plant 1 (the real process 3, not the simulation 20 used for training).

[0026] In each of at least some rounds of round 40, at least one adjustment 22 of model 29 for industrial process 3 is input (S1) to simulation 20, wherein the PV value of the simulation will depend on said adjustment 22.

[0027] In each of the multiple training rounds 40, the simulated PV value is input (S2) from simulation 20 of industrial process 3 to agent 2, which uses model 29 of said industrial process. The input (S2) is also an example DV value. Then, in response to the input (S2) simulated value and the example value, agent 2 outputs (S3) MV value. The output (S3) MV value is used in the simulation 20 of industrial process 3 (S4), where the simulation updates the PV value of the simulation used in that simulation. When using the MV value (S4), the cost of the simulated industrial process is estimated (S5). As feedback for the output (S3) MV value, the agent is fed (S6) a reward 21, where the reward is a function of the estimated (S5) cost.

[0028] In a more general aspect of the invention, a method is provided for training a machine learning agent 2 for controlling an industrial process in an industrial plant. The method includes: inputting simulated values ​​of process variables (PV) and example values ​​of disturbance variables (DV) from a simulation 20 of an industrial process, wherein the simulation is performed using a model 29 of the industrial process. An adjustment 22 is input to the simulation, wherein the simulated PV values ​​depend on the adjustment. The agent, in response to the simulated values ​​and example values, outputs values ​​of the manipulated variables (MV). Using the values ​​of MV in the simulation, the simulation updates the simulated PV values. While using the MV values, the cost of the simulated industrial process is estimated. A reward is fed to the agent as a function of the estimated cost.

[0029] The present invention has been described above primarily with reference to several embodiments. However, as will be readily understood by those skilled in the art, other embodiments besides those disclosed above are also possible within the scope of this disclosure as defined by the appended claims.

Claims

1. A method for training a machine learning agent (2) for controlling an industrial process (3) in an industrial plant (1) using a reinforcement learning algorithm, the method comprising: For each training round in the multiple training rounds (40): The simulated values ​​of process variables (PV) and example values ​​of disturbance variables (DV) are input to the agent (2), the simulated values ​​being derived from a simulation (20) of the industrial process (3) using a model (29) of the industrial process, the process variables being able to be sensed or estimated in the plant (1), and the disturbance variables being either sensed or not sensed in the plant; The agent (2) responds to the input (S2) simulation value and the example value, and outputs (S3) the value of the manipulated variable (MV), which can be actuated in the factory (1); In the simulation (20) of the industrial process (3), the value of the manipulated variable (MV) output by (S3) in (S4) is used, and the simulation updates the simulation value of the process variable (PV) used in the simulation. Estimate (S5) the cost of the simulated industrial process when using the value of the manipulated variable (MV) described in (S4); as well as The agent (2) is fed (S6) a reward (21) as a function of the estimated (S5) cost. Obtain (S7) the trained agent (2); The method further includes: in each of at least some of the multiple rounds (40), inputting (S1) at least one adjustment (22) of the model (29) of the industrial process (3) to the simulation (20), wherein the input (S2) simulated value of the process variable (PV) depends on the adjustment (22).

2. The method according to claim 1, wherein the at least one adjustment (22) includes adjusting the parameters in the model (29).

3. The method of claim 2, wherein the adjustment (22) of the parameter is based on an estimated probability assuming the parameter takes different values.

4. The method of claim 3, wherein the estimation of the probability comprises: The probability of assuming the parameter takes different values ​​is estimated based on historical data of the industrial process (3).

5. The method according to any one of the preceding claims, wherein the at least one adjustment (22) comprises adjustments performed by adding at least one equation to the model (29) and / or removing at least one equation from the model (29), and / or adding at least one term to the equation of the model (29) and / or removing at least one term from the equation of the model (29).

6. The method according to any one of the preceding claims further comprises: The trained agent (2) is used (S8) to control the industrial process (3) in the factory (1).

7. The method according to any one of the preceding claims, wherein the feeding (S6) of the reward (21) comprises: The reward is calculated based on an economic objective function (23) and a barrier function (24), wherein the economic objective function (23) relates to an economic objective for the industrial process (3) relative to the estimated (S5) cost, and the barrier function (24) relates to a predetermined constraint (25) for at least one of the process variables (PV) and / or a predetermined constraint (26) for at least one of the manipulated variables (MV).

8. A system (10) configured to train a machine learning agent (2) for controlling an industrial process (3) in an industrial plant (1) using a reinforcement learning algorithm, the system comprising: Processing circuit system (31); as well as Storage device (32) storing instructions (33) executable by the processing circuit system (31), wherein the system (10) operates to perform the method according to any of the preceding claims.

9. A computer program product (32) comprising a computer executable component (33) for causing the system (10) to perform the method according to any one of claims 1 to 7 when the computer executable component is run on a processing circuit system (31) included in the system.

Images