Method, control system and computer program

Abstract

A method for automatically reducing the dimensionality of a mathematical representation of a controlled application system is provided. The method includes receiving, in a control system, data corresponding to control activity variables and system state variables associated with the controlled application system, fitting a constrained reinforcement learning (CRL) model to the controlled application system based on the data corresponding to the control activity variables and the system state variables, and automatically identifying system state variables that drive the CRL model to recommend control activity variables of interest for each control activity variable of interest. The method also includes automatically performing a dimensionality reduction of a state space of the CRL model using a subset of the system state variables, estimating a transition probability matrix for a constrained Markov decision process (CMDP) of the controlled application system after the dimensionality reduction of the state space, and formulating the CMDP model as a linear programming (LP) problem using the transition probability matrix and several costs.

Description

[Background technology]

[0001] The present disclosure relates to the field of machine learning, and more particularly to optimizing control activities of a control system that directs the operation of a controlled application system by automatically reducing the dimensionality of a mathematical representation of the controlled application system. Summary of the Invention

[0002] According to embodiments described herein, a method is provided for automatically reducing the dimensionality of a mathematical representation of a controlled application system that is subject to one or more constraints. The method includes, at a control system that directs operation of the controlled application system, receiving data corresponding to control activity variables and system state variables associated with the controlled application system, and fitting, via a processor in the control system, a constrained reinforcement learning (CRL) model to the controlled application system based on the data corresponding to the control activity variables and the system state variables. The method also includes, via the processor, automatically identifying a subset of the system state variables by selecting control activity variables of interest, and for each control activity variable of interest, identifying a system state variable that drives the CRL model to recommend the control activity variable of interest. The method further includes, via the processor, automatically performing a dimensionality reduction of the state space of the CRL model using the subset of the system state variables; via the processor, estimating a transition probability matrix for a constrained Markov decision process (CMDP) of the controlled application system to which the one or more constraint values ​​are imposed after the dimensionality reduction of the state space; and via the processor, formulating the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs.

[0003] In another embodiment, a control system for automatically reducing the dimensionality of a mathematical representation of a controlled application system to which one or more constraint values ​​are imposed is provided. The control system includes an interface for receiving data corresponding to control activity variables and system state variables associated with the controlled application system. The control system also includes a processor and a computer-readable storage medium. The computer-readable storage medium includes program instructions that direct the processor to fit a constrained reinforcement learning (CRL) model to the controlled application system based on the data corresponding to the control activity variables and the system state variables, automatically identify, via the processor, a subset of the system state variables by selecting control activity variables of interest, and, for each control activity variable of interest, identify a system state variable that drives the CRL model to recommend the control activity variable of interest. The computer-readable recording medium also stores program instructions that instruct the processor to automatically perform a dimensionality reduction of the state space of the CRL model using the subset of the system state variables, estimate a transition probability matrix for a constrained Markov decision process (CMDP) of the controlled application system to which the one or more constraint values ​​are imposed after the dimensionality reduction of the state space, and formulate the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs.

[0004] In yet another embodiment, a computer program product is provided for automatically reducing the dimensionality of a mathematical representation of a controlled application system to which one or more constraint values ​​are imposed. The computer program product includes a computer-readable medium having program instructions embodied therein, the computer-readable medium being not itself a transient signal. The program instructions, when executed by the processor, cause the processor to fit a constrained reinforcement learning (CRL) model to the controlled application system based on data corresponding to the control action variables and the system state variables, the CRL model automatically identifying a subset of the system state variables by selecting control action variables of interest based on the data corresponding to the control action variables and the system state variables, and running the CRL model to recommend control action variables of interest for the control action variables of interest. The program instructions are also executed by the processor to cause the processor to automatically perform a dimensionality reduction of the state space of the CRL model using the subset of the system state variables, estimate a transition probability matrix for a constrained Markov decision process (CMDP) of the controlled application system to which the one or more constraint values ​​are imposed after the state space dimensionality reduction, and formulate the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a simplified block diagram of an example water distribution treatment plant (WWTP) in which the automatic state-space dimensionality reduction techniques described herein can be implemented;

[0006] [Figure 2]FIG. 2 is a process flow diagram of a method for automatically reducing the dimensionality of a mathematical representation of a controlled application system that is subject to one or more constraint values. DETAILED DESCRIPTION OF THE INVENTION

[0007] Machine learning techniques are commonly used to optimize the behavior of control systems associated with various different types of controlled application systems, such as wastewater treatment systems, agricultural systems, water resource management systems, query systems, epidemic initiation process systems, and robot motion planning systems. More specifically, machine learning techniques are used to generate models that mathematically describe the behavior of such controlled application systems. For example, reinforcement learning (RL) techniques can be used to learn policies that map the state of a controlled application system relative to its environment to control actions to be taken in a sequence that maximizes the significance of cumulative rewards by an operator of a control system associated with the controlled application system relative to that environment. An extension of RL, called deep reinforcement learning (DRL), utilizes deep neural networks to model the environmental behavior of a client / problem. Furthermore, when constraints are imposed on the behavior of a controlled application system, constrained RL or constrained deep reinforcement learning (CDRL), or both, can be used to learn policies that take such constraints into account. Additionally, Markov Decision Processes (MDPs) include complex frameworks where decisions use explicitly calculated / estimated transition probabilities between different system states in response to specific control actions. When Constrained Markov Decision Process (CMDP) techniques are used, the model is controlled to meet specific constraints associated with the controlled application system.

[0008] In operation, such controlled application systems often include an extremely large number of possible system states as well as an extremely large number of available control actions that can be taken by the respective control systems. For example, some controlled application systems include hundreds of possible system state variables as well as 100 possible control actions. As a result, implementing a constrained mathematical model for such a controlled application system may involve calculating upwards of millions of transition probability values ​​or training an extremely large neural network. However, the control systems for these applications often do not have sufficient processing power to calculate such a large number of transition probabilities in a reasonable amount of time, and many controlled application systems do not include enough sensors to collect enough data to perform calculations at this level. Furthermore, when one or more constraints are imposed on the controlled application system, the constrained mathematical model becomes even more complex and difficult to compute. Therefore, it is often desirable to reduce the size of such a constrained mathematical model.

[0009] This disclosure therefore describes techniques for automatically reducing the dimensionality of the state space of such constrained mathematical models by reducing the number of possible system states from thousands of variables to, for example, fewer than 10 or fewer than 20 variables. This is achieved using a combination of CMDP and CRL (or CDRL) techniques. The resulting simplified model allows a control system to quickly and effectively optimize the operation of a controlled application system while still maintaining one or more constraints imposed on the controlled application system.

[0010] The techniques described herein may be applied to any suitable type of controlled application system, such as wastewater treatment plants, health care systems, agricultural systems, water resource management systems, inquiry systems, epidemic initiation process systems, robotic motion planning systems, etc. However, for ease of discussion, the embodiments described herein relate to the application of such techniques to wastewater treatment plants.

[0011] FIG. 1 is a simplified block diagram of an example wastewater treatment plant (WWTP) 100 in which the automatic state-space dimensionality reduction techniques described herein can be implemented. WWTP 100 can be any suitable type of wastewater treatment unit, device, or system configured to treat incoming wastewater, referred to as influent 102. Influent 102 travels through a liquid line 104 within WWTP 100, which can include any number of screens, pumps, reactors, settling tanks, separation devices, air blowers, etc., for treating influent 102 to established levels in accordance with local regulations or protocols. The two main outputs of liquid line 104 are treated fluid, referred to as effluent 106, and treated biosolids, referred to as sludge 108. Sludge 108 then flows through sludge line 110, which can include any number of screens, pumps, reactors, settling tanks, separation devices, dewatering systems, aeration tanks, air blowers, etc., for treating sludge 108. The resulting treated sludge 112 may include filter cake, for example, which is then sent externally for treatment 114, but may also include activated sludge for use in the reactor, and the recycled effluent 106 is recycled back to the liquid line 104. Additionally, methane gas 118 separated from the sludge 108 in the sludge line 110 can be sent to a gas line 120 where it can be sold or used for various purposes.

[0012] The WWTP 100 also includes a control system 122 configured to control the function of various devices and systems within the WWTP 100, such as screens, pumps, reactors, settling tanks, separation devices, air blowers, dewatering systems, aeration tanks, etc., included in the liquid line 104, the sludge line 110, or both. In various embodiments, the control system 122 of the WWTP 100 comprises, for example, one or more servers, one or more general-purpose computing devices, one or more special-purpose computing devices, one or more virtual machines, or a combination thereof. Furthermore, the control system 122 includes one or more processors 124 as well as an interface 126 for receiving readings from multiple sensors within the WWTP 100, such as one or more sensors 128A connected to the liquid line 104 and one or more sensors 128B connected to the sludge line 110. Sensors 128A and 128B can be used to monitor various conditions associated with WWTP 100. For example, sensors 128A and 128B can be used to monitor flow variables, such as an effluent total nitrogen flow variable and an effluent total phosphorus flow variable. Readings can be received directly from the sensors or can be received via a proxy or input device.

[0013] Interface 126 can also be used to retrieve data from one or more databases 130 associated with WWTP 100 via a network 132, such as the Internet, a local area network (LAN), a wide area network (WAN), or a wireless network, or a combination thereof. Network 132 can include associated copper communication cables, optical communication fiber, wireless communication devices, routers, firewalls, switches, gateway computers, edge servers, etc. Data retrieved from database 130 can include, for example, information related to the operational requirements of WWTP 100, as well as historical operational data for WWTP 100 or for other similar WWTPs.

[0014] Control system 122 also includes a computer-readable storage medium(s) 134 containing program instructions that can be executed by processor(s) 124 to control the operation of WWTP 100, as described further herein. Computer-readable storage medium 134 may be integrated into control system 122 or may be an external device that is connected to control system 122 during use. Computer-readable storage medium 134 may include, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A more specific, non-exhaustive list of computer-readable storage media 134 includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, punch card, or mechanically encoded device having protruding structures within grooves that record instructions, and any suitable combination thereof. Furthermore, as used herein, the term "computer-readable storage media" is not to be construed as referring to electromagnetic waves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves, such as light pulses passing through a waveguide or other communication medium (e.g., optical pulses passing through a fiber optic cable), or electrical signals communicated through wires, which are themselves transitory signals. In some embodiments, the interface 126 includes a network adapter card or network interface that receives program instructions from the network 132 and transfers the program instructions for storage in a computer-readable storage medium 134 within the control system 122.

[0015] The program instructions may be either source code or object code written in any combination of programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or one or more conventional procedural programming languages, such as object-oriented programming languages ​​like Smalltalk®, C++, the “C” programming language, or similar programming languages. The program instructions may be executed entirely on control system 122, partially on control system 122 as a stand-alone software package, partially on control system 122 and partially on a remote computer connected to control system 122 via network 132, or entirely on a remote computer or server. In some embodiments, electrical circuitry, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer-readable program instructions using state information from the computer-readable program instructions to personalize the electrical circuitry to perform features of the present invention.

[0016] In operation, the characteristics of the effluent 106 and treated sludge 112 exiting the WWTP 100 must comply with regulatory constraints specific to the location of the WWTP 100. Such constraints may include, for example, limits on the total nitrogen and total phosphorus concentrations of the effluent 106 over a particular time period. An example of such a constraint is a 15 milligrams per liter (mg / L) upper limit on the monthly (or daily) average total nitrogen concentration in the effluent 106. To the extent that the effluent 106 and treated sludge 112 are treated, multiple steps are performed in the liquid line 104 and sludge line 110. For example, the liquid line 104 may include various different types of equipment for treating the influent 102 through biophysical or chemical subprocesses, or both, in addition to equipment for treating the influent 102 through any number of recursive subprocesses, with portions of the same influent passing through several subprocesses multiple times.

[0017] To comply with such constraints, the control system 122 may be used to control control action variables and system state variables associated with the WWTP 100. As used herein, the term "control action" refers to a possible action taken by the control system to control the operation of a controlled application system, while the term "system state" refers to a possible state of a controlled application system after the control system has taken one or more control actions. Examples of system state variables include: (1) low, medium, and high inlet flow rates (cubic meters per hour (m 3 / hr); (2) operating with one or more feedback or recycle loops set at an on or off setting; (3) operating with an upper limit on the total nitrogen concentration (in mg / L) or total phosphorus concentration (in mg / L) of the effluent 106, or both, set at a low, medium, or high setting; (4) operating with a time period associated with electrical costs set at a high, medium, or low setting; and (5) operating with a total operating cost set at a low, medium, or high setting. Examples of control action variables include: (1) increasing or decreasing the speed of one or more air blowers in the WWTP 100; (2) increasing or decreasing the amount of additive added to the liquid line 104 or the sludge line 110, or both, in the WWTP 100; (3) increasing or decreasing the pumping rate (m 3 / hr); and (4) controlling feedback devices, such as a feedback device in WWTP 100 used to send recycled effluent 116 back to liquid line 104.

[0018] As a result of this complexity, WWTPs typically have high operating costs. Total operating costs may include, for example, the cost of electricity required to operate components within the WWTP 100, the cost of various additives used in the wastewater treatment process, and disposal costs for the resulting treated sludge 112. Today, most WWTPs are operated in a conservative and inefficient, risk-adverse mode without the ability to quantify risk or truly optimize costs. Some subprocesses are locally optimized; however, local optimization of one subprocess may have a negative impact on one or more other subprocesses that were not considered. Such local optimization may even have a counterproductive effect on the process as a whole.

[0019] As a result, the computer-readable storage medium 134 of the control system 122 includes an operational optimization module 136 configured to generate a mathematical representation of the WWTP 100 that can be used to select control actions to maximize operational efficiency and minimize operating costs of the WWTP 100 while maintaining one or more imposed constraints.

[0020] In operation, the CMDP model associated with the WWTP 100 involves a high degree of complexity due to the vast number of variables associated with the wastewater treatment process. For example, the WWTP 100 may include hundreds of possible system state variables, in addition to over 100 possible control actions that can be taken by the control system 122. As a result, implementing a CMDP model for the WWTP 100 may involve calculating or estimating over 2^200 x 100 transition probability values. However, the control system 100 may not have sufficient processing power to calculate such a large number of transition probabilities within a reasonable amount of time. Furthermore, the WWTP 100 includes only a limited number of sensors 128A and 128B. For example, the entire WWTP 100 may include only 5-10 sensors. As a result, the control system 122 may not be able to collect enough data to perform calculations at this level. Furthermore, as constraints are imposed on the WWTP 100, the CMDP model becomes even more complex and difficult to compute.

[0021] As a result, according to embodiments described herein, the operation optimization module 136 in the computer-readable storage medium 134 of the control system 122 includes a sub-module referred to as an automatic state-space dimensionality reduction module 138. The automatic state-space dimensionality reduction module 138 includes instructions that cause the processor(s) 124 to automatically reduce the state-space dimensionality of the CRL model associated with the WWTP 100 by reducing the number of possible system states used in the CRL model from thousands of variables to fewer than 10 or fewer than 20 variables. This then simplifies the transition probability matrix for the CMDP model of the controlled application system, allowing the CMDP model to be formulated as a linear programming (LP) problem. The control system 122 can then use the LP problem to quickly and efficiently control the operation of the WWTP 100 while properly maintaining the imposed constraints.

[0022] The block diagram of FIG. 1 is not intended to indicate that WWTP 100 (or control system 122 within WWTP 100) must include all of the components shown in FIG. 1. Rather, WWTP 100 (or control system 122, or both) can include any number of additional or alternative components not shown in FIG. 1. For example, computer-readable storage medium 134 of control system 122 can include any number of additional modules for controlling and / or optimizing the operation of WWTP 100. Additionally, control system 122 can include and / or be connected to a simulation system that provides simulated data related to the operation of WWTP 100. In some embodiments, control system 122 can use a combination of simulated data and data obtained from sensors 128A and 128B to optimize the operation of WWTP 100.

[0023] We now provide a mathematical description of the CMDP technique that can be used by the embodiments described herein. CMDP is a 5-tuple

[0024]

number

[0025] where S is a finite set of system states,

[0026]

number

[0027] where U is a finite set of control activities,

[0028]

number

[0029] and P is the transition probability function

[0030]

number

[0031] and c is the cost function

[0032]

number

[0033] which contains a vector of costs for state-action pairs, and

[0034]

number

[0035] denotes the vector of costs associated with the constraints. A finite set of system states S is given by

[0036]

number

[0037] where the elements from S are vectors, and the finite set of control actions U is

[0038]

number

[0039] where U is a vector. The transition probability function P is

[0040]

number

[0041] Furthermore, s t is a random variable representing the state at time t, and u t is a random variable representing the activity at time t.

[0042] transition probability function,

[0043]

number

[0044] is one system state

[0045]

number

[0046] from another system state

[0047]

number

[0048] quantifies the transition probability when action u is selected to the state x. The cost associated with selecting action u when the system is in state x is given by

[0049]

number

[0050] Based on different probabilities and outcomes, a policy, π, is generated, where each policy contains a probability measure covering state-action pairs. The goal of the CMDP model is to minimize the overall cost,

[0051]

number

[0052] the total value of the constraints

[0053]

number

[0054] The goal is to find a policy π that minimizes π while ensuring that π is below a specified value.

[0055] CMDP discounted cost model parameters

[0056]

number

[0057] identifies the rate at which future costs will be reduced,

[0058] where:

[0059]

number

[0060] is an event

[0061]

number

[0062] The initial state of

[0063]

number

[0064] and the policy is

[0065]

number

[0066] The discounted occupancy measure according to this model is defined as shown in equation (1) below, where:

[0067]

number

[0068]

number

[0069] The expected cost of policy π at time t is defined as shown in equation (2).

[0070]

number

[0071] The cost is defined as shown in equation (3), where 0<β<1 (discount factor).

[0072]

number

[0073] constraints,

[0074]

number

[0075] is similarly defined as shown in equation (4).

[0076]

number

[0077] As mentioned above, the goal of CMDP technology is to

[0078]

number

[0079] The policy π that minimizes the total value of the constraints

[0080]

number

[0081] The goal is to find a policy π while ensuring that π is below a specified value. The resulting policy π can be used to optimize the operation of the controlled application system. However, as described in connection with the example shown in FIG. 1, many controlled application systems contain an excessively large number of possible system states and possible control actions. This large number of variables results in the creation of highly complex models that cannot be solved in a reasonable amount of time. Therefore, embodiments described herein provide an automatic state-space dimensionality reduction technique that can be used to quickly and efficiently reduce the size of CMDP models by automatically selecting a subset of the most important system state variables for each CMDP model. In various embodiments, this is achieved by combining CRL (or CDRL) and CMDP techniques, as described in more detail with respect to FIG. 2. Furthermore, after state-space dimensionality reduction, the CMDP model can be formulated as a linear programming (LP) problem that can be quickly and efficiently solved by a typical solver, such as the IBM CPLEX® optimizer.

[0082] FIG. 2 illustrates a process flow diagram of a method 200 for automatically reducing the dimensionality of a mathematical representation of a controlled application system subject to one or more constraints. The method 200 is implemented by a control system that directs the operation of the controlled application system. In some embodiments, the controlled application system is a WWTP, such as the WWTP 100 described in connection with FIG. 1, and the control system includes a processor and a computer-readable storage medium containing program instructions that direct the processor to perform the steps of the method 200. Additionally, in other embodiments, the controlled application system can be a health management system, an agricultural system, a water resource management system, an inquiry system, an epidemic early processing system, a robot motion planning system, or any other suitable type of controlled application system.

[0083] The method 200 begins at block 202, where a control system receives data corresponding to control activity variables and system state variables associated with a controlled application system. In some embodiments, at least a portion of the data is calculated via sensors coupled to the controlled application system. Further, in some embodiments, the control system calculates at least a portion of the data based on a simulation of the controlled application system.

[0084] As described herein, the control action variables include a large number of possible actions that the control system can take to control the operation of the controlled application system, and the system state variables include a large number of possible states of the controlled application system after the control system has performed one or more actions. In various embodiments, optimizing the control actions of the control system includes solving a large stochastic problem related to the relationship between the control action variables and the system state variables. Furthermore, the large stochastic problem considers a cost function associated with the controlled application system in addition to one or more constraints for the controlled application system.

[0085] In block 204, the control system processor fits a constrained reinforcement learning (CRL) model to the controlled application system based on data corresponding to the control action variables and system state variables. In other words, the control system processor considers a large-size stochastic problem within a machine learning framework and outputs a CRL model that best explains the behavior of the controlled system in terms of one or more of the constraints, control action variables, and system state variables that define the problem. In some embodiments, the CRL model is a constrained deep reinforcement learning (CDRL) model.

[0086] In various embodiments, the model fitting process may be performed using policies known in the art.

[0087]

number

[0088] This policy is a map that controls the control action u, providing a probability P between 1 and 0, and the environment of the controlled application system, the system state,

[0089]

number

[0090] A policy may be stationary, i.e., time-independent, or non-stationary, i.e., time-dependent, as known in the art. Furthermore, based on the policy, the joint probability of a particular pair of system state variables and control action variables, i.e., a state-action pair, is determined as follows:

[0091]

number

[0092] can be calculated as:

[0093]

number

[0094] is calculated based on data received by the control system.

[0095] In various embodiments, the policy for the CRL (or CDRL) model is implicit. Specifically, for a given system state as input for the CRL (or CDRL) model, a distribution of control actions (or most likely control actions) is provided as output. Because the total number of system states can be enormous or even infinite, the policy is typically not explicitly calculated. Instead, the CRL (and CDRL) process operates as a kind of "telltale." However, according to the embodiments described herein, probabilities can be estimated during a fitting process using data received as input, thus providing an implicit policy.

[0096] One particular process that can be used to generate a policy for a CRL (or CDRL) is referred to as the reward constrained policy optimization (RCPO) process. This process is described in a paper by Tessler, Mankowitz, and Mannor (2018) entitled “Reward Constrained Policy Optimization.” As described there, for a constrained optimization problem, the task is to find an objective function,

[0097]

number

[0098] the inequality constraints,

[0099]

number

[0100] The goal of the RCPO algorithm is to find a policy that maximizes the reward function while satisfying the constraints. The RCPO algorithm achieves this by incorporating the constraints into the reward function as penalty signals. The penalty signals guide the policy toward a solution that satisfies the constraints.

[0101] More specifically, for the RCPO algorithm, large-size stochastic problems are first formulated using Lagrangian relaxation techniques, which involve converting the constrained optimization problem into an equivalent unconstrained optimization problem. A penalty term is added for implausibility, and implausible solutions are suboptimal. As a result, for a given CMDP-based model, the unconstrained optimization problem is defined as shown in equation (5): where L is the Lagrangian and λ≧0 is the Lagrangian multiplier (penalty factor).

[0102]

number

[0103] The goal of equation (5) is a saddle point.

[0104]

number

[0105] where a reasonable solution is

[0106]

number

[0107] Next, the gradients are estimated. Specifically, the data relating to the control action variables and system state variables are calculated as shown in equations (6) and (7).

[0108]

number

[0109] It is used to define an algorithm for constrained optimization problems. In equations (6) and (7), Γ θ is a projection operator, which iterates over a compact, convex set of k Keep Γ stable. λ is the range of λ

[0110]

number

[0111]

number

[0112] is derived from equation (5), where

[0113]

number

[0114] The formulation for is derived using the log-likelihood trick shown in equations (8) and (9).

[0115]

number

[0116] Furthermore, in equations (6) and (7), η1(k) and η2(k) are step sizes that ensure that policy updates are performed on a timescale faster than that of the penalty coefficient λ. This leads to the assumption shown in equation (10), which is that the iterative

[0117]

number

[0118] will converge to a fixed point (i.e., a local minimum) that is a valid solution.

[0119]

number

[0120] The RCPO algorithm extends this process by using an actor-critic based approach, in which actors learn policies π and critics use differential learning, i.e., through the recursive Bellman equation. More specifically, under the RCPO algorithm, actors and critics are trained using an alternative guiding penalty, referred to as the discounted penalty. The discounted penalty is defined as shown in Equation (11).

[0121]

number

[0122] Additionally, the penalized reward function is defined as shown in equations (12) and (13).

[0123]

number

[0124] The penalized value of equation (13) can be estimated using the TD learning critic. The RCPO algorithm is a three-timescale (constrained actor-critic) process in which the actor and critic are updated after solving equation (13) and λ is updated after solving equation (6). The RCPO algorithm converges to a plausible solution under the following conditions: (1) The set of plausible solutions is

[0125]

number

[0126] (2) Locally minimal sets

[0127]

number

[0128] teeth,

[0129]

number

[0130] and (3)

[0131]

number

[0132] The RCPO algorithm then finds the most certain fixed point,

[0133]

number

[0134] Assume this is a reasonable solution.

[0135] In various embodiments, a separate actor-critic process can be used to fit a CRL model to a controlled application system. The steps a control system can follow to implement this process are summarized as follows: (1) learn Lagrangian multipliers that fit the mathematical representation of the controlled application system using gradient ascent, where steepest descent is used for the primary method (e.g., the actor-critic algorithm); and (2) use the learned Lagrangian multipliers to formulate the mathematical representation as a CRL model. This works due to the time-separation properties, i.e., slow and fast time scales, for steepest descent and gradient ascent. In various embodiments, this process can be applied to more generalized constraints using CDRL techniques. This process is further described in more detail in a paper by Bolker (2005) entitled "Actor-Critic Algorithms for Constrained Markov Decision Processes: Systems & Control Letters."

[0136] In block 206, the processor automatically identifies a subset of system state variables by selecting control action variables of interest, and for each control action variable of interest, identifies system state variables that drive the CRL model to recommend the control action variable of interest. In various embodiments, this step includes narrowing down thousands of variables to fewer than 10 or fewer than 20 variables, for example. The identified subset of system state variables is further described by one or more constraint values ​​and cost-related variables associated with the constrained mathematical model.

[0137] In various embodiments, automatically identifying a subset of system state variables in block 206 involves four steps. First, an occupancy measure is calculated for each pair of control action variables and system state variables, i.e., the state-action pairs described in connection with block 204. The occupancy measure is calculated as shown in equation (14).

[0138]

number

[0139] This calculation is based on the probabilities of the state-action pairs calculated during the model fitting process described in connection with block 204. The occupancy for each state-action pair represents the frequency, between 0 and 1, that the state-action pair is reached. In various embodiments, an occupancy measure can also be calculated during the model fitting process described in connection with block 204. Furthermore, the occupancy measure can be formulated with a discount factor β (which discounts future rewards) and an infinite horizon, as shown in equation (15).

[0140]

number

[0141] Second, the control system selects a number of control actions of interest, where each control action of interest is i In some embodiments, the control activity of interest is manually selected by an operator of the control system. For example, the user interface of the control system presents the operator with a list of control activities from which the operator can conveniently select, or the operator can manually type in a computer-readable identification value of the control activity of interest. In other embodiments, the control activity of interest is automatically selected by the control system in response to an automatically generated indication of a trigger event preprogrammed into the control system.

[0142] Third, the control system automatically determines the particular state-activity pairs that contain each control action of interest and have occupancy measures that comply with a predefined threshold. For example, for each control action of interest, it determines the D state-activity pairs (D≧1) that have the highest occupancy measures, i.e.,

[0143]

number

[0144] These state-action pairs can be used to determine the specific control action of interest, u i Fourth, the state-action pairs are isolated and analyzed to identify the specific subset of system state variables that have the most substantial impact on driving the CRL model to recommend each control action of interest.

[0145] In block 208, the processor automatically performs a dimensionality reduction of the state space of the CRL (or CDRL) model using the subset of system state variables, which includes incorporating the subset of system state variables identified in block 206 into the CRL model. In various embodiments, performing a dimensionality reduction of the state space of the CRL model in this manner transforms a complex, large-sized stochastic problem into a simpler stochastic problem that can be solved more quickly and efficiently by the control system.

[0146] In block 210, the processor estimates a transition probability matrix for a constrained Markov decision process (CMDP) model of the controlled system subject to one or more constraint values ​​after state-space dimensionality reduction. In various embodiments, this transition probability estimation process can be decomposed into three steps. First, the processor analyzes the state space and activity space with as much coverage as possible. In other words, the processor analyzes as many transitions as possible between different system states in response to different control actions. To this end, the processor can run simulations of possible transitions using a reduced set of control activity variables and system state variables of interest, focusing on the least frequently used control actions for each system state. Second, the processor estimates transition probabilities by dividing the number of transitions for each change in system state for a particular control action by the total number of control activity and system state variables in the subset of variables. Third, the processor performs interpolation to generate a transition probability matrix. This can be accomplished by using neighboring system states for activity-dependent or activity-independent system state variables, or both, or by implementing a method that ensures irreducibility. The resulting transition probability matrix represents a compact model of the behavior of the controlled application system (subject to one or more constraint values) based on the subset of variables.

[0147] Now, with reference to the mathematical description of transition probability estimation, the set of control activity variables is:

[0148]

number

[0149] where the number of control action variables is

[0150]

number

[0151] and the total number of control activity variables is

[0152]

number

[0153] The reduced set of system state variables is

[0154]

number

[0155] where the number of each system state variable is

[0156]

number

[0157] and the total number of system state variables in the reduced set is

[0158]

number

[0159] The number of events that occur in system state j due to control action i is

[0160]

number

[0161] and the number of transitions from system state j to system state k after control action i is executed is

[0162]

number

[0163] With these definitions, the basic estimation of the transition probability matrix is ​​given in equation (16).

[0164]

number

[0165] In various embodiments, system state variables are divided into two sets: controllable and uncontrollable. For embodiments in which the controlled application system is a WWTP, uncontrollable system state variables include, for example, influent flow, influent chemical load, and time period type due to electrical costs. Control actions and controllable system state variables do not affect transitions between uncontrollable system state variables. Controllable system state variables describe internal or effluent characteristics of the controlled application system and can be influenced by any type of past action and system state variable. Uncontrollable system state variables include, for example,

[0166]

number

[0167] where the system state variables with indices 1,...,I ​​are uncontrollable.

[0168]

number

[0169] are the state spaces corresponding to the uncontrollable and controllable system state variables, respectively. The total state space is the Cartesian product of these two spaces, i.e.,

[0170]

number

[0171] and their dimensions are, respectively,

[0172]

number

[0173]

number

[0174] The set of system states from the state space corresponding to state j from the uncontrollable state space,

[0175]

number

[0176]

number

[0177] In other words,

[0178]

number

[0179] The set of system states from the state space corresponding to state j from the controllable state space,

[0180]

number

[0181] teeth,

[0182]

number

[0183] In other words,

[0184]

number

[0185] Furthermore, if,

[0186]

number

[0187] If so,

[0188]

number

[0189]

number

[0190] represents the controllable part of state k. In other words, state k is a concatenation of states,

[0191]

number

[0192] For action-driven coverage, the current system state is

[0193]

number

[0194] Assume that the smallest selected control action so far, i.e.,

[0195]

number

[0196] If there are several such control actions, we randomly select among them according to a uniform distribution. Similarly, for state-driven coverage, if the current system

[0197]

number

[0198] The total number of times that the system reaches state k up to the current time is

[0199]

number

[0200] In addition,

[0201]

number

[0202]

number

[0203]

number

[0204] If there are several such control actions, select randomly among them according to a uniform distribution. State-driven coverage is

[0205]

number

[0206] In other words, select control actions that occur infrequently in system states.

[0207] In various embodiments, this process is repeated using action-driven coverage over a relatively long time period to obtain reasonable initial estimates of transition probabilities, and then the process is repeated interchangeably between action-driven and state-driven coverage.

[0208] For interpolation of the transition probability matrix, the first process involves interpolation based on neighboring system states:

[0209]

number

[0210] is the system state variable, all but one of which is

[0211]

number

[0212] and the difference by one system state variable is the system state equal to one adjacent interval.

[0213]

number

[0214] In addition to the set of system states in the neighborhood of

[0215]

number

[0216] Itself, L j It is expressed by L j A subset of

[0217]

number

[0218] is represented by

[0219]

number

[0220] contains the system states where control action i has been performed at least once.

[0221]

number

[0222] denotes the size of this set.

[0223] Parameters

[0224]

number

[0225] is used, and interpolation is performed for control actions and system states.

[0226]

number

[0227] A value of M=10 is typically used, and the interpolated probabilities are calculated as shown in equation (17).

[0228]

number

[0229] This process is performed if the number of arrivals to adjacent system states

[0230]

number

[0231]

number

[0232] The second process for interpolating the transition probability matrix involves ensuring the irreducibility of the transition probability matrix. This is important because if the transition probability matrix is ​​not irreducible for some control actions i, the subsequent optimal solution implies that some system states are not achieved under this control action, which is inappropriate for many controlled application systems. Therefore, an irreducibility guarantee method is used to address this issue. The irreducibility guarantee method uses a parameter ε, 0 ≦ ε ≦ 1, where ε = 0.01 is typically assumed. For a given initial system state s0, the irreducibility guarantee method includes the following steps: First, for all control actions i∈U, the values ​​of system states k and j are as shown in equations (18) and (19), respectively.

[0233]

number

[0234] A straightforward algorithm is then used to normalize the transition probability matrix, where the sum of all rows equals one.

[0235] In some embodiments, a third process is used for interpolation of the transition probability matrix. Specifically, the transition probabilities obtained via the first and second processes may contain significant distortions with respect to uncontrollable system state variables. Therefore, interpolation of the uncontrollable system state variables can be performed to correct this distortion. This process is further described in the paper by Zadorojniy, Shwartz, Wasserkrug, and Zeltyn (2016) entitled "Operational Optimization of Wastewater Treatment Plants: A CMDP-Based Decomposition Approach."

[0236] In block 212, the processor formulates the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs. In some embodiments, the LP problem can be formulated using, for example, Python / Optimization Programming Language (OPL) and then solved using IBM CPLEX® Optimizer. The mathematical description of block 212 is shown in equations (20)-(25), where non-zero entries correspond to initial system states.

[0237]

number

[0238] The block diagram of FIG. 2 is not intended to indicate that blocks 202-212 of method 200 should be performed in any particular order, or that all of blocks 202-212 of method 200 should be included in all cases. Furthermore, any number of additional blocks may be included within method 200, depending on the details of a particular implementation. For example, in some embodiments, method 200 includes analyzing an LP problem to determine one or more control actions that will optimize the operation of a controlled application system subject to one or more constraint values, and executing the one or more control actions via the control system. Furthermore, in some embodiments, method 200 is performed multiple times, and the resulting LP problem is used to analyze numerous different predictive scenarios related to the operation of the controlled application system. For example, method 200 can be used to provide predictions (or "what if" analyses) regarding the operation of a controlled application system under a variety of different constraint values ​​or a variety of different types of constraints, or combinations thereof.

[0239] The description of various embodiments of the present invention has been presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond those found in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for automatically reducing the dimensionality of a mathematical representation of a controlled application system to which one or more constraint values ​​are imposed, the method comprising: receiving, at a control system that directs operation of said controlled application system, data corresponding to control action variables and system state variables associated with said controlled application system; fitting a constrained reinforcement learning (CRL) model to the controlled application system based on data corresponding to the control action variables and the system state variables via a processor of the control system; automatically identifying, via the processor, a subset of the system state variables by selecting control action variables of interest and, for each control action variable of interest, identifying a system state variable that drives the CRL model to recommend each of the control action variables of interest based on a policy obtained for the fitted CRL model; via the processor, automatically performing a dimensionality reduction of the state space of the CRL model using the subset of the system state variables; estimating, via the processor, a transition probability matrix for a constrained Markov decision process (CMDP) model of the controlled application system, after dimensionality reduction of a state space, where the one or more constraint values ​​are imposed; and formulating, via the processor, the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs; A method comprising:

2. The method of claim 1 , wherein the CRL model comprises a constrained deep reinforcement learning (CDRL) model.

3. solving the LP problem via the processor to determine one or more actions that will optimize operation of the controlled application system subject to the one or more constraints; and executing said one or more control actions via said control system; The method of claim 1 , comprising:

4. at least, receiving at least a portion of data from a sensor coupled to the controlled application system; and calculating, via said processor, at least a portion of said data based on a simulation of said controlled application system; The method of claim 1 , comprising:

5. Fitting the CRL model to the controlled application system includes performing a rewarded constrained policy optimization (RCPO) process. The method of claim 1 , comprising:

6. Fitting the CRL model to the controlled application system includes: learning Lagrange multipliers that fit a mathematical representation of the controlled application system using a gradient ascent method; and Formulating the mathematical expression as a CRL model using the learned Lagrange multipliers. The method of claim 1 , comprising:

7. automatically identifying the subset of system state variables; calculating an occupancy measure associated with said control action variable and said system state variable pair; automatically determining specific pairs of control action variables and system state variables that include each control action variable of interest and have an occupancy measure that conforms to a predefined threshold; and analyzing the particular pairs of the control action variables and the system state variables to determine the subset of the system state variables that have the most substantial impact on driving the CRL model to recommend each control action variable of interest; The method of claim 1 , comprising:

8. estimating the transition probability matrix for the CMDP model comprises: analyzing a state space and an action space associated with said control action variables and said system state variables of interest; estimating the number of transitions for each change in a system state variable for a particular control action variable by dividing by the total number of control action variables and system state variables within said control action variable of interest and the subset of said system state variables; and interpolation to generate the transition probability matrix using a neighborhood system state for at least one of the activity-dependent or activity-independent system state variables; or Implementing irreducibility assurance methods, or both The method of claim 1 , comprising:

9. solving, via said processor, said LP problem to analyze a plurality of different predictive scenarios relating to the operation of said controlled application system; The method of claim 1 , comprising:

10. 1. A control system for directing the operation of a controlled application system to which one or more constraint values ​​are imposed, comprising: an interface for receiving data corresponding to control activity variables and system state variables associated with the controlled application system; a processor, the processor comprising: fitting a constrained reinforcement learning (CRL) model to the controlled application system based on data corresponding to the control action variables and the system state variables; automatically identifying, via the processor, a subset of the system state variables by selecting control action variables of interest and, for each control action variable of interest, identifying a system state variable that drives the CRL model to recommend each of the control action variables of interest based on a policy obtained for the fitted CRL model; automatically performing a dimensionality reduction of the state space of the CRL model using the subset of the system state variables; estimating a transition probability matrix for a constrained Markov decision process (CMDP) model of the controlled application system subject to the one or more constraint values ​​after dimensionality reduction of a state space; and Formulating the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs. a control system,

11. solving the LP problem to determine one or more actions that will optimize the operation of the controlled application system subject to the one or more constraints; and executing said one or more control actions via said control system; The control system of claim 10, comprising:

12. The control system of claim 10 , wherein the CRL model comprises a constrained deep reinforcement learning (CDRL) model.

13. 11. The control system of claim 10, wherein the controlled application system includes a wastewater treatment plant (WWTP), and the one or more constraint values ​​include at least one of a limit on total nitrogen concentration or a limit on phosphorus concentration of effluent generated by the WWTP.

14. 11. The control system of claim 10, storing program instructions directing fitting the CRL model to the controlled application system by executing a rewarded constrained policy optimization (RCPO) process.

15. fitting the CRL model to the controlled application system; learning Lagrange multipliers that fit a mathematical representation of the controlled application system using a gradient ascent method; and Formulating the mathematical expression as a CRL model using the learned Lagrange multipliers. The control system of claim 10, comprising:

16. Identifying the system state variables includes: calculating an occupancy measure associated with said control action variable and said system state variable pair; automatically determining specific pairs of control action variables and system state variables that include each control action variable of interest and have an occupancy measure that conforms to a predefined threshold; and analyzing the particular pairs of the control action variables and the system state variables to determine the subset of the system state variables that have the most substantial impact on driving the CRL model to recommend each control action variable of interest; The control system of claim 10, comprising:

17. estimating the transition probability matrix for the CMDP model comprises: analyzing a state space and an action space associated with said control action variables and said system state variables of interest; estimating the number of transitions for each change in a system state variable for a particular control action variable by dividing by the total number of control action variables and system state variables within said control action variable of interest and the subset of said system state variables; and interpolation to generate the transition probability matrix using a neighborhood system state for at least one of the activity-dependent or activity-independent system state variables; or Implementing irreducibility assurance methods, or both The control system of claim 10, comprising:

18. 11. The control system of claim 10, further comprising commanding the solving of the LP problem to analyze a plurality of different predictive scenarios related to the operation of the controlled application system.

19. The control system of claim 10 , further comprising calculating at least a portion of the data based on a simulation of the controlled application system.

20. 1. A computer program for automatically reducing the dimension of a mathematical representation of a controlled application system to which one or more constraints are imposed, the computer program comprising: fitting a constrained reinforcement learning (CRL) model to the controlled application system based on data corresponding to control action variables and system state variables; automatically identifying a subset of the system state variables by selecting control action variables of interest and, for each control action variable of interest, identifying a system state variable that drives the CRL model to recommend each of the control action variables of interest based on a policy obtained for the fitted CRL model; automatically performing a dimensionality reduction of the state space of the CRL model using the subset of the system state variables; estimating a transition probability matrix for a constrained Markov decision process (CMDP) model of the controlled application system subject to the one or more constraint values ​​after dimensionality reduction of a state space; and Formulating the CMDP model as a linear programming (LP) problem using the transition probability matrix, one or more cost objectives, and one or more constraint-related costs. A computer program that executes the following:

Images