Deep Causal Learning for Advanced Model Predictive Control

Abstract

A method for predictive control of a system having a subsystem. The method includes providing a signal injection related to the performance of the system. The signal injection includes various operational controls for the system or its subsystems. Response signals corresponding to the signal injection are received and the utility of the signals is measured. Based on the utility of the response signals, data related to the operational controls is modified to optimize the performance of the system via its subsystems.

Description

[Background technology]

[0001] Model predictive control (MPC) is an advanced method of process control used to control a process while satisfying a set of constraints. Multivariable control algorithms calculate optimal control actions using the following: an internal dynamic model of the process, a history of past control actions, and an optimization cost function J over a receding prediction horizon. The internal model is used to predict changes in the dependent variables of the modeled system caused by changes in the independent variables. Its precision and accuracy are important to achieve high value and performance. Summary of the Invention

[0002] A first method for predictive control of a system includes injecting a randomized controlled signal into a subsystem of the system and ensuring that the signal injection occurs within normal operating ranges and constraints. The method also includes monitoring performance of the system or subsystem in response to the controlled signal, calculating a confidence interval for a causal relationship between the performance of the system or subsystem and the controlled signal, and selecting an optimal signal for the performance of the system or subsystem based on the calculated confidence interval.

[0003] A second method for predictive control of a system includes providing a signal injection to a subsystem of the system and receiving a response signal corresponding to the signal injection. The method also includes measuring a utility of the response signal, accessing data related to operation of the system or subsystem, and modifying the data based on the utility of the response signal.

[0004] A third method for self-calibrating model predictive control of a system includes injecting N randomized controlled signals in subsystems of the system, ensuring that the signal injections occur within normal operating ranges and constraints, and monitoring M responses of the system or subsystems to the controlled signals. The method also includes calculating confidence intervals for first partial derivatives of the system response with respect to the signal injections, and predicting expected changes in performance caused by changes in the controlled signals based on an N by M matrix of first derivatives using a model predictive control algorithm to select optimal signals that iteratively improve system and subsystem performance. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 illustrates an advanced model predictive control of a system having subsystems. [Figure 2] 1 is a flowchart of the search space method of the system. [Figure 3] 1 is a flow chart of a signal injection method for the system. [Figure 4] 1 is a flow chart of a continuous training method for the system. [Figure 5] 1 is a flowchart of the memory management method of the system. DETAILED DESCRIPTION OF THE INVENTION

[0006] Deep causal learning (DCL) provides a robust prescriptive analytical platform with broad applicability to process control automation and optimization. DCL calculates causal relationships through randomized and controlled experiments and compares the difference in outcomes between distinct levels of one independent variable (action / setting / policy). If a system response surface is represented as a noisy vector-valued function F, where the input is a vector of settings for each of the system's independent variables and the output is a vector of values ​​representing the response of the system's dependent variables, DCL can then be interpreted as an active machine learning technique for estimating the values ​​of each element of the system's Jacobian matrix J, i.e., the first-order partial derivatives of the vector-valued function F. In addition, DCL can also quantify interaction effects between input variables and estimate the values ​​of second-order and even higher-order partial derivatives of the vector-valued function F, represented by an array of Hessian matrices. Finally, DCL is applicable to complex dynamical systems by providing a mechanism for identifying large time delays and higher-order dynamics, and can be used to estimate the system's time-dependent Jacobian matrix. Time-dependent (dynamic state) Jacobian and Hessian matrices are used in process control, particularly MPC, which is widely applicable to complex dynamic industrial systems and other systems with subsystems.

[0007] Examples of DCL algorithms and parameters are disclosed in WO 2020 / 188331, which is incorporated herein by reference as if fully set forth.

[0008] Embodiments of the present invention include methods in which DCL enables self-generating and self-calibrating causal models for advanced process control in the form of time-varying Jacobian and Hessian matrices whose matrix elements are evaluated in situ and in real time through randomized and controlled experiments by introducing small randomized perturbations to process control parameters.

[0009] FIG. 1 illustrates advanced MPC for a system 12 having subsystems 1-N. A processor 10 is electrically coupled to subsystems 14, 16, and 18 within the system 12. A data storage 20, such as an electronic memory, stores profiles and parameters 22, external data 24, and results 26. The results may include, for example, outcomes of injecting signals into the subsystems of the system 12. In use, the processor 10 uses the profiles and parameters 22 and, optionally, the external data 24 to inject signals into the subsystems 14, 16, and 18 to evaluate the performance of the system 12. The processor 10 stores responses to the signal injections as results 26 and can use those responses to optimize the performance of the system 12 via its subsystems 14, 16, and 18.

[0010] DCL is a method for determining the individual independent variable level (e.g., x) while holding all other independent variables constant. i,l , x i,l+1 ) and their respective outcomes (F i,l , F i,l+1 ) This information is expressed as dF / dx i =(F i,l+1 -F i,l ) / (x i,l+1 -x i,l ) in x i can be used to estimate the value of the first partial derivative of F with respect to , which represents one matrix element of the Jacobian matrix of the system shown below.

number

[0011] For each matrix element, DCL not only estimates its true value, but also the uncertainty surrounding that estimate as a confidence interval. As data accumulates over time, the confidence interval becomes narrower, corresponding to the increasing precision of the Jacobian matrix estimate. Furthermore, DCL can monitor the dependent variable over time after each change in the independent variable and calculate a time-varying Jacobian J(t) that captures the dynamics of the system response, such as time-varying causal effects, time delays, transient effects, and / or higher-order harmonics.

[0012] This time-varying Jacobian matrix and its associated confidence intervals can be used as an internal dynamic causal model in the MPC algorithm. Using Monte Carlo simulation, for example, a large set of different Jacobian matrices can be generated, with each matrix element randomly sampled from within its associated confidence interval. A confidence interval around the predicted outcome of any control action can be calculated by performing a statistical t-test on the set of predicted outcomes associated with this randomly generated set of Jacobian matrices. Unlike traditional MPC, where model uncertainty is not captured and / or unknown, the approach described herein enables risk-adjusted optimization of process control by providing an accurate quantification of the expected utility and variance associated with each possible process control action.

[0013] System response surfaces can often be nonlinear, and simple linear approximations are insufficient to accurately optimize process control decisions. While the Jacobian matrix provides a linear approximation of the system response, DCL can identify order, spatial, and / or temporal characteristics in the form of external variables (EVs) where the elements of the Jacobian matrix are statistically distinct. In that case, DCL initiates a clustering process similar to piecewise linear approximation, whereby different Jacobian matrices provide local linear approximations within each cluster. Clusters can be generated through various classification techniques, e.g., recursive partitioning algorithms such as conditional inference trees. Alternatively, regression models, e.g., Gaussian mixture regression models, can be used to contextually approximate coefficient values ​​as a function of EVs, allowing for a continuous set of coefficients and matrices across those environmental factors.

[0014] The internal model of an MPC can be further refined by calculating higher-order partial derivatives in the same way. As an example, a DCL is a linear regression model in which two independent variables x are varied. i and x j We can measure the causal effects associated with and calculate the matrix elements of the Hessian matrix of the system.

number

[0015] This set of matrices forms a comprehensive causal model of the underlying system that can be leveraged by several decision-making and process control algorithms, such as MPC. As is often the case in machine learning, greater system complexity can result in overfitting and poor performance in the real world. DCL utilizes both confidence intervals and baseline monitoring to assess the risk / reward of increasing the complexity of its internal model based on available data, and adjusts the model complexity based on evidence that it actually provides greater value in the real world. In addition to enabling advanced model-based process control, DCL's continuous testing of its internal causal model offers several advantages.

[0016] Accurate in-situ quantification of all elements of the Jacobian and Hessian matrices enables accurate determination of the optimal combination of process control actions with high external effectiveness. Often, the cross terms of these matrices may be poorly understood and / or uncharacterized because the complexity of the system means that no representative analytical model exists from which they can be derived numerically, and data-driven approaches cannot isolate partial derivative elements (i.e., the effect of a single variable while holding all others constant) simply by observing highly cross-correlated historical data. As a result, many current methods only provide suboptimal solutions that optimize the sum of local / direct effects (e.g., the block-diagonal Jacobian matrix is ​​equivalent to treating the system's subprocesses independently of each other), rather than providing true optimization that integrally exploits all interactions between the system's subprocesses.

[0017] Accurate in-situ quantification of time delays and other temporal characteristics further enables the timing of these combinations of process control actions to be optimized to minimize adverse effects such as instabilities, transient effects, and harmonics. Using direct estimation based on average matrix coefficients or Monte Carlo simulations by sampling within coefficient confidence intervals, MPCs can estimate the expected net outcome of combinations of process control adjustments over time and optimize the time delays between them. For example, an MPC can be programmed to maintain the temperature of a space, such as a data center, stable around a target value. As heat loads dynamically change within the space, the MPC adjusts fan speed settings to improve air mixing and minimize the presence of hot and cold spots. These adjustments induce transient periods during which airflow can be turbulent, creating localized high and low pressure and temperature points that are detrimental to the space, for example, impairing the operation of server racks. These effects can be predicted using a time-varying Jacobian matrix that calculates the system response at different time intervals and can be mitigated by optimizing the small time delays between fan speed adjustments that promote destructive interference, thereby allowing the overall system to reach steady state faster and with fewer negative side effects.

[0018] The sparsity of the Jacobian matrix provides an assessment of gaps and redundancies in the available controls (independent variables (IVs)) and sensors (dependent variables (DVs)) and can be used to estimate the marginal benefits of adding controls and sensors to improve performance, reduce variance, and / or minimize risk.

[0019] Monitoring matrix elements over time provides an indication of the stability of causal effects. DCL adjusts the data inclusion window used to calculate confidence intervals so that, when causal relationships change over time, only data representative of the system's current state is used in their estimation. Drift in the mean and / or width of the confidence intervals may indicate that the underlying physical causal relationships are changing. In some cases, these changes can map root causes, such as wear and tear of equipment or system failures, over time, thereby improving the accuracy of system diagnostics and the effectiveness of preventive maintenance. The amplitude of change in matrix elements can be used to estimate process gains associated with deploying resources to address the root causes of the changes and balance the benefits against costs, including the opportunity costs of deploying such resources.

[0020] 2-5 are flowcharts of DCL methods for model predictive control that optimize the performance of system 12, for example, via profiles and parameters for controlling subsystems of system 12. These methods may be implemented, for example, in software modules executed by processor 10.

[0021] Figure 2 is a flowchart of the search space method, which includes steps 30 of receiving control information (including costs), 32 of constructing a multidimensional space of all possible control states, 34 of constraining the space of potential control states, 36 of determining a normal / baseline sampling distribution, 38 of determining the sampling distribution with the highest utility, and 40 of performing automated control selection within the constrained space.

[0022] 3 is a flowchart of a signal injection method that includes steps of receiving 42 a set of potential signal injections, calculating 44 the spatial and temporal reach of the signal injections, adjusting 46 the signal injections in space and time, performing 48 the signal injections, collecting 50 response data, and associating 52 the response data with the signal injections.

[0023] Signal injections are profile and parameter changes to subsystems throughout the system. These injections do not need to be large changes and generally consist of small perturbations to control elements within the natural process noise. This allows the DCL to operate within normal operation without a significant increase in overall process variance. The range (i.e., search space) of these perturbations can be adjusted over time to reflect changes in process variance and / or operating objectives. The response to signal injection is typically the performance of the subsystem resulting from or related to the changes in profile and parameters from the signal injection.

[0024] For example, an algorithm can perturb values ​​in a lookup table representing profiles and parameters, then monitor and store the performance response of the corresponding subsystem. As another example, a DCL can perturb the gain values ​​of a PID controller, such as a thermostat, and monitor the response of a system under its control, such as a set of temperature sensors in a space. Temperature readings can be recorded at a single time (e.g., representing steady state) or at multiple time intervals to capture transient effects.

[0025] The temporal and spatial reach of signal injections relate to when and where to measure the response signals to signal injections used to calculate causal relationships, such as to minimize carryover and crossover effects between experiments. The DCL test for independence between repeated effect measurements automatically adjusts their reach to maximize both independence and statistical power. The cost of signal injections is typically related to how much the signal injections affect the overall system—for example, whether the signal injections may cause a degradation or deterioration of a subsystem's performance—and is controlled by the specific experimental scope. The queue for signal injections includes the order and priority of signal injections and relies on blocking and randomization to ensure high internal validity, even when optimizing utility. The utility of the response to signal injections includes another measure of the effectiveness or utility of the signal injections.

[0026] 4 is a flowchart of the continuous learning method, which includes steps of receiving 54 a set of potential signal injections, receiving 56 a current belief state, calculating 58 a learned value for the signal injections, receiving 60 a cost for the signal injections, selecting and adjusting 62 a signal injection, performing 64 a signal injection, collecting 66 response data, and updating 68 a belief state.

[0027] Belief states are a set of different models of the subsystem's performance in response to injected signals. In the case of MPC, belief states consist of a set of coefficients for the Jacobian and Hessian matrices. These belief states can be thought of as having attached uncertainty values ​​that reflect their likelihood of being accurate given the current set of tests and knowledge that may tend to confirm or falsify these various models. Information that can further confirm or falsify the model may be contained in this data or may be derived from fundamental properties of the particular model and the physical properties of the underlying system.

[0028] The learned value is a measurement where the knowledge generated as a result of signal injection can inform subsequent decision-making by the system, such as determining that a particular profile is more likely to be optimal. In the case of MPC, the reinforcement learning component of DCL controls the ratio of the exploration phase (random signal injection aimed at increasing the accuracy of the coefficients of the Jacobian and Hessian matrices) to the exploitation phase (signal injection aimed at improving system performance). In the exploration phase, DCL can prioritize reducing uncertainty in coefficients with the greatest impact, e.g., on the diagonal terms of the Jacobian and Hessian matrices. In the exploitation phase, DCL relies on MPC itself to drive decision-making by leveraging the causal model generated by DCL (i.e., model-based optimization). In other words, while DCL may be commonly used to drive decision making, the present application is not intended to replace MPC with DCL, but rather to maintain MPC in applications where it is already in use and augment it with DCL to continually test and improve the internal and external validity of the models used by the MPC (including accurate characterization of cross terms and time-varying terms), resulting in improved precision and accuracy over time. This approach may be particularly beneficial when the optimum is not expected to converge, but instead requires continuous iterative adjustment.

[0029] In the context of multi-objective optimization, the learned value may involve complex trade-offs between operational goals (e.g., performance vs. range), and optimality may change over time. The learned value may be calculated, for example, by predicting raw numerical data of belief states that may be falsified according to the predictions of a partially observable Markov decision process (POMDP) ​​or other statistical model, and the predicted impact of signal injection on the uncertainty level in the belief states in such a model, or experimental power analysis calculates the reduction in uncertainty and narrowing of confidence intervals based on an increase to the current sample size.

[0030] 5 is a flowchart of a memory management method. The memory management method includes steps 70 to receive a set of historical clusters, 72 to receive a set of historical signal injections, and 74 to calculate the temporal stability of the signal injections for the current cluster. If the signal injections from step 74 are stable (76), the memory management method performs steps 78 to receive a set of historical exogenous factor states, 80 to calculate the stability of the signal injections relative to the exogenous factor states, 82 to select two states and split the cluster only if there is sufficient variance across the two states and enough data within each state (after the split) to be able to drive a decision at each state (i.e., to calculate a confidence interval), and 84 to update the set of historical clusters.

[0031] Clusters are groups of experimental units that are exchangeable with respect to the causal effect measured. Within each cluster, effect measurements follow a normal distribution, free of bias and / or confounding effects from external factors, from which estimates of causal relationships can be derived (not just correlated). Clustering provides a mechanism for continuously optimizing the experimental design as new information about potential effect modifiers emerges, allowing DCL to operate as a self-organizing adaptive clinical trial method. Regression models, such as Gaussian mixture regression models, can further be used to approximate a continuous causal response surface across the generated clusters.

[0032] Table 1 provides an example algorithm for automatically generating and applying causal knowledge for model predictive control of a system having subsystems. The algorithm may be implemented in software or firmware executed by processor 10. [Table 1] In addition to the above embodiment, the following aspects are added. (Appendix 1) 1. A method for predictive control of a system, comprising: injecting a randomized and controlled signal into a subsystem of the system; ensuring that said signal injection occurs within normal operating ranges and constraints; monitoring the performance of the system or subsystem in response to the controlled signal; calculating a confidence interval for a causal relationship between the performance of the system or the subsystem and the controlled signal; using the calculated confidence interval to predict the expected change in performance caused by a change in the controlled signal; selecting an optimum signal that iteratively improves the performance of the system and subsystems. (Appendix 2) 2. The method of claim 1, wherein the controlled signal includes set points, time delays, and gain parameters of proportional controllers, integral controllers, derivative controllers, and combinations of controllers. (Appendix 3) 2. The method of claim 1, wherein the normal operating range comprises a multi-dimensional space of possible control states generated based on control information and operating constraints. (Appendix 4) 10. The method of claim 1, wherein the selecting step further comprises selecting the optimal signal based on external data. (Appendix 5) 2. The method of claim 1, wherein, at time T, the method predicts possible future states of the system at time T+t under different control signals, selects the optimal control signals at T+t that maximize system performance, and then repeats this process iteratively. (Appendix 6) 1. A method for predictive control of a system, comprising: providing signal injection for a subsystem of the system; receiving a response signal corresponding to the signal injection; measuring the utility of the response signal; accessing data relating to the operation of the system or a subsystem; and modifying the data based on the utility of the response signal. (Appendix 7) 7. The method of claim 6, wherein the signal injection includes set points, time delays, and gain parameters for proportional controllers, integral controllers, derivative controllers, and combinations of controllers. (Appendix 8) 7. The method of claim 6, wherein the accessing step includes accessing a lookup table. (Appendix 9) 7. The method of claim 6, wherein the signal injection has spatial reach. (Appendix 10) 7. The method of claim 6, wherein the signal injection has a temporal reach. (Appendix 11) 7. The method of claim 6, wherein the signal injection has multiple temporal reach ranges at different time intervals. (Appendix 12) 7. The method of claim 6, wherein the modifying step further comprises modifying the data based on external data. (Appendix 13) 7. The method of claim 6, wherein the data includes a causal model stored as a set of Jacobian and Hessian matrices. (Appendix 14) 14. The method of claim 13, wherein updating the model comprises modifying or updating coefficients of the matrix. (Appendix 15) 1. A method for self-calibrating model predictive control of a system, comprising: injecting N randomized and controlled signals in a subsystem of the system; ensuring that said signal injection occurs within normal operating ranges and constraints; monitoring the M responses of the system or subsystems to the controlled signals; calculating a confidence interval for a first partial derivative of the system response with respect to the signal injection; using a model predictive control algorithm to predict expected changes in performance caused by changes in the controlled signal based on the N×M matrix of first derivatives; selecting optimal signals that iteratively improve the performance of the system and subsystems based on expected changes in performance predicted by the model predictive control algorithm. (Appendix 16) 16. The method of claim 15, wherein the using step includes using an N×M matrix of second derivatives. (Appendix 17) 16. The method of claim 15, wherein the using step includes using an N×M matrix of Nth order derivatives. (Appendix 18) 16. The method of claim 15, wherein the using step includes using an N×M matrix of time-varying derivatives. (Appendix 19) 16. The method of claim 15, wherein the method optimally balances a search for updating the derivative estimates against utilization for causing the model predictive control algorithm to determine what action to take based on the current derivative estimates.

Claims

1. 1. A method for predictive control of a system using time-dependent Jacobian and Hessian matrices of the system estimated by deep causal learning for model predictive control, the system having a plurality of subsystems electrically connected to each other, the method comprising: injecting a randomized and controlled signal into the subsystem; ensuring that said signal injection occurs within normal operating ranges and constraints; monitoring the performance of the system or subsystem in response to the controlled signal; calculating a confidence interval for a causal relationship between the performance of the system or the subsystem and the controlled signal; using the calculated confidence interval to predict the expected change in performance caused by a change in the controlled signal; selecting an optimal signal that iteratively improves the performance of the system and subsystems; The method further comprises the step of predicting, at time T, possible future states of the system at time T+t under different control signals, and selecting the optimal control signals at T+t that maximize system performance, and then iteratively repeating the series of steps in the method.

2. The method of claim 1 , wherein the controlled signals include set points, time delays, and gain parameters for proportional controllers, integral controllers, derivative controllers, and combinations of these controllers.

3. The method of claim 1 , wherein the normal operating range comprises a multi-dimensional space of possible control states generated based on control information and operating constraints.

Images