Simulation of industrial equipment for control

Abstract

A method, system, and apparatus comprising a computer program encoded on a computer storage medium for simulating industrial equipment for control. One of the methods includes, at each of a plurality of time steps during a task episode, receiving, from a computer simulator of the industrial equipment, measurements representing the current state of the equipment; generating observations from the measurements; providing the observations as input to a control policy for controlling the equipment; receiving, as output, an action for controlling one or more set points of the equipment; generating, from the action, one or more control inputs for one or more set points of the equipment; and providing, as input to the simulator, (i) the control inputs and (ii) the current values of one or more configuration parameters of the simulator to cause the simulator to generate, as output, new measurements representing a new state of the equipment.

Description

Technical Field

[0006]

[0001] Cross - Reference to Related Applications This application claims the benefit of priority of U.S. Provisional Application No. 63 / 354,930, filed on June 23, 2022, which is hereby incorporated by reference in its entirety.

[0002] This specification relates to using a machine learning model to control industrial equipment.

Background Art

[0003] A neural network is a machine learning model that uses one or more layers of non - linear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer of the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values of each set of parameters.

Summary of the Invention

Means for Solving the Problems

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that simulates the operation of industrial equipment to enable a machine learning model to be trained to control the equipment.

[0005] The subject matter described in this specification may be implemented in certain embodiments to achieve one or more of the following advantages.

[0006] This specification describes techniques for training, evaluating, or both training, control policies for industrial equipment using computer simulations of industrial equipment. Once control policies are trained and / or evaluated in simulation, they can be deployed and used to control (real-world) industrial equipment.

[0007] More specifically, computer simulations of industrial equipment are deterministic; given the initial configuration, state of the industrial equipment, and control inputs, the computer simulation will constantly update the state of the industrial equipment in a similar manner. This can make existing frameworks for training control policies in simulations unsuitable choices for training control policies for industrial equipment. This is because the control of industrial equipment requires control policies that are robust to a number of real-world imperfections that can cause a given control input to have different effects on the state of the equipment. For example, the equipment's sensors may be noisy or malfunction, external conditions in the real-world environment of the equipment may change rapidly, setpoints may malfunction, etc. This specification describes frameworks for training control policies to be robust to such imperfections, or for evaluating control policies to determine whether a policy is robust to such imperfections, or both, without requiring modification of the simulator or RL agent being trained. In other words, this specification describes frameworks that enable deterministic simulators of industrial equipment to be used effectively to simulate real-world indeterminism. In detail, by using an environment subsystem to interface between the RL agent and the simulator, the system can incorporate various forms of nondeterminism into its interactions, for example, by introducing noise into control inputs, measurements, or both, or by changing the simulator's configuration parameters between and within task episodes. Furthermore, the same framework can be employed to introduce these varying degrees of nondeterminism to multiple different simulators for different equipment and to multiple different tasks. In detail, this framework enables highly expandable configurability, with each of the tasks, simulators, scenarios, and noises being independent axes of configurability, allowing users to combine them.

[0008] In one example described herein, a method performed by one or more computers includes, in each of several time steps in a task episode, receiving a measurement from a computer simulator of the industrial equipment that represents the current state of the industrial equipment; generating an observation from the measurement; providing the observation as input to a control policy for controlling the industrial equipment; receiving an action as output from the control policy to control one or more setpoints of the industrial equipment; generating one or more control inputs to one or more setpoints of the industrial equipment from the action; and providing, as input to the computer simulator, (i) one or more control inputs and (ii) the current values ​​of one or more configuration parameters of the computer simulator, so that the computer simulator generates as output a new measurement representing the new state of the industrial equipment for a subsequent time step.

[0009] Configuration parameters may specify additional information (in addition to control inputs) used by the computer simulator to represent the state of the industrial equipment. Several exemplary configuration parameters are described below.

[0010] Generating observations from measurements may involve adding noise to the measurements. Generating one or more control inputs to one or more setpoints of industrial equipment from actions may involve adding noise to one or more control inputs defined by the observations. The method may further involve identifying scenarios for task episodes. A scenario may specify, for each of several time steps, changes applied to one or more of the configuration parameters, one or more of the control inputs, or one or more of the measurements. A scenario may specify changes applied to one or more of the configuration parameters.

[0011] The method may further include sampling the configuration for task episodes, specifying an initial value for each of the configuration parameters. The method may also include, at each time step, applying the changes specified by the time step scenario to the initial values ​​of one or more configuration parameters in order to generate the current values ​​of the configuration parameters. The scenario may specify changes to be applied to one or more of the measurements. Generating observations from the measurements may include applying the changes specified by the time step scenario to each of the one or more measurements. The scenario may specify changes to be applied to one or more of the control inputs. Generating one or more control inputs from the actions may include applying the changes specified by the time step scenario to each of the one or more control inputs.

[0012] The computer simulator may be a deterministic simulator of the dynamics of industrial equipment. The method may further include training a control policy based at least on task episodes, and deploying the control policy to control the industrial equipment after training. The method may further include evaluating the control policy based at least on task episodes, and deploying the control policy to control the industrial equipment after evaluation.

[0013] The method may further include, after the deployment of a control policy, receiving measurements of the current state of the industrial equipment from the industrial equipment, generating a second observation from the measurements of the current state of the industrial equipment, providing the second observation as input to a control policy for controlling the industrial equipment, receiving a second action for controlling one or more setpoints of the industrial equipment as output from the control policy, generating a second one or more control inputs for one or more setpoints of the industrial equipment from the second action, and controlling one or more setpoints of the industrial equipment based on the second one or more control inputs.

[0014] The method may further include controlling a second industrial facility using a second control policy to generate a dataset, and a computer simulator of the industrial facility is configured to generate measurements representing the current new state of the industrial facility based on the dataset. The second industrial facility may be the same industrial facility as the first industrial facility.

[0015] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0016] [Figure 1] This is a diagram illustrating an exemplary simulation system. [Figure 2] This is a more detailed diagram of the simulation system. [Figure 3] This figure shows an example of how the simulation system behaves during a task episode. [Figure 4] This is an illustrative process flowchart for running a task episode using a simulator. [Modes for carrying out the invention]

[0017] Similar reference numbers and names in various drawings refer to the same elements.

[0018] This specification describes a system implemented as a computer program on one or more computers in one or more locations that simulates the operation of industrial equipment while the equipment is controlled by a control policy.

[0019] In detail, the control policy takes observational values ​​as input that describe the characteristics of the state of the industrial equipment, and accordingly generates actions that specify the respective settings for one or more setpoints of the industrial equipment. Each setpoint is a different controllable element of the industrial equipment. In other words, the control policy controls the equipment by repeatedly updating the settings for one or more setpoints of the industrial equipment.

[0020] For example, a control policy may be implemented as a neural network or other machine learning model, and the system may be used to train the control policy in a simulation before deploying it to control real-world industrial equipment. For instance, a control policy may be trained through reinforcement learning to maximize the reward received, which represents the policy's performance on a given task.

[0021] As another example, the system may be controlled using one control policy, such as a pre-trained neural network or a fixed or heuristic-based control policy, to generate a dataset. This dataset can then be used to train another control policy, for example, through offline reinforcement learning, without the need to use other control policies to control industrial equipment. Alternatively or additionally, the dataset can be used to evaluate the performance of another control policy, for example, to determine whether the control policy is suitable for deployment to control real-world industrial equipment.

[0022] Generally, industrial equipment includes one or more items of electronic devices, mechanical devices, or both, which are controllable by control policies. Control policies operate to control industrial equipment to perform specified tasks.

[0023] In some implementations, the facility is a service facility that includes multiple items of electronic equipment, such as a server farm or a data center, such as a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may include an auxiliary control device that controls the operating environment of the items of the device, such as an environmental control device for temperature control, such as a cooling device, or air flow control or an air conditioner. This device can include, for example, an air-cooled chiller, a water-cooled chiller, or both. The task may include a task that controls the use of resources, such as minimizing the power consumption or water consumption while the facility is operating. In some cases, the optimization may be subject to one or more constraints.

[0024] Generally, an action may be any action that affects the observed state of the environment, for example, an action configured to adjust any of the detected parameters described below. These may include actions that control the items of the device or the auxiliary control device or impose operating conditions on them, for example, actions that cause a change in the settings to adjust, control, or switch on or off the operation of the items of the device or the items of the auxiliary control device. As a specific example, the action may include an action that controls one or more chillers operating within the facility.

[0025] Generally, the observed value of the state of the environment may include any electronic signal representing the function of the facility or the devices within the facility. For example, the representation of the state of the environment may be an observed value created by a sensor detecting the state of the physical environment of the facility, or derived from an observed value created by a sensor detecting the state of one or more of the items of the device or one or more of the items of the auxiliary control device. These include sensors configured to detect electrical conditions such as current, voltage, power, or energy, the temperature of the facility, the flow, temperature, or pressure of a fluid within the facility or within the cooling system of the facility, or the physical configuration of the facility such as whether a vent is open or not.

[0026] Rewards or benefits may relate to metrics of task performance. For example, in the case of a task that controls the use of resources, such as a task that controls the use of power or water, e.g., minimizes it, the metric may include any metric of resource use.

[0027] In some implementations, the facility is a power generation facility, e.g., a renewable power generation facility such as a solar power plant or a wind power plant. The task may include a control task that controls the power generated by the facility, e.g., to meet demand, or to reduce the risk of mismatch between elements of the grid, or to maximize the power generated by the facility, e.g., a control task that controls the supply of power to the power distribution network. The action may include an action that controls the electrical or mechanical configuration of a power generator, such as the electrical or mechanical configuration of one or more renewable power generation elements, e.g., to control the configuration of a wind turbine or one or more solar panels or solar mirrors, or the electrical or mechanical configuration of a rotating power generator. The mechanical control action may include an action that controls the conversion of energy input to electrical energy output, e.g., the efficiency of the conversion or the degree of coupling of the energy input to the electrical energy output. The electrical control action may include an action that controls one or more of the voltage, current, frequency, or phase of the generated power.

[0028] Rewards or benefits may relate to metrics of task performance. For example, in the case of a task that controls the supply of power to the power distribution network, the metric may relate to the amount of power transmitted, or the amount of electrical mismatch between the power generation facility and the grid, such as voltage, current, frequency, or phase mismatch, or the amount of power or energy loss in the power generation facility. In the case of a task that maximizes the supply of power to the power distribution network, the metric may relate to the amount of power or energy transmitted to the grid, or the amount of power or energy loss in the power generation facility.

[0029] In general, observations of the environmental state may include any electronic signals that represent the electrical or mechanical function of the power generation equipment within a power generation facility. For example, the representation of the environmental state may be derived from observations created by sensors detecting the physical or electrical state of equipment within a power generation facility that is generating power, or the physical environment of such equipment, or the status of auxiliary equipment supporting the power generation equipment. Such sensors may include sensors configured to detect observations of the electrical status of equipment, such as current, voltage, power, or energy; temperature or cooling of the physical environment; fluid flow; or the physical configuration of equipment; and, for example, observations of the electrical status of the grid from local or remote sensors. Observations of the environmental state may also include one or more predictions about the future state of the operation of the power generation equipment, such as predictions of future wind levels or solar irradiance, or predictions of the future electrical status of the grid.

[0030] Figure 1 shows an example of a simulation system 100. The simulation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, which can implement the systems, components, and techniques described below.

[0031] System 100 is described as being used to control a heating, ventilating, and air conditioning (HVAC) system 120 of a simulated industrial facility 110.

[0032] However, more generally, system 100 may be used to control any mode of operation of any type of industrial equipment 110, for example, one of the modes described above.

[0033] The simulated industrial equipment 110 (also called simulator 110) is a computer simulation of real-world industrial equipment, that is, it models the state and dynamics of real-world industrial equipment as observed in various contexts using one or more computer programs. Specifically, simulator 110 is one or more software programs that maintain the state of real-world industrial equipment, for example, current readings of sensors in the equipment and possibly additional information, and takes as input (i) current values ​​of configuration parameters that specify the configuration of the simulator, and (ii) control inputs to one or more setpoints of the industrial equipment, and provides as output measured values, i.e., updated readings of sensors in the equipment that reflect the updated state of the equipment as a result of the control inputs.

[0034] System 100 can utilize any suitable computer simulator. For example, a user of the system can provide System 100 with access to a computer simulator of a real-world facility of interest to the user, for example, by enabling System 100 to access the simulator via an API or other interface, or by enabling System 100 to run the simulator.

[0035] Generally, the problem of controlling industrial equipment to perform a specified task can be structured as a constrained multi-objective optimization.

[0036] In an HVAC example, the controller 130 controls several setpoints that adjust the temperature exchange characteristics of the HVAC system 110 in order to perform tasks such as maintaining the equipment temperature at a constant level. In an HVAC example, the setpoints may include enabling and disabling selected chillers, and optionally setting the chiller leaving temperature.

[0037] HVAC components draw power from the grid, and therefore the next objective of the controller 130 may be to reduce power consumption. Thus, the overall task performed by the controller 130 may be structured as minimizing power consumption by the HVAC system 110 while satisfying one or more constraints regarding the equipment temperature.

[0038] If the controller fails in its task, it risks overheating the equipment, which can lead to disastrous consequences. For example, a failure in a computer component could result in data loss or downtime of electrical or mechanical components essential to the equipment's operation. To prevent this from happening, the manufacturer of the controller 130 implements a set of fail-safe constraints to prevent such events from occurring. Violation of these constraints not only compromises the controller's reliability but also typically results in the controller being disconnected from the equipment, making it impossible to optimize power consumption anymore.

[0039] System 100 may be used to provide a set of simulated scenarios (for example, control policies implemented as machine learning models) that can be used to safely and efficiently train and evaluate a controller. That is, System 100 may be used to train, evaluate, or both train a control policy on a simulator 110 that controls one or more setpoints specified by the controller 130.

[0040] More specifically, during the operation of system 100, control policy 150 (for example, a reinforcement learning agent) performs tasks in a closed-loop control system using simulator 110 as a ground truth model of equipment dynamics.

[0041] System 100 uses simulator 110 to evaluate the effect of the actions proposed by policy 150 with respect to the current state of the simulation.

[0042] The simulator 110 returns results in the form of measurements, which are a subset of the simulated state. The measurements can include current readings from any of the various sensors of the industrial equipment.

[0043] System 100 processes the measured values ​​into observed values ​​160, which are then provided as input to the control policy 150.

[0044] However, HVAC simulations are deterministic; that is, performing a given action in a given simulated state will always result in the same updated state. However, controlling a real-world HVAC system must account for various non-deterministic elements that may be encountered during operation and that can change how an action affects the state of the equipment. Examples of these non-deterministic elements (also called "imperfections") are described in more detail below with reference to Figures 2 and 3.

[0045] To introduce imperfection, system 100 can introduce noise into one or more of the various aspects of the control pipeline, such as the control inputs, simulation configurations, and observed values. This will be explained in more detail below with reference to Figures 2 and 3.

[0046] Figure 2 shows a more detailed diagram of the simulation system 100.

[0047] As shown in Figure 2, the simulation system 100 includes a simulator 110. In some implementations, the system 100 may also include a simulator data storage 210 that stores the specifications of several different simulators, so that, for example, an appropriate simulator can be selected for a given task to control a given real-world facility.

[0048] During operation, the simulation system 100 represents interaction with the simulator 110 as interaction with the environment subsystem 220.

[0049] The environment subsystem 220 is implemented as one or more computer programs and controls interaction with the simulator 110 by the RL agent 230. The RL agent 230 may include a control policy and related components for training the control policy through reinforcement learning based on the interaction of the control policy with the simulator 110.

[0050] As can be seen in Figure 2, the RL agent 230 takes observations as input and provides actions 234 as output to control one or more setpoints of the simulated equipment. The input observations include environmental observations 232 and, optionally, “task observations” 272 (for example, generated from environmental observations 232 according to several task parameters) which contain additional information specific to the task being performed.

[0051] The environmental subsystem 220 translates action 234 into control input 236 and provides control input 236 to the simulator 110. For example, translating action 234 may involve translating a high-level action (e.g., an indicator that the cooler should be disabled) into an instruction or other command that can be executed within the facility to perform the high-level action (e.g., a machine-readable instruction to disable the cooler).

[0052] The environment subsystem 220 also provides the simulator 110 with the values ​​of the configuration parameters 262 as input.

[0053] The configuration parameter 262 specifies additional information (in addition to the control input 236) required by the simulator 110 to adequately represent the state of real-world industrial equipment. In other words, the configuration parameter is a parameter necessary for initializing the simulator, i.e., to adequately represent the state of real-world equipment.

[0054] For example, configuration parameter 262 can specify a setpoint value that is not controlled by RL agent 250 but is required to be specified by the controller. For instance, if the setpoint includes enabling and disabling a selected cooler, as well as configuring the cooler outlet temperature, but RL agent 230 simply controls enabling and disabling the cooler, the configuration parameter specifies the cooler outlet temperature.

[0055] Configuration parameter 262 also specifies properties of the external environment of a real-world industrial facility. For example, configuration parameter 262 can specify the temperature, humidity, and precipitation of the external environment.

[0056] For example, before starting a task episode, the environment subsystem 220 can specify the initial values ​​for each of the configuration parameters, for example, by sampling a configuration from the configuration storage 264 that models a real-world configuration of an actual industrial facility. In some cases, as described below, the subsystem 220 can change the initial values ​​during the task episode, while in other cases, the subsystem 220 can maintain the initial values ​​throughout the entire task episode.

[0057] When the simulator 110 is configured as specified by the configuration parameter 238, it returns a measurement value 238 that reflects the updated state of the simulator 110 as a result of the control input 236 being applied.

[0058] The environmental subsystem 220 converts the measured values ​​238 into observed values ​​232, which are then provided as input to the RL agent 230. For example, the user can provide the system with specifications for the input to be received by the RL agent 230, namely which sensor measured values ​​are provided as input, the expected range of the sensor measured values, and the numerical format of the sensor measured values. The subsystem 220 can then standardize the measured values ​​238 so that they conform to the specifications of the observed values ​​232 provided by the user.

[0059] As described above, the RL agent 230 controls the simulator 110 to perform a specified task, for example, to optimize one or more performance metrics subject to one or more constraints. The constraints may include constraints on measurements, such as temperature not exceeding a threshold; constraints on actions, such as a given cooler not being enabled for longer than a continuous time window; or both.

[0060] To determine whether any constraints have been violated by a given action or measurement, system 100 includes a constraint evaluator 260 that maintains data specifying the current set of constraints for the task being performed by the RL agent 230. In detail, each configuration in storage 264 is associated with a set of constraints for a given task.

[0061] The evaluator 260 receives input including the current action, the current set of measurements, or both, and determines whether any of the current set of constraints specified by the configuration of the current task episode are violated. The evaluator 260 then provides data identifying whether any constraint violations have occurred to the environment subsystem 220, which can provide this information to the RL agent 230 as part of the corresponding observations.

[0062] The constraints on a given task may include soft constraints, hard constraints, or both.

[0063] Soft constraints are constraints that can be violated and simply negatively impact the performance evaluation of RL agent 230.

[0064] A hard constraint is a constraint that must not be violated; in other words, a violation of the constraint will result in the controller being disconnected from the equipment. When the evaluator 260 determines that a hard constraint has been violated, the environment subsystem 220 may terminate the current episode of control, i.e., provide an indication to the RL agent 230 that a hard constraint has been violated and that the RL agent 230 can no longer continue this instance of the simulation.

[0065] To train an RL agent 230, or to evaluate the performance of an already trained RL agent 230, the RL agent requests a training signal. In reinforcement learning, this is represented in the form of a set of rewards 272, which are numerical values ​​generated by the task subsystem 270 based on appropriate information, such as the results of constraint evaluations, measurements, and control inputs. From this information, a mapping to one or more numerical values ​​representing the rewards 272 can be specified by the user of system 100.

[0066] The RL agent 230 can use the reward 272, environmental observations 232, and action 234 to train a control policy using an appropriate reinforcement learning technique, such as an on-policy or off-policy reinforcement learning algorithm.

[0067] Alternatively, as described above, the RL agent 230 can store the reward 272, environmental observations 232, and action 234 for use when training a different policy through offline reinforcement learning, or for evaluating a different policy as described above.

[0068] Once a given policy is trained, it can be used to control real-world equipment simulated by the simulator 110.

[0069] Instead of directly providing action 234 as control input 236 and directly providing observation value 238 that results in a deterministic control loop, the environment subsystem 220 uses one of various components to introduce imperfection into the control process.

[0070] In detail, the environmental subsystem 220 can utilize one or more of the following: scenario 226 or noise generator 290.

[0071] As a specific example, the environment subsystem 220 may use noise generated by the noise generator 290 to add noise to the control input as part of converting actions to control inputs, to the observations as part of converting measurements to observations, or both. The noise is added to simulate sensor / pipeline imperfections and, in effect, diversify the distribution of simulated conditions (control noise) and observations (observation noise). The parameters of the noise generator 290, such as the noise distribution from which the noise is sampled, or the time and place to which the noise is applied, or both, may be specified by the user or sampled by the system 100 from a set of possible parameters.

[0072] Scenario 226 models a real-world scenario of interest, and the scenario 226 used for a given task episode may be specified by the user or sampled by system 100 from a set of scenarios. An example of a scenario 226 is one used to model environmental instability during the operation of a real-world facility that cannot be effectively captured by the operation of simulator 110. An example of such a real-world scenario is simulating changing weather conditions in the environment of a facility that may affect the effect of actions on the state of the facility.

[0073] More specifically, Scenario 226 is implemented as an input to the simulator, i.e., a change to the configuration parameters 262 used for a given task and / or a control input given to the simulator, a change to the output of the simulator 110, i.e., a change to a measurement generated by the simulator 110, or both.

[0074] As described above, the configuration parameter 262 contains information about the equipment status. When scenario 226, which modifies the configuration parameter, is selected, the environment subsystem 220 modifies the configuration parameter 262 using scenario 226 and then provides the configuration parameter 262 as input to the simulator 110. When scenario 226, which modifies the control input, is selected, the environment subsystem 220 modifies the control input using scenario 226 and then provides the control input as input to the simulator 110. When scenario 226, which modifies the measured value, is selected, the environment subsystem 220 modifies the measured value using scenario 226 and then converts the measured value into an observed value.

[0075] Therefore, in each episode step, in addition to sending a control input, the environment subsystem 220 also changes one or more of the selected configuration parameters 262 according to scenario 226, the selected control input itself, or the selected measured values ​​provided in response to the control input.

[0076] More specifically, Scenario 226 can be implemented as a time-dependent function that generates values ​​to be used as modifiers for one or more of the following: (i) one or more configuration parameters, (ii) one or more control inputs, or (iii) one or more measurements. That is, Scenario 226 maps the time index during an episode of control to the respective modifiers for one or more of (i), (ii), or (iii).

[0077] Several specific examples of Scenario 226 follow.

[0078] An example of Scenario 226 is a baseline scenario that does not utilize a configuration trajectory. Baseline scenarios can be used to test agent performance while controlling a simulated facility that is not perturbed, and to develop a fitness baseline for comparison with other tasks. Therefore, in this scenario, the initial parameter values ​​specified by the configuration are used throughout the entire episode.

[0079] Another example of Scenario 226 is the sensor drift scenario. This scenario introduces time-correlated noise into a selected set of measurement components. For example, the components may be randomly selected at the beginning of each episode. This scenario can test the agent's resilience to partially false information.

[0080] Another example of Scenario 226 is the frozen control scenario. This scenario freezes the value of a selected control for a random amount of time. That is, instead of applying a value for the control specified by the action, the environment subsystem 220 samples the length of a random time interval and provides the environment with the value of the selected control that was selected immediately before the time interval began, for the duration of that interval. For example, the control could be randomly selected at the beginning of each episode. This scenario can test the agent's ability to detect when the selected policy is not working and adapt by switching to an alternative.

[0081] Another example of Scenario 226 is a transient dynamics scenario. This scenario uses a set of configuration trajectories to modify selected simulation configuration parameters that represent aspects of the real-world environment of a real-world facility over the course of an episode. The configuration trajectories create changes to the selected parameters, which subsystem 220 adds to their baseline values ​​and then passes to simulator 110. Examples of such parameters include external ambient temperature, humidity, wind speed, and precipitation. This scenario can test the agent's resilience to constantly changing environmental conditions, building loads, and other variables outside the agent's control domain.

[0082] Another example of a scenario is the equipment degradation scenario. This scenario uses a set of configuration trajectories to modify selected simulation configuration parameters that represent the efficiency or other measures of performance of equipment within the facility, such as pumps, heat exchangers, cooling towers, and chillers. This scenario can test the agent's resilience to equipment performance degradation during equipment operation, for example, as a result of wear.

[0083] In this way, a given task episode is specified by the selection of simulator 110 from simulator storage 210, simulator configuration specifying initial configuration parameter values ​​262, scenario 226, and optionally noise parameters of noise generator 290. Once these are specified, for example, sampled by the system or specified by the user, system 100 can execute the task episode to generate training data for RL agent 230.

[0084] Figure 3 shows an example of the operation of the simulation system 100 during a task episode.

[0085] A task "episode" is a series of time steps in which agent 230 controls simulator 110. A "time step" is a time interval in which measurements are received from simulator 110 and control inputs are provided to the simulator in accordance with the measurements. A task episode may terminate, for example, if a predetermined number of time steps occur, if a hard constraint is violated, or if an error occurs in the simulator.

[0086] Before starting a task episode, the system selects a configuration 304. For example, the system may select a predetermined or randomly sampled initial configuration for the configuration parameters of the simulator 110.

[0087] The system also identifies a scenario, which is represented as a configuration trajectory 302 that assigns one or more values ​​or changes to one or more of the configuration parameters, one or more of the control inputs, or one or more of the measured values ​​at each time step in the episode. In other words, a scenario defines a time-dependent function for updating the initial configuration 304, measured values, and / or control inputs.

[0088] At each time step, agent 230 receives an observation 330 and selects an action 340 that specifies the value of one or more setpoints in simulator 110.

[0089] The action converter 350 converts the action 340 into the control input 354 of the simulator 110. As part of the conversion, the action converter 350 can add noise 352 to the control input.

[0090] The simulator receives a control input 354 and the values ​​of the configuration parameters generated by applying the trajectory 302 to the configuration 304. Given the configuration parameter values, the simulator generates a measurement 306 that includes the current values ​​of each of the equipment's sensors as a result of applying the control input 354.

[0091] The observation converter 310 (for example, part of the environment subsystem 220) converts the measurement 306 to the next observation 330 of agent 230. More specifically, as part of the observation conversion, the converter 310 can add observation noise 312 to one or more of the measurement 306. If the scenario requires that one of the sensors has sensor drift, the observation noise 312 can reflect a particular noisy reading of the selected sensor.

[0092] The constraint evaluator 260 then evaluates the control input generated from the action proposed by agent 230 and the observation generated from the measurement 306 to determine whether any of the constraints are violated. If any, information identifying which constraints are violated may be added to the next observation 330 before it is passed to agent 230.

[0093] Figure 4 is a flowchart of an exemplary process 400 for executing a task episode using a simulator. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations. For example, a simulation system appropriately programmed according to this specification, such as the simulation system 100 shown in Figure 1, can perform process 400.

[0094] Before a task episode, the system can select a simulator configuration and scenario. For example, the system can randomly sample a configuration from a set of possible configurations that model the real-world operating conditions of the equipment, and accept a scenario as user input. As another example, the system can randomly sample both the configuration and the scenario.

[0095] The system then performs the following steps at each time step during a task episode:

[0096] The system receives, as output from the simulator, measurements representing the current state of the industrial equipment being modeled by the simulator (step 402).

[0097] The system converts the measured values ​​into observed values ​​(step 404).

[0098] The system provides observations as input to the control policy (step 406). For example, the control policy may be a policy trained by the RL agent.

[0099] The system receives the action as output from the control policy (step 408).

[0100] The system translates the action into a control input for the simulator (step 410).

[0101] The system provides control inputs and current values ​​of configuration parameters as inputs to the simulator, i.e., to be used when generating new measurements that represent the next state of the industrial equipment (412).

[0102] In this specification, the term “configured” is used in relation to system and computer program components. “Configured” for one or more computer systems to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action when in operation. “Configured” for one or more computer programs to perform a particular operation or action means that the programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action.

[0103] The subject matter and functional operating embodiments described herein may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-temporary storage medium for execution by or control of the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable memory board, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0104] The term "data processing device" refers to data processing hardware and encompasses all types of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A device may be a dedicated logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit), or may include these. In some cases, in addition to hardware, a device may include code that creates an execution environment for computer programs, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more of these.

[0105] Computer programs, sometimes called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, such as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not have to, correspond to a file in a file system. A program can be stored in a single file dedicated to it, in part of a file containing other programs or data, such as one or more scripts stored in a markup language document, or in multiple collaborative files, such as a file storing one or more modules, subprograms, or parts of code. A computer program can be deployed to run on one computer, or on multiple computers located in one site or distributed across multiple sites and interconnected by a data communication network.

[0106] In this specification, the term “database” is used broadly to refer to any collection of data, which does not need to be structured in any particular way, or does not need to be structured at all, and may be stored in one or more locations on a storage device. Thus, for example, an indexed database may contain multiple collections of data, each of which may be organized and accessed differently.

[0107] Similarly, in this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Generally, an engine is implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers may be dedicated to a particular engine, while in other cases, multiple engines may be installed on and running on the same computer.

[0108] The processes and logic flows described herein may be implemented by one or more programmable computers executing one or more computer programs to perform their functions by operating on input data and generating outputs. The processes and logic flows may also be implemented by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0109] A computer suitable for running computer programs can be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or other types of central processing units. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for issuing or executing instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or integrated into dedicated logic circuits. Generally, a computer is coupled to one or more mass storage devices for storing data, including magnetic disks, magneto-optical disks, or optical disks, or is operable to receive data from or transfer data to one or more mass storage devices, or both. However, a computer is not required to have such devices. Furthermore, a computer may be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a Universal Serial Bus (USB) flash device.

[0110] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks, and all forms of non-volatile memory, media, and memory devices.

[0111] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user, as well as a keyboard and a pointing device such as a mouse or trackball to which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from that web browser. The computer can also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0112] A data processing device for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for handling the training or fabrication of the machine learning model, i.e., inference, the common parts of the workload, and the computationally intensive parts.

[0113] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework.

[0114] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component such as an application server, or a frontend component such as a client computer having a graphical user interface, a web browser, or an application on which a user can interact with the implementation of the subject matter described herein, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0115] A computing system can include clients and servers. Clients and servers are generally geographically separated from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device, for example, to display data to a user interacting with a device acting as a client, and to receive user input from the user. Data generated on the user device, such as the results of user interaction, may be received from the device by the server.

[0116] This specification includes many specific details of implementation, but these should not be interpreted as limitations on the scope of the invention or the scope of what may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Some features described herein in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately in multiple embodiments or in any suitable partial combination. Furthermore, features may be described above as functioning in a certain combination, and may be initially claimed as such, but one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may cover a partial combination or a variation of a partial combination.

[0117] Similarly, while operations are shown in the drawings and described in the claims in a specific order, this should not be understood as requiring such operations to be performed in a specific order shown, or sequentially, or that all illustrated operations must be performed to achieve a desired result. In some environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0118] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order and still achieve the desired results. As an example, the process shown in the accompanying diagram does not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. [Explanation of symbols]

[0119] 100 Simulation Systems 110 Simulated Industrial Equipment / Simulators 120 Heating, ventilation, and air conditioning systems 130 Controllers 150 Control Policies 160 observations 210 Simulator Data Storage 220 Environmental Subsystems 226 Scenarios 230 RL Agent 232 Environmental observation values 234 Action 236 Control Inputs 238 Configuration Parameters / Measurements 250 RL Agent 260 Constraint Evaluator 262 Configuration Parameters 264 configuration storage 270 Task Subsystems 272 Task Observations / Rewards 290 Noise Generator 302 Trajectory 304 Configuration 306 measurement values 310 Observation Converter 312 Observation noise 330 Observations 340 Actions 350 Action Converters 352 Noise 354 Control Input

Claims

1. A method performed by one or more computers, In each of the multiple time steps within a task episode, Receiving measured values ​​representing the current state of the industrial equipment from a computer simulator of the industrial equipment, From the aforementioned measured values, generate observed values, The observed values ​​are provided as input to a control policy for controlling the aforementioned industrial equipment, The output from the control policy is to receive an action to control one or more setpoints of the industrial equipment, From the aforementioned action, one or more control inputs are generated for one or more setpoints of the industrial equipment, To cause the computer simulator to generate new measurements as output representing the new state of the industrial equipment for a subsequent time step, the inputs to the computer simulator are: (i) the one or more control inputs, and (ii) the current values ​​of one or more configuration parameters of the computer simulator. Methods that include...

2. To generate observed values ​​from the aforementioned measured values, Adding noise to the aforementioned measurement values The method according to claim 1, including the method described in claim 1.

3. From the aforementioned action, one or more control inputs to the one or more setpoints of the industrial equipment are generated. Adding noise to one or more control inputs defined by the aforementioned observed values. The method according to claim 1, including the method described in claim 1.

4. Identifying a scenario for the task episode, wherein the scenario is such that for each of the plurality of time steps, One or more of the above configuration parameters, One or more of the aforementioned control inputs, or One or more of the above measured values, Specify, identify, each change that applies to one or more of the following: The method according to claim 3, further comprising:

5. The scenario specifies a change that applies to one or more of the configuration parameters, and the method is Sampling the configuration for the task episode, specifying an initial value for each of the configuration parameters, and at each time step, For each of the one or more configuration parameters, the change specified by the scenario of the time step is applied to the initial value of the configuration parameter in order to generate the current value of the configuration parameter. The method according to claim 4, further comprising:

6. The aforementioned scenario specifies a change applied to one or more of the aforementioned measurements, and generates an observation from the aforementioned measurements. For each of the one or more measurements, apply the change specified by the scenario of the time step to the measurement. The method according to claim 4, including the method described in claim 4.

7. The aforementioned scenario specifies a change applied to one or more of the control inputs, and generates one or more control inputs from the action, For each of the one or more control inputs, apply the change specified by the scenario of the time step to the control input. The method according to claim 4, including the method described in claim 4.

8. The method according to claim 1, wherein the computer simulator is a deterministic simulator of the dynamics of the industrial equipment.

9. Training the control policy based at least on the aforementioned task episodes, After the training, the control policy for controlling the industrial equipment will be deployed. The method according to claim 1, further comprising:

10. Evaluating the control policy based at least on the aforementioned task episode, After the evaluation, the control policy for controlling the industrial equipment is deployed. The method according to claim 1, further comprising:

11. After the deployment of the control policy, the system receives measured values ​​of the current state of the industrial equipment from the industrial equipment, To generate a second observed value from the measured values ​​of the current state of the industrial equipment, The second observed value is provided as an input to the control policy for controlling the industrial equipment, The output from the control policy is to receive a second action for controlling one or more setpoints of the industrial equipment, From the second action described above, generate a second one or more control inputs to the one or more setpoints of the industrial equipment, Controlling the one or more setpoints of the industrial equipment based on the second one or more control inputs. The method according to claim 9, further comprising:

12. To generate the dataset, use a second control policy to control a second industrial facility. It further includes, The computer simulator of the industrial equipment is configured to generate the measured values ​​that represent the current new state of the industrial equipment based on the dataset. The method according to claim 1.

13. A system comprising one or more computers and one or more storage devices that store instructions, when executed by the one or more computers, causing the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 12.

14. One or more computer-readable storage media that, when executed by one or more computers, stores instructions causing the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 12.

Images