Simulating industrial facility for control

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for simulating an industrial facility for control. A method includes: at each of a plurality of time steps during a round of a task: receiving a measurement from a computer simulator of an industrial facility representative of a current state of the facility; generating observations from the measurements; providing the observations as input to a control strategy for controlling the facility; receiving as output an action for controlling one or more setpoints of the facility; generating one or more control inputs for one or more setpoints of the facility from the action; and providing (i) the control input and (ii) a current value of one or more configuration parameters for the simulator as inputs to the simulator to cause the simulator to generate as an output a new measurement representative of a new state of the facility.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 63 / 354,930, filed on June 23, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification relates to using machine learning models to control industrial facilities. Background Art

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs from received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network, that is, the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0005] This specification describes a system implemented as a computer program on one or more computers in one or more locations that simulates the operation of an industrial facility to allow a machine learning model to be trained to control the facility.

[0006] The subject matter described in this specification can be implemented in specific embodiments to realize one or more of the following advantages.

[0007] This specification describes techniques for training, evaluating, or training and evaluating a control strategy for an industrial facility using a computer simulation of an industrial facility. Once a control strategy has been trained and / or evaluated in a simulation, the control strategy can be deployed and used to control a (real world) industrial facility.

[0008] More specifically, given an initial configuration, the state of the industrial facility, and control inputs, the computer simulation of the industrial facility is deterministic, and the computer simulation will always update the state of the industrial facility in the same way. This can make existing frameworks for training control strategies in simulations poor choices for training control strategies for industrial facilities, because controlling industrial facilities requires control strategies that are robust to any number of real-world imperfections that can cause given control inputs to have different effects on the state of the facility. For example, sensors for the facility can be noisy or can fail, external conditions in the environment of the real-world facility can change rapidly, set points can fail, etc. The present specification describes a framework for training control strategies to be robust to such imperfections, for evaluating control strategies to determine whether the strategy is robust to such imperfections, or both, without modifying the simulator or RL agent that is performing the training. That is, the present specification describes a framework that allows deterministic simulators of industrial facilities to be effectively used to simulate the non-determinism of the real world. Specifically, by using an environment subsystem to interface between the RL agent and the simulator, the system is able to incorporate various aspects of nondeterminism into the interaction, for example, by introducing noise into the control inputs, measurements, or both, or by modifying the simulator's configuration parameters between and within mission rounds. Furthermore, the same framework can be employed to introduce these varying degrees of nondeterminism for multiple different simulators at different facilities and multiple different tasks. In particular, the framework allows for very extensive configurability - allowing users to combine tasks, simulators, scenarios, and noise, where each of these is an independent axis of configurability.

[0009] In one example described herein, a method performed by one or more computers includes: at each of a plurality of time steps during a mission round: receiving a measurement representing a current state of the industrial facility from a computer simulator of the industrial facility; generating an observation from the measurement; providing the observation as input to a control strategy for controlling the industrial facility; receiving as output an action for controlling one or more set points of the industrial facility from the control strategy; generating from the action one or more control inputs for the one or more set points of the industrial facility; and providing as input to the computer simulator (i) one or more control inputs and (ii) current values ​​of one or more configuration parameters for the computer simulator so that the computer simulator generates as output a new measurement representing a new state of the industrial facility for a subsequent time step.

[0010] Configuration parameters (in addition to specifying control inputs) may specify additional information used by the computer simulator to represent the state of the industrial facility. Some example configuration parameters are described below.

[0011] Generating observations from the measurements may include adding noise to the measurements. Generating one or more control inputs for one or more set points of the industrial facility from the actions may include adding noise to the one or more control inputs defined by the observations. The method may further include identifying a scenario for the task round. The scenario may specify, for each time step in a plurality of time steps, a corresponding modification to be applied to one or more of the following: one or more configuration parameters in the configuration parameters, one or more control inputs in the control inputs, or one or more measurement results in the measurements. The scenario may specify a modification to be applied to one or more configuration parameters in the configuration parameters.

[0012] The method may further include sampling a configuration for the task round, the configuration specifying a corresponding initial value for each of the configuration parameters. The method may include at each time step: for each of the one or more configuration parameters, applying the modification specified by the scenario for the time step to the initial value of the configuration parameter to generate a current value of the configuration parameter. The scenario may specify a modification to be applied to one or more of the measurement results. Generating observations from the measurement results may include: for each of the one or more measurement results, applying the modification specified by the scenario for the time step to the measurement results. The scenario may specify a modification to be applied to one or more of the control inputs. Generating one or more control inputs from an action may include: for each of the one or more control inputs, applying the modification specified by the scenario for the time step to the control input.

[0013] The computer simulator may be a deterministic simulator of the dynamics of the industrial facility. The method may further include training the control strategy based at least on the mission rounds; and after training, deploying the control strategy for controlling the industrial facility. The method may further include evaluating the control strategy based at least on the mission rounds; and after evaluating, deploying the control strategy for controlling the industrial facility.

[0014] The method may further include: after deploying the control strategy and based on the industrial facility, receiving a measurement result of a current state of the industrial facility; generating a second observation based on the measurement result of the current state of the industrial facility; providing the second observation as an input to the control strategy for controlling the industrial facility; receiving a second action for controlling one or more set points of the industrial facility as an output from the control strategy; generating a second one or more control inputs for one or more set points of the industrial facility based on the second action; and controlling the one or more set points of the industrial facility based on the second one or more control inputs.

[0015] The method may further include controlling a second industrial facility using a second control strategy to generate a data set; wherein the computer simulator of the industrial facility is configured to generate measurements representing a current state and a new state of the industrial facility based on the data set. The second industrial facility may be the same industrial facility as the industrial facility.

[0016] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 An example simulation system is shown.

[0018] Figure 2 A more detailed view of the simulated system is shown.

[0019] Figure 3 An example of the operation of the simulation system during a mission round is shown.

[0020] Figure 4 is a flow chart of an example process for executing a mission round using a simulator.

[0021] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0022] This specification describes a system implemented as a computer program on one or more computers in one or more locations that simulates the operation of an industrial facility while the industrial facility is being controlled by a control strategy.

[0023] In particular, the control strategy receives as input an observation characterizing a state of the industrial facility and, in response, generates an action that specifies a corresponding setting of one or more set points of the industrial facility. Each set point is a different controllable element of the industrial facility. That is, the control strategy controls the facility by repeatedly updating the settings of one or more set points of the industrial facility.

[0024] For example, a control policy can be implemented as a neural network or other machine learning model, and the system can be used to train the control policy in simulation before deploying the control policy for controlling a real-world industrial facility. For example, a control policy can be trained through reinforcement learning to maximize the reward received that represents the policy's performance of some specified task.

[0025] As another example, a control strategy, such as a trained neural network or a fixed or heuristic-based control strategy, can be used to control a system to generate a data set. The data set can then be used to train another control strategy, such as through offline reinforcement learning, without using the other control strategy to control an industrial facility. Alternatively or additionally, the data set can be used to evaluate the performance of another control strategy, such as to determine whether the control strategy is suitable for deployment in controlling a real-world industrial facility.

[0026] Typically, an industrial facility is an industrial facility that includes one or more of electronic equipment, mechanical equipment, or both that can be controlled by a control strategy. The control strategy operates to control the industrial facility to perform a specified task.

[0027] In some embodiments, the facility is a service facility including a plurality of electronic equipment items, such as a server farm or a data center, such as a telecommunications data center, or a computer data center for storing or processing data, or any service facility. The service facility may also include auxiliary control equipment for controlling the operating environment of the equipment item, such as environmental control equipment, such as temperature control, such as cooling equipment, or air flow control or air conditioning equipment. The equipment may include, for example, an air-cooled chiller, a water-cooled chiller, or both. Tasks may include tasks for controlling, for example, minimizing, the use of resources, such as tasks for controlling power consumption or water consumption when the facility is in operation. Optionally, optimization may be subject to one or more constraints.

[0028] In general, an action may be any action that has an effect on an observed state of an environment, for example, an action configured to adjust any of the sensed parameters described below. These may include actions for controlling an item of equipment or an auxiliary control item of equipment or for imposing operating conditions on an item of equipment or an auxiliary control item of equipment, such as actions that result in a change to a setting to adjust, control, or turn on or off the operation of an item of equipment or an auxiliary control item of equipment. As a specific example, an action may include an action to control one or more chillers operating within a facility.

[0029] Typically, observations of the state of the environment may include any electronic signal that represents the function of the facility or equipment in the facility. For example, representations of the state of the environment may be derived from observations made by any sensor that senses the state of the physical environment of the facility or by any sensor that senses the state of one or more items of equipment or assists in controlling the state of one or more items of equipment. These sensors include sensors configured to sense electrical conditions, such as current, voltage, power, or energy; the temperature of the facility; the flow, temperature, or pressure of fluid within the facility or within the cooling system of the facility; or physical facility configurations, such as whether a vent is open.

[0030] The reward or reward may relate to a measure of the performance of the task. For example, in the case of a task to control - e.g. minimise - the use of a resource - such as a task to control the use of electricity or water, the measure may include any measure of the use of the resource.

[0031] In some embodiments, the facility is a power generation facility, such as a renewable power generation facility, such as a solar farm or a wind farm. The task may include control tasks that control the power generated by the facility, for example, controlling the delivery of power to a distribution network, for example, to meet demand or reduce the risk of mismatches between elements of the grid, or to maximize the power generated by the facility. These actions may include actions that control the electrical or mechanical configuration of a generator, such as the electrical or mechanical configuration of one or more renewable power generation elements, for example, for controlling the configuration of a wind turbine or solar panel or multiple panels or mirrors, or the electrical or mechanical configuration of a rotating generator. Mechanical control actions may, for example, include actions that control the conversion of energy input to electrical energy output, such as the efficiency of the conversion or the degree of coupling of energy input to electrical energy output. Electrical control actions may, for example, include actions that control one or more of the voltage, current, frequency, or phase of the generated power.

[0032] The reward or reward may involve a measure of the performance of the task. For example, in the case of a task to control the delivery of electricity to a distribution network, the measure may involve a measure of the power transmitted, or a measure of an electrical mismatch between the generating facility and the grid, such as a voltage, current, frequency, or phase mismatch, or a measure of power or energy lost in the generating facility. In the case of a task to maximize the delivery of electricity to a distribution network, the measure may involve a measure of power or energy delivered to the grid, or a measure of power or energy lost in the generating facility.

[0033] In general, observations of environmental conditions may include any electronic signal representing the electrical or mechanical function of a power generating device in a power generating facility. For example, representations of environmental conditions may be derived from observations made by any sensor sensing the physical or electrical state of a device in a power generating facility that is generating electricity, or the physical environment of such a device, or the condition of an auxiliary device supporting the power generating device. Such sensors may include sensors configured to sense electrical conditions of the device, such as current, voltage, power, or energy; temperature or cooling of the physical environment; fluid flow; or the physical configuration of the equipment; and observations of electrical conditions of the power grid, such as from local or remote sensors. Observations of environmental conditions may also include one or more predictions of future conditions regarding the operation of the power generating equipment, such as predictions of future wind levels or solar irradiance or predictions of future electrical conditions of the power grid.

[0034] Figure 1is a diagram of an example simulation system 100. Simulation system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below may be implemented.

[0035] The system 100 will be described as being used to control a heating, ventilation, and air conditioning (HVAC) system 120 of a simulated industrial facility 110 .

[0036] More generally, however, the system 100 may be used to control any aspect of the operation of any type of industrial facility 110, such as one of the aspects described above.

[0037] The simulated industrial facility 110 (also referred to as simulator 110) is a computer simulation of a real-world industrial facility, i.e., the state and dynamics of the real-world industrial facility that would be observed in various situations are modeled using one or more computer programs. That is, the simulator 110 is one or more software programs that maintain the state of the real-world industrial facility, such as the current readings of sensors within the facility and optionally, such as additional information, and receives as input (i) current values ​​of configuration parameters specifying the configuration of the simulator and (ii) control inputs of one or more set points of the industrial facility, and provides as output measurement results reflecting an updated state of the facility as a result of the control inputs, i.e., updated readings of sensors within the facility.

[0038] System 100 may utilize any suitable computer simulator. For example, a user of the system may provide system 100 with access to a computer simulator of a real-world facility of interest to the user, such as by allowing system 100 to access the simulator through an API or other interface or by allowing system 100 to execute the simulator.

[0039] Typically, the problem of controlling an industrial facility to perform a specified task can be formulated as a constrained multi-objective optimization.

[0040] In the HVAC example, the controller 130 controls a plurality of set points that adjust the temperature exchange characteristics of the HVAC system 110 to perform a task, for example, attempting to maintain the facility temperature at a specific level. In the HVAC example, the set points may include enabling and disabling selected chillers, and optionally configuring the chiller leave temperature.

[0041] HVAC components draw power from the grid, so the next goal of controller 130 may be to reduce power consumption. Thus, the overall task performed by controller 130 may be structured to minimize power consumption of HVAC system 110 while satisfying one or more constraints on facility temperature.

[0042] If a controller fails at its task, the controller risks overheating the facility, which can lead to serious consequences, such as failure of computer components leading to data loss or downtime of electrical or mechanical components that are critical to the operation of the facility. To prevent such situations from occurring, manufacturers of controllers 130 introduce a set of failsafe constraints that prevent such events from occurring. Violating the constraints not only undermines the reliability of the controller, but also typically results in the controller being disconnected from the facility and no longer being able to optimize power consumption.

[0043] The system 100 can be used to provide a collection of simulation scenarios that can be used to safely and efficiently train and evaluate a controller (e.g., a control strategy implemented as a machine learning model). That is, the system 100 can be used to train, evaluate, or train and evaluate a control strategy that controls one or more of the set points specified by the controller 130 for the simulator 110.

[0044] More specifically, during operation of the system 100 , the control strategy 150 (eg, a reinforcement learning agent) uses the simulator 110 as a true value model of the facility dynamics to perform tasks in a closed-loop control system.

[0045] System 100 uses simulator 110 to evaluate the effects of actions proposed by policy 150 on the current state of the simulation.

[0046] The simulator 110 returns results in the form of measurements, which are a subset of the simulated state. The measurements may include current readings from any of a variety of sensors in the industrial facility.

[0047] The system 100 processes the measurements into observations 160 that are provided as input to the control strategy 150 .

[0048] However, HVAC simulations are deterministic, i.e., executing a given action in a given simulated state will always result in the same updated state. However, control of a real-world HVAC system requires consideration of any of the various non-deterministic elements that may be encountered during operation and that may modify how an action affects the state of the facility. Figure 2 and 3 Examples of these non-deterministic elements (also called "defects") are described in more detail.

[0049] To introduce defects, the system 100 may introduce noise into various aspects of the control pipeline, such as one or more of the control inputs, simulation configurations, and observations. This will be referred to below Figure 2 and 3 Describe in more detail.

[0050] Figure 2A more detailed view of simulation system 100 is shown.

[0051] like Figure 2 As shown, simulation system 100 includes simulator 110. In some embodiments, system 100 may also include simulator data storage 210 that stores specifications for a plurality of different simulators, for example, so that an appropriate simulator may be selected for a given task to control a given real-world facility.

[0052] During operation, simulation system 100 represents interactions with simulator 110 as interactions with environment subsystem 220 .

[0053] The environment subsystem 220 is implemented as one or more computer programs and controls the interaction of the RL agent 230 with the simulator 110. The RL agent 230 can include a control policy and associated components for training the control policy through reinforcement learning based on the interaction of the control policy with the simulator 110.

[0054] from Figure 2 As can be seen, the RL agent 230 receives observations as input and provides actions 234 as output for controlling one or more set points of the simulated facility. The input observations include environmental observations 232 and optionally include "task observations" 272, which include additional information specific to the task being performed (e.g., additional information generated from environmental observations 232 based on some task parameters).

[0055] Environmental subsystem 220 translates action 234 into control input 236 and provides control input 236 to simulator 110. For example, translating action 234 may include translating a high-level action (e.g., an indicator that a chiller should be disabled) into instructions or other commands that can be executed within the facility to perform the high-level action (e.g., machine-readable instructions to disable a chiller).

[0056] Environment subsystem 220 also provides values ​​of configuration parameters 262 as input to simulator 110 .

[0057] Configuration parameters 262 specify (in addition to specifying control inputs 236) additional information required to fully represent the state of the real-world industrial facility by simulator 110. That is, configuration parameters are parameters necessary to initialize the simulator, ie, to fully represent the state of the real-world facility.

[0058] As an example, configuration parameters 262 may specify the value of a set point that is not controlled by RL agent 250 but needs to be specified by the controller. For example, when the set point includes enabling and disabling selected chillers and configuring the chiller leave temperature, but RL agent 230 only controls enabling and disabling the chiller, the configuration parameter specifies the chiller leave temperature of the chiller.

[0059] Configuration parameters 262 also specify properties of the external environment of the real-world industrial facility. For example, configuration parameters 262 can specify the temperature of the external environment, the humidity of the external environment, the precipitation rate of the external environment, and the like.

[0060] For example, prior to beginning a mission round, the environment subsystem 220 can sample a configuration of respective initial values ​​for each of the configuration parameters of a specified configuration, e.g., configuration parameters that model a real-world configuration of a real industrial facility, from the configuration storage 264. In some cases, as will be described below, the subsystem 220 can modify the initial values ​​during the course of the mission round, while in other cases, the subsystem 220 can maintain the initial values ​​throughout the mission round.

[0061] When in the configuration specified by configuration parameters 238 , simulator 110 returns measurements 238 that reflect the updated state of simulator 110 as a result of applying control input 236 .

[0062] Environmental subsystem 220 then converts measurements 238 into observations 232 that are provided as input to RL agent 230. For example, a user may provide the system with specifications for the inputs received by RL agent 230, i.e., which sensor measurements are provided as input, the expected range of sensor measurements, the numeric format of sensor measurements, etc. Subsystem 220 can then normalize measurements 238 so that they fit the specifications of observations 232 provided by the user.

[0063] As described above, RL agent 230 controls simulator 110 to perform a specified task, e.g., optimize one or more metrics of performance according to one or more constraints. Constraints can include constraints on measurements, e.g., a temperature that does not exceed a threshold, constraints on actions, e.g., a given cooler is not enabled for more than a continuous time window, or both.

[0064] To determine whether a given action or measurement violates any constraints, system 100 includes constraint evaluator 260, which maintains data specifying the current set of constraints for the tasks performed by RL agent 230. In particular, each configuration in storage 264 is associated with a set of constraints for a given task.

[0065] Evaluator 260 receives inputs including the current action, the current set of measurements, or both, and determines whether any of the constraints in the current set of constraints specified by the configuration of the current task round have been violated. Evaluator 260 then provides data identifying whether any constraint violations have occurred to environment subsystem 220, which can provide this information to RL agent 230 as part of the corresponding observation.

[0066] The constraints for a given task can include soft constraints, hard constraints, or both.

[0067] Soft constraints are constraints that can be violated and only result in a negative impact on the evaluation of the performance of the RL agent 230 .

[0068] A hard constraint is a constraint that cannot be violated, i.e., violating the constraint causes the controller to be disconnected from the facility. When evaluator 260 determines that a hard constraint has been violated, environment subsystem 220 can terminate the current control round, i.e., provide an indication to RL agent 230 that the hard constraint has been violated and RL agent 230 can no longer continue this instance of the simulation.

[0069] In order to train the RL agent 230 or evaluate the performance of an already trained RL agent 230, the RL agent requires a training signal. In reinforcement learning, this is represented by a reward set 272, which is a numerical value generated by the task subsystem 270 based on any appropriate information, such as the results of constraint evaluations, measurements, and control inputs. The mapping from this information to one or more numerical values ​​representing rewards 272 can be specified by a user of the system 100.

[0070] RL agent 230 can use rewards 272 , environment observations 232 , and actions 234 to train a control policy using any suitable reinforcement learning technique—e.g., an on-policy or off-policy reinforcement learning algorithm.

[0071] Alternatively, as described above, RL agent 230 can store rewards 272, environment observations 232, and actions 234 for use in training another policy through offline reinforcement learning, or for evaluating another policy as described above.

[0072] Once a given policy has been trained, the given policy can be used to control a real-world facility simulated by the simulator 110 .

[0073] Instead of directly providing actions 234 as control inputs 236 and measurements 238 as observations 232 that would result in a deterministic control loop, environmental subsystem 220 uses any of a variety of components to introduce imperfections into the control process.

[0074] In particular, environment subsystem 220 may utilize one or more of scene 226 or noise generator 290 .

[0075] As a specific example, the environment subsystem 220 can use the noise generated by the noise generator 290 to add noise to control inputs as part of converting actions to control inputs, to observations as part of converting measurements to observations, or both. Noise is added to simulate sensor / pipeline imperfections and, in effect, to diversify the distribution of simulated states (control noise) and observations (observation noise). The parameters of the noise generator 290 - e.g., the parameters of the noise distribution from which the noise is sampled, the time and location at which the noise is applied, or both - can be specified by the user or sampled by the system 100 from a set of possible parameters.

[0076] The scenarios 226 model real-world scenarios of interest, and the scenarios 226 to be used for a given mission round can be specified by the user or sampled from a set of scenarios by the system 100. An example of a scenario 226 is a scenario used to model environmental instabilities during operation of a real-world facility that are not effectively captured by the operation of the simulator 110. An example of such a real-world scenario is changing weather conditions in a simulated facility environment, which can have an impact on the effects of actions on the state of the facility.

[0077] More specifically, scenarios 226 are implemented as modifications to the inputs to the simulator, i.e., configuration parameters 262 for a given task and / or control inputs provided to the simulator, modifications to the outputs of simulator 110, i.e., modifications to measurements generated by simulator 110, or both.

[0078] As described above, the configuration parameters 262 include information about the state of the facility. When a scenario 226 is selected to modify the configuration parameters, the environment subsystem 220 uses the scenario 226 to modify the configuration parameters 262 before providing the configuration parameters 262 to the simulator 110 as input. When a scenario 226 is selected to modify the control inputs, the environment subsystem 220 uses the scenario 226 to modify the control inputs before providing the control inputs to the simulator 110 as input. When a scenario 226 is selected to modify the measurement results, the environment subsystem 220 uses the scenario 226 to modify the measurement results before converting the measurement results into observations.

[0079] Thus, at each turn step, in addition to sending a control input, the environment subsystem 220 changes the value of one or more of: a selected configuration parameter 262 according to the scenario 226, the selected control input itself, or a selected measurement provided in response to the control input.

[0080] Specifically, scenario 226 can be implemented as a time-dependent function that produces a value used as a modifier for one or more of: (i) one or more configuration parameters, (ii) one or more control inputs, or (iii) one or more measurement results. That is, scenario 226 maps a time index during a control round to a corresponding modifier for one or more of (i), (ii), or (iii).

[0081] Some specific examples of scenario 226 follow.

[0082] An example of a scenario 226 is a baseline scenario that does not utilize a configuration trajectory. The baseline scenario can be used to test the performance of the agent while controlling an unperturbed simulation facility and developing a suitable baseline for comparison with other tasks. Therefore, in this scenario, the initial parameter values ​​specified by the configuration are used throughout the rounds.

[0083] Another example of scenario 226 is a sensor drift scenario. This scenario introduces temporally correlated noise into a set of selected measurement result components. For example, the components can be randomly selected at the beginning of each round. This scenario can test the resilience of the agent to partial false information.

[0084] Another example of a scenario 226 is a freeze control scenario. The scenario freezes the value of a selected control for a random amount of time. That is, instead of applying the value of the control specified by the action, the environment subsystem 220 instead samples a random time interval length and during that time interval provides the environment with the value of the selected control that was selected immediately before the start of the time interval. For example, a control may be randomly selected at the beginning of each round. This scenario can test the agent's ability to detect when the selected strategy fails and adapt by switching to an alternative.

[0085] Another example of a scenario 226 is a non-stationary dynamic scenario. The scenario uses a collection of configuration trajectories to modify selected simulation configuration parameters that represent aspects of the real-world environment of a real-world facility over the course of a round. The configuration trajectories produce changes to the selected parameters, which the subsystem 220 adds to their baseline values ​​and then passes to the simulator 110. Examples of such parameters include external ambient temperature, humidity, wind speed, precipitation, etc. This scenario can test the resilience of the agent to changing environmental conditions, building loads, and other variables outside the control domain of the agent.

[0086] Another example of a scenario is a degradation of equipment scenario. This scenario uses a collection of configuration trajectories to modify selected simulation configuration parameters that represent the efficiency or other measures of the performance of equipment in a facility, such as pumps, heat exchangers, cooling towers, chillers, etc. This scenario can test the resilience of the agent to equipment performance degradation, such as due to wear and tear during facility operation.

[0087] Thus, a given mission episode is specified by selecting simulator 110 from simulator storage 210, configuring the simulator to specify initial configuration parameter values ​​262, selecting a scenario 226, and optionally selecting noise parameters for noise generator 290. Once these are specified (e.g., sampled by the system or specified by a user), system 100 is able to execute the mission episode in order to generate training data for RL agent 230.

[0088] Figure 3 An example 300 of the operation of the simulation system 100 during a mission round is shown.

[0089] A task "episode" is a sequence of time steps in which the agent 230 controls the simulator 110. A "time step" is a time interval during which measurements are received from the simulator 110 and control inputs are provided to the simulator in response to the measurements. For example, a task episode can terminate if a predetermined number of time steps have occurred, if a hard constraint has been violated, or if an error occurs in the simulator 110.

[0090] Before initiating a mission round, the system selects a configuration 304. For example, the system can select a predetermined or randomly sampled initial configuration for the configuration parameters of the simulator 110.

[0091] The system also identifies a scenario, represented as a configuration track 302, which assigns a corresponding value or a corresponding modification to one or more of the following at each time step during the round: one or more of the configuration parameters, one or more of the control inputs, or one or more of the measurement results. That is, the scenario defines a time-dependent function for updating the initial configuration 304, the measurement results, and / or the control inputs.

[0092] At each time step, agent 230 receives observations 330 and selects actions 340 that specify values ​​for one or more set points of simulator 110 .

[0093] The motion converter 350 converts the motion 340 into a control input 354 for the simulator 110. As part of the conversion, the motion converter 350 can add noise 352 to the control input.

[0094] The simulator receives control input 354 and values ​​of configuration parameters generated by applying trajectory 302 to configuration 304 and generates measurements 306 including respective current values ​​of each sensor in a set of sensors of the facility resulting from applying control input 354 given the configuration parameter values.

[0095] An observation converter 310 (e.g., part of the environment subsystem 220) converts the measurement 306 into a next observation 330 for the agent 230. Specifically, as part of converting the observation, the converter 310 can add observation noise 312 to one or more of the measurements 306. If the scenario requires that one of the sensors have sensor drift, the observation noise 312 can reflect a specified noise reading for the selected sensor.

[0096] Constraint evaluator 260 then evaluates the control inputs generated from the actions proposed by agent 230 and the observations generated from measurements 306 to determine if any constraints are violated. Information specifying which constraints (if any) are violated can then be added to the next observation 330 before the observation 330 is passed to agent 230.

[0097] Figure 4 4 is a flow chart of an example process 400 for executing a mission round using a simulator. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a simulation system appropriately programmed according to the present specification, such as Figure 1 The simulation system 100 depicted in , is capable of performing the process 400 .

[0098] Prior to a mission round, the system can select a simulator configuration and a scenario. For example, the system can randomly sample a configuration from a set of possible configurations that model the real-world operating conditions of the facility, and can receive a scenario as user input. As another example, the system can randomly sample both configurations and scenarios.

[0099] The system then performs the following steps at each time step during a task round.

[0100] The system receives as output from the simulator measurements representing the current state of the industrial facility modeled by the simulator (step 402).

[0101] The system converts the measurements into observations (step 404).

[0102] The system provides the observations to the control policy as input (step 406). For example, the control policy can be a policy trained by an RL agent.

[0103] The system receives an action as an output from the control strategy (step 408).

[0104] The system converts the motions into control inputs for the simulator (step 410).

[0105] The system provides the current values ​​of the control inputs and configuration parameters to the simulator as input (412), ie, for generating new measurements representing the next state of the industrial facility.

[0106] This specification uses the term "configured" in connection with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0107] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions encoded on a tangible non-transitory storage medium, for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical signal, optical signal, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0108] The term "data processing apparatus" refers to data processing hardware and covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or a plurality of processors or computers. The apparatus can also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0109] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, such as files storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on one or more computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0110] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way or at all, and it can be stored on a storage device in one or more locations. Thus, for example, an index database can include multiple data collections, each of which can be organized and accessed differently.

[0111] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and run on the same computer or on multiple computers.

[0112] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or by a combination of a dedicated logic circuit and one or more programmed computers.

[0113] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Typically, the computer will also include or be operably coupled to receive data from one or more large-capacity storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or transmit data or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0114] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0115] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and receiving a response message from the user in return.

[0116] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing common and computationally intensive portions of machine learning training or production—i.e., inference—workloads.

[0117] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework).

[0118] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component, such as an application server, or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0119] The computing system can include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, for displaying data to a user interacting with the device acting as a client and receiving user input therefrom. Data generated at the user device, such as the result of a user interaction, can be received from the device at the server.

[0120] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope that can be claimed, but as descriptions of features that can be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination can be removed from the combination, and the claimed combination can involve a sub-combination or a variation of the sub-combination.

[0121] Similarly, although operations are depicted in a particular order in the drawings and recited in the claims, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0122] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: At each of a number of time steps during a mission round: receiving, from a computer simulator of an industrial facility, measurements representing a current state of the industrial facility; generating observations from the measurements; providing the observations as input to a control strategy for controlling the industrial facility; receiving as output from the control strategy an action for controlling one or more set points of the industrial facility; generating from the actions one or more control inputs for the one or more set points of the industrial facility; and (i) the one or more control inputs and (ii) current values ​​of one or more configuration parameters of the computer simulator are provided as input to the computer simulator so that the computer simulator generates as output new measurements representing a new state of the industrial facility at a subsequent time step.

2. A method according to any preceding claim, wherein: Generating observations from the measurements includes: Noise is added to the measurement.

3. A method according to any preceding claim, wherein: Generating one or more control inputs for the one or more set points of the industrial facility from the actions includes: Noise is added to one or more control inputs defined by the observations.

4. The method according to any preceding claim, further comprising: A scenario for the mission round is identified, wherein the scenario specifies, for each of the plurality of time steps, a respective modification to be applied to one or more of: One or more of the configuration parameters, one or more of the control inputs, or One or more of the measurements.

5. The method according to claim 4, wherein: The scenario specifies modifications to be applied to one or more of the configuration parameters, wherein the method further comprises: A configuration of the task round is sampled, the configuration specifying a respective initial value of each of the configuration parameters, and at each time step: For each configuration parameter of the one or more configuration parameters, a modification specified by the scenario for the time step is applied to the initial value of the configuration parameter to generate the current value of the configuration parameter.

6. The method according to claim 4 or 5, wherein: The scenario specifies a modification to be applied to one or more of the measurements, and wherein generating observations from the measurements comprises: For each of the one or more measurements, the modification specified by the scenario for that time step is applied to that measurement.

7. The method according to any one of claims 4 to 6, wherein: The scenario specifies a modification to be applied to one or more of the control inputs, and wherein generating one or more control inputs from the action comprises: For each of the one or more control inputs, the modification specified by the scenario for the time step is applied to the control input.

8. A method according to any preceding claim, wherein: The computer simulator is a deterministic simulator of the dynamics of the industrial facility.

9. The method according to any preceding claim, further comprising: training the control strategy based at least on the mission rounds; as well as After the training, the control strategy is deployed for controlling the industrial facility.

10. The method according to any preceding claim, further comprising: evaluating the control strategy based at least on the mission round; as well as After the evaluating, the control strategy is deployed for controlling the industrial facility.

11. The method according to claim 9 or 10, further comprising: After deploying the control strategy, receiving a measurement of a current state of the industrial facility from the industrial facility; generating a second observation from the measurements of the current state of the industrial facility; providing the second observation as input to the control strategy for controlling the industrial facility; receiving as an output from the control strategy a second action for controlling the one or more set points of the industrial facility; generating, from the second action, second one or more control inputs for the one or more set points of the industrial facility; as well as The one or more set points of the industrial facility are controlled based on the second one or more control inputs.

12. The method according to any preceding claim, further comprising: controlling a second industrial facility using a second control strategy to generate a set of data; Wherein the computer simulator of the industrial facility is configured to generate measurements representing a current state and a new state of the industrial facility based on the data set.

13. A system comprising one or more computers and one or more storage devices storing instructions, which when executed by the one or more computers cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1 to 12.

14. One or more computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the respective method of any one of claims 1 to 12.

Images