Hybrid reinforcement learning (RL) for controlling water distribution network

Abstract

A method and system controls a water distribution network. A database of previous states is maintained based on residential water demand, reservoir water level, and energy price. And determining the current state of the water distribution pipe network. The rewards are determined and include reservoir level constraints, energy costs, and switching counts. The query-based model is used to determine a set of control points for controlling the first previous state. The RL agent performs training based on the previous state and the reward. The RL agent determines a control setpoint (a control setpoint that varies the pump speed) that maintains the reservoir water level, minimizes energy cost, and complies with the switching count. The RL agent determines time slots and selects one of the time slots. A mixing setpoint is generated to control the water distribution network within the selected time slot.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. Section 119(e) of the following co-pending and commonly assigned U.S. provisional patent applications, which are incorporated herein by reference:

[0003] Provisional application serial number 63 / 512,795, filed on July 10, 2023, inventors are Yuan Zhou, Shu Wang, Harsh Bakulchandra Patel, Michael Pennell, Jieliang Luo and Alex Lamb, titled "Controlling Water Distribution Network with AI-Powered Controllers by Hybrid Reinforcement Learning (RL)", and attorney file number is 30566.0611USP1. Background of the Invention Technical Field

[0005] The present invention relates generally to water distribution networks, and more particularly to a method, apparatus, system, and article of manufacture for controlling a water distribution network using a hybrid approach.

[0006] 2. Description of Related Technology

[0007] Water distribution networks play a critical role in ensuring citizens have access to clean drinking water. These networks consist of a series of pipes, valves, and pumps that carry water from the source to the end user. The water distribution network must be able to adjust its capacity to meet changes in demand over time. This process is called optimization and involves finding the best solution that maximizes benefits while minimizing energy consumption costs and / or negative consequences of violating constraints.

[0008] Optimization solutions aim to provide operators with real-time optimization recommendations to follow. Generating the best solution in real-time is one of the challenges often faced. Prior art solutions are both (1) expensive (in terms of processing power and time) and (2) do not guarantee convergence.

[0009] Prior art combines clustering and genetic algorithms (GA) to reduce the number of iterations required for convergence (i.e., optimal solution). However, since the state of the plant may be complex and difficult to capture through clustering, this approach cannot always guarantee (GA) convergence. In this regard, the clustering / GA / evolutionary algorithms based on the prior art are based on population-based optimization, which requires a lot of computing power and a large number of iterations and training to identify the optimal control. In other words, the optimization system of the prior art requires a large amount of time (e.g., more than 10 hours) to converge to a feasible / optimal solution. This excessive processing time is in direct conflict with the desire for real-time recommendations for controlling the water distribution network. In addition, the prior art system is unstable and there is no guarantee that the solution will correctly converge to a feasible solution.

[0010] In addition to the fact that prior art techniques are (1) expensive and (2) do not guarantee convergence, prior art systems attempt to find a solution that makes sense for a given simulation model but may produce results that are significantly different from typical controls in the database. This significant difference leads to a reluctance to adopt the solution. In other words, the optimization produced by the prior art may be an improvement over methods used in the past, but may be significantly different (from previous uses) so that the user is reluctant to adopt the suggestion. In this regard, the optimization recommended by the prior art may not be easily accepted / understood by the user, and therefore, the user's adoption of the prior art suggestion may not be accepted and / or may be significantly delayed (e.g., the user may be reluctant to adopt the system suggestion due to the extreme difference, or may feel that if adopted, the water distribution system may fail / damage).

[0011] Therefore, we need to be able to minimize the expense of identifying an optimal solution that guarantees convergence, while also increasing user adoption. Summary of the invention

[0012] Embodiments of the present invention provide an artificial intelligence-driven controller that can handle the complexity and uncertainty of real-time disturbances by utilizing a warm-start strategy and reinforcement learning (RL).

[0013] In this regard, RL-based optimization according to embodiments of the present invention can accommodate uncertainty and train the agent's reasoning time to be within a desired time frame (e.g., within a few seconds and / or below a specified threshold time limit). Embodiments of the present invention provide an AI (artificial intelligence) driven controller by utilizing a warm start strategy and RL-based optimization to gradually introduce optimal control, which enables the final control set point to be better resistant to potential simulation errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Referring now to the drawings, wherein like reference numerals indicate corresponding parts, in which:

[0015] Figure 1 A reinforcement learning (RL) framework for a water distribution network according to one or more embodiments of the present invention is shown;

[0016] Figure 2 The structure of an Anglian water distribution network according to one or more embodiments of the present invention is shown.

[0017] Figure 3 The present invention shows one or more embodiments of the Figure 2 A more general / simplified structure of the water distribution network;

[0018] Figure 4 indicating convergence of a reward function according to one or more embodiments of the present invention;

[0019] Figure 5 shows examples of RL agent actions according to one or more embodiments of the present invention;

[0020] Figure 6 illustrates a monthly savings report from a RL agent according to one or more embodiments of the present invention;

[0021] Figure 7 shows a workflow for hybrid RL deployment according to one or more embodiments of the present invention;

[0022] Figure 8 A time slot selection strategy for hybrid RL according to one or more embodiments of the present invention is shown;

[0023] Fig. 9 A query pool update workflow according to one or more embodiments of the present invention is shown;

[0024] Fig.10 shows a general logic flow for controlling a water distribution network according to one or more embodiments of the present invention;

[0025] Fig.11 is an exemplary hardware and software environment for implementing one or more embodiments of the present invention; and

[0026] Fig.12 A typical distributed / cloud-based computer system utilized in accordance with one or more embodiments of the present invention is schematically illustrated. DETAILED DESCRIPTION

[0027] In the following description, reference is made to the accompanying drawings which form a part hereof and in which are shown by way of illustration several embodiments of the invention. It should be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the present invention.

[0028] method

[0029] The optimization problem to be solved is how to provide the operator with the best control set point based on previous experience and observation. This problem needs to be converted into a form suitable for reinforcement learning (RL). In this regard, in order to solve this problem, the embodiment of the present invention formulates the problem as a Markov decision process (MDP) and utilizes the RLGYM framework. GYM is an open source PYTHON library for developing and comparing reinforcement learning algorithms by providing a standard application programming interface (API) to communicate between the learning algorithm and the environment and a set of standard environments that conform to the API.

[0030] Figure 1 An RL framework for a water distribution network according to one or more embodiments of the present invention is shown. As shown, an RL agent 102 receives input in the form of rewards 104 and actions 106, and outputs various states 108. In turn, the states 108 are utilized by a simulator 110, which updates the rewards 104 and actions 106 fed back to the RL agent 102.

[0031] In this regard, the states 108 used by the operator to make decisions in the observation space need to be determined and may include reservoir levels and the latest pump states, forecasts of future water demand (i.e., residential water demand), and energy rate structures. Based on the desired goals, embodiments of the present invention attempt to ensure that the following three (3) goals / constraints / rewards are met in addition to system constraints: (1) desired reservoir levels; (2) maximized energy cost savings; and (3) a limit on the number of control changes utilized on a defined time basis (e.g., daily). Rewards 104 are established to accomplish / specify such constraints. Actions 106 (e.g., pump speed) are defined to control the system. RL (via RL agent 102) is trained to work with simulator 110 to maintain the current state of the site / distribution network and determine the optimal control for the desired output.

[0032] In view of the above, in the context of this problem, the reward system of GYM enables rewards 104 to be represented for different actions 106 (e.g., pump speeds) and states 108. Positive rewards can be assigned to actions 106 that lead to desired results (such as reaching a target set point or avoiding system failure). Conversely, negative rewards can be assigned to suboptimal actions. In addition, embodiments of the present invention (e.g., via GYM) enable constraints to be incorporated as conditions that must be met during the learning process. For example, the range of values ​​for a particular parameter can be limited or the frequency of certain actions can be limited. Through the flexible interface of GYM, discrete or continuous action spaces can be used to represent operator choices and adjust parameters. By utilizing GYM, optimization problems can be modeled as MDPs, representing various aspects of rewards 104, constraints, and the problem domain. This allows the development and evaluation of reinforcement learning algorithms to provide operators with optimal control set points based on their experience and observations.

[0033] In the MDP framework, the observation space contains various factors / states 108, such as reservoir water level, pump status, demand forecast (e.g., residential water demand), and energy rates. These observations / states 108 guide the decision-making process of the RL agent 102. In addition, the reward 104 in the MDP formulation of embodiments of the present invention can include reservoir water level constraints, energy costs, and switching counts, thereby providing a comprehensive evaluation of the agent's performance.

[0034] In view of the above, Figure 1 The detailed structure of the RL agent 102 is shown, which is intended to navigate the MDP environment and generate optimal control set points.

[0035] Given the distinct daily and seasonal patterns observed in energy rates and demand, control horizons can be designed / evaluated on a daily basis. This means that the optimal solution should also follow the same daily frequency. Each control set point is determined at fifteen (15) minute intervals, and the RL agent 102 interacts with the environment ninety-six (96) times to provide recommendations for the next twenty-four (24) hours.

[0036] To train the RL agent 102, two different RL algorithms can be employed: Proximal Policy Optimization (PPO) and R2D2. PPO is an online policy RL algorithm known for its stable performance, which helps to evaluate hypotheses and preliminary results. Once a minimum satisfaction threshold for the initial results is reached, embodiments of the present invention can transition to Region-based Recursive Doppler Anti-aliasing (R2D2), a distributed offline policy RL algorithm, which provides improved sampling efficiency, thereby achieving optimal performance.

[0037] environment

[0038] Figure 2 An environment for training an RL agent 102 is shown. In this regard, Figure 2 The structure of an Anglian water distribution network is shown in accordance with one or more embodiments of the present invention. As shown, there are more than four (4) control variables 202 (consisting of pump / valve stations (P0-P2 and V1)), four (4) reservoirs / storage tanks 204 (S0-S3), and several demand / demand zones 206. In addition, a measurement channel 208 and a prediction channel 210 are shown. Embodiments of the present invention provide machine learning based simulations for this complex water distribution network structure.

[0039] As shown in the figure, the total demand 212 is composed of Peter old main line 212A + Peter new main line 212B + Morcott to Peter 212C + total Beanfield flow 212D. Figure 2 In the example, the control variables 202 (P0-P2 and V1) control the flow / distribution of water in the pipe network. Therefore, Figure 2 The structure of the Anglian water distribution network is shown, where all demand areas 206 consume water and use reservoirs 204 to store water to meet demand. The entire network is designed to pump water upstream from the source on the upper left and provide water to the demand areas 206 at the end without violating operational constraints.

[0040] Based on Figure 2 Given the complexity of the problem shown in the water distribution network of FIG. , it is desirable to simplify / generalize the problem / network based on three objectives (e.g., constraint violation, energy savings, and switch count). Figure 3 The present invention shows one or more embodiments of the Figure 2 A more general / simplified structure of a water distribution network. The generalized version of the complex structure represents the combination of all storage tanks / reservoirs 204 in a single reservoir 302 (total network capacity) and all demands 206 in a single demand 304 (total network outlets / network demand). The control set point is the total inlet flow (Morcott inlet 306 (P1) + Wing inlet 308 (P2)), representing the main source of plant inlet.

[0041] There are three (3) main objectives to consider when training the RL agent 102:

[0042] 1. Constraint violation: keep the water level below the upper / lower limit;

[0043] 2. Energy savings: energy optimization with respect to rate structures; and

[0044] 3. Switch Count: Keep the change count of the control set point below the defined threshold.

[0045] Evaluation and Results

[0046] After training the RL agent 102, a series of experiments are prepared to evaluate its performance in terms of energy savings, constraint violations, and switch counts. The experiments maintain perspective by introducing various initial conditions and abnormal demand patterns.

[0047] In one or more embodiments, the AWS (AMAZON WEB SERVICES) SAGEMAKER RL estimator can be used to perform experiments. The GYM framework can be used to prepare custom environments. The following results are generated using a PPO (Proximal Policy Optimization) agent from a library compatible with the GYM framework (e.g., RLLIB). The reward function can be prepared so that the agent can correctly reinforce in the correct way for all three (3) goals.

[0048] Figure 4 represents the convergence of the reward function according to one or more embodiments of the present invention. Specifically, this set of experiments is for a simpler version of the pipe network (i.e., Figure 3 ) is executed. It can be observed that the RL training converges well and is stable. As shown in the figure, the horizontal axis provides the time range, while the vertical axis reflects the number of units of reward. As time goes by, the maximum episode reward 402 and the average episode reward 404 are close to convergence (for example, starting from around 15:35 on December 2).

[0049] Figure 5 An example of RL agent actions according to one or more embodiments of the present invention is shown. Specifically, Figure 5 The graphs shown in represent an example date analysis for May 29, 2021. The graphs are used to represent volume (graph 502), rate (graph 508), demand (graph 516), pump (graph 518), and rate channel (graph 520) as well as measurement data. Figure 5 The diagram is used to show the control situation after the model is trained and the comparison of different methods that can be used.

[0050] The first graph 502 shows a comparison between measured historical data 504 and data (representing reservoir volume) controlled by the RL 506. Overall, the RL driven water level remains within the defined constraints (140,000-180,000).

[0051] The second chart 508 shows the rate cost values ​​(i.e., energy prices) for specific dates (May 29, 2021 and May 30, 2021). It clearly shows three different rate levels: low 510, medium 512, and high 514. It is desirable to use more energy during low rate time periods than during high rate time periods because it costs less.

[0052] The third chart 516 shows the demand pattern (i.e., the amount of water required to meet residential demand) for the same dates (May 29, 2021 and May 30, 2021).

[0053] Turning to the fourth and fifth graphs 518 and 520, they depict a comparison between measured historical data 522 of the control set points for the Morcott (graph 518) and Wing Pump (graph 520) and the RL control data 524. Clearly, the RL adjusts pumping accordingly, thereby reducing pumping during high rate time periods and increasing pumping during low rate time periods.

[0054] In chart 526, a cost comparison between the measured data (measured data cost 528) and the RL driven solution (RL driven solution cost 530) can be observed. The results indicate that the overall cost of the RL driven solution is lower than the measured data, while the end state remains relatively unchanged. This indicates that RL effectively utilizes the rate structure to achieve cost savings.

[0055] A more thorough cost savings analysis was conducted for each day in 2020. Figure 6 A monthly savings report from an RL agent is shown in accordance with one or more embodiments of the present invention. A time series graph 602 represents the cost savings per day in 2020, while a bar graph 604 represents the total savings per month. The average annual savings percentage is approximately 4.5% and is evenly distributed across the months.

[0056] In view of the above, Figure 5 and Figure 6 The graphs show that RL optimization works well for both energy and constraint compliance.

[0057] deploy

[0058] In one or more embodiments, it may be desirable to utilize hybrid RL to smooth the experience of taking suggested controls. This hybrid RL based approach may be referred to as a warm start model / policy. In other words, embodiments of the invention may utilize a deployment pipeline designed to apply an RL agent in near real-time suggestions via a combination of a warm start query based policy and RL (referred to herein as a hybrid RL pipeline). Figure 7 A workflow for a hybrid RL deployment according to one or more embodiments of the present invention is shown.

[0059] The Hybrid-RL pipeline has three sections 702 to 706, the first section (referred to as the hot start pipeline 702) is used to find the K most recent set points that the client previously applied to the water distribution network state that is close to the current state of the water distribution network. In the hot start pipeline 702, the state variables 708 are fed to the query-based model 710 (which is a model separate from the simulator or RL model) to determine K control set points 712. In other words, the current measured state 708 is passed to the query-based model 710. The query-based model 710 provides a best guess, which attempts to estimate whether the operator will make a decision in a similar situation (i.e., what control the operator is likely to apply in a similar situation).

[0060] The second part is called hybrid RL 704, which uses the best guess (i.e., k control set points 712 from the query-based model 710) to determine which controls to provide to the operator within a defined time period (e.g., the next 24 hours). The hybrid RL 704 will first perform a time slot selection 714, where a subset of the next 24-hour time period is selected based on different criteria for the RL agent to generate set points to combine with the results obtained from the hot start pipeline 702. In other words, the time slot selection 714 provides for selecting a time slot (i.e., time / time period) and the RL agent generates a hybrid set point 716. In this regard, based on the K control set points 712 (i.e., the best guess), an algorithm is applied to determine which portion (e.g., time slot) is most valuable to change. Under the direct RL model, everything will change. However, in hybrid RL, the time slot / portion that is most likely to provide the most value to optimize savings can be selected. Therefore, embodiments of the present invention identify and select the most valuable time slot / portion.

[0061] In addition to the above, different threshold limits may be utilized to limit the search for vulnerable time slots. For example, embodiments of the present invention may limit the start of controls to a maximum of two (2) hours that are easily acceptable / adopted by the operator. After the initial time slot / period, a single different control schedule may be utilized for the subsequent 24 hours. In other words, the most vulnerable controls that are likely to be accepted / adopted by the operator are identified and provided during the initial time period. Thereafter, embodiments of the present invention may gradually transition to providing more suggestions from the RL system (to slowly get the user to adopt the suggested changes / controls).

[0062] Figure 8 An example of criteria that may be used to select and combine set points for a 1:96 time period is shown. More specifically, Figure 8 A time slot selection strategy for hybrid RL according to one or more embodiments of the present invention is shown. Figure 8The time slot strategy shown in provides a gradual introduction of more and more changes that are offered / proposed / suggested to the operator. For example, only 1-2 changes may be suggested at the beginning, and over time, more and more changes are suggested / proposed. As shown at 802, the 1:5 time slot is selected using a time period based on a constraint time slot selection strategy (e.g., keeping the reservoir water level below an upper / lower limit) (6:96 of the total time slot is left). At 804, the 21:45 time slot is selected using a savings-based time period selection strategy (e.g., optimizing energy / cost savings for the tariff structure), and at 806, the 41:59 time slot is selected using a switching count-based time slot selection strategy (e.g., keeping the change count of the control set point below a defined threshold). Finally, embodiments of the present invention can place hybrid control set points into a simulation engine and use an objective function to select the optimal set point for real-time recommendation (i.e., dynamically recommended when controlling the distribution network).

[0063] Once the deployment pipeline is operational, the system will monitor compliance with set points in a real-time / dynamic manner based on data (e.g., the actual time an event occurs, such as distributing water through the distribution network and / or using different controls when distributing water).

[0064] Fig. 9 FIG. 2 shows a query pool update workflow according to one or more embodiments of the present invention. Fig. 9 and Figure 7 , the seed pool 902 used by the query model in the warm start pipeline 702 may be updated with the previous hybrid RL 704 setpoints (i.e., if the client follows the final setpoints 722). In these cases, the system can automatically / autonomously increase the portion of the setpoint 722 subset from the RL 704, thereby improving the user experience because the new setpoint pattern / new setpoints 722 from the RL 704 are slowly introduced based on what the user / client is already familiar with. In other words, as the operator / client begins to apply the recommended solution / control, embodiments of the present invention will continuously and automatically / autonomously slowly introduce more and more changes (e.g., at a defined rate and / or based on a schedule / progressive schedule) to increase the operator / client's comfort with the recommended changes.

[0065] Embodiments of the invention may also enable different goals to be specified for the algorithm (e.g., reducing risk to the system and / or saving energy). Different goals / objectives may have different impacts and may establish priorities / preferences. For example, if the focus is on a reservoir level constraint, and the system determines that a certain time period (such as 1-2 p.m.) may breach an upper limit on the reservoir level under a certain control, embodiments of the invention may focus on the control during that time period as the most vulnerable portion and provide recommendations to reduce the likelihood of breaching the limit during that time period.

[0066] General logic flow

[0067] In view of the above, Fig.10 A general logic flow for controlling a water distribution network according to one or more embodiments of the present invention is shown.

[0068] At step 1002, a database is maintained. The database contains / includes two or more previous states of the water distribution network. The two or more previous states are based on: (1) residential water demand; (2) reservoir water level; and (3) an energy rate that defines the energy cost of using energy during a specified time period.

[0069] At step 1004, the current state of the water distribution network is determined.

[0070] At step 1006, one or more rewards are determined. The reward includes / contains (1) a reservoir level constraint specifying a minimum or maximum reservoir level; (2) an energy cost; (3) a switching count of the number of control set point changes. As used herein, a control set point controls a pump speed.

[0071] At step 1008, a determination is made via a query-based model accessing a database. The determination includes: (1) determining a first previous state from two or more previous states based on a comparison with a current state; and (2) a set of control set points that a previous user applied to the first previous state.

[0072] At step 1010, a reinforcement learning (RL) agent is trained based on two or more previous states and one or more rewards. The RL agent is trained to determine control set points in real time when utilizing water in a water distribution network: (i) maintaining a reservoir water level to comply with a reservoir water level constraint; (ii) minimizing energy costs based on energy rates and residential water demand; and (iii) complying with a switching count. It may also be noted that the control set points change pump speeds to transition the water distribution network to a new state and result in one or more of the one or more rewards.

[0073] In addition to the above, the RL agent is trained to (1) determine two or more time slots within a defined time period (where each of the two or more time slots is based on a different criterion); (2) select a first time slot (i.e., the most valuable time slot / portion as described above) of the two or more time slots based on a set of control set points; and (3) generate one or more hybrid set points to control the water distribution network within the first time slot (where the hybrid set points are based on the set of control points). In one or more embodiments, the first time slot is selected based on determining which of the two or more time slots is most likely to result in a maximum optimization of one or more rewards. Furthermore, the RL agent may select the first time slot within a defined threshold time range.

[0074] The training of the RL agent may also include the ability to enable the selection of one or more different objectives. Each of the one or more selectable different objectives focuses on a different set of one or more rewards (e.g., a combination of residential water demand and reservoir water level, a combination of reservoir water level and energy rates, based only on residential water demand, etc.).

[0075] After the first time slot, a schedule may be utilized to provide different hybrid set points. The schedule may define a gradual transition to provide more and more hybrid set points generated by the RL agent.

[0076] At step 1012, the water distribution network is controlled based on one or more mixing set points.

[0077] The system can also monitor compliance of one or more hybrid set points to one or more rewards in real-time based on data (e.g., from the control of the water distribution network). Based on the compliance, a previously user-applied set of control set points in the database can be updated by adding one or more hybrid set points (i.e., adding to the set of control points in the database). Then, a portion of the one or more hybrid set points generated by the RL agent (for controlling the water distribution network) can be autonomously increased (i.e., a larger portion / proportion of the hybrid set points generated by the RL agent can be provided as recommended actions / control points).

[0078] The control of the water distribution network may also include inputting one or more mixing set points into a simulation engine. The simulation engine simulates the impact of using the one or more mixing set points on the water distribution network. Based on the simulation, an objective function may be used to dynamically recommend in real time when water flows through the water distribution network one of the one or more mixing set points. Such an objective function may be based on compliance with one or more rewards.

[0079] Control of the water distribution network may also include dynamically recommending mixing set points to users in real time as water flows through the water distribution network. Usage of the recommended mixing set points may then be monitored. A database may then be updated based on the usage, such that the RL agent may utilize the updated database to further control the water distribution network.

[0080] Hardware Environment

[0081] Fig.11 1 is an exemplary hardware and software environment 1100 (referred to as a computer-implemented system and / or computer-implemented method) for implementing one or more embodiments of the present invention. The hardware and software environment includes a computer 1102 and may include peripheral devices. The computer 1102 may be a user / client computer, a server computer, or may be a database computer. The computer 1102 includes a hardware processor 1104A and / or a dedicated hardware processor 1104B (hereinafter collectively referred to as processor 1104 in the alternative) and a memory 1106, such as a random access memory (RAM). The computer 1102 may be coupled to and / or integrated with other devices, including input / output (I / O) devices, such as a keyboard 1114, a cursor control device 1116 (e.g., a mouse, a pointing device, a pen and tablet computer, a touch screen, a multi-touch device, etc.), and a printer 1128. In one or more embodiments, the computer 1102 may be coupled to or may include a portable or media viewing / listening device 1132 (e.g., an MP3 player, an iPod, a NOOK, a portable digital video player, a cellular device, a personal digital assistant, etc.). In yet another embodiment, the computer 1102 may include a multi-touch device, a mobile phone, a gaming system, an Internet-enabled television, a television set-top box, or other Internet-enabled device executing on a variety of platforms and operating systems.

[0082] In one embodiment, computer 1102 operates by hardware processor 1104A executing instructions defined by computer program 1110 (e.g., a computer-aided design [CAD] application) under the control of operating system 1108. Computer program 1110 and / or operating system 1108 may be stored in memory 1106 and may interface with users and / or other devices to accept input and commands and, based on such input and commands and the instructions defined by computer program 1110 and operating system 1108, provide output and results.

[0083] The output / result may be presented on the display 1122 or provided to another device for presentation or further processing or action. In one embodiment, the display 1122 includes a liquid crystal display (LCD) having a plurality of individually addressable liquid crystals. Alternatively, the display 1122 may include a light emitting diode (LED) display having a red, green, and blue diode cluster driven together to form a full-color pixel. Each liquid crystal or pixel of the display 1122 changes to an opaque or translucent state in response to the processor 1104 applying the instructions of the computer program 1110 and / or the operating system 1108 to the data or information generated by the input and command to form a part of the image on the display. The image may be provided by a graphical user interface (GUI) module 1118. Although the GUI module 1118 is depicted as a separate module, the instructions for executing the GUI functions may reside or be distributed in the operating system 1108, the computer program 1110, or be implemented with a dedicated memory and processor.

[0084] In one or more embodiments, the display 1122 is integrated with / integrated into the computer 1102 and includes a multi-touch device having a touch sensing surface (e.g., a track pod or a touch screen) having the ability to recognize the presence of two or more contact points in contact with the surface. Examples of multi-touch devices include mobile devices (e.g., IPHONE, NEXUS S, DROID devices, etc.), tablet computers (e.g., IPAD, HP TOUCHPAD, SURFACE devices, etc.), portable / handheld game / music / video player / console devices (e.g., IPOD TOUCH, MP3 players, NINTENDO SWITCH, PLAYSTATION PORTABLE, etc.), touch tables, and touch walls (e.g., where images are projected through acrylic and / or glass and then illuminated with LED backlights).

[0085] Some or all of the operations performed by the computer 1102 according to the computer program 1110 instructions may be implemented in a special purpose processor 1104B. In this embodiment, some or all of the computer program 1110 instructions may be implemented via firmware instructions stored in a read-only memory (ROM), a programmable read-only memory (PROM), or a flash memory within the special purpose processor 1104B or stored in the memory 1106. The special purpose processor 1104B may also be hardwired by circuit design to perform some or all of the operations to implement the present invention. In addition, the special purpose processor 1104B may be a hybrid processor that includes dedicated circuits for performing a subset of functions and other circuits for performing more general functions (such as responding to computer program 1110 instructions). In one embodiment, the special purpose processor 1104B is an application specific integrated circuit (ASIC).

[0086] The computer 1102 may also implement a compiler 1112 that allows an application or computer program 1110 written in a programming language such as C, C++, assembly language, SQL, PYTHON, PROLOG, MATLAB, RUBY, RAILS, HASKELL, or other language to be translated into processor 1104 readable code. Alternatively, the compiler 1112 may be an interpreter that directly executes instruction / source code, translates source code into an intermediate representation that is executed, or executes stored precompiled code. Such source code may be written in a variety of programming languages ​​such as JAVA, JAVASCRIPT, PERL, BASIC, etc. Upon completion, the application or computer program 1110 uses the relationships and logic generated using the compiler 1112 to access and manipulate data received from the I / O devices and stored in the memory 1106 of the computer 1102.

[0087] Computer 1102 may also optionally include external communications means such as a modem, satellite link, Ethernet card, or other means for accepting input from other computers 1102 and providing output to other computers.

[0088] In one embodiment, the instructions implementing the operating system 1108, computer program 1110, and compiler 1112 are tangibly embodied in a non-transitory computer-readable medium (e.g., data storage device 1120), which may include one or more fixed or removable data storage devices, such as a zip drive, a floppy disk drive 1124, a hard drive, a CD-ROM drive, a tape drive, etc. In addition, the operating system 1108 and computer program 1110 consist of computer program 1110 instructions that, when accessed, read, and executed by a computer 1102, cause the computer 1102 to perform the steps necessary to implement and / or use the present invention, or load a program of instructions into the memory 1106, thereby creating a special data structure, thereby causing the computer 1102 to operate as a specially programmed computer to perform the method steps described herein. The computer program 1110 and / or operating instructions may also be tangibly embodied in the memory 1106 and / or the data communication device 1130, thereby making a computer program product or article of manufacture according to the present invention. Thus, the terms "article of manufacture," "program storage device," and "computer program product" as used herein are intended to encompass a computer program accessible from any computer-readable device or media.

[0089] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices, may be used with the computer 1102 .

[0090] Fig.12 A typical distributed / cloud-based computer system 1200 is schematically shown, which uses a network 1204 to connect a client computer 1202 to a server computer 1206. A typical combination of resources may include: a network 1204, which includes the Internet, a LAN (local area network), a WAN (wide area network), an SNA (system network architecture) network, etc.; a client 1202, which is a personal computer or workstation (such as Fig.11 ); and server 1206, which is a personal computer, workstation, minicomputer or mainframe computer (such as Fig.11 However, it may be noted that different networks, such as a cellular network (e.g., GSM [Global System for Mobile Communications] or other), a satellite-based network, or any other type of network, may be used to connect the client 1202 and the server 1206 according to embodiments of the present invention.

[0091] A network 1204 (such as the Internet) connects the client 1202 to the server computer 1206. The network 1204 may utilize Ethernet, coaxial cable, wireless communication, radio frequency (RF), etc. to connect the client 1202 and the server 1206 and provide communication therebetween. In addition, in a cloud-based computing system, the client 1202, the server computer 1206, and users may share resources (e.g., storage devices, processors, applications, memory, infrastructure, etc.) in the client 1202 and the server computer 1206 across one or more networks. Resources may be shared by multiple users and may be dynamically reallocated as needed. In this regard, cloud computing may be referred to as a model that enables access to a shared pool of configurable computing resources.

[0092] The client 1202 can execute a client application or a web browser and communicate with a server computer 1206 executing a web server 1210. Such a web browser is typically a program such as MICROSOFT INTERNET EXPLORER / EDGE, MOZILLA FIREFOX, OPERA, APPLE SAFARI, GOOGLE CHROME, etc. In addition, the software executed on the client 1202 can be downloaded from the server computer 1206 to the client computer 1202 and installed as a plug-in or ACTIVEX control of the web browser. Therefore, the client 1202 can use the ACTIVEX component / component object model (COM) or distributed COM (DCOM) component to provide a user interface on the display of the client 1202. The web server 1210 is typically a program such as MICROSOFT'S INTERNET INFORMATION SERVER.

[0093] The web server 1210 can host an active server page (ASP) or Internet server application programming interface (ISAPI) application 1212 that can execute scripts. The script calls the object (called business object) that executes the business logic. Then, the business object manipulates the data in the database 1216 through the database management system (DBMS) 1214. Alternatively, the database 1216 may be part of the client 1202 or directly connected to the client, rather than communicating / obtaining information from the database 1216 across the network 1204. When developers encapsulate business functions into objects, the system can be called a component object model (COM) system. Therefore, the script executed on the web server 1210 (and / or application 1212) calls the COM object that implements the business logic. In addition, the server 1206 can use MICROSOFT'S TRANSACTION SERVER (MTS) to access the required data stored in the database 1216 via an interface (such as ADO (active data objects), OLE DB (object linking and embedding database) or ODBC (open database connection)).

[0094] In general, these components 1200-1216 include logic and / or data embedded in and / or retrievable from a device, medium, signal, or carrier (e.g., a data storage device, a data communication device, a remote computer or a device coupled to a computer via a network or via another data communication device, etc.) Further, such logic and / or data, when read, executed, and / or interpreted, causes the steps required to implement and / or use the present invention to be performed.

[0095] Although the terms "user computer," "client computer," and / or "server computer" are referred to herein, it should be understood that such computers 1202 and 1206 may be interchangeable and may also include thin client devices with limited or full processing capabilities, portable devices (such as cell phones, notebook computers, pocket computers, multi-touch devices), and / or any other device with suitable processing, communication, and input / output capabilities.

[0096] Of course, those skilled in the art will recognize that any combination of the above components, or any number of different components, peripherals, and other devices may be used with computers 1202 and 1206. Embodiments of the present invention are implemented as a software / CAD application on a client 1202 or server computer 1206. Furthermore, as described above, the client 1202 or server computer 1206 may comprise a thin client device or a portable device with a multi-touch based display.

[0097] in conclusion

[0098] This concludes the description of the preferred embodiments of the present invention. Some alternative embodiments for implementing the present invention are described below. For example, any type of computer (such as a mainframe, minicomputer, or personal computer) or computer configuration (such as a time-sharing mainframe, a local area network, or a stand-alone personal computer) can be used with the present invention. In summary, embodiments of the present invention provide a hybrid RL-based method for providing real-time optimal control set points for water distribution. By using the initial water level and demand forecast as the observation space, embodiments of the present invention can find the optimal set points that save costs while maintaining constraints. In terms of deployment, a query-based hot start method is combined with an RL agent to gradually introduce the optimal RL set points based on its current operating routine, which also makes the final set points more robust to potential simulation errors.

[0099] The foregoing description of the preferred embodiments of the present invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive of the present invention or to limit the present invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The scope of the present invention is intended not to be limited by this specific embodiment, but rather by the appended claims.

Claims

1. A computer-implemented method for controlling a water distribution network, comprising: (a) maintaining a database of two or more previous states of the water distribution network, wherein the two or more previous states are based on: (1) Residential water demand; (2) water level in the reservoir; and (3) an energy rate that defines the energy cost of using energy during a specified period of time; (b) determining a current state of the water distribution network; (c) determine one or more awards, including: (1) Reservoir water level constraint, specifying the minimum or maximum water level of the reservoir; (2) the energy cost; and (3) a switching count of the number of control set point changes, wherein the control set point controls the pump speed; (d) determining via a query-based model accessing said database: (1) determining a first previous state from the two or more previous states based on a comparison with the current state; (2) the set of control set points previously applied by a user to the first previous state; (e) training a reinforcement learning (RL) agent based on the two or more previous states and the one or more rewards, wherein: (1) When water in the water distribution network is utilized, the RL agent determines the control set point in real time: (i) maintaining the water level of the reservoir to comply with the water level constraints of the reservoir; (ii) minimizing the energy costs based on the energy rates and the residential water demand; and (iii) complying with the switching count; wherein the control set point changes the pump speed to cause the water distribution network to transition to a new state and result in one or more of the one or more rewards; and (2) RL Agent: (1) determining two or more time slots within a defined time period, wherein each of the two or more time slots is based on a different criterion; (2) selecting a first time slot of two or more time slots based on the set of control set points; and (3) generating one or more hybrid set points to control the water distribution network during the first time slot, wherein the hybrid set points are based on the set of control points; and (f) controlling the water distribution network based on the one or more mixing set points.

2. The computer-implemented method of claim 1 , wherein the selection of the first time slot is based on a determination of which of the two or more time slots is most likely to result in a maximum optimization of the one or more rewards.

3. The computer-implemented method of claim 1 , wherein: The RL agent selects the first time slot within a defined threshold time range.

4. The computer-implemented method of claim 1 , further comprising: After the first time slot, a schedule is utilized to provide a different mixing set point.

5. The computer-implemented method of claim 4, wherein the schedule defines a gradual transition to provide an increasing number of hybrid set points generated by the RL agent.

6. The computer-implemented method of claim 1 , further comprising: monitoring compliance of the one or more blending set points with the one or more rewards in real time based on the data; updating the set of control set points previously applied by the user in the database by adding the one or more mixing set points based on the compliance condition; Autonomously increasing a portion of the one or more mixing set points generated by the RL agent for controlling the water distribution network.

7. The computer-implemented method of claim 1 , further comprising: Enabling selection of one or more different objectives, wherein each of the one or more different objectives focuses on a different set of the one or more rewards.

8. The computer-implemented method of claim 1, further comprising: inputting the one or more mixing set points into a simulation engine; The simulation engine simulates the effects of using the one or more mixing set points on the water distribution network; Based on the simulation, one of the one or more mixing set points is selected to dynamically recommend in real time as water flows through the water distribution network using an objective function, wherein the objective function is based on compliance with the one or more rewards.

9. The computer-implemented method of claim 1, wherein said controlling said water distribution network comprises: dynamically recommending the mixing set point to a user in real time as water flows through the water distribution network; Monitors usage of recommended mix set points; updating the database based on the usage; and The RL agent further controls the water distribution network using the updated database.

10. A computer-implemented system for controlling a water distribution network, comprising: (a) a computer having a memory; (b) a processor executing on said computer; (c) the memory stores an instruction set, wherein the instruction set, when executed by the processor, causes the processor to perform operations, the operations comprising: (1) maintaining a database of two or more previous states of the water distribution network, wherein the two or more previous states are based on: (i) Residential water demand; (ii) water levels in reservoirs; and (iii) an energy rate that defines the energy cost of using energy during a specified period of time; (2) determining the current state of the water distribution network; (3) Determine one or more awards, including: (i) a reservoir water level constraint, specifying a minimum water level or a maximum water level of the reservoir; (ii) the cost of the energy; and (iii) a switching count of the number of times a control set point changes, wherein the control set point controls a pump speed; (2) Determining via a query-based model accessing the database: (i) determining a first previous state from the two or more previous states based on a comparison with the current state; (ii) said set of control set points previously applied by a user to said first previous state; (3) training a reinforcement learning (RL) agent based on the two or more previous states and the one or more rewards, wherein: (i) The RL agent determines the control set point in real time as water in the water distribution network is utilized: (A) maintaining the water level of the reservoir to comply with the water level constraints of the reservoir; (B) minimizing said energy costs based on said energy rates and said residential water demand; and (C) complying with the switching count; wherein the control set point changes the pump speed to cause the water distribution network to transition to a new state and result in one or more of the one or more rewards; and (ii) the RL agent: (A) determining two or more time slots within a defined time period, wherein each of the two or more time slots is based on a different criterion; (B) selecting a first time slot of the two or more time slots based on the set of control set points; and (C) generating one or more hybrid set points to control the water distribution network during the first time slot, wherein the hybrid set points are based on the set of control points; and (4) Controlling the water distribution network based on the one or more mixing set points.

11. The computer-implemented system of claim 10, wherein the selection of the first time slot is based on a determination of which of the two or more time slots is most likely to result in a maximum optimization of the one or more rewards.

12. The computer-implemented system of claim 10, wherein: The RL agent selects the first time slot within a defined threshold time range.

13. The computer implemented system of claim 10, wherein the operations further comprise: After the first time slot, a schedule is utilized to provide a different mixing set point.

14. The computer-implemented system of claim 13, wherein the schedule defines a gradual transition to provide an increasing number of hybrid set points generated by the RL agent.

15. The computer implemented system of claim 10, wherein the operations further comprise: monitoring compliance of the one or more blending set points with the one or more rewards in real time based on the data; updating the set of control set points previously applied by the user in the database by adding the one or more mixing set points based on the compliance condition; Autonomously increasing a portion of the one or more mixing set points generated by the RL agent for controlling the water distribution network.

16. The computer implemented system of claim 10, wherein the operations further comprise: Enabling selection of one or more different objectives, wherein each of the one or more different objectives focuses on a different set of the one or more rewards.

17. The computer implemented system of claim 10, wherein the operations further comprise: inputting the one or more mixing set points into a simulation engine; The simulation engine simulates the effects of using the one or more mixing set points on the water distribution network; Based on the simulation, one of the one or more mixing set points is selected to dynamically recommend in real time as water flows through the water distribution network using an objective function, wherein the objective function is based on compliance with the one or more rewards.

18. The computer implemented system of claim 10, wherein said controlling said water distribution network comprises: dynamically recommending the mixing set point to a user in real time as water flows through the water distribution network; Monitors usage of recommended mix set points; updating the database based on the usage; and The RL agent further controls the water distribution network using the updated database.

Images