Urban water service dynamic scheduling optimization method and device based on hierarchical reinforcement learning

Abstract

The application provides a city water affair dynamic scheduling optimization method and device based on hierarchical reinforcement learning, relates to the field of intelligent water affair scheduling control, and aims to improve the processing efficiency and environmental benefits of city water affair dynamic scheduling. The method comprises the following steps: acquiring city water affair data, and determining city summary data of a whole basin and local water affair data corresponding to a plurality of sub-basins included in the whole basin; inputting the city summary data of the whole basin into a global coordination layer in a city water affair scheduling model, and outputting scheduling coefficients, target water level intervals and target water quality intervals corresponding to the plurality of sub-basins respectively; inputting the local water affair data, the scheduling coefficients, the target water level intervals and the target water quality intervals into a local execution layer in the city water affair scheduling model, and outputting sewage plant pump start-stop, aeration intensity adjustment, effluent flow setting, pipe network regional pump station start-stop, valve opening degree control, river gate opening and closing and storage tank inflow and outflow control corresponding to the plurality of sub-basins respectively.

Description

Technical Field

[0001] This application relates to the field of smart water management scheduling and control, and more specifically, to a method and apparatus for optimizing urban water dynamic scheduling based on hierarchical reinforcement learning. Background Technology

[0002] Urban water management scheduling refers to the dynamic allocation and control of water resources in water supply and drainage by coordinating various water-related facilities to ensure urban water security and smooth drainage. Related technologies often employ hierarchical management and independent control (e.g., sewage treatment plants, pipeline pumping stations, and river gates are independently controlled), and rely on fixed rules or linear optimization models. This can lead to problems such as delayed overflows, insufficient local water storage, or excessive energy consumption at pumping stations under complex dynamic conditions (e.g., rainfall fluctuations, nonlinear hydrodynamic responses). Summary of the Invention

[0003] This application provides a method and apparatus for dynamic scheduling optimization of urban water affairs based on hierarchical reinforcement learning, aiming to overcome the above-mentioned problems or at least partially solve them.

[0004] The first aspect of this application provides a method for dynamic scheduling optimization of urban water resources based on hierarchical reinforcement learning, including: Acquire urban water affairs data, and based on the urban water affairs data, determine the overall urban data for the entire basin and the local water affairs data corresponding to each of the multiple sub-basins included in the entire basin. The urban water affairs data includes at least: influent and effluent flow rates of sewage treatment plants, water quality concentration, water level at pipeline nodes, pump power, gate opening, river flow velocity, river water level, and rainfall. The city aggregate data of the entire basin is input into the global coordination layer of the pre-trained urban water affairs scheduling model, and the scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-basins are output. The local water affairs data, scheduling coefficients, target water level ranges and target water quality ranges corresponding to the multiple sub-basins are input into the local execution layer of the pre-trained urban water affairs scheduling model, and the corresponding sewage treatment plant pump start-up and shutdown, aeration intensity adjustment, effluent flow setting, pipeline area pump station start-up and shutdown, valve opening control, river gate opening and closing and storage tank inflow and outflow control are output for the multiple sub-basins. The urban water affairs scheduling model adopts a hierarchical reinforcement learning architecture, which divides the dynamic scheduling of urban water affairs into a global coordination layer and a local execution layer. The global coordination layer is responsible for the goals of water level safety, pollution reduction and minimizing water affairs operation energy consumption in the whole basin, while the local execution layer is responsible for the control actions in the scheduling strategy.

[0005] In one alternative implementation, the global coordination layer is constructed according to the following steps: Define a first state of the global coordination layer, which includes at least: total water inflow, total system energy consumption, river water level margin, and pollution load from the aggregated urban data of the entire watershed; Define the first action of the global coordination layer, which includes at least: the scheduling coefficient, target water level and target water quality range of each sub-basin; Define the first update cycle of the global coordination layer; Construct a first agent for the global coordination layer, the first agent being used to select a first action based on the current first state; Establish the first reward function for the first agent of the global coordination layer: ; Among them, R H First reward; H r H represents the real-time water level of the river channel. r tar C is the safe water level for the river channel. f E represents the concentration of wastewater effluent from the wastewater treatment plant. total The total energy consumption of the system is denoted as α, β, and γ, which are pre-set target weight coefficients. The overall objective of the reward function of the global coordination layer is to minimize energy consumption while maintaining a safe water level in the river and reducing pollution.

[0006] In one alternative implementation, the local execution layer is constructed according to the following steps: Define a second state for the local execution layer, the second state including at least: local water resources data for each sub-basin; Define the actions of the local execution layer, the second action including at least: starting and stopping the sewage treatment plant pumps in each sub-basin, adjusting the aeration intensity, setting the effluent flow rate, starting and stopping the pumping stations in the pipeline area, controlling the valve opening degree, opening and closing the river gates and controlling the inflow and outflow of the storage tank; A second update cycle is defined for the local execution layer, which is shorter than the first update cycle of the global coordination layer. Construct multiple second agents in the local execution layer, each of which selects its own second action based on its current second state. Establish a second reward function for each second agent in the local execution layer: ; in, The second reward is the second reward for the i-th second agent corresponding to the i-th sub-basin; Let be the node water level of the i-th sub-basin. The target water level for the i-th sub-basin; Let be the local effluent concentration of the i-th sub-basin; The water quality compliance threshold for the i-th sub-basin; ω represents the local pump power consumption of the i-th sub-basin; i ω is the scheduling coefficient of the i-th sub-basin output by the global coordination layer. i，H ω i，C ω i，E The sub-scheduling coefficient is the output of the i-th sub-basin of the global coordination layer.

[0007] In one alternative implementation, the urban water management model is trained according to the following steps: The first agent of the global coordination layer selects a first action based on the current first state; The multiple second agents in the local execution layer select a second action based on their respective current second state and first action; Input the current first state, the current second state, and the second action into the agent environment model, and output the next first state and second state, as well as the first reward parameter and the second reward parameter; The first agent of the global coordination layer calculates the first reward based on the first reward parameter and the first reward function; The multiple second agents in the local execution layer calculate their respective second rewards based on their respective second reward parameters and second reward functions; Based on the first reward and the second reward, the total loss is calculated according to the total loss function, and the network parameters of the first agent and the network parameters of the plurality of second agents are updated according to the total loss to obtain the trained urban water management scheduling model. The total loss function incorporates multiple constraints, including at least: pump power constraints, river water level constraints, and wastewater treatment plant effluent compliance constraints. The total loss function is as follows: ; Where L is the total loss; L RL For -J _RL J _RL P represents the cumulative reward; λ represents the weighted gain; p For pump power, P max For pump power limits; H r It is the real-time water level of the river channel, H r tar This is the safe water level of the river channel; C f C represents the concentration of the wastewater effluent from the wastewater treatment plant. f tar This is the threshold for water quality to meet standards.

[0008] In one optional implementation, the agent environment model includes a multi-layer long short-term memory network and a graph neural network; the agent environment model is trained according to the following steps: Offline simulations of a small number of typical working conditions were performed using a mechanistic hydrodynamic water quality model to obtain training samples. The training samples include: the current state of the sample, the second action of the sample, the state of the next sample, and the reward of the next sample. The agent environment model to be trained is trained based on the current state of the sample and the second action of the sample in the training samples. Based on the predicted state and predicted reward output by the agent environment model to be trained for a training sample, as well as the corresponding state and reward for the next sample, the model parameters of the agent environment model to be trained are updated to obtain the trained agent environment model.

[0009] In one optional implementation, the method further includes: Real-time monitoring of target indicators, including: system energy consumption, effluent water quality, and water level fluctuation; When one or more of the target indicators are greater than the preset thresholds corresponding to each indicator, the urban water management scheduling model is retrained until all target indicators are lower than their respective preset thresholds and the total loss of the model converges and stabilizes, thus obtaining the updated urban water management scheduling model.

[0010] A second aspect of this application provides a dynamic scheduling optimization device for urban water affairs based on hierarchical reinforcement learning, comprising: The acquisition module is used to acquire urban water affairs data and, based on the urban water affairs data, determine the urban summary data of the entire basin and the local water affairs data corresponding to each of the multiple sub-basins included in the entire basin. The urban water affairs data includes at least: sewage treatment plant influent and effluent flow rate, water quality concentration, water level at pipeline nodes, pump power, gate opening, river flow velocity, river water level, and rainfall. The first processing module is used to input the urban aggregate data of the entire basin into the global coordination layer of the pre-trained urban water affairs scheduling model, and output the scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-basins. The second processing module is used to input the local water affairs data, scheduling coefficients, target water level ranges and target water quality ranges corresponding to the multiple sub-basins into the local execution layer of the pre-trained urban water affairs scheduling model, and output the sewage treatment plant pump start-up and shutdown, aeration intensity adjustment, effluent flow setting, pipeline area pump station start-up and shutdown, valve opening control, river gate opening and closing and storage tank inflow and outflow control corresponding to the multiple sub-basins. The urban water affairs scheduling model adopts a hierarchical reinforcement learning architecture, which divides the dynamic scheduling of urban water affairs into a global coordination layer and a local execution layer. The global coordination layer is responsible for the goals of water level safety, pollution reduction and minimizing water affairs operation energy consumption in the whole basin, while the local execution layer is responsible for the control actions in the scheduling strategy.

[0011] According to a third aspect of this application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning described in the first aspect.

[0012] A fourth aspect of this application provides a readable storage medium storing a program or instructions thereon, which, when executed by a processor, implements the steps of the urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning described in the first aspect.

[0013] According to a fifth aspect of this application, a program product is provided, including a program that, when executed by a processor, implements the steps of the urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning described in the first aspect.

[0014] Beneficial technical effects: This application employs a hierarchical reinforcement learning architecture for urban water management scheduling decisions, enabling dynamic and interconnected control of urban water treatment plants, networks, and rivers, thereby improving treatment efficiency and environmental benefits. The reinforcement learning approach allows for interactive learning with the environment, automatically adapting to complex conditions such as rainfall fluctuations and nonlinear hydrodynamic changes, effectively reducing delayed overflows and localized water shortages caused by response lags or inappropriate decisions. Furthermore, the complex joint scheduling of water treatment plants, networks, and rivers is broken down into high-level global coordination and low-level local execution—the global coordination layer and local execution layer of the urban water management scheduling model—reducing the dimensions of states and actions, and improving model training efficiency, stability, and real-time scheduling capabilities. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the steps of a dynamic scheduling optimization method for urban water resources based on hierarchical reinforcement learning proposed in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of an urban water dynamic scheduling optimization device based on hierarchical reinforcement learning proposed in an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device according to this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] Reference Figure 1 , Figure 1 This is a flowchart illustrating the steps of a dynamic urban water management scheduling optimization method based on hierarchical reinforcement learning proposed in an embodiment of this application. Figure 1 As shown, specifically, the method includes the following steps S1~S3: Step S1: Obtain urban water affairs data, and based on the urban water affairs data, determine the urban summary data for the entire basin and the local water affairs data corresponding to each of the multiple sub-basins included in the entire basin. The urban water affairs data includes at least: sewage treatment plant influent and effluent flow rates, water quality concentration, water level at pipeline nodes, pump power, gate opening, river flow velocity and water level, and rainfall.

[0019] In this embodiment, real-time and historical water management data are collected through an urban water management monitoring system to obtain urban water management data. The urban water management monitoring system includes various sensors and monitoring instruments deployed on facilities such as sewage treatment plants, pipeline pumping stations, and river gates. Urban water management data includes: influent and effluent flow rates from sewage treatment plants. Water quality concentration Pipeline node water level Pump power Gate opening River flow velocity River water level and rainfall The urban water affairs data is time-standardized and missing data is imputed to form a state vector, thus obtaining the processed urban water affairs data. Statistical analysis of the processed urban water affairs data yields summary urban data reflecting the overall operation of the entire watershed, such as total inflow, total system energy consumption, river water level margin, and pollution load. Based on the physical topology of the urban water affairs monitoring system, the entire urban watershed is divided into multiple sub-watersheds. For each sub-watershed, its corresponding local water affairs data is extracted from the processed urban water affairs data.

[0020] Step S2: Input the aggregated urban data of the entire watershed into the global coordination layer of the pre-trained urban water management scheduling model, and output the scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-watersheds.

[0021] In this embodiment, the urban water management scheduling model adopts a hierarchical reinforcement learning architecture, dividing the dynamic scheduling of urban water affairs into a global coordination layer and a local execution layer. The global coordination layer is responsible for the goals of water level safety, pollution reduction, and minimizing water operation energy consumption across the entire basin. It makes decisions based on the macroscopic state of the entire basin through a first intelligent agent. The aggregated urban data of the entire basin is input into the pre-trained global coordination layer of the urban water management scheduling model, which outputs the scheduling coefficients, target water level ranges, and target water quality ranges for each of the multiple sub-basins. These can serve as macroscopic goals to guide the local execution layer in performing fine-grained control.

[0022] Step S3: Input the local water affairs data, scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-basins into the local execution layer of the pre-trained urban water affairs scheduling model, and output the sewage treatment plant pump start-up and shutdown, aeration intensity adjustment, effluent flow setting, pipeline area pump station start-up and shutdown, valve opening control, river gate opening and closing and storage tank inflow and outflow control corresponding to each of the multiple sub-basins.

[0023] In this embodiment, the local execution layer of the urban water management scheduling model is responsible for the control actions in the scheduling strategy (such as starting and stopping sewage treatment plant pumps and opening and closing river gates). The local execution layer includes multiple second agents, each corresponding to a sub-basin and responsible for the specific equipment control within that sub-basin. The local water management data, scheduling coefficients, target water level ranges, and target water quality ranges corresponding to each of the multiple sub-basins are input into the pre-trained local execution layer of the urban water management scheduling model, outputting a dynamic urban water management scheduling strategy. The dynamic urban water management scheduling strategy includes: starting and stopping sewage treatment plant pumps, adjusting aeration intensity, setting effluent flow rate, starting and stopping pump stations in the pipeline network area, controlling valve opening, opening and closing river gates, and controlling the inflow and outflow of storage tanks, etc., for each of the multiple sub-basins.

[0024] This application employs a hierarchical reinforcement learning architecture for urban water management scheduling decisions, enabling dynamic and interconnected control of urban water treatment plants, networks, and rivers, thereby improving treatment efficiency and environmental benefits. The reinforcement learning approach allows for interactive learning with the environment, automatically adapting to complex conditions such as rainfall fluctuations and nonlinear hydrodynamic changes, effectively reducing delayed overflows and localized water shortages caused by response lags or inappropriate decisions. Furthermore, the complex joint scheduling of water treatment plants, networks, and rivers is broken down into high-level global coordination and low-level local execution—the global coordination layer and local execution layer of the urban water management scheduling model—reducing the dimensions of states and actions, and improving model training efficiency, stability, and real-time scheduling capabilities.

[0025] In one implementation, the "global coordination layer in the urban water management model" in step S2 above is constructed according to the following steps S21-S25: Step S21: Define the first state of the global coordination layer, which includes at least: the total inflow, total system energy consumption, river water level margin, and pollution load in the urban aggregate data of the entire watershed; Step S22: Define the first action of the global coordination layer, the first action including at least: the scheduling coefficient, target water level and target water quality range of each sub-basin; Step S23: Define the first update cycle of the global coordination layer; Step S24: Construct the first agent of the global coordination layer, the first agent being used to select a first action based on the current first state; Step S25: Establish the first reward function for the first agent of the global coordination layer: ; Among them, R H First reward; H r H represents the real-time water level of the river channel. r tar C is the safe water level for the river channel. f E represents the concentration of wastewater effluent from the wastewater treatment plant. total The total energy consumption of the system is denoted as α, β, and γ, which are pre-set target weight coefficients. The overall objective of the reward function of the global coordination layer is to minimize energy consumption while maintaining a safe water level in the river and reducing pollution.

[0026] In this embodiment, the global coordination layer in the urban water management scheduling model is defined, including its state input, action output, reward function, and update cycle, as well as the construction of an intelligent agent. State Input (First State): This is the input information upon which the global coordination layer intelligent agent makes decisions. It consists of aggregated urban data for the entire watershed, such as total inflow, total system energy consumption, river water level margin, and pollution load. Action Output (First Action): The scheduling coefficient, target water level, and target water quality range for each sub-watershed. First Intelligent Agent: Constructed using a deep reinforcement learning algorithm, such as Proximal Policy Optimization (PPO), it is responsible for macro-level scheduling decisions across the entire watershed and outputs the first action based on the current input first state. First Update Cycle: The time interval for iteratively updating the network parameters of the first intelligent agent. The first update cycle is relatively long, such as TH=30min. First Reward Function: This drives the global coordination layer to make decisions towards the global optimum. Its expression is as follows: ; Among them, R H First reward; H r H represents the real-time water level of the river channel. r tar C is the safe water level for the river channel. f E represents the concentration of wastewater effluent from the wastewater treatment plant.total The total energy consumption of the system is denoted as α, β, and γ, which are pre-set target weight coefficients. The overall objective of the reward function of the global coordination layer is to minimize energy consumption while maintaining a safe water level in the river and reducing pollution.

[0027] In one implementation, the "local execution layer in the urban water management model" in step S3 above is constructed according to the following steps S31 to S35: Step S31: Define the second state of the local execution layer, the second state including at least: local water resources data of each sub-basin; Step S32: Define the actions of the local execution layer, the second action including at least: starting and stopping sewage treatment plant pumps, adjusting aeration intensity, setting effluent flow rate, starting and stopping pump stations in the pipeline area, controlling valve opening, opening and closing river gates and controlling the inflow and outflow of the regulating tank; Step S33: Define a second update cycle for the local execution layer, wherein the second update cycle of the local execution layer is less than the first update cycle of the global coordination layer; Step S34: Construct multiple second agents in the local execution layer, wherein the multiple second agents are used to select their respective second actions according to their current second states; Step S35: Establish the second reward function for each second agent in the local execution layer: ; in, The second reward is the second reward for the i-th second agent corresponding to the i-th sub-basin; Let be the node water level of the i-th sub-basin. The target water level for the i-th sub-basin; Let be the local effluent concentration of the i-th sub-basin; The water quality compliance threshold for the i-th sub-basin; ω represents the local pump power consumption of the i-th sub-basin; i ω is the scheduling coefficient of the i-th sub-basin output by the global coordination layer. i，H ω i，C ω i，E The sub-scheduling coefficient is the output of the i-th sub-basin of the global coordination layer.

[0028] In this embodiment, for the local execution layer in the urban water management scheduling model, its state input, action output, reward function, and update cycle are defined, and intelligent agents are constructed. State input (second state): This is the input information upon which each sub-agent in the local execution layer bases its decisions; it consists of local water management data corresponding to multiple sub-basins. Action output (second action): This includes the start / stop of sewage treatment plant pumps, aeration intensity adjustment, effluent flow setting, start / stop of regional pumping stations in the pipeline network, valve opening control, river gate opening / closing, and inflow / outflow control of regulating reservoirs for each sub-basin. Multiple second agents: Constructed using deep reinforcement learning algorithms, such as the Deep Deterministic Policy Gradient (DDPG) algorithm. Each second agent corresponds to a sub-basin and is responsible for the specific equipment control within that sub-basin, outputting a second action based on the current input second state. Second update cycle: The time interval for iterative updates of the second agent network parameters. The second update cycle is relatively short, such as TH = 1 / 5 min. Because the local hydraulic processes change rapidly and require high-frequency control, the second update cycle of the local execution layer is set to be shorter than the first update cycle of the global coordination layer. The second reward function drives the sub-agents of the local execution layer to achieve their control objectives; its expression is as follows: ; in, The second reward is the second reward for the i-th second agent corresponding to the i-th sub-basin; Let be the node water level of the i-th sub-basin. The target water level for the i-th sub-basin; Let be the local effluent concentration of the i-th sub-basin; The water quality compliance threshold for the i-th sub-basin; ω represents the local pump power consumption of the i-th sub-basin; i ω is the scheduling coefficient of the i-th sub-basin output by the global coordination layer. i，H ω i，C ω i，E The sub-scheduling coefficient is the output of the i-th sub-basin of the global coordination layer.

[0029] In one implementation, the "urban water management model" in steps S2 and S3 above is trained according to the following steps: The first agent of the global coordination layer selects a first action based on the current first state; The multiple second agents in the local execution layer select a second action based on their respective current second state and first action; Input the current first state, the current second state, and the second action into the agent environment model, and output the next first state and second state, as well as the first reward parameter and the second reward parameter; The first agent of the global coordination layer calculates the first reward based on the first reward parameter and the first reward function; The multiple second agents in the local execution layer calculate their respective second rewards based on their respective second reward parameters and second reward functions; Based on the first reward and the second reward, the total loss is calculated according to the total loss function, and the network parameters of the first agent and the network parameters of the plurality of second agents are updated according to the total loss to obtain the trained urban water management scheduling model. The total loss function incorporates multiple constraints, including at least: pump power constraints, river water level constraints, and wastewater treatment plant effluent compliance constraints. The total loss function is as follows: ; Where L is the total loss; L RL For -J _RL J _RL P represents the cumulative reward; λ represents the weighted gain; p For pump power, P max For pump power limits; H r It is the real-time water level of the river channel, H r tar This is the safe water level of the river channel; C f C represents the concentration of the wastewater effluent from the wastewater treatment plant. f tar This is the threshold for water quality to meet standards.

[0030] In this embodiment, at the start of training, the network parameters of the global coordination layer and the local execution layer are initialized. The current aggregated urban data for the entire watershed is input as the current first state into the global coordination layer of the urban water management model. The first agent in the global coordination layer outputs a first action based on the current first state. The local water management data of each sub-watershed is used as the current second state, and the first action and the current second state are input into the local execution layer of the urban water management model. Each second agent in the local execution layer outputs a second action for that sub-watershed based on the second state and the state and action for that sub-watershed in the first action. The second actions of all sub-watersheds, i.e., the urban water management dynamic scheduling strategy, along with the current first and second states, are input into the agent environment model, outputting the next first and second states, as well as the first and second reward parameters. The first reward parameters refer to the parameters required to calculate the first reward function, including real-time river water level, wastewater treatment plant effluent concentration, and total system energy consumption. Similarly, the second reward parameters include node water level, local effluent concentration, and local pump power consumption.

[0031] The first agent in the global coordination layer calculates its first reward for the current step using its first reward function based on the first reward parameters returned by the agent environment model. Each second agent in the local execution layer calculates its second reward for the current step using its second reward function based on the second reward parameters of the corresponding sub-basin returned by the agent environment model. Based on the first and second rewards, the total loss is calculated according to the total loss function, and the network parameters of the first agent and multiple second agents are updated according to the total loss to obtain the trained urban water management scheduling model.

[0032] This application takes into account that reinforcement learning algorithms aim to maximize long-term cumulative reward, denoted as J. _RL To facilitate training using gradient descent, we take the negative of the result to obtain the reinforcement learning loss L. _RL =-J _RL This is used to adjust the policy network parameters of the agent. Based on this, to ensure physical and water quality safety, constraints such as power, water level, and water quality are directly incorporated into the differentiable loss function, maintaining these constraints during training without requiring post-training pruning. The network loss function is embedded for the following constraints: Pump power constraint: Pp <= Pmax; River water level constraint Hr <= Hsafe; Wastewater treatment plant effluent compliance constraint Cf <= Cstd; The total loss function is: ; Where L is the total loss; LRL is -J_RL, and J_RL is the cumulative reward; λ is the weight gain; Pp is the pump power, and Pmax is the pump power limit; Hr is the real-time water level of the river, and Hrtar is the safe water level of the river; Cf is the effluent concentration of the wastewater treatment plant, and Cftar is the water quality compliance threshold. The gradients of the above constraint terms participate in the backpropagation process, forming a constraint-driven gradient update mechanism: during the training phase, the network parameters respond instantly to violations, and the policy update direction is pushed towards the physical and water quality feasible regions, avoiding invalid searches and accelerating convergence.

[0033] In one implementation, the agent environment model includes a multi-layer long short-term memory network and a graph neural network. The aforementioned "agent environment model" is trained according to the following steps: Offline simulations of a small number of typical working conditions were performed using a mechanistic hydrodynamic water quality model to obtain training samples. The training samples include: the current state of the sample, the second action of the sample, the state of the next sample, and the reward of the next sample. The agent environment model to be trained is trained based on the current state of the sample and the second action of the sample in the training samples. Based on the predicted state and predicted reward output by the agent environment model to be trained for a training sample, as well as the corresponding state and reward for the next sample, the model parameters of the agent environment model to be trained are updated to obtain the trained agent environment model.

[0034] In this embodiment, considering the long computation time of mechanistic hydrodynamic water quality models, such as the SWMM (Storm Water Management Model), a proxy environment model is constructed to replace the mechanistic hydrodynamic water quality model, thereby improving the training speed of the urban water management model. The proxy environment model includes a multilayer long short-term memory (LSTM) network and a graph neural network (GNN). The GNN is used to capture the spatial topological features of the pipe network, sewage treatment plant, and river nodes, while the LSTM network is used to capture the dynamic temporal features of the node states changing over time. Representative typical operating conditions are selected, such as scenarios under different rainfall intensities (light rain, heavy rain, torrential rain) and different seasons (dry season, rainy season). Offline simulations of a small number of typical operating conditions are performed using the mechanistic hydrodynamic water quality model to obtain training samples. Each training sample includes: the current state of the sample, the second action of the sample, the state of the next sample, and the reward of the next sample. The current state of a sample includes: the current first state and the current second state; the next sample state includes: the next sample first state and the next sample second state; the next sample reward includes: the next sample first reward and the next sample second reward. The current state and the second action of the training samples are input into the agent environment model to be trained. Based on the predicted state and predicted reward output by the agent environment model for a training sample, as well as the corresponding next sample state and next sample reward, the model parameters of the agent environment model to be trained are updated until the model prediction error converges or the maximum number of training iterations is reached, resulting in a trained agent environment model. In the online phase, the agent model can receive industrial data streams for rolling updates, achieving continuous learning.

[0035] In one implementation, the urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning further includes the following steps S5-S6: Step S5: Monitor target indicators in real time, including: system energy consumption, effluent water quality and water level fluctuation; Step S6: When one or more of the target indicators are greater than the preset thresholds corresponding to each indicator, the urban water management scheduling model is retrained until all target indicators are lower than their respective preset thresholds and the total loss of the model converges and stabilizes, thus obtaining the updated urban water management scheduling model.

[0036] In this embodiment, the optimal urban water management scheduling strategy output by the trained urban water management scheduling model—namely, the start / stop of sewage treatment plant pumps, adjustment of aeration intensity, setting of effluent flow rate, start / stop of pipeline area pumping stations, valve opening control, opening and closing of river gates, and control of inflow and outflow of regulating reservoirs for each of the multiple sub-basins—is sent to the sewage treatment plant control system, pipeline pumping stations, and river gates, while real-time monitoring of target indicators. Target indicators include: system energy consumption, effluent quality, and water level fluctuations. The real-time monitored target indicators are compared with preset thresholds corresponding to each indicator. When one or more target indicators exceed their respective preset thresholds, it indicates that the current scheduling strategy may no longer be able to adapt to changes in system status, automatically triggering retraining of the urban water management scheduling model. Retraining continues until all target indicators are below their respective preset thresholds, and the model's total loss function value converges and stabilizes, at which point an updated urban water management scheduling model is obtained. Through online real-time monitoring and adaptive retraining mechanisms, the model is continuously updated based on measured data, automatically adapting to changes in basin conditions during long-term operation, ensuring the long-term stability and effectiveness of the scheduling strategy.

[0037] Based on the same inventive concept, one embodiment of this application provides a dynamic scheduling optimization device for urban water affairs based on hierarchical reinforcement learning, such as... Figure 2 As shown, the device includes: The acquisition module is used to acquire urban water affairs data and, based on the urban water affairs data, determine the urban summary data of the entire basin and the local water affairs data corresponding to each of the multiple sub-basins included in the entire basin. The urban water affairs data includes at least: sewage treatment plant influent and effluent flow rate, water quality concentration, water level at pipeline nodes, pump power, gate opening, river flow velocity, river water level, and rainfall. The first processing module is used to input the urban aggregate data of the entire basin into the global coordination layer of the pre-trained urban water affairs scheduling model, and output the scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-basins. The second processing module is used to input the local water affairs data, scheduling coefficients, target water level ranges and target water quality ranges corresponding to the multiple sub-basins into the local execution layer of the pre-trained urban water affairs scheduling model, and output the sewage treatment plant pump start-up and shutdown, aeration intensity adjustment, effluent flow setting, pipeline area pump station start-up and shutdown, valve opening control, river gate opening and closing and storage tank inflow and outflow control corresponding to the multiple sub-basins. The urban water affairs scheduling model adopts a hierarchical reinforcement learning architecture, which divides the dynamic scheduling of urban water affairs into a global coordination layer and a local execution layer. The global coordination layer is responsible for the goals of water level safety, pollution reduction and minimizing water affairs operation energy consumption in the whole basin, while the local execution layer is responsible for the control actions in the scheduling strategy.

[0038] Based on the same inventive concept, one embodiment of this application provides an electronic device, such as... Figure 3 As shown. The electronic device includes a memory and a processor, which are connected via a bus. The memory stores programs or instructions that can be executed on the processor to implement the steps in the urban water dynamic scheduling optimization method based on hierarchical reinforcement learning described in any of the above embodiments of this application.

[0039] Based on the same inventive concept, this disclosure also provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps in the urban water dynamic scheduling optimization method based on hierarchical reinforcement learning described in any of the above embodiments of this application.

[0040] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0041] Based on the same inventive concept, this disclosure also provides a program product, including a program that, when executed by a processor, can perform the steps in the urban water dynamic scheduling optimization method based on hierarchical reinforcement learning described in any of the above embodiments of this application.

[0042] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0043] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0044] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0045] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0046] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0047] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0048] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0049] The above provides a detailed description of the urban water dynamic scheduling optimization method and apparatus based on hierarchical reinforcement learning provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for dynamic scheduling and optimization of urban water resources based on hierarchical reinforcement learning, characterized in that, include: Acquire urban water affairs data, and based on the urban water affairs data, determine the overall urban data for the entire basin and the local water affairs data corresponding to each of the multiple sub-basins included in the entire basin. The urban water affairs data includes at least: influent and effluent flow rates of sewage treatment plants, water quality concentration, water level at pipeline nodes, pump power, gate opening, river flow velocity, river water level, and rainfall. The city aggregate data of the entire basin is input into the global coordination layer of the pre-trained urban water affairs scheduling model, and the scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-basins are output. The local water affairs data, scheduling coefficients, target water level ranges and target water quality ranges corresponding to the multiple sub-basins are input into the local execution layer of the pre-trained urban water affairs scheduling model, and the corresponding sewage treatment plant pump start-up and shutdown, aeration intensity adjustment, effluent flow setting, pipeline area pump station start-up and shutdown, valve opening control, river gate opening and closing and storage tank inflow and outflow control are output for the multiple sub-basins. The urban water affairs scheduling model adopts a hierarchical reinforcement learning architecture, which divides the dynamic scheduling of urban water affairs into a global coordination layer and a local execution layer. The global coordination layer is responsible for the goals of water level safety, pollution reduction and minimizing water affairs operation energy consumption in the whole basin, while the local execution layer is responsible for the control actions in the scheduling strategy.

2. The urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning according to claim 1, characterized in that, The global coordination layer is constructed according to the following steps: Define a first state of the global coordination layer, which includes at least: total water inflow, total system energy consumption, river water level margin, and pollution load from the aggregated urban data of the entire watershed; Define the first action of the global coordination layer, which includes at least: the scheduling coefficient, target water level and target water quality range of each sub-basin; Define the first update cycle of the global coordination layer; Construct a first agent for the global coordination layer, the first agent being used to select a first action based on the current first state; Establish the first reward function for the first agent of the global coordination layer: ； Among them, R H First reward; H r H represents the real-time water level of the river channel. r tar C is the safe water level for the river channel. f E represents the concentration of wastewater effluent from the wastewater treatment plant. total The total energy consumption of the system is denoted as α, β, and γ, which are pre-set target weight coefficients. The overall objective of the reward function of the global coordination layer is to minimize energy consumption while maintaining a safe water level in the river and reducing pollution.

3. The urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning according to claim 2, characterized in that, The local execution layer is constructed according to the following steps: Define a second state for the local execution layer, the second state including at least: local water resources data for each sub-basin; Define the actions of the local execution layer, the second action including at least: starting and stopping the sewage treatment plant pumps in each sub-basin, adjusting the aeration intensity, setting the effluent flow rate, starting and stopping the pumping stations in the pipeline area, controlling the valve opening degree, opening and closing the river gates and controlling the inflow and outflow of the storage tank; A second update cycle is defined for the local execution layer, which is shorter than the first update cycle of the global coordination layer. Construct multiple second agents in the local execution layer, each of which selects its own second action based on its current second state. Establish a second reward function for each second agent in the local execution layer: ； in, The second reward is the second reward for the i-th second agent corresponding to the i-th sub-basin; Let be the node water level of the i-th sub-basin. The target water level for the i-th sub-basin; Let be the local effluent concentration of the i-th sub-basin; The water quality compliance threshold for the i-th sub-basin; ω represents the local pump power consumption of the i-th sub-basin; i ω is the scheduling coefficient of the i-th sub-basin output by the global coordination layer. i，H ω i，C ω i，E The sub-scheduling coefficient is the output of the i-th sub-basin of the global coordination layer.

4. The urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning according to claim 3, characterized in that, Train the urban water management model according to the following steps: The first agent of the global coordination layer selects a first action based on the current first state; The multiple second agents in the local execution layer select a second action based on their respective current second state and first action; Input the current first state, the current second state, and the second action into the agent environment model, and output the next first state and second state, as well as the first reward parameter and the second reward parameter; The first agent of the global coordination layer calculates the first reward based on the first reward parameter and the first reward function; The multiple second agents in the local execution layer calculate their respective second rewards based on their respective second reward parameters and second reward functions; Based on the first reward and the second reward, the total loss is calculated according to the total loss function, and the network parameters of the first agent and the network parameters of the plurality of second agents are updated according to the total loss to obtain the trained urban water management scheduling model. The total loss function incorporates multiple constraints, including at least: pump power constraints, river water level constraints, and wastewater treatment plant effluent compliance constraints. The total loss function is as follows: ； Where L is the total loss; L RL For -J _RL J _RL P represents the cumulative reward; λ represents the weighted gain; p For pump power, P max For pump power limits; H r It is the real-time water level of the river channel, H r tar This is the safe water level of the river channel; C f C represents the concentration of the wastewater effluent from the wastewater treatment plant. f tar This is the threshold for water quality to meet standards.

5. The urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning according to claim 4, characterized in that, The agent environment model includes a multi-layer long short-term memory network and a graph neural network; Train the agent environment model according to the following steps: Offline simulations of a small number of typical working conditions were performed using a mechanistic hydrodynamic water quality model to obtain training samples. The training samples include: the current state of the sample, the second action of the sample, the state of the next sample, and the reward of the next sample. The agent environment model to be trained is trained based on the current state of the sample and the second action of the sample in the training samples. Based on the predicted state and predicted reward output by the agent environment model to be trained for a training sample, as well as the corresponding state and reward for the next sample, the model parameters of the agent environment model to be trained are updated to obtain the trained agent environment model.

6. The urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning according to claim 4, characterized in that, The method further includes: Real-time monitoring of target indicators, including: system energy consumption, effluent water quality, and water level fluctuation; When one or more of the target indicators are greater than the preset thresholds corresponding to each indicator, the urban water management scheduling model is retrained until all target indicators are lower than their respective preset thresholds and the total loss of the model converges and stabilizes, thus obtaining the updated urban water management scheduling model.

7. A dynamic scheduling and optimization device for urban water affairs based on hierarchical reinforcement learning, characterized in that, include: The acquisition module is used to acquire urban water affairs data and, based on the urban water affairs data, determine the urban summary data of the entire basin and the local water affairs data corresponding to each of the multiple sub-basins included in the entire basin. The urban water affairs data includes at least: sewage treatment plant influent and effluent flow rate, water quality concentration, water level at pipeline nodes, pump power, gate opening, river flow velocity, river water level, and rainfall. The first processing module is used to input the urban aggregate data of the entire basin into the global coordination layer of the pre-trained urban water affairs scheduling model, and output the scheduling coefficients, target water level ranges and target water quality ranges corresponding to each of the multiple sub-basins. The second processing module is used to input the local water affairs data, scheduling coefficients, target water level ranges and target water quality ranges corresponding to the multiple sub-basins into the local execution layer of the pre-trained urban water affairs scheduling model, and output the sewage treatment plant pump start-up and shutdown, aeration intensity adjustment, effluent flow setting, pipeline area pump station start-up and shutdown, valve opening control, river gate opening and closing and storage tank inflow and outflow control corresponding to the multiple sub-basins. The urban water affairs scheduling model adopts a hierarchical reinforcement learning architecture, which divides the dynamic scheduling of urban water affairs into a global coordination layer and a local execution layer. The global coordination layer is responsible for the goals of water level safety, pollution reduction and minimizing water affairs operation energy consumption in the whole basin, while the local execution layer is responsible for the control actions in the scheduling strategy.

8. An electronic device, characterized in that, It includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein when the program or instructions are executed by the processor, they implement the steps of the urban water dynamic scheduling optimization method based on any one of claims 1-5.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions, which, when executed by a processor, implement the steps of the urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning as described in any one of claims 1-5.

10. A program product, characterized in that, The program, when executed by a processor, implements the steps of the urban water affairs dynamic scheduling optimization method based on hierarchical reinforcement learning as described in any one of claims 1-5.

Images