A city real-time drainage scheduling method and system based on deep reinforcement learning

Abstract

The present application relates to the technical field of urban drainage and waterlogging prevention, and particularly relates to a real-time urban drainage scheduling method and system based on deep reinforcement learning. The method comprises the following steps: constructing a double-objective reward function and constraint conditions for minimizing the total amount of surface waterlogging and the overflow frequency of combined sewer outfall; simulating rainfall runoff by using a hydrodynamic mechanism model combined with the basic data of the target area; establishing a multi-agent model based on deep reinforcement learning; constructing the interactive coupling of the decision action and the simulated environment state; obtaining the updated environment state and the drainage scheduling reward value through the coupling; driving the multi-agent model to generate a strategy network; predicting the drainage and waterlogging prevention engineering action based on real-time rainstorm information; and dynamically updating the action by using the periodically generated strategy network. The present application ensures that the multi-objective optimization decision is quickly realized within the response time, achieves the effect of waterlogging relief, enhances the urban rainstorm resilience, and meets the real-time demand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of urban drainage and waterlogging prevention, and in particular to a real-time urban drainage scheduling method and system based on deep reinforcement learning. BACKGROUND

[0002] Implementing the storage and drainage joint regulation of the urban rainwater drainage system is a key measure to alleviate urban waterlogging, and the accuracy and efficiency of the regulation largely determine the overall rainwater resilience of the city. However, the regulation is limited by complex factors such as scale, facility correlation (pipe network drainage, pump station pumping, storage tank storage), hydraulic characteristics and economic cost, and the timeliness requirement of the regulation, making it difficult to develop a scientific joint regulation strategy within a limited response time.

[0003] Currently, most of the calculation methods for the optimal scheduling of urban rainwater drainage systems rely on the combination of numerical simulation and non-gradient optimization, and in the case of complex multi-objective scheduling problems, the optimization may fall into a local optimum and there may be few feasible solutions. In addition, with the advancement of the "four pre" intelligent water conservancy system, real-time optimal scheduling of the rainwater drainage system has become a development trend, which puts forward higher requirements for timeliness. Therefore, it is urgent to use new technologies to perform real-time optimal scheduling of the urban rainwater drainage system. SUMMARY

[0004] The technical problem to be solved by the embodiments of the present application is to provide a real-time urban drainage scheduling method and system based on deep reinforcement learning, to solve the problem that the prior art lacks optimal scientific decisions for waterlogging relief, cost reduction and efficiency improvement, etc. in the optimal scheduling of urban rainwater drainage systems.

[0005] The present application discloses a real-time urban drainage scheduling method based on deep reinforcement learning, comprising:

[0006] Minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined drainage outlet are dual objectives, and the state and action of the drainage scheduling are determined, the reward function and the constraint condition of the drainage scheduling are constructed;

[0007] Obtain the basic data of the target area, and based on the constructed reward function and constraint condition, use a hydrodynamic mechanism model to simulate rainfall runoff in the target area based on the obtained basic data, and obtain drainage scheduling data generated in the rainfall runoff simulation process to establish an offline database;

[0008] A multi-agent model is established based on a deep reinforcement learning algorithm, and the data of the offline database is used to train the multi-agent model;

[0009] According to the training process of the multi-agent model, an interactive coupling relationship between an environment state and an output decision action of the multi-agent model in a rainfall runoff simulation process is established, and an updated environment state and a drainage scheduling reward value in the rainfall runoff simulation process are obtained through the interactive coupling relationship, and the multi-agent model dynamically generates a policy network according to the updated environment state and the drainage scheduling reward value;

[0010] A real-time storm prediction scheduling scheme is obtained, the storm prediction scheduling scheme is optimized through the periodically generated policy network, a control instruction is generated according to the optimized scheduling scheme to execute a drainage and waterlogging prevention engineering action, and the policy network, the control instruction and the drainage and waterlogging prevention engineering action are added to the offline database for online updating.

[0011] Optionally, the double objectives of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined system drainage outlet are determined, and the state and action of the drainage scheduling are determined, and a reward function and a constraint condition of the drainage scheduling are constructed, including:

[0012] The state of the drainage scheduling is determined, and the corresponding drainage scheduling action is determined according to the current state of the drainage scheduling;

[0013] The constraint condition is set according to the regulation and control effect of the determined drainage scheduling action on the total amount of surface waterlogging and the overflow frequency, and the double objectives of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined system drainage outlet are constructed, and a reward function of the drainage scheduling is constructed, and the function expression of the reward function is:

[0014]

[0015] In the formula, is a reward value of the drainage scheduling, is a waterlogging point number, is a total number of waterlogging points, is a maximum waterlogging amount of the i th waterlogging point, is a combined system overflow drainage outlet number, is a total number of combined system overflow drainage outlets; is an overflow frequency of the j th combined system overflow drainage outlet, and are weighting coefficients.

[0016] Optionally, the urban real-time drainage scheduling method further includes a method for rainfall runoff simulation of the target area, including:

[0017] The boundary initial condition of the hydrodynamic mechanism model is set according to the reward function and the constraint condition;

[0018] The basic data is substituted into the water power mechanism model, and the water power mechanism model is configured based on an open source library to simulate rainfall runoff of the target area;

[0019] Drainage scheduling data generated in a rainfall runoff simulation process is acquired, and the offline database is established according to the drainage scheduling data and the basic data.

[0020] Optionally, the multi-agent model is established based on a deep reinforcement learning algorithm, and the multi-agent model is trained using data of the offline database, including:

[0021] The multi-agent model is established based on a deep reinforcement learning algorithm;

[0022] The water power mechanism model is used as an interactive environment, and drainage scheduling data in multiple environment states in a rainfall runoff simulation process is extracted from the offline database as a training set;

[0023] Model parameters of the multi-agent model are set, the multi-agent model is trained using the training set, and a decision action of global scheduling is generated by all agents in the multi-agent model during the training process of the multi-agent model.

[0024] Optionally, the urban real-time drainage scheduling method further includes a method of interactive coupling of an environment state in a rainfall runoff simulation process and a decision action output by the multi-agent model, including:

[0025] According to the training process of the multi-agent model, drainage scheduling data at an initial environment state in a rainfall runoff simulation process is acquired by monitoring;

[0026] The multi-agent model generates a decision action and a policy network according to the drainage scheduling data at the initial environment state, and drives the water power mechanism model to run rainfall runoff simulation of a single time step through the generated decision action;

[0027] According to rainfall runoff simulation of a single time step of the water power mechanism model, drainage scheduling data at a current environment state is acquired by monitoring, and a drainage scheduling reward value at the current environment state is acquired through the constructed reward function;

[0028] The multi-agent model dynamically updates the policy network according to the drainage scheduling data at the current environment state and the drainage scheduling reward value.

[0029] Optionally, the multi-agent model dynamically updates the policy network according to the drainage scheduling data at the current environment state and the drainage scheduling reward value, including:

[0030] The Critic network and the Actor network in the multi-agent model are called, the Critic network evaluates the flood control value benefit by receiving global environment state data, and the Actor network generates an independent control strategy through local observation data;

[0031] The drainage scheduling data in the current environment state are input into the Critic network to calculate the deviation of the drainage scheduling reward value and the expected flood control benefit, and a policy gradient correction vector is generated;

[0032] The local control strategy deviation output by the Actor network is adjusted according to the policy gradient correction vector, and the weight matrix and the bias parameter of the policy network are updated periodically according to the engineering decision period.

[0033] Optionally, the urban real-time drainage scheduling method further comprises a method for online updating of drainage and waterlogging prevention engineering actions, comprising

[0034] Real-time rainstorm information of a target area is monitored and acquired, and rainstorm prediction data are generated based on a meteorological prediction model;

[0035] According to the rainstorm prediction data, an initial rainstorm prediction scheduling scheme is generated in combination with river and lake boundary data of the target area, scheduling hierarchical rules and engineering state data;

[0036] The current environment state of the target area is extracted from the rainstorm prediction scheduling scheme and input into the policy network, and the rainstorm prediction scheduling scheme is optimized based on the current environment state of the target area through the periodically generated policy network;

[0037] In each engineering decision period, a control instruction is generated according to the optimized scheduling scheme, and a drainage and waterlogging prevention engineering action is performed;

[0038] The periodically generated policy network, the optimized control instruction and the drainage and waterlogging prevention engineering action are added to the offline database, and the drainage and waterlogging prevention engineering action is online updated through training of the multi-agent model.

[0039] The application also discloses a drainage scheduling system adopting the urban real-time drainage scheduling method based on deep reinforcement learning.

[0040] A reward function construction module is configured to minimize the total amount of surface waterlogging and the overflow frequency of combined drainage and discharge outlets as double objectives, and to determine the state and action of drainage scheduling, and to construct a reward function and constraint condition of drainage scheduling;

[0041] The rainfall runoff simulation module is configured to acquire basic data of a target area, simulate rainfall runoff of the target area based on the acquired basic data by using a hydrodynamic mechanism model according to a constructed reward function and constraint condition, and acquire drainage scheduling data generated in a rainfall runoff simulation process to establish an offline database.

[0042] The model training module is configured to establish a multi-agent model based on a deep reinforcement learning algorithm, and train the multi-agent model by using data of the offline database.

[0043] The policy network generation module is configured to establish interaction coupling between an environment state in a rainfall runoff simulation process and a decision action output by the multi-agent model according to a training process of the multi-agent model, acquire an updated environment state and a drainage scheduling reward value in the rainfall runoff simulation process by the interaction coupling, and dynamically generate a policy network by the multi-agent model according to the updated environment state and the drainage scheduling reward value.

[0044] The engineering action update module is configured to acquire a real-time rainstorm prediction scheduling scheme, optimize the rainstorm prediction scheduling scheme by the periodically generated policy network, generate a control instruction to execute a drainage and waterlogging prevention engineering action according to the optimized scheduling scheme, and add the policy network, the control instruction and the drainage and waterlogging prevention engineering action to the offline database for online update.

[0045] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the urban real-time drainage scheduling method based on deep reinforcement learning.

[0046] The application further discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the urban real-time drainage scheduling method based on deep reinforcement learning when executing the computer program.

[0047] Compared with the prior art, the urban real-time drainage scheduling method and system based on deep reinforcement learning have the following advantages:

[0048] By constructing a double-objective reward function and constraint condition of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of combined sewer overflow, combining the hydrodynamic mechanism model to simulate the rainfall runoff of the basic data, based on the deep reinforcement learning algorithm, a multi-agent model is established, the interaction coupling of the environment state and the decision action is constructed, and then the strategy network is dynamically generated according to the updated environment state and the drainage scheduling reward value, and the generation and update of the drainage action are driven by the real-time rainstorm information, which effectively solves the problems of optimization falling into local optimum and few feasible solutions in the existing optimization scheduling, significantly improves the scheduling accuracy and efficiency, ensures the realization of multi-objective optimization decision in the response time, achieves the dual effects of waterlogging relief and cost reduction and efficiency improvement, enhances the urban rainwater resilience and meets the real-time demand. BRIEF DESCRIPTION OF DRAWINGS

[0049] The technical solutions of the present application will be further described in detail below with reference to the accompanying drawings and embodiments. In the drawings:

[0050] Figure 1 The step schematic block diagram of the urban real-time drainage scheduling method based on deep reinforcement learning provided by the embodiments of the present application is shown in the figure.

[0051] Figure 2 The schematic diagram of the framework of the urban real-time drainage scheduling provided by the embodiments of the present application is shown in the figure.

[0052] Figure 3 The comparison chart of the flood and waterlogging risk before and after the urban real-time drainage scheduling provided by the embodiments of the present application is shown in the figure.

[0053] Figure 4 The comparison chart of the total amount of waterlogging and the overflow frequency of combined sewer overflow before and after the urban real-time drainage scheduling provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0054] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. Now, the preferred embodiments of the present application will be described in detail with reference to the accompanying drawings.

[0055] The present application discloses a kind of urban real-time drainage scheduling method based on deep reinforcement learning, as shown in Figure 1 And Figure 2 It includes:

[0056] S1, with the double target of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of combined sewer overflow, and determining the state and action of drainage scheduling, constructing the reward function and constraint condition of drainage scheduling;

[0057] S2, acquire the basic data of the target area, and simulate rainfall runoff of the target area based on the acquired basic data by using a hydrodynamic mechanism model according to a constructed reward function and a constraint condition, and acquire drainage scheduling data generated in a rainfall runoff simulation process to establish an offline database;

[0058] S3, establish a multi-agent model based on a deep reinforcement learning algorithm, and train the multi-agent model by using data of the offline database;

[0059] S4, according to a training process of the multi-agent model, establish an interactive coupling relationship between an environment state in the rainfall runoff simulation process and a decision action output by the multi-agent model, acquire an updated environment state and a drainage scheduling reward value in the rainfall runoff simulation process through the interactive coupling relationship, and dynamically generate a policy network by the multi-agent model according to the updated environment state and the drainage scheduling reward value;

[0060] S5, acquire a real-time rainstorm prediction scheduling scheme, optimize the rainstorm prediction scheduling scheme by using the periodically generated policy network, generate a control instruction according to the optimized scheduling scheme to execute a drainage and waterlogging prevention engineering action, and add the policy network, the control instruction and the drainage and waterlogging prevention engineering action to the offline database for online updating.

[0061] By implementing the above-mentioned urban real-time drainage scheduling method embodiment, the reward function and the constraint condition of drainage and waterlogging prevention are constructed with the double targets of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of combined drainage system, providing a quantitative decision basis for waterlogging relief, directly reducing the problems of surface water accumulation and overflow, and improving the scientific basis of system scheduling. Secondly, by obtaining the basic data of the target area (such as pipe network data, land use data, scheduling engineering data, etc.) and using the hydrodynamic mechanism model to simulate rainfall runoff, the reward function and the constraint condition are ensured to accurately reflect the hydraulic characteristics and facility correlation in the simulation process, providing a reliable environmental basis for subsequent optimization. Based on the deep reinforcement learning algorithm, a multi-agent model is established and the interaction coupling between the environment state in the rainfall runoff simulation process and the output decision action of the multi-agent model is constructed. This mechanism enables the hydrodynamic mechanism model to capture the dynamic correlation of multiple facilities such as pipe network drainage, pump station pumping, and storage tank storage in the simulation, overcoming the limitations of scale and complexity, and obtaining updated environmental state and drainage scheduling reward value through interaction coupling. The multi-agent model dynamically generates a strategy network based on this, iteratively optimizes the double targets in the reward function, and forms a globally optimal decision scheme, effectively avoiding the defects of local optimization and few feasible solutions in traditional methods, and improving the scheduling accuracy. Finally, according to the real-time rainstorm prediction scheduling scheme, the drainage and waterlogging prevention engineering action is generated, and the strategy network is updated based on the periodic generation of the strategy network. Through the closed-loop feedback mechanism, dynamic prediction and strategy optimization are combined, the scheduling efficiency is greatly improved to meet the requirements of limited response time, and the strategy network is adjusted in real time to adapt to the changes of rainstorm, thereby efficiently formulating a multi-target joint scheduling strategy, achieving the dual goals of waterlogging relief and cost reduction and efficiency improvement, and enhancing the urban rainwater resilience. Therefore, the urban real-time drainage scheduling method embodiment from the double target construction to the historical data simulation, multi-agent interaction, strategy network generation and real-time action updating, cooperates with the adaptive ability of the deep reinforcement learning algorithm to overcome the constraints of facility correlation, hydraulic characteristics and economic cost on scheduling, ensuring the generation of optimal scientific decisions in rapid response, and promoting the development of real-time optimization scheduling of rainwater drainage system.

[0062] Further, the reward function and the constraint condition of drainage scheduling are constructed with the double targets of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of combined drainage system, and the state and action of drainage scheduling are determined, including:

[0063] The state of drainage scheduling is determined, and the corresponding drainage scheduling action is determined according to the current state of drainage scheduling;

[0064] The constraint condition is set according to the regulation and control effect of the determined drainage scheduling action on the total amount of surface waterlogging and the overflow frequency, and the reward function of drainage scheduling is constructed with the double targets of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of combined drainage system, and the function expression of the reward function is:

[0065]

[0066] wherein, is a reward value of drainage scheduling, is a number of waterlogging points, is a total number of waterlogging points, is a maximum waterlogging amount of the i-th waterlogging point, is a number of combined sewer overflow outfall, is a total number of combined sewer overflow outfall; is an overflow frequency of the j-th combined sewer overflow outfall, and are weighting coefficients.

[0067] Through the implementation of the above-mentioned embodiment of the urban real-time drainage scheduling method, the state of the drainage scheduling is accurately defined, such as the water level of the pump station, the water storage level of the storage tank, the current opening degree of the gate, the current flow and water level of the drainage pipe network measuring point, etc., so that the entire drainage scheduling can real-time perceive the change of the hydraulic state. And according to the drainage scheduling state, the corresponding drainage scheduling action is determined, such as the start-stop control water level of the front pool of the pump station, the opening-closing control water level of the gate of the storage tank, etc., the response logic between the control means and the state is clear. The constraint conditions are set based on the regulation and control effect of the drainage scheduling action on the total amount of surface waterlogging and the overflow frequency, including the maximum volume of the storage tank, the maximum volume of the front pool of the pump station, the maximum / minimal control water level of the pump station, the maximum / minimal change amount of the gate, etc., which quantifies the physical boundary and the operation safety rule as the decision boundary, and guarantees the feasibility of the scheduling action. A structured reward function is constructed, and the comprehensive influence of the scheduling state recognition, the scheduling action decision and its regulation and control on the waterlogging points and the overflow control of the outfall are unified into a quantitative evaluation system through the weighting coefficients 、 to dynamically balance the priority of the double-objective optimization of minimizing the total amount of surface waterlogging and minimizing the overflow frequency, so as to accurately generate a joint scheduling strategy that can simultaneously reduce the total amount of surface waterlogging and the overflow frequency of the combined sewer outfall. Thus, under the restriction of the constraint conditions, the scheduling action directly adjusts the drainage intensity of each waterlogging point and the prevention and control level of each combined sewer overflow outfall, realizes the collaborative optimization of flood control and pollution control, and improves the urban rainwater resilience.

[0068] Further, the urban real-time drainage scheduling method further comprises a method for simulating rainfall runoff in a target area, comprising:

[0069] setting the boundary initial conditions of the water dynamic mechanism model according to the reward function and the constraint conditions;

[0070] substituting the basic data into the water dynamic mechanism model, and simulating rainfall runoff in the target area based on the open source library configuration of the water dynamic mechanism model;

[0071] Obtain drainage scheduling data generated by the rainfall runoff simulation process, and establish an offline database according to the drainage scheduling data and the basic data.

[0072] Through the implementation of the above-mentioned urban real-time drainage scheduling method embodiment, the reward function and the constraint condition are converted into the setting basis of the boundary initial condition of the water dynamic mechanism model, so that the constraint values such as the maximum volume of the regulation and storage reservoir and the control water level of the pump station are accurately quantified as the model running boundary, ensuring that the simulation process meets the actual physical operation rules. The basic data of the target area are substituted into the water dynamic mechanism model, and the rainfall runoff drainage process simulation is performed based on the configuration of the open source library (such as the open source PySWMM code), so that the simulation process can truly reflect the facility correlation such as the pipe network drainage dynamics, the pumping capacity of the pump station and the water storage characteristics of the regulation and storage tank. The PySWMM code is a Python library based on the SWMM (Storm Water Management Model) of the United States Environmental Protection Agency, which is used to simulate and evaluate the performance of the urban drainage system.

[0073] As described above, the drainage scheduling data (such as the flow of each node, the pipe fullness, the surface water depth, etc.) generated by the water dynamic mechanism model in the simulation process is used to cooperatively build an offline database with the basic data, which completely covers the change process of key parameters such as the water level and flow of the pipe network measuring point, the water level of the pump station and the water storage level of the regulation and storage tank under different scheduling states, providing high-fidelity basic data for subsequent analysis. Based on the configuration of the open source library, the water dynamic mechanism model is given strong compatibility with complex pipe network topology and land use penetration characteristics, ensuring accurate reproduction of the simulation results of the hydraulic characteristics. Therefore, the urban real-time drainage scheduling method of the embodiment of the present application is driven by the boundary conditions and the historical data, so that the drainage scheduling data output by the water dynamic mechanism model has high reliability, provides sufficient environmental interaction samples for the subsequent multi-agent model, and provides structured data support for the subsequent strategy network training by the offline database, laying a data foundation for global optimization scheduling.

[0074] Further, a multi-agent model is established based on a deep reinforcement learning algorithm, and the data of the offline database is used to train the multi-agent model, including:

[0075] A multi-agent model is established based on a deep reinforcement learning algorithm;

[0076] The water dynamic mechanism model is used as an interactive environment, and drainage scheduling data in multiple environmental states in the rainfall runoff simulation process are extracted from the offline database as a training set;

[0077] The model parameters of the multi-agent model are set, the multi-agent model is trained using the training set, and the decision actions of global scheduling are generated by all agents in the multi-agent model during the training process of the multi-agent model.

[0078] Further, the urban real-time drainage scheduling method further comprises a method for coupling the environmental state and the multi-agent model output decision action in the rainfall runoff simulation process, comprising:

[0079] According to the training process of the multi-agent model, drainage scheduling data at the initial environmental state in the rainfall runoff simulation process is obtained by monitoring;

[0080] The multi-agent model generates decision actions and policy networks according to the drainage scheduling data at the initial environmental state, and drives the hydrodynamic mechanism model to run the rainfall runoff simulation of a single time step through the generated decision actions;

[0081] According to the rainfall runoff simulation of a single time step of the hydrodynamic mechanism model, drainage scheduling data at the current environmental state is obtained by monitoring, and a drainage scheduling reward value at the current environmental state is obtained through a constructed reward function;

[0082] The multi-agent model dynamically updates the policy network according to the drainage scheduling data and the drainage scheduling reward value at the current environmental state.

[0083] Through the implementation of the above-mentioned urban real-time drainage scheduling method embodiment, a multi-agent model is established based on a deep reinforcement learning algorithm, that is, the MADDPG algorithm (Multi-Agent Deep Deterministic Policy Gradient) is preferably used to generate decision actions of global scheduling through a policy set to cooperate with all agents, solving the defect of insufficient facility correlation cooperation in the traditional method. The model parameters in the multi-agent model are set, such as the number of pump stations, the discount factor, the update coefficient, etc., and the hydrodynamic mechanism model is used as the interactive environment to establish an interactive coupling mechanism between the decision actions and the operation of the hydrodynamic mechanism model through the interface developed by Python language. Python is a widely used high-level programming language. The coupling mechanism makes the decision actions (such as the start-stop control water level of the pump station front pool and the gate opening-closing control water level of the storage reservoir) output by each agent drive the hydrodynamic mechanism model to run the rainfall runoff simulation of a single time step, ensuring that the action response real-time and the model calculation granularity match.

[0084] As mentioned above, the rainfall runoff simulation process is trained based on the reward function and the constraint condition, and the multi-agent model generates decision actions and policy network according to the drainage scheduling data in the initial environment state obtained by monitoring. The feedback decision action triggers the hydrodynamic mechanism model to run the rainfall runoff simulation of a single time step, and outputs the drainage scheduling reward value after the environment state is updated (from environment state t to t+n). By continuously monitoring the drainage scheduling data in the current environment state, the multi-agent model dynamically corrects the weight parameters of the policy network according to the change of the environment state and the drainage scheduling reward value, forming a closed-loop learning mechanism of “action output-simulation update-reward evaluation-policy optimization”. This mechanism enables the policy network to break through the local optimal limit in the iteration, effectively improves the multi-facility coordinated scheduling accuracy, and meets the real-time response timeliness requirement. The entire interactive coupling process relies on the PyTorch framework to realize efficient calculation of the algorithm, ensuring the speed of single-step simulation and policy update, and finally providing core support for scientific joint debugging strategy generation. The PyTorch framework is an open-source machine learning library mainly used for deep learning and neural networks.

[0085] Further, the multi-agent model dynamically updates the policy network according to the drainage scheduling data and the drainage scheduling reward value in the current environment state, including:

[0086] The Critic network and the Actor network in the multi-agent model are called, the Critic network evaluates the flood control value benefit by receiving global environment state data, and the Actor network generates independent control strategies through local observation data;

[0087] The drainage scheduling data in the current environment state is input into the Critic network to calculate the deviation between the drainage scheduling reward value and the expected flood control benefit, and a policy gradient correction vector is generated;

[0088] The local control strategy deviation output by the Actor network is adjusted according to the policy gradient correction vector, and the weight matrix and bias parameters of the policy network are updated periodically according to the engineering decision period.

[0089] Through the implementation of the above-mentioned urban real-time drainage scheduling method embodiment, the Critic network in the multi-agent model receives global environmental state data for centralized evaluation of flood control value benefits, and the Actor network generates independent control strategies for each facility based on local observation data, forming a global and local collaborative decision architecture. The updated drainage scheduling data of the environmental state is input into the Critic network to calculate the deviation amount of the actual drainage scheduling reward value and the expected flood control benefit, accurately quantifying the difference between the decision effect and the target. Based on the deviation amount, a policy gradient correction vector is generated to direct the Actor network to adjust the local control strategy deviation, ensuring that each independent control strategy is always iterated in the direction of optimizing the cumulative reward within the next 3T (T is a single time step). Finally, the weight matrix and bias parameter of the policy network are periodically updated according to the actual decision period of the project (such as T=30min or T=60min), enabling the Actor network to real-time correct the parameter configuration of the control strategy according to the change of the environmental state. Thus, a closed-loop feedback link of "evaluation-deviation calculation-gradient correction-parameter update" is established, making the global value evaluation of the Critic network provide an objective benchmark for action effect, the policy gradient correction vector converts the reward deviation into a specific parameter adjustment direction, and the periodic weight matrix and bias parameter update gives the policy network the ability to dynamically adapt to new environmental states, thereby continuously improving the adaptability of the control strategy in real-time scheduling, effectively coordinating the facility-related actions of pipe network drainage, pump station pumping, and storage pool water storage, and ultimately enhancing the accuracy and timeliness of the drainage system's storage and drainage joint regulation. The Critic network and the Actor network together constitute a deep reinforcement learning algorithm, which is an algorithm framework that combines value function and policy optimization.

[0090] As mentioned above, the statistical correlation between each rainfall characteristic parameter (such as rainfall peak, maximum 30min rainfall, maximum 60min rainfall, total rainfall, etc.) and the maximum water accumulation of the waterlogging point , overflow frequency of combined system overflow outlet can be quantified by Pearson correlation analysis. Based on the strong correlation threshold of Pearson correlation coefficient greater than 0.8, the key rainfall characteristic parameters (such as rainfall peak, maximum 30min rainfall, etc.) are selected as the rainfall characteristic input parameters of the multi-agent model, in order to provide a scientific basis for the training of the multi-agent model. Among them, the selected strong correlation rainfall characteristic parameters can accurately represent the core influencing factors of the rainfall process on the total amount of surface waterlogging and overflow frequency, so that the environmental state input has the feature dimension reduction ability with clear physical meaning. After entering the training stage of the multi-agent model, these parameters and global state data such as pipe network key node flow and water level, pump station state, etc. are input into the Critic network together to drive it to accurately evaluate the current state and action on the maximum water accumulation of the waterlogging point and the overflow frequency of the combined system overflow outlet long-term impact value. At the same time, as an important part of local observation data, it guides the Actor network to generate independent control strategies for pump station start-stop control water level and reservoir gate opening and closing control water level, ensuring that the decision action is strongly coupled with the mechanism of rainfall-induced waterlogging.

[0091] Further, the urban real-time drainage dispatching method further includes an online updating method for drainage waterlogging prevention engineering actions, including

[0092] Monitoring and obtaining real-time rainstorm information of the target area, and generating rainstorm prediction data based on a meteorological prediction model;

[0093] According to the rainstorm prediction data, in combination with the river and lake boundary data of the target area, the dispatching hierarchical rules and the engineering state data, an initial rainstorm prediction dispatching scheme is generated;

[0094] The current environmental state input strategy network of the target area is extracted from the rainstorm prediction dispatching scheme, and the rainstorm prediction dispatching scheme is optimized based on the current environmental state of the target area through the periodically generated strategy network;

[0095] In each engineering decision period, control instructions are generated according to the optimized dispatching scheme, and drainage waterlogging prevention engineering actions are performed;

[0096] The periodically generated strategy network, the optimized control instructions and the drainage waterlogging prevention engineering actions are added to the offline database, and the drainage waterlogging prevention engineering actions are updated online through the training of the multi-agent model.

[0097] Through the implementation of the above-described urban real-time drainage scheduling method, real-time rainstorm information of the target area is monitored and acquired, and rainstorm forecast data is generated based on a meteorological prediction model, providing timely rainfall characteristic input for scheduling. An initial rainstorm forecast scheduling scheme (such as specific pump station start / stop commands and reservoir gate openings) is generated by combining river and lake boundary data of the target area (e.g., river water level thresholds), scheduling classification rules (e.g., pump station start / stop priorities under different rainfall levels), and engineering status data (e.g., current water storage capacity of the storage tank and pump station operating status). This scheme ensures that the initial decision conforms to hydrological boundaries and operating rules through multi-source data fusion. The current environmental state of the target area is extracted from the rainstorm forecast scheduling scheme to input the strategy network. During each engineering decision period (e.g., T=30min), the rainstorm forecast scheduling scheme is optimized based on the current environmental state of the target area using the periodically generated strategy network (e.g., adjusting the control water level of the pump station forebay). The controller executes the optimized scheduling scheme to perform corresponding drainage and flood control engineering actions (e.g., modifying the gate opening increment), forming a closed-loop control chain of "environmental state input - strategy network decision - engineering action optimization". Therefore, the urban real-time drainage scheduling method in this embodiment of the invention utilizes the real-time response capability of the strategy network to transform rainstorm forecast data, river and lake boundary constraints, and changes in engineering status into optimized time-series control commands, ensuring that drainage and flood control engineering actions always adapt to the dynamic environment. Furthermore, by combining an online learning mechanism (adding the periodically generated strategy network, optimized control commands, and drainage and flood control engineering actions to an offline database after each rainstorm), the strategy network continuously optimizes the mapping relationship between decision-making actions and environmental states, ultimately updating drainage and flood control engineering actions online. This achieves precise matching between engineering actions and the spatiotemporal changes of rainstorms, improving flood control and drainage efficiency.

[0098] Preferably, after each rainstorm event, the optimized drainage and flood control engineering actions, along with actual rainstorm information (such as peak rainfall and maximum 30-minute rainfall) and environmental conditions (such as flow and water level at pipe network nodes), are combined to form a new sample, which is then added to the offline database as the core content of "adding new data." The updated offline database provides an incremental training set for subsequent state interactions of the environmental module (environmental state t to t+n), driving the agent's decision-making algorithm to optimize the policy network parameters using the new sample in subsequent offline training. This ensures that the multi-agent model can learn from the engineering action optimization experience verified in rainstorm events by combining online updates and offline training, ultimately strengthening the deep reinforcement learning model's ability to dynamically adapt to environmental changes and meeting the sustainable development needs of real-time optimized scheduling under the smart water conservancy system.

[0099] like Figure 3 As shown, Figure 3 This is a comparison chart of flood inundation risk before and after using the urban real-time drainage scheduling method of this invention. Figure 3As can be seen from the data, before the scheduling, there were continuously distributed areas of severe and moderate risk in the northern and central parts of the city, indicating that these areas were prone to deep water accumulation or overflow pollution events under heavy rain conditions. After the scheduling, the severe risk areas almost disappeared, and the moderate risk areas shrank significantly, turning into areas with no risk and slight risk. This proves that the urban real-time drainage scheduling method of the present invention, through dynamic control of the start and stop water levels of pumping stations and the operation of reservoir gates, accurately shaving peaks and filling valleys, and suppressing the formation of high-risk waterlogging points.

[0100] like Figure 4 As shown, Figure 4 This is a comparison chart of the total accumulated water volume and the number of overflows at combined sewer outlets before and after the implementation of the urban real-time drainage scheduling method of this invention. Figure 4 As can be seen, under all rainstorm recurrence periods (10a-100a), the total waterlogging volume after scheduling is lower than before scheduling, and the overflow frequency curve after scheduling is always below that before scheduling. Moreover, the optimization range of both increases with the increase of rainstorm intensity. This proves that the urban real-time drainage scheduling method of the present invention, under the constraints of pump station control water level and storage reservoir volume, dynamically coordinates the pump station start-stop water level and gate opening action, and simultaneously achieves the global optimal decision of minimizing the total surface waterlogging volume and the overflow frequency of combined sewer overflow outlets. It still maintains high efficiency and coordination under extreme rainfall scenarios.

[0101] This invention also discloses a drainage scheduling system, which employs the aforementioned deep reinforcement learning-based real-time urban drainage scheduling method. The drainage scheduling system includes:

[0102] The reward function construction module is used to construct the reward function and constraints for drainage scheduling with the dual objectives of minimizing the total surface waterlogging and minimizing the frequency of combined sewer overflows, and to determine the state and actions of drainage scheduling.

[0103] The rainfall-runoff simulation module is used to acquire basic data of the target area. Based on the constructed reward function and constraints, it uses a hydrodynamic mechanism model to simulate rainfall-runoff in the target area using the acquired basic data, and acquires drainage scheduling data generated during the rainfall-runoff simulation process to establish an offline database.

[0104] The model training module is used to build a multi-agent model based on deep reinforcement learning algorithms and to train the multi-agent model using data from an offline database.

[0105] The policy network generation module is used to establish the interactive coupling between the environmental state and the output decision actions of the multi-agent model during the rainfall-runoff simulation process based on the training process of the multi-agent model. It obtains the updated environmental state and drainage scheduling reward value during the rainfall-runoff simulation process through interactive coupling. The multi-agent model dynamically generates the policy network based on the updated environmental state and drainage scheduling reward value.

[0106] An engineering action updating module is configured to obtain a real-time rainstorm prediction scheduling scheme, optimize the rainstorm prediction scheduling scheme through a periodically generated strategy network, generate a control instruction according to the optimized scheduling scheme to execute a drainage and waterlogging prevention engineering action, and add the strategy network, the control instruction and the drainage and waterlogging prevention engineering action to an offline database for online updating.

[0107] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is characterized in that the computer program is executed by a processor to implement the urban real-time drainage scheduling method based on deep reinforcement learning.

[0108] The application further discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer device is characterized in that the processor executes the computer program to implement the urban real-time drainage scheduling method based on deep reinforcement learning.

[0109] The application is described according to flowcharts and / or block diagrams of the method, device (system) and computer program product according to the specific embodiments. It should be understood that each flow and / or block in the flowcharts and / or block diagrams and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. The computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0110] The computer program instructions can also be stored in a computer readable memory capable of guiding a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a product comprising instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0111] The computer program instructions can also be loaded into a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to produce a computer implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus for performing the functions specified in one or more flows and / or blocks.

[0112] It should be understood that the above examples are only used to illustrate the technical solutions of the present application, but not limit the technical solutions of the present application. Those skilled in the art can modify the technical solutions described in the above examples, or make equivalent replacements to some of the technical features. All these modifications and replacements shall belong to the protection scope of the present application.

Claims

1. A city real-time drainage scheduling method based on deep reinforcement learning, characterized in that, The urban real-time drainage scheduling method comprises: With the double objectives of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of combined drainage outlets, the state and action of drainage scheduling are determined, the reward function and constraint condition of drainage scheduling are constructed; Obtain the basic data of the target area, and based on the obtained basic data, simulate the rainfall runoff of the target area by using the hydrodynamic mechanism model according to the constructed reward function and constraint condition, and obtain the drainage scheduling data generated in the rainfall runoff simulation process to establish an offline database; Based on the deep reinforcement learning algorithm, a multi-agent model is established, and the data of the offline database is used to train the multi-agent model; According to the training process of the multi-agent model, the interactive coupling relationship between the environment state in the rainfall runoff simulation process and the decision action output by the multi-agent model is established, and the updated environment state and drainage scheduling reward value in the rainfall runoff simulation process are obtained through the interactive coupling relationship, and the multi-agent model dynamically generates a policy network according to the updated environment state and drainage scheduling reward value; Obtain the real-time rainstorm prediction scheduling scheme, optimize the rainstorm prediction scheduling scheme through the periodically generated policy network, generate control instructions according to the optimized scheduling scheme to execute drainage and flood control engineering actions, and add the policy network, the control instructions and the drainage and flood control engineering actions to the offline database for online updating; The interactive coupling between the environment state in the rainfall runoff simulation process and the decision action output by the multi-agent model comprises: According to the training process of the multi-agent model, the drainage scheduling data at the initial environment state in the rainfall runoff simulation process is obtained by monitoring; The multi-agent model generates a decision action and a policy network according to the drainage scheduling data under the initial environment state, and drives the hydrodynamic mechanism model to run a single time step of rainfall runoff simulation through the generated decision action; According to the single time step of rainfall runoff simulation of the hydrodynamic mechanism model, the drainage scheduling data under the current environment state is obtained by monitoring, and the drainage scheduling reward value under the current environment state is obtained through the constructed reward function; The multi-agent model dynamically updates the policy network according to the drainage scheduling data under the current environment state and the drainage scheduling reward value; The multi-agent model dynamically updates the policy network according to the drainage scheduling data under the current environment state and the drainage scheduling reward value, comprising: Call the Critic network and the Actor network in the multi-agent model, the Critic network evaluates the flood control value benefit by receiving global environment state data, and the Actor network generates an independent control strategy by local observation data; Input the drainage scheduling data under the current environment state into the Critic network to calculate the deviation amount of the drainage scheduling reward value and the expected flood control benefit, and generate a strategy gradient correction vector; According to the strategy gradient correction vector, adjust the deviation of the local control strategy output by the Actor network, and update the weight matrix and bias parameter of the policy network periodically according to the engineering decision period.

2. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The double targets of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined drainage system are determined, and the state and action of the drainage scheduling are determined, the reward function and the constraint condition of the drainage scheduling are constructed, including: determining the state of the drainage scheduling, and determining the corresponding drainage scheduling action according to the current state of the drainage scheduling; constraint conditions are set according to the regulation and control effect of the determined drainage scheduling action on the total amount of surface waterlogging and the overflow frequency, and the double targets of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined drainage system are constructed, and the reward function of the drainage scheduling is constructed, and the function expression of the reward function is: wherein, is the reward value of water drainage scheduling, is the number of waterlogging points, is the total number of waterlogging points, is the maximum waterlogging amount of the ith waterlogging point, is the number of combined sewer overflow outfall, is the total number of combined sewer overflow outfall; is the overflow frequency of the jth combined sewer overflow outfall, and are weighting coefficients.

3. The real-time urban drainage scheduling method based on deep reinforcement learning according to claim 2, characterized in that, The urban real-time drainage scheduling method further includes a method for simulating rainfall runoff in the target area, including: setting the boundary initial conditions of the water dynamic mechanism model according to the reward function and the constraint condition; substitute the basic data into the water dynamic mechanism model, and configure the water dynamic mechanism model based on the open source library to simulate rainfall runoff in the target area; obtain the drainage scheduling data generated in the rainfall runoff simulation process, and establish the offline database according to the drainage scheduling data and the basic data.

4. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 3, characterized in that, The multi-agent model is established based on the deep reinforcement learning algorithm, and the data of the offline database is used to train the multi-agent model, including: establishing the multi-agent model based on the deep reinforcement learning algorithm; the water dynamic mechanism model is used as an interactive environment, and the drainage scheduling data of multiple environmental states in the rainfall runoff simulation process are extracted from the offline database as a training set; set the model parameters of the multi-agent model, train the multi-agent model using the training set, and generate the decision action of the global scheduling in cooperation with all agents in the multi-agent model during the training process of the multi-agent model.

5. The real-time urban drainage scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The urban real-time drainage scheduling method further includes a method for online updating of drainage and waterlogging prevention engineering actions, including monitoring and obtaining real-time heavy rain information of the target area, and generating heavy rain prediction data based on a weather prediction model; generate an initial heavy rain prediction scheduling scheme according to the heavy rain prediction data, combined with the river and lake boundary data of the target area, the scheduling classification rules and the engineering state data; input the current environmental state of the target area into the policy network from the heavy rain prediction scheduling scheme, and optimize the heavy rain prediction scheduling scheme based on the current environmental state of the target area through the periodically generated policy network; generate control instructions according to the optimized scheduling scheme and execute drainage and waterlogging prevention engineering actions at each engineering decision period; add the periodically generated policy network, the optimized control instructions and the drainage and waterlogging prevention engineering actions to the offline database, and update the drainage and waterlogging prevention engineering actions online through the training of the multi-agent model.

6. A drainage scheduling system employing the real-time urban drainage scheduling method based on deep reinforcement learning of any one of claims 1-5, characterized in that, The drainage scheduling system includes: a reward function construction module for determining the state and action of the drainage scheduling, and constructing the reward function and the constraint condition of the drainage scheduling with the double targets of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined drainage system; The rainfall runoff simulation module is configured to acquire basic data of a target area, simulate rainfall runoff of the target area based on the acquired basic data by using a hydrodynamic mechanism model according to a constructed reward function and constraint condition, and acquire drainage scheduling data generated in a rainfall runoff simulation process to establish an offline database; The model training module is configured to establish a multi-agent model based on a deep reinforcement learning algorithm, and train the multi-agent model by using data of the offline database; The strategy network generation module is configured to establish interaction coupling between an environment state in the rainfall runoff simulation process and a decision action output by the multi-agent model according to a training process of the multi-agent model, acquire an updated environment state and a drainage scheduling reward value in the rainfall runoff simulation process by the interaction coupling, and dynamically generate a strategy network by the multi-agent model according to the updated environment state and the drainage scheduling reward value. The engineering action updating module is configured to acquire a real-time rainstorm prediction scheduling scheme, optimize the rainstorm prediction scheduling scheme by the periodically generated strategy network, generate a control instruction to execute a drainage and waterlogging prevention engineering action according to the optimized scheduling scheme, and add the strategy network, the control instruction and the drainage and waterlogging prevention engineering action to the offline database for online updating.

7. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the deep reinforcement learning-based urban real-time drainage scheduling method of any one of claims 1-5.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the deep reinforcement learning-based urban real-time drainage scheduling method of any one of claims 1-5.

Images