Urban real-time drainage scheduling method and system based on deep reinforcement learning

Abstract

The invention relates to the technical field of urban drainage and waterlogging prevention, in particular to an urban real-time drainage scheduling method and system based on deep reinforcement learning. The method comprises the steps of constructing a dual-target reward function and constraint conditions for minimizing the total amount of surface waterlogging and the overflow frequency of a combined system discharge port, performing rainfall runoff simulation by adopting a hydrodynamic mechanism model and combining basic data of a target area, and establishing a multi-agent model based on deep reinforcement learning. The method comprises the following steps: constructing interactive coupling of decision actions and simulated environment states, obtaining updated environment states and drainage scheduling reward values through the coupling, driving a multi-agent model to generate a strategy network, predicting and generating drainage waterlogging prevention engineering actions based on real-time rainstorm information, and dynamically updating the actions by utilizing the periodically generated strategy network. According to the method, it is ensured that multi-objective optimization decision is rapidly achieved within response time, the waterlogging relieving effect is achieved, urban rainfall flood toughness is enhanced, and the real-time requirement is met.

Description

Technical Field

[0001] The present invention relates to the field of urban drainage and flood prevention technology, and in particular to a method and system for urban real-time drainage scheduling based on deep reinforcement learning. Background Art

[0002] Implementing coordinated regulation of urban stormwater drainage systems is a key measure for alleviating urban flooding. The accuracy and efficiency of these regulations largely determine a city's overall resilience to flooding. However, these regulations are constrained by complex factors such as scale, facility dependencies (drainage pipe networks, pumping stations, and reservoirs), hydraulic characteristics, and economic costs. Furthermore, the timeliness of these regulations makes it difficult to develop a sound coordinated regulation strategy within limited response times.

[0003] Currently, computational methods for optimizing the scheduling of urban stormwater drainage systems mostly rely on a combination of numerical simulation and non-gradient optimization. This can lead to optimization stuck in local optima and a lack of feasible solutions in complex multi-objective scheduling problems. Furthermore, with the advancement of the "Four Prediction" smart water conservancy system, real-time optimization and scheduling of stormwater drainage systems has become a trend, placing higher demands on timeliness. Therefore, new technologies are urgently needed to optimize the scheduling of urban stormwater drainage systems in real time. Summary of the Invention

[0004] The technical problem to be solved by the embodiments of the present invention is to provide a real-time urban drainage scheduling method and system based on deep reinforcement learning, so as to solve the problem in the existing technology of the lack of optimal scientific decision-making for urban rainwater drainage system optimization scheduling in terms of alleviating urban waterlogging, reducing costs and increasing efficiency.

[0005] The present invention discloses a method for real-time urban drainage scheduling based on deep reinforcement learning, comprising: Taking minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflow as the dual objectives, the state and action of drainage scheduling are determined, and the reward function and constraints of drainage scheduling are constructed; Obtaining basic data of the target area, performing rainfall runoff simulation on the target area using a hydrodynamic mechanism model based on the obtained basic data according to the constructed reward function and constraints, and obtaining drainage scheduling data generated during the rainfall runoff simulation process to establish an offline database; Establishing a multi-agent model based on a deep reinforcement learning algorithm, and training the multi-agent model using data from the offline database; According to the training process of the multi-agent model, an interactive coupling relationship between the environmental state during the rainfall runoff simulation and the decision action output by the multi-agent model is established, and the updated environmental state and drainage scheduling reward value during the rainfall runoff simulation are obtained through the interactive coupling relationship. The multi-agent model dynamically generates a strategy network based on the updated environmental state and drainage scheduling reward value; A real-time rainstorm forecast scheduling plan is obtained, the rainstorm forecast scheduling plan is optimized through the periodically generated strategy network, control instructions are generated according to the optimized scheduling plan to execute drainage and flood control engineering actions, and the strategy network, the control instructions and the drainage and flood control engineering actions are added to the offline database for online updating.

[0006] Optionally, the dual objectives of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflows, determining the state and action of drainage scheduling, and constructing the reward function and constraints of drainage scheduling include: Determine the status of drainage scheduling and determine the corresponding drainage scheduling action according to the current status of drainage scheduling; Constraints are set based on the regulatory effects of the determined drainage scheduling actions on the total amount of surface waterlogging and the frequency of overflows. A reward function for drainage scheduling is constructed with the dual objectives of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflows. The functional expression of the reward function is:

[0007] Where, is the reward value for drainage scheduling, Number the waterlogging points. is the total number of waterlogging points, is the maximum amount of water accumulation at the i-th waterlogging point, Number the combined sewer overflow outlet. is the total number of combined sewer overflow outlets; is the overflow frequency of the jth combined sewer overflow outlet, and are all weighting coefficients.

[0008] Optionally, the urban real-time drainage scheduling method further includes a method for simulating rainfall runoff in the target area, including: Setting the boundary initial conditions of the hydrodynamic mechanism model according to the reward function and the constraint conditions; Substituting the basic data into the hydrodynamic mechanism model, and configuring the hydrodynamic mechanism model based on an open source library to perform rainfall runoff simulation on the target area; Drainage scheduling data generated by the rainfall runoff simulation process is acquired, and the offline database is established according to the drainage scheduling data and the basic data.

[0009] Optionally, establishing a multi-agent model based on a deep reinforcement learning algorithm and training the multi-agent model using data from the offline database includes: Establishing the multi-agent model based on a deep reinforcement learning algorithm; Using the hydrodynamic mechanism model as an interactive environment, extracting drainage scheduling data under multiple environmental states during rainfall runoff simulation from the offline database as a training set; The model parameters of the multi-agent model are set, the multi-agent model is trained using the training set, and during the multi-agent model training process, all agents in the multi-agent model are coordinated to generate a global scheduling decision action. Optionally, the urban real-time drainage scheduling method further includes a method for interactively coupling the environmental state and the decision-making actions output by the multi-agent model during the rainfall runoff simulation process, including: According to the training process of the multi-agent model, drainage scheduling data at the initial environmental state during the rainfall runoff simulation process is obtained through monitoring; The multi-agent model generates decision actions and a strategy network according to drainage scheduling data under the initial environmental state, and drives the hydrodynamic mechanism model to run a rainfall runoff simulation of a single time step through the generated decision actions; According to the rainfall runoff simulation of a single time step of the hydrodynamic mechanism model, drainage scheduling data under the current environmental state is obtained through monitoring, and a drainage scheduling reward value under the current environmental state is obtained through the constructed reward function; The multi-agent model dynamically updates the strategy network according to the drainage scheduling data and the drainage scheduling reward value under the current environment state.

[0010] Optionally, the multi-agent model dynamically updates the policy network according to the drainage scheduling data and the drainage scheduling reward value under the current environmental state, including: Invoking a critic network and an actor network in the multi-agent model, wherein the critic network evaluates the value and benefit of flood control by receiving global environmental status data, and the actor network generates an independent control strategy based on local observation data; Input the drainage scheduling data under the current environmental state into the Critic network to calculate the deviation between the drainage scheduling reward value and the expected flood control benefit, and generate a policy gradient correction vector; The local control policy deviation output by the Actor network is adjusted according to the policy gradient correction vector, and the weight matrix and bias parameters of the policy network are updated periodically according to the engineering decision period.

[0011] Optionally, the urban real-time drainage scheduling method also includes a method for online updating of drainage and flood control engineering actions, including: Monitor and obtain real-time rainstorm information in the target area, and generate rainstorm forecast data based on meteorological forecast models; Generate an initial rainstorm forecast scheduling plan based on the rainstorm forecast data, combined with river and lake boundary data, scheduling classification rules, and project status data in the target area; Extracting the current environmental state of the target area from the rainstorm forecast scheduling scheme and inputting it into the strategy network, and optimizing the rainstorm forecast scheduling scheme based on the current environmental state of the target area through the periodically generated strategy network; During each engineering decision period, control instructions are generated based on the optimized scheduling plan and drainage and flood control engineering actions are executed; The periodically generated strategy network, the optimized control instructions and the drainage and flood control engineering actions are added to the offline database, and the drainage and flood control engineering actions are updated online through the training of the multi-agent model.

[0012] The present invention also discloses a drainage scheduling system, which adopts the above-mentioned urban real-time drainage scheduling method based on deep reinforcement learning. The drainage scheduling system includes: The reward function construction module is used to minimize the total amount of surface waterlogging and the frequency of combined sewer overflows as the dual objectives, determine the state and action of drainage scheduling, and construct the reward function and constraints of drainage scheduling; A rainfall runoff simulation module is used to obtain basic data of the target area, and according to the constructed reward function and constraints, use a hydrodynamic mechanism model to simulate rainfall runoff in the target area based on the obtained basic data, and obtain drainage scheduling data generated during the rainfall runoff simulation process to establish an offline database; A model training module, configured to establish a multi-agent model based on a deep reinforcement learning algorithm and train the multi-agent model using data from the offline database; a policy network generation module for establishing, based on the training process of the multi-agent model, an interactive coupling between the environmental state during the rainfall runoff simulation and the decision action output by the multi-agent model, and obtaining, through the interactive coupling, an updated environmental state and drainage scheduling reward value during the rainfall runoff simulation, wherein the multi-agent model dynamically generates a policy network based on the updated environmental state and drainage scheduling reward value; The engineering action update module is used to obtain a real-time rainstorm forecast scheduling plan, optimize the rainstorm forecast scheduling plan through the periodically generated strategy network, generate control instructions according to the optimized scheduling plan to execute drainage and flood control engineering actions, and add the strategy network, the control instructions and the drainage and flood control engineering actions to the offline database for online updating.

[0013] The present invention also discloses a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned urban real-time drainage scheduling method based on deep reinforcement learning.

[0014] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the computer program, the above-mentioned urban real-time drainage scheduling method based on deep reinforcement learning is implemented.

[0015] Compared with the existing technology, the urban real-time drainage scheduling method and system based on deep reinforcement learning provided by the embodiments of the present invention have the following beneficial effects: By constructing a dual-objective reward function and constraint conditions of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflow, combining the hydrodynamic mechanism model to simulate rainfall runoff on basic data, establishing a multi-agent model based on the deep reinforcement learning algorithm, and constructing the interactive coupling of environmental status and decision-making actions, the strategy network is dynamically generated according to the updated environmental status and drainage scheduling reward value, and the generation and update of drainage and waterlogging prevention engineering actions are driven by real-time rainstorm information. This effectively solves the problems of optimization falling into local optimality and few feasible solutions in existing optimization scheduling, significantly improves the scheduling accuracy and efficiency, ensures that multi-objective optimization decisions are quickly made within the response time, achieves the dual effects of alleviating urban waterlogging and reducing costs and increasing efficiency, enhances urban rain and flood resilience and meets real-time needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments, in which: Figure 1 A schematic block diagram of the steps of a method for real-time urban drainage scheduling based on deep reinforcement learning provided by an embodiment of the present invention; Figure 2 A schematic diagram of the framework of real-time urban drainage scheduling provided by an embodiment of the present invention; Figure 3 A comparison chart of flood and inundation risks before and after real-time urban drainage scheduling provided by an embodiment of the present invention; Figure 4 A comparison chart of the total amount of accumulated water and the number of overflows from the combined sewer overflow outlet before and after real-time urban drainage scheduling provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. Now, in conjunction with the accompanying drawings, the preferred embodiments of the present invention will be described in detail.

[0018] The present invention discloses a real-time urban drainage scheduling method based on deep reinforcement learning. Figure 1 and Figure 2 As shown, including: S1. Minimizing the total amount of surface waterlogging and the frequency of combined sewer overflows are the dual objectives. The state and actions of drainage scheduling are determined, and the reward function and constraints of drainage scheduling are constructed. S2. Obtain basic data for the target area. Based on the constructed reward function and constraints, use a hydrodynamic mechanism model to simulate rainfall and runoff in the target area based on the acquired basic data. Acquire drainage scheduling data generated during the rainfall and runoff simulation process to establish an offline database. S3. Build a multi-agent model based on deep reinforcement learning algorithm and train it using data from offline database. S4. Based on the training process of the multi-agent model, an interactive coupling relationship is established between the environmental state during the rainfall runoff simulation and the decision-making actions output by the multi-agent model. The updated environmental state and drainage scheduling reward value during the rainfall runoff simulation are obtained through the interactive coupling relationship. The multi-agent model dynamically generates a policy network based on the updated environmental state and drainage scheduling reward value. S5. Obtain a real-time rainstorm forecast scheduling plan, optimize the rainstorm forecast scheduling plan through a periodically generated strategy network, generate control instructions based on the optimized scheduling plan to execute drainage and flood control engineering actions, and add the strategy network, control instructions, and drainage and flood control engineering actions to the offline database for online updating.

[0019] By implementing the above-mentioned real-time urban drainage scheduling method, a reward function and constraints for drainage and flood prevention are constructed with the dual objectives of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflows. This provides a quantitative decision-making basis for waterlogging mitigation, directly and targetedly reduces surface waterlogging and overflow issues, and enhances the scientific basis for system scheduling. Secondly, by obtaining basic data for the target area (such as pipeline network data, land use data, and scheduling project data) and using a hydrodynamic mechanism model to simulate rainfall and runoff, the reward function and constraints ensure that the simulation process accurately reflects hydraulic characteristics and facility relevance, providing a reliable environmental basis for subsequent optimization. A multi-agent model based on a deep reinforcement learning algorithm is established, and the interactive coupling between environmental states and the decision actions output by the multi-agent model during rainfall and runoff simulation is constructed. This mechanism enables the hydrodynamic mechanism model to capture the dynamic connections between multiple facilities, such as pipe network drainage, pumping station drainage, and reservoir storage, in real time during the simulation, overcoming scale and complexity limitations. Through interactive coupling, the model obtains updated environmental states and drainage scheduling rewards. Based on these, the multi-agent model dynamically generates a policy network to iteratively optimize the dual objectives in the reward function to form a globally optimal decision solution. This effectively avoids the optimization pitfalls of traditional methods, which often lead to local optima and a lack of feasible solutions, and improves scheduling accuracy. Finally, drainage and flood control project actions are generated based on real-time rainstorm forecast scheduling plans, and these actions are updated based on the periodically generated policy network. This closed-loop feedback mechanism combines dynamic prediction with policy optimization, significantly improving scheduling efficiency to meet limited response time requirements. The policy network is driven in real time to adapt to rainstorm changes, thereby efficiently formulating multi-objective coordinated scheduling strategies, achieving the dual goals of alleviating urban flooding and reducing costs and increasing efficiency, thereby enhancing urban stormwater resilience. Therefore, the urban real-time drainage scheduling method of the embodiment of the present invention synergistically enhances the adaptive ability of the deep reinforcement learning algorithm from dual-objective construction to historical data simulation, multi-agent interaction, strategy network generation and real-time action update, so as to overcome the constraints of complex factors such as facility correlation, hydraulic characteristics and economic costs on scheduling, ensure the generation of optimal scientific decisions in rapid response, and promote the development of real-time optimization scheduling of rainwater drainage systems.

[0020] Furthermore, with the dual objectives of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflows, the state and actions of drainage scheduling are determined, and the reward function and constraints of drainage scheduling are constructed, including: Determine the status of drainage scheduling and determine the corresponding drainage scheduling action according to the current status of drainage scheduling; Constraints are set based on the regulatory effects of the determined drainage scheduling actions on the total amount of surface waterlogging and overflow frequency. Taking minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflow as the dual objectives, a reward function for drainage scheduling is constructed. The functional expression of the reward function is:

[0021] Where, is the reward value for drainage scheduling, Number the waterlogging points. is the total number of waterlogging points, is the maximum amount of water accumulation at the i-th waterlogging point, Number the combined sewer overflow outlet. is the total number of combined sewer overflow outlets; is the overflow frequency of the jth combined sewer overflow outlet, and are all weighting coefficients.

[0022] Through the implementation of the above-mentioned urban real-time drainage scheduling method embodiment, the state of drainage scheduling is accurately defined, such as the water level of the pump station, the water level of the regulating reservoir, the current opening of the gate, the current flow and water level of the drainage network measuring point, etc., so that the entire drainage scheduling can perceive the changes in hydraulic state in real time. And according to this drainage scheduling state, the corresponding drainage scheduling action is determined, such as the start and stop control water level of the pump station front pool, the opening and closing control water level of the regulating reservoir gate, etc., to clarify the response logic between the control means and the state. Based on the control effect of the drainage scheduling action on the total amount of surface waterlogging and the overflow frequency, set constraints, including the maximum volume of the regulating reservoir, the maximum volume of the pump station front pool, the maximum / minimum control water level of the pump station, the maximum / minimum change of the gate, etc., quantify the physical boundaries and operation safety rules into decision boundaries to ensure the feasibility of the scheduling action. Construct a structured reward function, through the weighting coefficient 、 By dynamically balancing the dual-objective optimization priorities of minimizing total surface waterlogging and minimizing overflow frequency, this system integrates dispatching state identification, dispatching action decisions, and their combined impact on waterlogging control at waterlogging points and outlet overflow control into a quantitative assessment system. This allows for the precise generation of a coordinated dispatching strategy that simultaneously reduces both total surface waterlogging and the frequency of combined sewer overflows. Ultimately, within the constraints of these constraints, dispatching actions directly adjust the drainage intensity of each waterlogging point and the control level of each combined sewer overflow outlet, achieving coordinated optimization of flood control and drainage with pollution control, and enhancing urban stormwater resilience.

[0023] Furthermore, the urban real-time drainage scheduling method also includes a method for simulating rainfall runoff in a target area, including: Set the boundary initial conditions of the hydrodynamic mechanism model according to the reward function and constraints; Substitute the basic data into the hydrodynamic mechanism model, and configure the hydrodynamic mechanism model based on the open source library to simulate rainfall runoff in the target area; The drainage scheduling data generated by the rainfall runoff simulation process is obtained, and an offline database is established based on the drainage scheduling data and basic data.

[0024] By implementing the above-mentioned embodiment of the real-time urban drainage scheduling method, the reward function and constraints are converted into the basis for setting the boundary initial conditions of the hydrodynamic mechanism model. This allows the constraint values ​​such as the maximum volume of the regulating reservoir and the controlled water level of the pumping station to be accurately quantified as the model's operating boundaries, ensuring that the simulation process meets the actual physical operating rules. The basic data of the target area is substituted into the hydrodynamic mechanism model, and the rainfall runoff drainage process is simulated based on the configuration based on open source libraries (such as the open source PySWMM code). The simulation process can truly reflect the correlation between facilities such as the drainage dynamics of the pipeline network, the drainage capacity of the pumping station, and the water storage characteristics of the regulating reservoir. Among them, the PySWMM code is a Python library based on the US Environmental Protection Agency's SWMM (Storm Water Management Model, an urban surface water model) for simulating and evaluating the performance of urban drainage systems.

[0025] As described above, the drainage scheduling data (such as node flow, pipe fullness, surface water depth, etc.) generated by the hydrodynamic mechanism model during the simulation process is used in conjunction with basic data to construct an offline database, which fully covers the changes in key parameters such as the water level flow at the pipe network measurement points, the water level at the pump station, and the water level in the regulating reservoir under different scheduling conditions, providing high-fidelity basic data for subsequent analysis. Furthermore, based on the configuration of the open source library, the hydrodynamic mechanism model is given strong compatibility with complex pipe network topologies and land use permeability characteristics, ensuring that the simulation results accurately reproduce the hydraulic characteristics. Therefore, the urban real-time drainage scheduling method of the embodiment of the present invention, through the dual drive of boundary conditions and historical data, makes the drainage scheduling data output by the hydrodynamic mechanism model highly reliable, provides sufficient environmental interaction samples for the subsequent multi-agent model, and provides structured data support for subsequent policy network training by the offline database, laying the data foundation for global optimization scheduling.

[0026] Furthermore, a multi-agent model is established based on a deep reinforcement learning algorithm and trained using data from an offline database, including: Establish a multi-agent model based on deep reinforcement learning algorithm; The hydrodynamic mechanism model is used as an interactive environment, and drainage scheduling data under multiple environmental conditions during rainfall-runoff simulation are extracted from the offline database as a training set. Set the model parameters of the multi-agent model, use the training set to train the multi-agent model, and coordinate all agents in the multi-agent model to generate global scheduling decision actions during the multi-agent model training process.

[0027] Furthermore, the urban real-time drainage scheduling method also includes a method for interactively coupling the environmental state and the decision-making actions output by the multi-agent model during the rainfall runoff simulation process, including: According to the training process of the multi-agent model, the drainage scheduling data at the initial environmental state during the rainfall runoff simulation is obtained through monitoring; The multi-agent model generates decision actions and a policy network based on the drainage scheduling data under the initial environmental state, and drives the hydrodynamic mechanism model to run a single time step rainfall runoff simulation through the generated decision actions. Based on the rainfall-runoff simulation of a single time step of the hydrodynamic mechanism model, the drainage scheduling data under the current environmental state is obtained through monitoring, and the drainage scheduling reward value under the current environmental state is obtained through the constructed reward function; The multi-agent model dynamically updates the strategy network based on the drainage scheduling data and drainage scheduling reward value under the current environmental state.

[0028] Through the implementation of the aforementioned real-time urban drainage scheduling method, a multi-agent model is established based on a deep reinforcement learning algorithm. Specifically, the MADDPG algorithm (Multi-Agent Deep Deterministic Policy Gradient) is preferably used to coordinate all agents through a set of policies to generate global scheduling decisions. This addresses the drawback of traditional methods, which often lack coordination across facilities. Model parameters within the multi-agent model, such as the number of pumping stations, discount factors, and update coefficients, are set. Using the hydrodynamic mechanism model as an interactive environment, an interactive coupling mechanism is established between decision actions and the operation of the hydrodynamic mechanism model through an interface developed in Python, a widely used high-level programming language. This coupling mechanism enables the decision actions output by each agent (such as starting and stopping the water level in the pump station forebay or opening and closing the gates of the regulating reservoir to control the water level) to drive the hydrodynamic mechanism model to run a single time-step rainfall-runoff simulation, ensuring real-time action response and matching the model's computational granularity.

[0029] As described above, during the training phase, drainage scheduling data under multiple environmental states, extracted from an offline database, is extracted. Based on a reward function and constraints, the multi-agent model generates decision actions and a policy network based on the drainage scheduling data obtained under the initial environmental state, obtained through monitoring. The feedback decision action triggers the hydrodynamic mechanism model to run a single time-step rainfall-runoff simulation and outputs the drainage scheduling reward value for the updated environmental state (from state t to state t+n) in real time. By continuously monitoring drainage scheduling data under the current environmental state, the multi-agent model dynamically adjusts the policy network weight parameters based on environmental state changes and drainage scheduling reward values. This forms a closed-loop learning mechanism of "action output - simulation update - reward evaluation - policy optimization." This allows the policy network to break through local optimality during iteration, effectively improving the accuracy of multi-facility coordinated scheduling and meeting real-time response requirements. The entire interactive coupling process relies on the PyTorch framework for efficient algorithm computation, ensuring the speed of single-step simulation and policy updates, ultimately providing core support for scientific coordinated strategy generation. PyTorch is an open-source machine learning library primarily used for deep learning and neural network applications.

[0030] Furthermore, the multi-agent model dynamically updates the strategy network based on the drainage scheduling data and drainage scheduling reward value under the current environmental state, including: The Critic network and Actor network in the multi-agent model are called. The Critic network evaluates the value and benefits of flood control by receiving global environmental status data, and the Actor network generates independent control strategies based on local observation data. The drainage scheduling data under the current environmental state is input into the Critic network to calculate the deviation between the drainage scheduling reward value and the expected flood control benefit, and generate a policy gradient correction vector; Adjust the local control policy deviation output by the Actor network according to the policy gradient correction vector, and update the weight matrix and bias parameters of the policy network according to the engineering decision period.

[0031] Through the implementation of the above-described real-time urban drainage scheduling method, a critic network in a multi-agent model is invoked to receive global environmental state data for a centralized assessment of flood control value and benefits. Simultaneously, an actor network generates independent control strategies for each facility based on local observation data, forming a collaborative global and local decision-making architecture. Drainage scheduling data, after updating the environmental state, is fed into the critic network to calculate the deviation between the actual drainage scheduling reward and the expected flood control benefit, accurately quantifying the difference between the decision effect and the target. Based on this deviation, a policy gradient correction vector is generated to guide the actor network in adjusting the local control policy deviation, ensuring that each independent control strategy consistently iterates towards optimizing the cumulative reward within the optimal future timeframe of 3T (where T is a single time step). Finally, the weight matrix and bias parameters of the policy network are periodically updated according to the actual project decision period (e.g., T = 30 minutes or T = 60 minutes), enabling the actor network to adjust the control strategy parameter configuration in real time based on environmental state changes. This establishes a closed-loop feedback loop of "evaluation-bias calculation-gradient correction-parameter update," enabling the Critic network's global value assessment to provide an objective benchmark for action effectiveness. The policy gradient correction vector converts reward bias into specific parameter adjustment directions, and periodic weight matrix and bias parameter updates empower the policy network to dynamically adapt to new environmental states. This enables the multi-agent model to continuously improve the adaptability of control strategies in real-time scheduling, effectively coordinating the interrelated actions of pipeline drainage, pumping station drainage, and reservoir storage, ultimately enhancing the accuracy and timeliness of the drainage system's combined storage and drainage coordination. The Critic network and the Actor network together constitute a deep reinforcement learning algorithm, an algorithmic framework that combines value functions and policy optimization.

[0032] As mentioned above, the Pearson correlation analysis can be used to quantify the relationship between various rainfall characteristic parameters (such as rainfall peak, maximum 30-minute rainfall, maximum 60-minute rainfall, total rainfall, etc.) and the maximum water accumulation at the waterlogging point. , Overflow frequency of combined sewer overflow outlet Statistical correlation. Based on the strong correlation threshold of the Pearson correlation coefficient greater than 0.8, key rainfall characteristic parameters (such as rainfall peak, maximum 30-minute rainfall, etc.) are screened and used as rainfall characteristic input parameters of the multi-agent model to provide a scientific basis for the training of the multi-agent model. Among them, the screened strongly correlated rainfall characteristic parameters can accurately characterize the core influencing factors of the rainfall process on the total amount of surface waterlogging and overflow frequency, so that the environmental state input has the ability of feature dimensionality reduction with clear physical meaning. After entering the training stage of the multi-agent model, these parameters are input into the Critic network together with the global state data such as the flow and water level of key nodes of the pipeline network and the status of the pump station to drive it to accurately evaluate the current state and action on the maximum water volume of the waterlogging point Overflow frequency of combined sewer overflow outlet As an important component of local observation data, it guides the Actor network to generate independent control strategies for controlling water levels at pump stations and reservoir gates, ensuring strong coupling between decision-making actions and rainfall-induced waterlogging mechanisms.

[0033] Furthermore, the urban real-time drainage scheduling method also includes a method for online updating of drainage and flood control engineering actions, including Monitor and obtain real-time rainstorm information in the target area, and generate rainstorm forecast data based on meteorological forecast models; Generate an initial rainstorm forecast scheduling plan based on rainstorm forecast data, combined with river and lake boundary data, scheduling classification rules, and project status data in the target area; The current environmental state of the target area is extracted from the rainstorm forecast scheduling scheme and input into the strategy network. The rainstorm forecast scheduling scheme is optimized based on the current environmental state of the target area through the periodically generated strategy network. During each engineering decision period, control instructions are generated based on the optimized scheduling plan and drainage and flood control engineering actions are executed; The periodically generated policy network, optimized control instructions, and drainage and flood control engineering actions are added to the offline database, and the drainage and flood control engineering actions are updated online through the training of a multi-agent model.

[0034] By implementing the above-described embodiment of the real-time urban drainage scheduling method, real-time rainstorm information in the target area is monitored and acquired, and rainstorm forecast data is generated based on a meteorological forecast model, providing highly timely rainfall feature input for scheduling. An initial rainstorm forecast scheduling plan (e.g., specific pump station start / stop instructions and reservoir gate opening) is generated by combining the target area's river and lake boundary data (e.g., river water level thresholds), scheduling classification rules (e.g., pump station start / stop priorities at different rainfall levels), and project status data (e.g., current reservoir water capacity and pump station operating status). This plan utilizes multi-source data fusion to ensure that initial decisions comply with hydrological boundaries and operating rules. The current environmental state of the target area is extracted from the rainstorm forecast scheduling plan and inputted into the strategy network. During each project decision period (e.g., T = 30 minutes), the periodically generated strategy network optimizes the rainstorm forecast scheduling plan based on the current environmental state of the target area (e.g., adjusting the control water level in the pump station forebay). The optimized scheduling plan is then executed by a controller to perform corresponding drainage and flood control project actions (e.g., modifying the gate opening increment), forming a closed-loop control chain of "environmental state input - strategy network decision - project action optimization." Therefore, the real-time urban drainage scheduling method of this embodiment leverages the real-time response capabilities of a policy network to transform rainstorm forecast data, river and lake boundary constraints, and project state changes into optimized time-series control instructions, ensuring that drainage and flood control project actions consistently adapt to the dynamic environment. Combined with an online learning mechanism (after each rainstorm, the periodically generated policy network, optimized control instructions, and drainage and flood control project actions are added to an offline database), the policy network continuously optimizes the mapping between decision-making actions and environmental conditions. Ultimately, drainage and flood control project actions are updated online, achieving precise alignment between project actions and the spatiotemporal variations of rainstorms, thereby improving flood control and drainage efficiency.

[0035] Optimally, after each rainstorm event, optimized drainage and flood control engineering actions, along with actual rainstorm information (e.g., peak rainfall, maximum 30-minute rainfall), and environmental conditions (e.g., flow and water levels at network nodes) are combined to form new samples, which are then added to the offline database as the core content of "adding new data." This updated offline database provides an incremental training set for subsequent state interactions with environmental modules (from environmental states t to t+n), driving the agent's decision-making algorithm to optimize policy network parameters using these new samples during subsequent offline training. This ensures that the multi-agent model can learn from the engineering action optimization experience validated during rainstorm events through a combination of online updates and offline training. Ultimately, this strengthens the deep reinforcement learning model's ability to dynamically adapt to environmental changes, thereby meeting the sustainable development needs of real-time optimized scheduling within a smart water conservancy system.

[0036] like Figure 3 As shown, Figure 3 This is a comparison chart of flood and submergence risks before and after the real-time urban drainage scheduling method of the present invention is used. Figure 3It can be seen that before the scheduling: there are continuously distributed heavy risk and moderate risk areas in the north and central parts of the city, indicating that the area is prone to deep waterlogging or overflow pollution incidents under heavy rain conditions; and after the scheduling: the heavy risk area almost disappeared, and the moderate risk area shrank significantly, turning into mainly no risk and slight risk, proving that the urban real-time drainage scheduling method of the present invention can accurately cut peaks and fill valleys by dynamically regulating the start and stop water levels of pump stations and the operation of storage reservoir gates, thereby suppressing the formation of high-risk waterlogging points.

[0037] like Figure 4 As shown, Figure 4 This is a comparison chart of the total amount of accumulated water and the number of combined sewer overflows before and after the real-time urban drainage scheduling method of the present invention is used. Figure 4 It can be seen from the data that in all rainstorm return period scenarios (10a-100a), the total amount of waterlogging after scheduling is lower than that before scheduling, and the overflow frequency curve after scheduling is always below that before scheduling. Moreover, the optimization range of the two increases with the increase of rainstorm intensity. This proves that the urban real-time drainage scheduling method of the present invention dynamically coordinates the start and stop water levels of the pumping station and the gate opening under the constraints of maintaining the control water level of the pumping station and the volume of the regulating reservoir, and simultaneously achieves the global optimal decision of minimizing the total amount of surface waterlogging and minimizing the overflow frequency of the combined sewer outlet, and still maintains efficient coordination under extreme rainfall scenarios.

[0038] The present invention also discloses a drainage scheduling system, which adopts the above-mentioned urban real-time drainage scheduling method based on deep reinforcement learning. The drainage scheduling system includes: The reward function construction module is used to minimize the total amount of surface waterlogging and the frequency of combined sewer overflows as the dual objectives, determine the state and action of drainage scheduling, and construct the reward function and constraints of drainage scheduling; The rainfall runoff simulation module is used to obtain basic data of the target area. According to the constructed reward function and constraints, the hydrodynamic mechanism model is used to simulate rainfall runoff in the target area based on the obtained basic data. The drainage scheduling data generated during the rainfall runoff simulation process is obtained to establish an offline database. The model training module is used to build a multi-agent model based on the deep reinforcement learning algorithm and train the multi-agent model using data from the offline database; The policy network generation module is used to establish the interactive coupling between the environmental state and the decision-making actions output by the multi-agent model during the rainfall runoff simulation process based on the training process of the multi-agent model. The updated environmental state and drainage scheduling reward value during the rainfall runoff simulation are obtained through the interactive coupling. The multi-agent model dynamically generates a policy network based on the updated environmental state and drainage scheduling reward value. The engineering action update module is used to obtain real-time rainstorm forecast scheduling plans, optimize the rainstorm forecast scheduling plans through a periodically generated strategy network, generate control instructions based on the optimized scheduling plan to execute drainage and flood control engineering actions, and add the strategy network, control instructions, and drainage and flood control engineering actions to the offline database for online updating.

[0039] The present invention also discloses a computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, it implements the above-mentioned urban real-time drainage scheduling method based on deep reinforcement learning.

[0040] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, characterized in that when the processor executes the computer program, the above-mentioned urban real-time drainage scheduling method based on deep reinforcement learning is implemented.

[0041] The present invention is described based on flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to specific embodiments. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0042] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0044] It should be understood that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Those skilled in the art may modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents; and all these modifications and replacements should fall within the scope of protection of the present invention.

Claims

1. A real-time urban drainage scheduling method based on deep reinforcement learning, characterized in that: The urban real-time drainage scheduling method comprises: Taking minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflow as the dual objectives, the state and action of drainage scheduling are determined, and the reward function and constraints of drainage scheduling are constructed; Obtaining basic data of the target area, performing rainfall runoff simulation on the target area using a hydrodynamic mechanism model based on the obtained basic data according to the constructed reward function and constraints, and obtaining drainage scheduling data generated during the rainfall runoff simulation process to establish an offline database; Establishing a multi-agent model based on a deep reinforcement learning algorithm, and training the multi-agent model using data from the offline database; According to the training process of the multi-agent model, an interactive coupling relationship between the environmental state during the rainfall runoff simulation and the decision action output by the multi-agent model is established, and the updated environmental state and drainage scheduling reward value during the rainfall runoff simulation are obtained through the interactive coupling relationship. The multi-agent model dynamically generates a strategy network based on the updated environmental state and drainage scheduling reward value; A real-time rainstorm forecast scheduling plan is obtained, the rainstorm forecast scheduling plan is optimized through the periodically generated strategy network, control instructions are generated according to the optimized scheduling plan to execute drainage and flood control engineering actions, and the strategy network, the control instructions and the drainage and flood control engineering actions are added to the offline database for online updating.

2. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The dual objectives of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflows are determined, and the state and action of drainage scheduling are determined. The reward function and constraints of drainage scheduling are constructed, including: Determine the status of drainage scheduling and determine the corresponding drainage scheduling action according to the current status of drainage scheduling; Constraints are set based on the regulatory effects of the determined drainage scheduling actions on the total amount of surface waterlogging and the frequency of overflows. A reward function for drainage scheduling is constructed with the dual objectives of minimizing the total amount of surface waterlogging and minimizing the frequency of combined sewer overflows. The functional expression of the reward function is: Where, is the reward value for drainage scheduling, Number the waterlogging points. is the total number of waterlogging points, is the maximum amount of water accumulation at the i-th waterlogging point, Number the combined sewer overflow outlet. is the total number of combined sewer overflow outlets; is the overflow frequency of the jth combined sewer overflow outlet, and are all weighting coefficients.

3. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 2 is characterized in that: The urban real-time drainage scheduling method also includes a method for simulating rainfall runoff in the target area, including: Setting the boundary initial conditions of the hydrodynamic mechanism model according to the reward function and the constraint conditions; Substituting the basic data into the hydrodynamic mechanism model, and configuring the hydrodynamic mechanism model based on an open source library to perform rainfall runoff simulation on the target area; Drainage scheduling data generated by the rainfall runoff simulation process is acquired, and the offline database is established according to the drainage scheduling data and the basic data.

4. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 3 is characterized in that: The method of establishing a multi-agent model based on a deep reinforcement learning algorithm and training the multi-agent model using data from the offline database includes: Establishing the multi-agent model based on a deep reinforcement learning algorithm; Using the hydrodynamic mechanism model as an interactive environment, extracting drainage scheduling data under multiple environmental states during rainfall runoff simulation from the offline database as a training set; The model parameters of the multi-agent model are set, the multi-agent model is trained using the training set, and during the multi-agent model training process, all agents in the multi-agent model are coordinated to generate a global scheduling decision action.

5. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 4 is characterized in that: The urban real-time drainage scheduling method also includes a method for interactively coupling the environmental state and the decision-making actions output by the multi-agent model during the rainfall runoff simulation process, including: According to the training process of the multi-agent model, drainage scheduling data at the initial environmental state during the rainfall runoff simulation process is obtained through monitoring; The multi-agent model generates decision actions and a strategy network according to drainage scheduling data under the initial environmental state, and drives the hydrodynamic mechanism model to run a rainfall runoff simulation of a single time step through the generated decision actions; According to the rainfall runoff simulation of a single time step of the hydrodynamic mechanism model, drainage scheduling data under the current environmental state is obtained through monitoring, and the drainage scheduling reward value under the current environmental state is obtained through the constructed reward function; The multi-agent model dynamically updates the strategy network according to the drainage scheduling data and the drainage scheduling reward value under the current environment state.

6. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 5 is characterized in that: The multi-agent model dynamically updates the strategy network according to the drainage scheduling data and the drainage scheduling reward value under the current environment state, including: Invoking a critic network and an actor network in the multi-agent model, wherein the critic network evaluates the value and benefit of flood control by receiving global environmental status data, and the actor network generates an independent control strategy based on local observation data; Input the drainage scheduling data under the current environmental state into the Critic network to calculate the deviation between the drainage scheduling reward value and the expected flood control benefit, and generate a policy gradient correction vector; The local control policy deviation output by the Actor network is adjusted according to the policy gradient correction vector, and the weight matrix and bias parameters of the policy network are updated periodically according to the engineering decision period.

7. The urban real-time drainage scheduling method based on deep reinforcement learning according to claim 5 is characterized in that: The urban real-time drainage scheduling method also includes a method for online updating of drainage and flood control engineering actions, including: Monitor and obtain real-time rainstorm information in the target area, and generate rainstorm forecast data based on meteorological forecast models; Generate an initial rainstorm forecast scheduling plan based on the rainstorm forecast data, combined with river and lake boundary data, scheduling classification rules, and project status data in the target area; Extracting the current environmental state of the target area from the rainstorm forecast scheduling scheme and inputting it into the strategy network, and optimizing the rainstorm forecast scheduling scheme based on the current environmental state of the target area through the periodically generated strategy network; During each engineering decision period, control instructions are generated based on the optimized scheduling plan and drainage and flood control engineering actions are executed; The periodically generated strategy network, the optimized control instructions and the drainage and flood control engineering actions are added to the offline database, and the drainage and flood control engineering actions are updated online through the training of the multi-agent model.

8. A drainage scheduling system, using the urban real-time drainage scheduling method based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: The drainage scheduling system includes: The reward function construction module is used to minimize the total amount of surface waterlogging and the frequency of combined sewer overflows as the dual objectives, determine the state and action of drainage scheduling, and construct the reward function and constraints of drainage scheduling; A rainfall runoff simulation module is used to obtain basic data of the target area, and according to the constructed reward function and constraints, use a hydrodynamic mechanism model to simulate rainfall runoff in the target area based on the obtained basic data, and obtain drainage scheduling data generated during the rainfall runoff simulation process to establish an offline database; A model training module, configured to establish a multi-agent model based on a deep reinforcement learning algorithm and train the multi-agent model using data from the offline database; a policy network generation module for establishing, based on the training process of the multi-agent model, an interactive coupling between the environmental state during the rainfall runoff simulation and the decision action output by the multi-agent model, and obtaining, through the interactive coupling, an updated environmental state and drainage scheduling reward value during the rainfall runoff simulation, wherein the multi-agent model dynamically generates a policy network based on the updated environmental state and drainage scheduling reward value; The engineering action update module is used to obtain a real-time rainstorm forecast scheduling plan, optimize the rainstorm forecast scheduling plan through the periodically generated strategy network, generate control instructions according to the optimized scheduling plan to execute drainage and flood control engineering actions, and add the strategy network, the control instructions and the drainage and flood control engineering actions to the offline database for online updating.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the urban real-time drainage scheduling method based on deep reinforcement learning according to any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the urban real-time drainage scheduling method based on deep reinforcement learning according to any one of claims 1 to 7 is implemented.

Images