Rainwater pipe network optimization method based on deep deterministic policy gradient algorithm

Abstract

The present application relates to a kind of rainwater pipe network optimization methods based on deep deterministic policy gradient algorithm, comprising: step S1: definition environment and model parameters, wherein state includes the water depth of each node of pipe network, the flow rate of each pipeline and current rainfall, action includes changing pipeline diameter, changing pipeline depth etc.;Step S2: build and initialize Actor and Critic deep neural network;Step S3: define physical and economic constraints and process by reward function;Step S4: utilize urban hydrology model SWMM to design training process, while using experience replay and target network technology to train and test;Step S5: the model is evaluated and iteratively optimized to meet the preset performance standards.The present application provides an efficient, self-adapting optimization tool for urban flood control and drainage facilities design, which has important value for improving urban flood control capacity, reducing investment cost and operation and maintenance cost, and protecting urban residents' quality of life.

Description

Technical Field

[0001] This invention relates to the field of information technology, and in particular to a stormwater drainage network optimization method based on the Deep Deterministic Policy Gradient (DDPG) algorithm. Background Technology

[0002] Urban stormwater drainage systems are critical infrastructure for urban flood control and drainage. Their design optimization involves multiple objectives and constraints, such as flood control capacity, construction cost, and operating costs. Therefore, effective optimization models are essential for improving urban flood control capabilities, reducing investment and operation and maintenance costs, and are also an important means of ensuring the quality of life for urban residents.

[0003] Traditional stormwater drainage network design optimization methods mainly rely on experience and engineering specifications, such as the weighted method and the constraint method. These methods are effective in handling single-objective optimization problems, but they often fall short when dealing with multi-objective optimization and multi-constraint problems. Furthermore, these methods have limitations in handling the constraints of the drainage network system, often requiring manual adjustments, and the optimization process lacks dynamism and adaptability.

[0004] To address these issues, researchers have introduced evolutionary algorithms, such as genetic algorithms and particle swarm optimization, in an attempt to achieve breakthroughs in handling multi-objective optimization problems and constraints. However, these methods face difficulties in dealing with continuous states and action spaces, have limited effectiveness in optimizing large-scale problems, and may require significant computational time and resources.

[0005] Specifically, the optimization design problem of stormwater drainage networks is a typical multi-objective, multi-constraint continuous state and action space problem. The state space mainly consists of factors such as the water depth at each node of the network, the flow velocity in each pipe, and the current rainfall, while the action space mainly involves adjusting the diameter and burial depth of a pipe, and even measures such as adding new drainage nodes. Traditional evolutionary algorithms are mainly suitable for discrete state and action spaces. For continuous state and action spaces, these algorithms often require complex encoding and decoding processes, and may also suffer from inefficiency during the search process.

[0006] Based on this, this invention proposes a stormwater drainage network optimization model based on the Deep Deterministic Policy Gradient Algorithm (DDPG). DDPG is an advanced deep reinforcement learning algorithm capable of handling continuous state and action spaces, making it highly applicable to solving stormwater drainage network optimization problems. DDPG can directly learn the optimal policy in the continuous state and action space without complex encoding and decoding processes, greatly improving optimization efficiency and accuracy.

[0007] This new model provides an effective tool for the optimal design of urban stormwater drainage networks. It not only effectively handles multi-objective optimization problems and multiple constraints, but also better adapts to and addresses problems involving continuous states and action spaces, thus demonstrating advantages in dealing with large-scale urban stormwater drainage network optimization problems. In today's rapidly accelerating urbanization process, the proposal of this model has significant theoretical and practical value. Summary of the Invention

[0008] This invention provides a stormwater drainage network optimization method based on a deep deterministic policy gradient algorithm. Combining deep learning and reinforcement learning methods, it can effectively handle multi-objective optimization problems and multiple constraints. It also exhibits good optimization performance in continuous state and action space. It provides an efficient and adaptive optimization tool for the design of urban flood control and drainage facilities, which is of great value for improving urban flood control capabilities, reducing investment and operation and maintenance costs, and ensuring the quality of life of urban residents.

[0009] This invention provides a stormwater drainage network optimization method based on a deep deterministic strategy gradient algorithm, specifically including the following steps: The method includes the following steps:

[0010] Step S1: Define the environment and parameters of the model. Define the water depth of each node in the rainwater pipe network, the flow velocity of each pipe, and the current rainfall as the state. Change the pipe diameter, change the pipe burial depth, or add a new drainage node as the action. Finally, set the reward function by considering flood control capacity, cost, and design constraints.

[0011] Step S2: Construct the Actor network and the Critic network. The Actor network takes the current state as input and outputs the action. The Critic network takes the current state and the action as input and outputs the action value. At the same time, the network weights are initialized.

[0012] Step S3: Define physical and economic constraints, including limitations on pipe diameter and burial depth, cost constraints, and the ability to cope with rainstorms of a certain intensity, and incorporate these constraints into the reward function. If any constraint is violated, a negative reward will be given.

[0013] Step S4: Using the urban hydrological model SWMM, simulate rainwater flow and pipe network performance to generate training data. Update the Actor and Critic networks through training, and stabilize the training using experience replay and target network techniques. At the same time, conduct tests under various possible environments to ensure the model's generalization ability.

[0014] Step S5: Test the model under different rainstorm intensities to evaluate its performance and generalization ability. If the model performs poorly, adjust the model parameters and return to step S4 for iterative training until the model reaches the preset performance standard or the maximum number of iterations.

[0015] Further, step S1 includes the following steps:

[0016] Step S11: Collect and organize pipeline network information, including pipeline network layout, pipe size, and drainage node location, while preparing and processing rainfall data;

[0017] Step S12: Define a state space based on the data collected in step S11. The state space includes node water depth, pipe flow velocity, and current rainfall intensity.

[0018] Step S13: Define the action space of the model, including changing the pipe diameter, changing the burial depth, or adding new drainage nodes, and these actions are all within the actual operable range.

[0019] Step S14: Construct the reward function based on flood control capacity, cost, and basic design constraints;

[0020] Step S15: Use the urban hydrological model SWMM to establish a simulation environment. This simulation environment can output a new simulated state based on the model's actions and return corresponding rewards.

[0021] Further, step S2 includes the following steps:

[0022] Step S21: Construct an Actor network with the current state as input to determine the optimal action to be performed for the current state. Specifically, the state space s consists of the water depth h of each node in the pipe network, the flow velocity v of each pipe, and the current rainfall r, denoted as s = {h, ​​v, r}. The action space a consists of the pipe diameter d, the pipe burial depth l, and the newly added drainage node n, denoted as a = {d, l, n}. The Actor network can be viewed as a mapping function from the state space to the action space. This mapping function is implemented using a deep neural network. The Actor network has N layers, where the weights and biases of the i-th layer are W and W, respectively. i and b i The activation function ReLU is expressed as f(x). Assuming the Actor network input is the state space s and the network output is a, i.e., the pipeline configuration decision, then the Actor network can be represented as:

[0023] a = f N (W N f N-1 (W N-1 ...f2(W2f1(W1s+b1)+b2)...+b N-1 )+b N (1)

[0024] Where f1, f2, ... f N-1 f NLet W1, W2, ... Wn represent the activation functions of layers 1, 2, ..., N-1 and N of the Actor network, respectively. N-1 W N Let b1, b2, ..., bn represent the weights of layers 1, 2, ..., N-1 and N of the Actor network, respectively. N-1 b N These represent the bias values ​​of the 1st, 2nd, ..., N-1th and Nth layers of the Actor network, respectively.

[0025] Step S22: Construct a Critic network. This network accepts the current state and the actions output by the Actor network as input, and outputs the value of the corresponding action, i.e., the expected system performance evaluation. Specifically, the goal of the Critic network is to evaluate the value of taking a certain action in the current state, i.e., to evaluate the effectiveness of the selected stormwater pipe network configuration. The Critic network can be viewed as a mapping function from the state-action space to the action value. Assume that the Actor network has M layers, where the weights and biases of the j-th layer are W′ respectively. j and b′ j The activation function ReLU is expressed as g(x), the network input is {s, a}, which represents the current state and action, and the network output is named Q, which represents the value of the action. The Critic network is represented as follows:

[0026] Q = g M (W′ M g M-1 (W M-1 ...f2(W′2g1(W′1s+b′1)+b′2)...+b′ M-1 )+b′ M (2)

[0027] Where g1, g2, ... g M-1 g M Let W′1, W′2, ..., W′ be the activation functions of layers 1, 2, ..., N-1 and N of the Critic network, respectively. M-1 , W′ M Let b′1, b′2, ..., b′ be the weights of layers 1, 2, ..., N-1 and N of the Critic network, respectively. N-1 b′ N These represent the bias values ​​of the Critic network's layers 1, 2, ..., N-1 and N, respectively.

[0028] Step S23: Before training begins, the weights of the Actor and Critic networks are initialized using a normal distribution to set the initial state for subsequent network optimization;

[0029] Step S24: Define the network optimizer and loss function. This model uses the Adam optimizer and sets the loss function to the mean squared error between the predicted action value and the actual reward of the Critic network, so as to optimize the network based on the feedback of the loss function.

[0030] Further, step S3 includes the following steps:

[0031] Step S31: Define physical constraints, including limitations on pipe diameter and burial depth, as well as the connectivity of pipe flow;

[0032] Step S32: Define economic constraints based on economic considerations in practical applications;

[0033] Step S33: Define flood control capacity constraints to ensure that the optimized stormwater network can cope with rainstorms of preset intensity;

[0034] Step S34: The model handles the various constraints mentioned above by including constraints in the reward function and giving negative rewards for actions that violate the constraints. Specifically, the reward function R is defined as:

[0035] R=r1-λ1*C-λ2*V-λ3*D (3)

[0036] Where r1 is the basic reward, usually set to a positive value; C is the portion of the actual cost exceeding the budget, if it does not exceed the budget, then C = 0; V is the portion of the flood overflow exceeding the tolerance value, if it does not exceed the tolerance value, then V = 0; D is the degree of violation of physical constraints; λ1, λ2, and λ3 are the corresponding penalty coefficients.

[0037] Step S35: During training, these constraints are incorporated into the model, and at each step, it is checked whether the new state satisfies the constraints to ensure that the optimized stormwater network meets the physical, economic and flood control constraints.

[0038] Further, step S4 includes the following steps:

[0039] Step S41: Construct an urban flooding model using the Urban Hydrological Model (SWMM). This model simulates the flow of rainwater in the pipe network and generates corresponding new states and rewards based on the given actions.

[0040] Step S42: Model training. The Actor network and Critic network are trained using the new state and reward generated in step S41. The goal is to maximize the flood control capacity of the pipeline network and minimize the cost while satisfying various constraints.

[0041] Step S43: Employ experience replay and target network techniques. Experience replay breaks down data correlations and increases training stability by saving and randomly sampling past experiences, including states, actions, and rewards, during training. Target network techniques provide a stable target Q-value by creating a network with slower parameter updates, addressing the instability issue in Q-learning training. Specifically, in experience replay, the stormwater network optimization model saves a series of past experiences e = (s, a, r, s′), where s is the current state, a is the action performed, r is the reward received, and s′ is the new state after performing action a. These experiences are stored in the experience replay buffer D of the stormwater network optimization model, and a small batch of experiences is randomly sampled to update the network parameters during each training session. In target network techniques, the model has two networks with identical structures: Actor network A and its corresponding target network A′, and Critic network C and its corresponding target network C′. The parameters of networks A and C are updated using gradient descent, while the parameters of networks A′ and C′ are updated using soft updates, expressed as follows:

[0042] θ′ A =τθ A +(1-τ)θ′ A (4)

[0043] θ′ C =τθ C +(1-τ)θ′ C (5)

[0044] Where, θ A and θ C These are the parameters of networks A and C, θ′ A and θ′ C These are the parameters of networks A′ and C′, and τ is the parameter for soft updates;

[0045] Step S44: Conduct model testing to ensure the model's generalization ability. Specifically, test the model under different rainstorm conditions. At the same time, remove the noise added in step S42 during the testing phase to accurately evaluate the model's performance and make predictions based on the model.

[0046] Furthermore, in step S42, to handle continuous states and action spaces, noise is added during training to encourage the model to explore; during the prediction phase, noise is removed so that the model outputs the optimal action; specifically, this includes the following steps:

[0047] Step S421: Set the parameters of the Ornstein-Uhlenbeck process, which is used to generate random noise in the continuous action space. The Ornstein-Uhlenbeck process is a stochastic process, and its update formula in discrete time is:

[0048] X t+1 =X t +θ(μ-X t )Δt+σsqrt(Δt)W t (6)

[0049] Among them, X t The noise value X at time t. t+1 Here, θ is the noise value at time t+1, θ is the velocity parameter that determines the speed at which the noise value regresses to its long-term mean μ, where μ is the long-term mean (usually set to 0), σ is the standard deviation of the noise, which determines the amplitude of the noise, Δt is the time step, and W... t It is a random variable sampled from the standard normal distribution, and sqrt(Δt) is...;

[0050] Step S422: During the model training phase, the random noise generated in step S421 is added to the output actions of the Actor network to encourage the model to explore more possible solutions. Specifically, the action space is a = {d, l, n}. According to the Ornstein-Uhlenbeck process, noise is generated for each action parameter, that is, noise X is generated for d, l, and n respectively. d，t+1 X l，t+1 and X n，t+1 Therefore, the action a′ after adding noise is represented as:

[0051] a′=a+X t+1 ={d+X d,t+1 ,l+X l，t+1 , n+X n,t+1} (7)

[0052] In this way, the noise generated by the Ornstein-Uhlenbeck process directly affects each decision variable of the stormwater network, including the pipe diameter d, burial depth l, and the newly added drainage node n, thus providing the model with a better way to explore the solution space. In addition, the noise added to each decision variable needs to be adjusted σ according to the characteristics and dimensions of these decision variables so that the influence of noise on all decision variables is balanced.

[0053] Step S423: Adjust the random noise based on the model training effect. If the model explores too little or predicts too conservatively, increase the noise standard deviation; conversely, if the model behaves too randomly or predicts too aggressively, decrease the noise standard deviation to ensure that the model performs well under various rainfall conditions.

[0054] Further, step S5 includes the following steps:

[0055] Step S51: The model makes predictions under simulated rainstorms of varying intensities to evaluate the performance of the stormwater drainage network;

[0056] Step S52: Based on the prediction results, evaluate the model's performance in terms of flood control capacity, economic cost, and compliance with basic design constraints;

[0057] Step S53: If the model performance does not meet the preset standard or the generalization ability is insufficient, the model hyperparameters or network structure need to be adjusted, and then training should be restarted.

[0058] Step S54: If the model performance reaches the preset standard or the maximum number of iterations is reached, end the optimization and record the current model parameters for subsequent application.

[0059] As can be seen from the above technical solution, the technical effect of the stormwater pipe network optimization model based on the deep deterministic strategy gradient algorithm provided by the present invention is as follows:

[0060] 1. The model enables automated optimization design of stormwater pipe networks, effectively improving the efficiency and accuracy of network design. The DDPG-based algorithm can learn from historical data and predict the optimal stormwater pipe network design scheme, which is faster and more accurate than traditional design methods based on human experience. Furthermore, through continuous learning and iteration, the model's performance will continuously improve over time.

[0061] 2. The model considers various constraints in actual engineering projects, making the optimization results more practical. These include physical constraints (such as limitations on pipe diameter and burial depth, and water flow connectivity), economic constraints (such as construction cost budget), and flood control capacity constraints, ensuring that the model's optimization results reach the optimal level while meeting these basic conditions.

[0062] 3. The model employs a continuous action space strategy optimization method, resulting in more refined optimization results and the ability to handle more complex real-world problems. By introducing the Ornstein-Uhlenbeck noise process, the model can better explore the solution space during training, thereby finding optimal solutions. Through continuous evaluation and optimization of the model's performance, it demonstrates good performance under various environments (such as rainstorms of varying intensities). Through iterative training, the model's generalization ability can be significantly improved. Attached Figure Description

[0063] Figure 1 The flowchart of the stormwater pipe network optimization model based on the deep deterministic strategy gradient algorithm provided in the embodiment of the present invention is shown.

[0064] Figure 2The diagram illustrates a specific application of the stormwater pipe network optimization model based on the deep deterministic strategy gradient algorithm provided in this embodiment of the invention.

[0065] Figure 3 The flowchart of the algorithm for defining and processing various constraints in this invention is shown.

[0066] Figure 4 This diagram illustrates the training process of the rainwater pipe network optimization model provided in an embodiment of the present invention.

[0067] Figure 5 The results of rainwater pipe network optimization under different rainstorm recurrence cycles provided by the embodiments of the present invention are shown. Detailed Implementation

[0068] The embodiments of the technical solution of the present invention will now be described and explained in detail with reference to the accompanying drawings:

[0069] according to Figure 1 As shown, a stormwater drainage network optimization model based on a deep deterministic strategy gradient algorithm specifically includes the following steps:

[0070] Step S1: Define the environment and model parameters. This includes states, actions, and rewards. States include water depth at each node of the pipe network, flow velocity in each pipe, and current rainfall. Actions include changing the diameter of a pipe, changing the burial depth of a pipe, and adding a new drainage node. The reward function needs to comprehensively consider multiple factors such as flood control capacity, cost, and meeting basic design constraints.

[0071] Step S2: Build the model. Construct an Actor network, with the current state as input and the action as output. Construct a Critic network, with the current state and action as input and the action value as output. Initialize the network weights.

[0072] Step S3: Define and process constraints. Define physical and economic constraints for actions and states, including limitations on pipe diameter and burial depth, cost constraints, and the ability to withstand heavy rain of a certain intensity. Constraints can be processed by including them in the reward function; if a constraint is violated, a negative reward is given.

[0073] Step S4: Design the training process. Utilize the SWMM (Storm Water Management Model) to construct an urban flooding model, simulating rainwater flow and pipe network performance. Continuously update the state and rewards, obtain multiple sets of data, train the Actor and Critic networks, and employ Experience Replay and Target Network techniques to stabilize the training process. Simultaneously, test the model under various possible environments (such as different rainstorm conditions) to ensure its generalization ability.

[0074] Step S5: Processing the continuous state and action space. During training, noise is added to encourage the model to explore. In the prediction phase, noise is removed, and the optimal action is output.

[0075] Step S6: Evaluate and iteratively optimize the model. Test the model under various rainfall intensities to evaluate its performance and generalization ability. If the model performs poorly, adjust the model parameters and return to step S4. Continue from step S4 to step S6 until the model reaches the preset performance standard or reaches the maximum number of iterations.

[0076] In this embodiment, the stormwater drainage network of a frequently flooded area in a residential community in Nanjing is optimized. This area includes 39 stormwater pipelines and 40 stormwater pipe points. Other data within the area includes a DEM (Digital Image Model) and basic geographic data such as buildings and roads. Using the storm intensity formula for Nanjing, four storm events were designed in this embodiment, each lasting two hours, with return periods of 1 year, 5 years, 10 years, and 20 years, respectively. The implementation steps are as follows: Figure 2 As shown.

[0077] In this embodiment, the stormwater drainage network optimization problem involves numerous complex factors, including environment, climate, geography, and economy. Therefore, the goal of this invention is to optimize the drainage network design intelligently to improve flood control capabilities, reduce costs, and simultaneously meet basic design constraints. To achieve this goal, it is necessary to construct an environment that accurately simulates the real world and define a reward function that encompasses these objectives. To this end, this invention collects and organizes the geographical information and rainfall data of the urban watershed's stormwater drainage network required for optimization, defines a state space and action space reflecting real-world conditions, and constructs a reward function that reflects multiple objectives such as flood control capabilities and costs, as well as design constraints. Finally, a simulation environment is built using tools such as urban hydrological models. This environment can simulate corresponding new states based on the model's action outputs and return corresponding rewards. Thus, through accurate simulation of the real world and reasonable reward design, the model can continuously optimize itself during the learning process, thereby achieving the goals of optimizing the stormwater drainage network, improving flood control capabilities, and reducing costs. Therefore, step S1 includes the following steps:

[0078] Step S11: Collect and organize data. Collect geographic information of the stormwater drainage network in the urban watershed that requires stormwater drainage network optimization, including the network layout, pipe length, diameter, burial depth, and location of drainage nodes. Simultaneously, historical and predicted rainfall data, including data on rainstorms of varying intensities and frequencies, also needs to be obtained.

[0079] Step S12: Define the state space. Based on the collected data, define the state space, which may include the water depth at each node of the pipeline network, the flow velocity in each pipe, and the current rainfall intensity (including rainstorms of various intensities and frequencies).

[0080] Step S13: Define the action space. Determine the actions that the model can perform, which may include changing the diameter of a pipe, changing the burial depth of a pipe, adding a new drainage node, etc. These actions should be within the scope of realistic operability.

[0081] Step S14: Construct the reward function. The reward function should comprehensively consider multiple factors such as flood control capacity, cost, and compliance with basic design constraints. For example, a positive reward should be given when floodwaters do not overflow, a negative reward should be given when the total cost of the pipeline exceeds the budget, and a negative reward should also be given when design constraints are not met (such as the upstream pipe diameter being greater than or equal to the downstream pipe diameter). In addition, corresponding reward functions should be set for rainstorms of different intensities and frequencies to ensure that the optimized pipeline network can cope with rainfall conditions of varying intensities.

[0082] Step S15: Build the simulation environment. Using the urban hydrological model SWMM, build a simulation environment for the stormwater drainage network based on the defined states and actions, and the corresponding reward function. This environment should be able to simulate the corresponding new states based on the model's action output and return the corresponding rewards.

[0083] In this embodiment, the S2 step design of the stormwater drainage network optimization model based on the deep deterministic policy gradient algorithm mainly aims to establish two deep neural networks, namely the Actor network and the Critic network, to form strategies and evaluate their value. These two networks collaborate to achieve effective optimization of the stormwater drainage network. First, the Actor network acts as a policy generator, its basic principle being to learn and determine the optimal action to take under specific environmental conditions. In the stormwater drainage network optimization problem, these environmental conditions include current rainfall conditions, network status, urban topography, etc., while actions include measures such as adjusting pipe diameters and changing network connection methods. Through repeated learning and experimentation, the Actor network gradually understands the relationship between environmental conditions and actions, thereby learning to formulate the best network adjustment strategy under specific circumstances. Next, the Critic network acts as a policy evaluator, its basic principle being to assign a value to each state-action pair to evaluate the quality of the Actor network's strategies. In the stormwater drainage network optimization problem, the Critic network needs to comprehensively consider various influencing factors, such as flood control capacity, construction and maintenance costs, to comprehensively evaluate the strategies. The design of the Critic network is particularly important in multi-objective optimization problems because it needs to accurately evaluate the trade-offs between different objectives. Finally, the weights of the Actor and Critic networks are initialized to give the networks an initial learning state at the start of training. These weights will be continuously updated during training to gradually approach the optimal solution. Therefore, step S2 includes the following steps:

[0084] Step S21: Construct the Actor Network. This is a deep neural network whose input is the current state, including information such as water depth at each node of the pipe network, flow velocity in each pipe, current rainfall, and various possible rainstorms. This information is combined to form a state description of the stormwater pipe network. The Actor Network needs to learn from these states and determine the most appropriate actions, including changing the diameter of a pipe, changing the burial depth of a pipe, and adding new drainage nodes.

[0085] Specifically, the state space consists of factors such as the water depth h at each node of the pipe network, the flow velocity v in each pipe, and the current rainfall r, denoted as s = {h, ​​v, r}. The action space consists of decisions such as the pipe diameter d, the pipe burial depth l, and the addition of a new drainage node n, denoted as a = {d, l, n}. The Actor network can be viewed as a mapping function from the state space to the action space. This mapping function can be implemented using a deep neural network, defined as an N-layer network, where the weights and biases of the i-th layer are W and W, respectively. i and b iThe activation function is ReLU, expressed as f(x). Assuming the network input is named s (network state) and the network output is named a (network configuration decision), then the Actor network can be represented as:

[0086] a = f N (W N f N-1 (W N-1 ...f2(W2f1(W1s+b1)+b2)...+b N-1 )+b N (1)

[0087] Where f1, f2, ... f N-1 ,f N Let W1, W2, ... Wn represent the activation functions of layers 1, 2, ..., N-1 and N of the Actor network, respectively. N-1 W N Let b1, b2, ..., bn represent the weights of layers 1, 2, ..., N-1 and N of the Actor network, respectively. N-1 b N These represent the bias values ​​of the 1st, 2nd, ..., N-1th and Nth layers of the Actor network, respectively.

[0088] Step S22: Construct the Critic network. The Critic network is also a deep neural network. Its input is the current state and the action output by the Actor network, and its output is the corresponding action value. The action value represents the expected performance of the stormwater drainage network system after taking a certain action in a given state. This performance includes a comprehensive evaluation of multiple factors such as flood control capability, cost, and compliance with basic design constraints.

[0089] Specifically, the goal of a Critic network is to evaluate the value of taking a certain action in the current state, i.e., to evaluate the effectiveness of the selected stormwater drainage network configuration. Given a state space of s = {h, ​​v, r} (including factors such as water depth h at each node, flow velocity v in each pipe, and current rainfall r) and an action space of a = {d, l, n} (including decisions such as changing the diameter d of a pipe, changing the burial depth l of a pipe, and adding a new drainage node n), a Critic network can be viewed as a mapping function from the state-action space to the value of an action. Assuming the neural network has M layers, where the weights and biases of the j-th layer are W′,... j and b′ j The activation function is ReLU, denoted as g(x). The network input is {s, a}, representing the current state and action. The network output is named Q, representing the value of the action. The Critic network can be represented as follows:

[0090] Q = g M (W′M g M-1 (W M-1 ...f2(W′2g1(W′1s+b′1)+b′2)...+b′ M-1 )+b′ M (2)

[0091] Where g1, g2, ... g M-1 g M Let W′1, W′2, ..., W′ be the activation functions of layers 1, 2, ..., N-1 and N of the Critic network, respectively. M-1 , W′ M Let b′1, b′2, ..., b′ be the weights of layers 1, 2, ..., N-1 and N of the Critic network, respectively. N-1 b′ N These represent the bias values ​​of the Critic network's layers 1, 2, ..., N-1 and N, respectively.

[0092] Step S23: Initialize network weights. Before training begins, the weights of the Actor and Critic networks need to be initialized. This determines the network's state at the start of training. The initial values ​​of the weights can be chosen based on the characteristics of the problem; this invention uses a normal distribution.

[0093] Step S24: Define the network optimizer and loss function. In this model, the Adam optimizer is used, and the loss function is typically defined as the mean squared error between the predicted action value and the actual reward of the Critic network. This allows the network to be continuously optimized based on the feedback from the loss function, enabling it to better adapt to and solve stormwater drainage network optimization problems.

[0094] This embodiment carefully defines and handles constraints to ensure that the optimization process conforms to actual conditions and needs. These constraints mainly include physical constraints, economic constraints, and flood control constraints. Physical constraints, including limitations on pipe diameter and burial depth, as well as the connectivity of water flow in the pipe network, are set based on the actual physical characteristics and engineering requirements of the stormwater pipe network. In actual stormwater pipe network design, basic principles such as the upstream pipe section burial depth elevation needing to be greater than the downstream pipe section elevation, and the upstream pipe section diameter needing to be greater than or equal to the downstream pipe section diameter, must be met. These constraints are reflected in the deep learning model by setting certain penalty terms. When an action violates these physical constraints, the model receives a negative reward, thus gradually avoiding such actions during the learning process. Economic constraints mainly refer to the construction and maintenance costs of the stormwater pipe network, as economic factors are often key constraints in actual engineering applications. This constraint is also achieved by setting penalty terms; if the construction and maintenance costs resulting from the optimization exceed the predetermined budget, the model will receive a negative reward. Flood control constraints are set based on the main functions of the stormwater pipe network, namely drainage and flood control. Stormwater drainage networks need to be able to withstand heavy rainfall of a certain intensity; this is a basic requirement for flood control. Similarly, when the optimization results fail to meet the flood control requirements, the model will receive a negative reward.

[0095] After defining the above constraints, this invention incorporates these constraints into the model's reward function by introducing a penalty term, ensuring that the model consistently adheres to these constraints during the learning process. Specifically, in each training step, it checks whether the new state satisfies the constraints; if not, a negative reward is given, guiding the model's learning direction. The purpose of this design is to tightly integrate actual engineering constraints with the deep learning model's training process, enabling the model to learn optimization strategies that meet practical application needs. During the deep learning model's learning process, these constraints play a clear guiding role, ensuring that the model not only finds problem-solving strategies but also that these strategies conform to actual engineering constraints, thereby improving the model's practicality and application value. Therefore, step S3 is as follows... Figure 3 As shown, it includes the following steps:

[0096] Step S31: Define physical constraints. This includes pipe diameter constraints (the upstream pipe segment diameter must be greater than or equal to the downstream pipe segment diameter), burial depth constraints (the burial depth elevation of the upstream pipe segment and pipe point must be greater than the downstream pipe segment and pipe point elevation), and connectivity constraints. Generally, the burial depth elevation of the upstream pipe segment needs to be greater than the downstream pipe segment elevation, and the upstream pipe segment diameter needs to be greater than or equal to the downstream pipe segment diameter.

[0097] Step S32: Define economic constraints. Considering that the construction and maintenance of rainwater pipe networks require certain economic costs in practical applications, cost constraints need to be set in the model, and the cost of the pipe network needs to be kept within the budget as much as possible.

[0098] Step S33: Define flood control capacity constraints. The stormwater drainage network needs to be able to withstand rainstorms of a certain intensity; this is its basic flood control requirement. Therefore, corresponding constraints need to be set in the model. In this embodiment, the stormwater drainage network needs to be able to withstand four types of rainstorms: once-in-a-year, once-in-five-year, once-in-ten-year, and once-in-twenty-year. During these rainstorms, the stormwater drainage network should not overflow.

[0099] Step S34: Handling Constraints. After defining the various constraints mentioned above, the model designs a mechanism to handle these constraints, ensuring that the optimization result satisfies them; that is, the constraints are included in the reward function. When an action violates a constraint, it can be penalized by giving a negative reward. If the new state does not meet the flood control capacity constraint, i.e., the flood overflow predicted by the model exceeds the specified tolerance value, then a negative reward can be given.

[0100] Specifically, the reward function can be defined as:

[0101] R=r1-λ1*C-λ2*V-λ3*D (3)

[0102] Where r1 is the basic reward, usually set to a positive value; C is the portion of the actual cost exceeding the budget, if it does not exceed the budget, then C = 0; V is the portion of the flood overflow exceeding the tolerance value, if it does not exceed the tolerance value, then V = 0; D is the degree of violation of physical constraints; λ1, λ2, and λ3 are the corresponding penalty coefficients, which can be adjusted according to the actual situation.

[0103] Step S35: Implement constraint handling. Incorporate the defined constraints into the training of the deep learning model for stormwater pipe network optimization. Specific operations may require checking whether the new state satisfies the constraints at each step of model training, or checking whether the constraints are violated after each action is executed.

[0104] This embodiment focuses on addressing issues related to model construction, training, testing, evaluation, and optimization. The model constructs an urban flooding model using the SWMM urban hydrological model, providing a realistic simulation environment that allows the model to learn the optimal pipe network design scheme under diverse rainfall conditions. Simultaneously, the model incorporates experience replay and target network techniques to break down the correlations between data, improve training stability, and provide a stable target Q-value, avoiding potential instability during training. After model training, the model is tested with different rainfall intensities and frequencies to ensure good generalization ability and practicality. Finally, by evaluating the test results, if the model performance does not meet expectations, further optimization can be performed, including adjusting the network structure and learning rate. This invention handles the optimization process step-by-step, considering the balance between global and local factors, achieving the goal of optimizing flood control capacity and economic costs while ensuring the model's stability and practicality. Specifically, as... Figure 4 As shown, step S4 includes the following steps:

[0105] Step S41: Model Building. The urban flooding model is built using the Urban Hydrological Model (SWMM). This model simulates the flow of rainwater in the pipe network and generates new states and rewards based on given actions (i.e., decisions).

[0106] Step S42: Model Training. The Actor network (decision network) and Critic network (value assessment network) are trained using the generated data. The training objective is to maximize the cumulative reward, that is, to maximize the flood control capacity of the pipeline network while minimizing costs, while ensuring that constraints are met.

[0107] Step S43: Employing Experience Replay and Target Network Techniques. To make the training process more stable, this invention employs Experience Replay and Target Network techniques. Experience Replay involves saving past experiences (states, actions, rewards, etc.) and randomly selecting a portion for use during training. This breaks down the correlation between data and improves training stability. The Target Network addresses the instability problem in Q-learning caused by the use of the same parameters for the target Q-value and the actual Q-value during the update process. It provides a stable target Q-value by creating a network with the same structure as the original network but with slower parameter updates.

[0108] Specifically, in experience replay, the stormwater drainage network optimization model stores a series of past experiences e = (s, a, r, s′), where s is the current state, a is the action performed, r is the reward received, and s′ is the new state after performing action a. These experiences are stored in the experience replay buffer D of the stormwater drainage network optimization model, and a small batch of experiences is randomly selected from it to update the network parameters during each training iteration. In the target network technique, the model has two networks with identical structures: Actor network A and its corresponding target network A′, and Critic network C and its corresponding target network C′. The parameters of networks A and C are updated using gradient descent, while the parameters of networks A′ and C′ are updated using soft updates, which can be expressed as:

[0109] θ′ A =τθ A +(1-τ)θ′ A (4)

[0110] θ′ C =τθ C +(1-τ)θ′ C (5)

[0111] Where, θ A and θ C These are the parameters of networks A and C, θ′ A and θ′ C These are the parameters of networks A′ and C′, and τ is the parameter for soft updates, which can be a small value.

[0112] Step S44: Model Testing. To ensure the model's generalization ability, it needs to be tested under various possible environments (such as different rainfall conditions). This step is done by setting different rainfall intensities and frequencies.

[0113] This embodiment involves the application of the Ornstein-Uhlenbeck noise process in the multi-objective optimization problem of stormwater pipe networks. It is necessary to define the parameters of the Ornstein-Uhlenbeck noise process to accommodate the continuous action space involved in stormwater pipe network optimization, such as pipe diameter, burial depth, and the location of drainage nodes. Introducing this noise process can increase the model's exploration ability during the training phase. Therefore, during training, this noise is added to the model's output actions to encourage the model to explore more possible solutions. However, when the model makes decisions or predictions in practical applications, this noise needs to be removed so that the model can output what it considers the optimal solution without being disturbed by random noise. Furthermore, the parameters of the noise process also need to be dynamically adjusted. If the model's exploration during training is insufficient, or the prediction results are too conservative, the standard deviation of the noise can be appropriately increased; conversely, if the model's behavior is too random, or the prediction results are too aggressive, the standard deviation of the noise should be appropriately decreased. Through such adjustments, the model can perform well under different rainfall conditions, further optimizing the stormwater pipe network to meet flood control and economic needs. Specifically, step S5 includes the following steps:

[0114] Step S51: Define the parameters of the Ornstein-Uhlenbeck noise process. In the optimization problem of stormwater pipe networks, since the action space is continuous, including pipe diameter, pipe burial depth, and the location of drainage nodes, the Ornstein-Uhlenbeck process can be used to generate continuous random noise. The parameters of this process (such as the standard deviation and theta value of the noise) will affect the amplitude and frequency of the noise and need to be set according to the actual optimization problem. The Ornstein-Uhlenbeck process is a stochastic process, and its update formula in discrete time is:

[0115] X t+1 =X t +θ(μ-X t )Δt+σsqrt(Δt)W t (6)

[0116] Among them, X t θ is the noise value at time t, θ is the velocity parameter that determines the speed at which the noise value regresses to its long-term mean μ, μ is the long-term mean, usually set to 0, σ is the standard deviation of the noise, which determines the amplitude of the noise, Δt is the time step, and W t It is a random variable sampled from the standard normal distribution.

[0117] Step S52: Add noise during training. During model training, Ornstein-Uhlenbeck noise generated in step S51 is added to the actions output by the Actor network. This means that when determining the pipe diameter, burial depth, and drainage node location, some random variations are added to the model's suggestions, encouraging the model to explore more possible solutions, which can help find a better stormwater pipe network configuration.

[0118] Specifically, the action space of this invention includes the pipe diameter d, the pipe burial depth l, and the newly added drainage node n, and the action vector is a = {d, l, n}. According to the Ornstein-Uhlenbeck process, we can generate noise for each action parameter, that is, generate noise X for d, l, and n respectively. d，t+1 X l，t+1 and X n，t+1 Therefore, the action a′ after adding noise can be expressed as:

[0119] a′=a+X t+1 ={d+X d,t+1 ,l+X l，t+1 , n+X n,t+1} (7)

[0120] In this way, the noise generated by the Ornstein-Uhlenbeck process directly affects each decision variable of the stormwater network, including the pipe diameter d, burial depth l, and the number of new drainage nodes n, thus providing the model with a better way to explore the solution space. Furthermore, the noise added to each decision variable needs to be adjusted σ according to the characteristics and dimensions of these variables to keep the influence of the noise balanced across all decision variables.

[0121] Step S53: Remove noise during the prediction phase. After the model training is complete, the noise added in step S52 should be removed when using the model for prediction or decision-making. This way, in practical applications, the optimal stormwater drainage network configuration as determined by the model can be obtained without being interfered with by noise.

[0122] Step S54: Adjust the noise parameters based on the training results. If the model's exploration during training is insufficient, or the predictions are too conservative, the standard deviation of the noise can be appropriately increased; conversely, if the model's behavior is too random, or the predictions are too aggressive, the standard deviation of the noise can be appropriately decreased. This ensures that the model performs well under various rainfall conditions.

[0123] In this embodiment, the model needs to be evaluated and optimized. Different intensities of rainstorms, such as once-in-ten-year or once-in-twenty-year events, can be simulated to predict the performance of the stormwater drainage network under various rainstorm intensities, thus evaluating the model's effectiveness. Based on the prediction results, the model's flood control capacity, economic efficiency, and whether it meets basic design constraints are evaluated. This evaluation can be conducted by comparing the prediction results with actual conditions or based on preset evaluation indicators. Furthermore, the model needs to undergo iterative optimization. If the model's performance fails to meet preset standards, or if the model's ability to handle rainstorms of different intensities is uneven, further optimization is required. Optimization methods may include adjusting the hyperparameters of the deep deterministic gradient algorithm, such as the learning rate and discount factor, or adjusting the network structure, such as the number of layers and nodes. Specifically, step S6 includes the following steps:

[0124] Step S61: Model Evaluation. Simulate different rainfall intensities (such as once-in-ten-year rainfall, once-in-twenty-year rainfall, etc.) using SWMM, make predictions using the current model, and obtain the model's prediction results for the performance of the stormwater drainage network under various rainfall intensities.

[0125] Step S62: Performance Evaluation. Based on the prediction results, evaluate the model's performance, including flood control capacity, economic cost, and compliance with basic design constraints. This can be done by comparing with actual conditions or based on preset evaluation indicators.

[0126] Step S63: Iterative Optimization Judgment. If the model's performance does not meet the preset standard, or the model's generalization ability is insufficient, such as uneven handling of rainstorms of different intensities, then further optimization of the model is required. This can be done by adjusting the hyperparameters of DDPG, including the learning rate and discount factor, or by adjusting the network structure, including the number of layers and nodes. Then return to step S4 for a new round of training.

[0127] Step S64: Update the model. If the model's performance reaches the preset standard or the maximum number of iterations is reached, then this optimization ends. Record the model parameters at this point for future applications.

[0128] The optimized stormwater drainage network in this embodiment can handle four types of rainfall events with return periods of one year, five years, ten years, and twenty years, respectively. The spatial distribution map of the modified pipe sections and pipe points after optimization is shown below. Figure 5 As shown, the dashed lines represent the optimized pipeline network, while the solid lines represent the original pipeline network that does not require optimization.

[0129] In summary, the stormwater drainage network optimization model based on the Deep Deterministic Policy Gradient Algorithm (DDPG) is an advanced model combining deep learning and reinforcement learning methods. This model effectively handles multi-objective optimization problems and multiple constraints by defining clear environment and model parameters, including states, actions, and rewards. In this model, deep neural networks are used to construct Actor and Critic networks. The Actor network determines the optimal action, while the Critic network evaluates the Actor network's actions. Simultaneously, the model sets physical and economic constraints on states and actions, enabling the handling of continuous state and action spaces. The model training process employs a Swing Model of Scale (SWMM), simulating stormwater flow and drainage network performance to update states and rewards for training the Actor and Critic networks. Experience replay and target network techniques are also used to ensure training stability. To increase the model's exploratory nature, noise is added to the output actions of the Actor network during training. Through continuous iterative optimization, the model can be tested under various storm intensities to evaluate its performance and generalization ability. The model demonstrates excellent performance in handling multi-objective optimization problems and multiple constraints, as well as in continuous state and action spaces, thus providing an efficient and adaptive optimization tool for the design of urban flood control and drainage facilities.

[0130] The above description is merely a general procedure of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A stormwater drainage network optimization method based on a deep deterministic strategy gradient algorithm, characterized in that, The method includes the following steps: Step S1: Define the environment and parameters of the model. Define the water depth of each node in the rainwater pipe network, the flow velocity of each pipe, and the current rainfall as the state. Change the pipe diameter, change the pipe burial depth, or add a new drainage node as the action. Finally, set the reward function by considering flood control capacity, cost, and design constraints. Step S2: Construct the Actor network and the Critic network. The Actor network takes the current state as input and outputs the action. The Critic network takes the current state and the action as input and outputs the action value. At the same time, the network weights are initialized. Step S3: Define physical and economic constraints, including limitations on pipe diameter and burial depth, cost constraints, and the ability to cope with rainstorms of a certain intensity, and incorporate these constraints into the reward function. If any constraint is violated, a negative reward will be given. Step S4: Using the urban hydrological model SWMM, simulate rainwater flow and pipe network performance to generate training data. Update the Actor and Critic networks through training, and stabilize the training using experience replay and target network techniques. At the same time, conduct tests under various possible environments to ensure the model's generalization ability. Step S5: Test the model under different rainstorm intensities to evaluate the model's performance and generalization ability. If the model performs poorly, adjust the model parameters and return to step S4 for iterative training until the model reaches the preset performance standard or reaches the maximum number of iterations. Step S2 includes the following steps: Step S21: Construct an Actor network with the current state as input to determine the optimal action to be performed for the current state. Specifically, the state space s consists of the water depth h of each node in the pipe network, the flow velocity v of each pipe, and the current rainfall r, denoted as s = {h, ​​v, r}. The action space a consists of the pipe diameter d, the pipe burial depth l, and the newly added drainage node n, denoted as a = {d, l, n}. The Actor network can be viewed as a mapping function from the state space to the action space. This mapping function is implemented using a deep neural network. The Actor network has N layers, where the weights and biases of the i-th layer are respectively... and The activation function ReLU is expressed as f(x). Assuming the Actor network input is the state space s and the network output is a, i.e., the pipeline configuration decision, then the Actor network can be represented as: Where f1, f2,…f N-1 ,f N These represent the first, second, ... layers of the Actor network, respectively. Layer, number Activation functions of layers; W1, W2, ... W N-1 W N These represent the first, second, ... layers of the Actor network, respectively. Layer, number Layer weights, b1, b2, ... b N-1 ,b N These represent the first, second, ... layers of the Actor network, respectively. Layer, number Layer bias value; Step S22: Construct a Critic network. This network accepts the current state and the actions output by the Actor network as input, and outputs the value of the corresponding action, i.e., the expected system performance evaluation. Specifically, the goal of the Critic network is to evaluate the value of taking a certain action in the current state, i.e., to evaluate the effectiveness of the selected stormwater pipe network configuration. The Critic network can be viewed as a mapping function from the state-action space to the action value. Assume that the Actor network has M layers, where the weights and biases of the j-th layer are respectively... and The activation function ReLU is expressed as g(x), the network input is {s, a}, which represents the current state and action, and the network output is named Q, which represents the value of the action. The Critic network is represented as follows: Among them, g1, g2, ... g M-1 ,g M These represent the Critic network layers 1, 2, ..., 6, 7, ... Layer, number The activation functions of the layers, W'1, W'2, ..., W' M-1 ,W' M These represent the Critic network layers 1, 2, ..., 6, 7, ... Layer, number Layer weights, b'1, b'2, ..., b' N-1 ,b' N These represent the Critic network layers 1, 2, ..., 6, 7, ... Layer, number Layer bias value; Step S23: Before training begins, the weights of the Actor and Critic networks are initialized using a normal distribution to set the initial state for subsequent network optimization; Step S24: Define the network optimizer and loss function. This model uses the Adam optimizer and sets the loss function to the mean squared error between the predicted action value and the actual reward of the Critic network, so as to optimize the network based on the feedback of the loss function.

2. The stormwater pipe network optimization method based on a deep deterministic strategy gradient algorithm according to claim 1, characterized in that, Step S1 includes the following steps: Step S11: Collect and organize pipeline network information, including pipeline network layout, pipe size, and drainage node location, while preparing and processing rainfall data; Step S12: Define a state space based on the data collected in step S11. The state space includes node water depth, pipe flow velocity, and current rainfall intensity. Step S13: Define the action space of the model, including changing the pipe diameter, changing the burial depth, or adding new drainage nodes, and these actions are all within the actual operable range. Step S14: Construct the reward function based on flood control capacity, cost, and basic design constraints; Step S15: Use the urban hydrological model SWMM to establish a simulation environment. This simulation environment can output a new simulated state based on the model's actions and return corresponding rewards.

3. The stormwater pipe network optimization method based on a deep deterministic strategy gradient algorithm according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Define physical constraints, including limitations on pipe diameter and burial depth, as well as the connectivity of pipe flow; Step S32: Define economic constraints based on economic considerations in practical applications; Step S33: Define flood control capacity constraints to ensure that the optimized stormwater network can cope with rainstorms of preset intensity; Step S34: The model handles the various constraints mentioned above by including constraints in the reward function and giving negative rewards for actions that violate the constraints. Specifically, the reward function... Defined as: in, This is the base reward, which is usually set to a positive value; This refers to the portion of the actual cost that exceeds the budget. If it does not exceed the budget, then... =0; This refers to the portion of the flood overflow that exceeds the tolerance limit. If it does not exceed the tolerance limit, then... =0; It refers to the degree to which physical constraints are violated; , , That is the corresponding penalty coefficient; Step S35: During training, these constraints are incorporated into the model, and at each step, it is checked whether the new state satisfies the constraints to ensure that the optimized stormwater network meets the physical, economic and flood control constraints.

4. The stormwater pipe network optimization method based on deep deterministic strategy gradient algorithm according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Construct an urban flooding model using the Urban Hydrological Model (SWMM). This model simulates the flow of rainwater in the pipe network and generates corresponding new states and rewards based on the given actions. Step S42: Model training. The Actor network and Critic network are trained using the new state and reward generated in step S41. The goal is to maximize the flood control capacity of the pipeline network and minimize the cost while satisfying various constraints. Step S43: Employ experience replay and target network techniques. Experience replay breaks down data correlations and increases training stability by saving and randomly sampling past experiences, including states, actions, and rewards, during training. Target network techniques provide a stable target Q-value by creating a network with slower parameter updates, addressing the instability issue in Q-learning training. Specifically, in experience replay, the stormwater network optimization model saves a series of past experiences e = (s, a, r, s'), where s is the current state, a is the action performed, r is the reward received, and s' is the new state after performing action a. These experiences are stored in the experience replay buffer D of the stormwater network optimization model, and a small batch of experiences is randomly sampled to update the network parameters during each training session. In the target network technique, the model has two networks with identical structures: Actor network A and its corresponding target network A', and Critic network C and its corresponding target network C'. The parameters of networks A and C are updated using gradient descent, while the parameters of networks A' and C' are updated using soft updates, expressed as follows: in, and These are the parameters of networks A and C. and These are the parameters of networks A' and C'. These are parameters for soft updates; Step S44: Conduct model testing to ensure the model's generalization ability. Specifically, test the model under different rainstorm conditions. At the same time, remove the noise added in step S42 during the testing phase to accurately evaluate the model's performance and make predictions based on the model.

5. The stormwater drainage network optimization method based on a deep deterministic strategy gradient algorithm according to claim 1, characterized in that, In step S42, in order to handle continuous states and action space, noise is added during training to encourage the model to explore; during the prediction phase, noise is removed so that the model outputs the optimal action. Specifically, the following steps are included: Step S421: Set the parameters of the Ornstein-Uhlenbeck process, which is used to generate random noise in the continuous action space. The Ornstein-Uhlenbeck process is a stochastic process, and its update formula in discrete time is: in, It is in time The noise value, It is in time The noise value, The velocity parameter determines whether the noise value reverts to its long-term mean. speed, This is the long-term mean, which is usually set to 0. The standard deviation of the noise level determines its amplitude. It is the time step. It is a random variable sampled from the standard normal distribution, #imgpt47# is...; Step S422: During the model training phase, the random noise generated in step S421 is added to the output actions of the Actor network to encourage the model to explore more possible solutions. Specifically, the action space is a = {d, l, n}. According to the Ornstein-Uhlenbeck process, noise is generated for each action parameter, i.e., noise #imgpt48#, #imgpt49#, and #imgpt50# are generated for d, l, and n respectively. Then, the action #imgpt51# after adding noise is represented as: In this way, the noise generated by the Ornstein-Uhlenbeck process directly affects each decision variable of the stormwater network, including the pipe diameter d, burial depth l, and the newly added drainage node n, thus providing the model with a better way to explore the solution space; in addition, the noise added to each decision variable needs to be adjusted according to the characteristics and dimensions of these decision variables to keep the influence of noise balanced among all decision variables. Step S423: Adjust the random noise based on the model training effect. If the model explores too little or predicts too conservatively, increase the noise standard deviation; conversely, if the model behaves too randomly or predicts too aggressively, decrease the noise standard deviation to ensure that the model performs well under various rainfall conditions.

6. The stormwater pipe network optimization method based on a deep deterministic strategy gradient algorithm according to claim 1, characterized in that, Step S5 includes the following steps: Step S51: The model makes predictions under simulated rainstorms of varying intensities to evaluate the performance of the stormwater drainage network; Step S52: Based on the prediction results, evaluate the model's performance in terms of flood control capacity, economic cost, and compliance with basic design constraints; Step S53: If the model performance does not meet the preset standard or the generalization ability is insufficient, the model hyperparameters or network structure need to be adjusted, and then training should be restarted. Step S54: If the model performance reaches the preset standard or the maximum number of iterations is reached, end the optimization and record the current model parameters for subsequent application.

Images