A raw water system scheduling and risk early warning method based on deep reinforcement learning

Abstract

The present application relates to a kind of raw water system scheduling and risk early warning method based on deep reinforcement learning, comprising the following steps: establishing the Markov game model of waterworks water intake pump station total energy consumption minimization and minimum sediment concentration problem, and the corresponding state space, action space and reward function are designed;Obtain historical operation data and hydrological data, based on the preset hydraulic model and MLP neural network, construct scheduling environment model;Through scheduling environment model and agent PPO algorithm, the deep reinforcement learning agent is trained, and based on domain randomization improves generalization performance, obtains agent strategy;Build seq2seq model, based on historical operation data, respectively on river sediment content and reservoir water supply is multi-step forecast, combined with the agent strategy obtained by training, generate risk early warning model.Compared with prior art, the present application has strong generalization performance, has the advantages such as risk early warning function.

Description

Technical Field

[0001] This invention relates to the field of raw water dispatching, and in particular to a raw water system dispatching and risk early warning method based on deep reinforcement learning. Background Technology

[0002] Water is the foundation of sustainable economic and social development. Optimizing the scheduling of raw water systems and reducing the energy consumption of water intake pumping stations while ensuring water supply security and meeting water demand is of great significance for reducing pumping station operating costs, alleviating reservoir siltation, and reducing carbon dioxide emissions.

[0003] In existing technologies, there are studies on the optimal allocation of water supply from pump stations in multi-source water supply systems, aiming to save water supply costs. The invention with application number CN202211475230.4 achieves energy-saving scheduling of water intake pump stations at water plants based on long short-term memory networks and a multi-agent actor-attention-commentator reinforcement learning algorithm. The invention with application number CN202010227091.8 determines the objective functions and constraints for the first and second levels of optimization scheduling in the raw water system, respectively, and solves them using an accelerated genetic algorithm to obtain a scheduling strategy that achieves energy saving and consumption reduction.

[0004] As can be seen from the above research, current research on the optimal scheduling of water resources mainly focuses on the macro-allocation of water resources in river basins, the optimal scheduling of reservoirs, and the optimal scheduling of urban water supply networks. There is a lack of research and practice specifically on the joint scheduling mode and technology of water sources, regulating reservoirs, and water plants, and few studies have paid attention to the generalization performance of scheduling models and risk warning. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology in terms of lack of model generalization performance and risk warning, and to provide a raw water system scheduling and risk warning method based on deep reinforcement learning.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A method for scheduling and risk early warning of raw water systems based on deep reinforcement learning includes the following steps:

[0008] S1: Establish a Markov game model for minimizing the total energy consumption of the water intake pumping station and minimizing the sediment content of the water intake. The constraints are water balance and reservoir level within a safe range. Design the corresponding state space, action space and reward function.

[0009] S2: Acquire historical operational and hydrological data, and construct a scheduling environment model based on a pre-set hydraulic model and MLP neural network;

[0010] S3: By scheduling the environment model and agent PPO algorithm, the deep reinforcement learning agent is trained based on the Markov game model, state space, action space and reward function, and the generalization performance is improved by domain randomization to obtain the agent policy.

[0011] S4: Construct a seq2seq model, and based on historical operational data, make multi-step predictions on river sediment content and reservoir water supply, and combine the trained agent strategy to generate a risk warning model;

[0012] S5: Deploy the trained agent strategy and risk warning model into the actual system.

[0013] Furthermore, the state space includes reservoir water volume, cumulative reservoir water volume, historical average water intake sediment concentration, historical average river sediment concentration, daily river sediment concentration, and reservoir outflow, while the action space represents the combination of pumps.

[0014] Furthermore, the expression for the reward function is:

[0015] r t+1 =c1r t+1,water +c2r t+1,rule +c3r t+1,reservoir +c4r t+1,power

[0016] In the formula, r t+1 Let r be the reward value at time t+1. t+1,water r t+1,ruke r t+1,reservoir r t+1,power These are used to control the sediment content of the water intake, the reservoir level, the remaining water in the reservoir, and energy consumption, respectively.

[0017] Furthermore, the scheduling environment model mainly includes a prediction model and a hydraulic model, and it also calculates the reservoir outflow at time t+1 based on the current user water usage model. t+1 The expressions for the prediction model and the hydraulic model are as follows:

[0018]

[0019] L t+1 =H(a) t )

[0020] In the formula, u t =(m,s d1 ,s d2 ,r d1 ,s u1 ,r u1 (p1, p2), where m is the month of the date to be predicted, and s d1 The sediment concentration of the river one day prior to the downstream forecast date, sd2 The sediment content of the river downstream in the previous two days, r d1 The downstream river flow rate s u1 The sediment content of the upstream river, r u1 p1 represents the upstream river flow, p2 represents the rainfall of the previous day, and p2 represents the rainfall of the previous two days; P is the sediment concentration of the river on the current day, which is predicted using real historical data. MLP neural network model; a t H represents the action taken from the t-th step to the (t+1)-th step; H represents the hydraulic model; L represents the action taken from the t-th step to the (t+1)-th step. t+1 =(v t+1 ,g 1+1 ,w 1+1 ,h t+1 ); where v t+1 Let g be the water volume in the reservoir at time t+1. 1+1 w represents the cumulative water intake at time t+1. t+1 h represents the historical average sediment content of water intake at time t+1. t+1 The historical average river sediment concentration at time t+1.

[0021] Furthermore, the hydraulic model includes pump characteristic curves and pipeline system characteristic curves.

[0022] Furthermore, step S3 includes the following steps:

[0023] S31: Based on the historical operation data of the water intake pumping station, and by randomizing the information or physical parameters in the simulation environment or introducing disturbances into the simulation environment through domain randomization, the current environmental state is obtained;

[0024] S32: The agent outputs the action to be taken based on the current environmental state;

[0025] S33: Apply the action to the simulated environment. The action will affect part of the state at the next moment. The state that is changed due to the action is calculated using a hydraulic model. The state that is not related to the action is predicted using a prediction model. And construct the environmental state and reward at the next moment.

[0026] S34: Store the current environment state, actions, the state at the next moment, and the reward;

[0027] S35: Use the PPO algorithm to iterate and obtain the agent's policy.

[0028] Furthermore, the objective function expression of the PPO algorithm is:

[0029]

[0030]

[0031] In the formula, r t (θ) is the ratio between the new strategy and the old strategy; It is the advantage function; ∈ is a hyperparameter that controls the difference between the old and new strategies, thereby adjusting the magnitude of the strategy update.

[0032] Furthermore, the basic features of the seq2seq model input include the sediment content of the downstream river, the sediment content of the upstream river, the flow rate of the downstream river, the flow rate of the upstream river, the rainfall, the temperature, and the monthly data. The evaluation indicators include the root mean square error and the mean absolute error.

[0033] Furthermore, the calculation expression for the risk warning model is as follows:

[0034]

[0035]

[0036] In the formula, The estimated reservoir flow rate for day j within the current time period i is calculated, with α as an intermediate variable. Let q be the historical average outflow of the reservoir on day k in the previous time period i-1. i-1,k This represents the actual outflow from the reservoir on day k within the previous time period i-1.

[0037] Furthermore, step S4 also includes inputting the predicted sediment content and reservoir outflow into the predictive control model to continuously output the future proposed strategies, calculating the number of days of water availability based on the reservoir capacity corresponding to the pump start-up action, and classifying the risk level accordingly. The expression for calculating the number of days of water availability is as follows:

[0038]

[0039] In the formula, D t R represents the number of days that the reservoir's water volume can meet water supply needs while the pumps are off; t-1 Q represents the remaining water volume in the reservoir the previous day; t This represents the daily water intake; O t E represents the reservoir's outflow volume for the day. t This represents the daily evaporation from the reservoir. This represents the average water outflow from the reservoir during the current period.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1) This invention combines a prediction model with a control model to endow the agent with predictive capabilities, making the scheduling system more forward-looking. It also uses deep reinforcement learning to enable the agent to learn and adapt to constantly changing hydrological conditions, achieving more flexible and intelligent scheduling decisions, thus breaking through the traditional scheduling methods.

[0042] 2) This invention introduces domain randomization technology, which enhances the agent's ability to generalize to different environments, making it more flexible and reliable in practical applications.

[0043] 3) This invention designs a risk warning model based on deep learning, performs multi-step prediction of sediment content through a seq2seq model, and dynamically models and predicts reservoir outflow based on real-time hydrological data, making the system detect potential risks more timely and helping to take necessary scheduling measures in advance. Attached Figure Description

[0044] Figure 1 This is a flowchart of the method of the present invention.

[0045] Figure 2 This is the predictive control model framework of the present invention.

[0046] Figure 3 This is the risk warning model framework in this invention.

[0047] Figure 4 To test the results of multi-step prediction of annual sediment content.

[0048] Figure 5 To test the results of multi-step prediction of reservoir outflow throughout the year.

[0049] Figure 6a The number of pumps started daily in a test year under a perfect sediment concentration prediction model.

[0050] Figure 6b This represents the number of pumps turned on daily during the test year under a manual strategy.

[0051] Figure 6c To predict the number of pumps started daily during the test year under the scheduling strategy.

[0052] Figure 7a The number of pumps started each month during the test year under different strategies.

[0053] Figure 7b This is a summary of the number of pumps started each month during the test year under different strategies.

[0054] Figure 8 To test the annual changes in reservoir water volume under different strategies.

[0055] Figure 9 The test was conducted to assess the annual average sediment concentration per unit of water intake under different strategies.

[0056] Figure 10a The test aimed to measure the change in unit water intake energy consumption during the year under different strategies.

[0057] Figure 10b This is a summary of the changes in unit water intake energy consumption during the test year under different strategies. Detailed Implementation

[0058] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0059] Example 1

[0060] like Figure 1 As shown, this invention is a method for scheduling and risk early warning of raw water systems based on deep reinforcement learning, comprising the following steps:

[0061] S1: Establish a Markov game model for minimizing the total energy consumption of the water intake pumping station and minimizing the sediment content of the water intake. The constraints are water balance and reservoir level within a safe range. Design the corresponding state space, action space and reward function.

[0062] To ensure two fundamental premises—first, maintaining the reservoir water level within a safe range; and second, guaranteeing water supply demand—the goal is to minimize pump station energy consumption and sediment content in the water intake. This minimization problem is transformed into a Markov game, involving corresponding environmental states, behaviors, and reward functions. The three fundamental premises are constrained by setting the reward function. The expression for the first premise is:

[0063]

[0064] In the formula, v t+1 Let v be the reservoir capacity in the (t+1)th step. max For maximum storage capacity, v min This represents the safe storage capacity. The practical significance of this reward function is to ensure that the reservoir capacity remains between the safe and maximum capacity at any given moment.

[0065] Regarding the second premise, its expression is:

[0066]

[0067]

[0068]

[0069] In the formula, ε1 is the interval length; ε2∈(0,1) is the low water volume penalty factor; the reservoir surplus index α represents the approximate number of days that the remaining water in the reservoir can be used by residents. t+1 This represents the average level of recent surface water consumption. To ensure the reliability of water supply when the available water in the reservoir is low, the penalty caused by higher sediment content in water intake is no longer considered when α < 15.

[0070] For the scheduling objective, namely minimizing pump station energy consumption and water intake sediment content, control is also achieved through the design of a reward function. For energy consumption control, the expression is:

[0071]

[0072] In the formula, P threshold The energy consumption threshold is set at 110 kWh / km in this invention. 3 ;P t+1 Q represents the daily energy consumption for water intake. t+1 This represents the daily water intake.

[0073] For sediment concentration control, the expression is:

[0074]

[0075] In the formula, w t+1 The historical average sediment content of water taken before step t+1. Q represents the current sediment content of the river. t+1 This represents the water intake at the current step size. The practical significance of this reward function is that when... Less than w t+1 At times, the more water you collect, the greater the reward; conversely, when... Greater than w t+1 The more water you take, the greater the penalty you will receive.

[0076] The behavior of the agent is represented by the number of pumps activated per step, expressed as:

[0077]

[0078] In the formula, v t The water volume of the reservoir; g t To accumulate reservoir water volume; w t The historical average sediment content of water intake; h t This refers to the historical average sediment content of rivers. The sediment content of the river on that day; t This refers to the outflow rate of the reservoir.

[0079] S2: Acquire historical operational and hydrological data, and construct a scheduling environment model based on a pre-set hydraulic model and MLP neural network;

[0080] S3: By scheduling the environment model and agent PPO algorithm, the deep reinforcement learning agent is trained based on the Markov game model, state space, action space and reward function, and the generalization performance is improved by domain randomization to obtain the agent policy.

[0081] S31: Based on the historical operation data of the water intake pumping station, and by randomizing the information or physical parameters in the simulation environment or introducing disturbances into the simulation environment through domain randomization, the current environmental state is obtained;

[0082] Specifically, during the construction of the raw water system simulation environment, disturbances are introduced into the environment, and Gaussian noise is added to randomize data such as hydrological environment and urban water consumption. The environmental state component o... t Use Gaussian noise (|μ|≤μ threshold ,σ≤σ threshold ,μ threshold =2.5×10 4 m 3 / d,σ threshold =2×10 4 m 3 Daily surface water usage data ( / d) were used for training, where μ threshold With σ threshold All parameters are Gaussian thresholds. Meanwhile, the hydrological conditions of rivers and the urban water demand vary significantly from year to year. Therefore, when generating the initial environmental conditions, the initial reservoir storage, changes in the hydrological environment of the current trajectory, and changes in water consumption are randomly sampled and combined to generate diverse training samples.

[0083] The initial simulation environment reset process after adding domain randomization is as follows: randomly select a year from the hydrological training dataset as the year y of this sampling segment. r Randomly select a year from the water consumption training dataset as the year y for this sampling segment. w Randomly sample the initial reservoir water volume v0∈(v min ,v max Randomly sampled water usage Gaussian noise parameter μ∈(-μ 1hreshold ,μ threshold ), σ∈(-σ threshold ,σ threshold ).

[0084] S32: The agent outputs the action to be taken based on the current environmental state;

[0085] The calculation process is as follows: Reset the environment to its initial state, and reset the counting parameter t to 1; when t is less than or equal to the total time step T step Time (T) step =365 or 366): Based on the current state s t Sample an action a from the control policy network. t ;

[0086] Perform the selected action and apply it to the simulation environment;

[0087] S33: Apply the action to the simulated environment. The action will affect part of the state at the next moment. The state that is changed due to the action is calculated using a hydraulic model. The state that is not related to the action is predicted using a prediction model. And construct the environmental state and reward at the next moment.

[0088] Next state s t+1 The calculation is as follows:

[0089]

[0090] v t+1 =v t +Q t -o t

[0091] g t+1 =g t +Q t

[0092]

[0093]

[0094]

[0095] o t+1 =o t +X~N(μ,σ 2 )

[0096] Among them, Q t Let be the water intake at time t; μ and σ are Gaussian noise parameters.

[0097] S34: Store the current environment state, actions, the state at the next moment, and the reward;

[0098] S35: Use the PPO algorithm to iterate and obtain the agent's policy.

[0099] like Figure 2 As shown, S2 and S3 together construct the core predictive control system of this invention. For the MLP, it consists of an input layer, a hidden layer, and an output layer. In this embodiment, for predicting sand content during the non-freezing period, the hidden layer of the neural network has 12 neurons; for predicting sand content during the freezing period, the hidden layer has 10 neurons. The LeakyReLU function is used as the activation function of the hidden layer to calculate the output of each neuron. For the input features, mutual information (MI) and Pearson correlation coefficient are used for feature selection, ultimately determining the output of u. t1 =(m,s d1 ,s d2 ,rd1 ,s u1 ,r u1 (p1, p2) are used as input features for the non-freezing period sediment concentration prediction model; since p1 and p2 are 0 during the freezing period, u t2 =(m,s d1 ,s d2 ,r d1 ,s u1 ,r u1 These features serve as input features for the sediment concentration prediction model during the freezing period. To eliminate the influence of the dimensionality of these features, feature normalization is also required. Furthermore, the Adam optimizer is used during model training, with a learning rate of 0.001 and a weight decay rate of 0.01. The root mean square error (MSE) and mean absolute error (MAE) are used as evaluation metrics to assess the model's performance.

[0100] The core of the PPO algorithm is to use the following strategy loss function:

[0101]

[0102]

[0103] In the formula, For estimating the advantage function, following the suggestion in the original PPO paper, GAE (generalized advantage estimation) was used. The pruning parameter ∈ was set to 0.2. Model construction parameters were set as follows: total number of iterations 300k, step size per update 840, batch size 420, and learning rate 1×10⁻⁶. 4 .

[0104] S4: Construct a seq2seq model, and based on historical operational data, make multi-step predictions on river sediment content and reservoir water supply, and combine the trained agent strategy to generate a risk warning model;

[0105] Risk model framework such as Figure 3 As shown, a seq2seq model is first used to perform multi-step prediction of river suspended solids concentration. For the input features, the sediment content of the downstream river (s) is used to predict the concentration of suspended solids. d ), upstream river sediment content (s) u ), downstream river flow (r) d ), upstream river flow (r) uUsing rainfall (p), temperature (T), and monthly data (m) as basic features, the sediment content of downstream rivers for the next 10 days is predicted using the basic features from the previous 20 days. Due to the significant fluctuations in sediment content data, a data stabilization process is applied to reduce the impact of these fluctuations on the model's predictive performance. The expression is as follows:

[0106] s = ln(s) + Min{ln(s)}

[0107] In the formula, s is the set of sediment concentration data.

[0108] To eliminate the influence of these dimensions, the features are normalized. The RMSE and MAE metrics are used to evaluate the model's performance. Figure 4 The error between the multi-step prediction results and the actual values ​​of sediment concentration during the test year (2021) is shown. The error calculation expression is as follows:

[0109]

[0110] In the formula, The error (kg / m³) between the predicted and actual sediment content on day j within the current time period i. 3 ); The predicted sediment content (kg / m³) for day j in the current time period i. 3 );s i,j The actual sediment content (kg / m³) on day j within the current time period i. 3 ).

[0111] A multi-step prediction of reservoir outflow over the next 10 days is made using a method that corrects historical data in real time. The specific expression is as follows:

[0112]

[0113]

[0114] In the formula, Estimate the reservoir flow (in ten thousand m³) for day j within the current time period i. 3 ), The historical average outflow of the reservoir on day k in the previous time period i-1 (in ten thousand m³) 3 ), q i-1,k The actual outflow of the reservoir on day k in the previous time period i-1 (in ten thousand m³) 3 ).

[0115] Figure 5 The error between the multi-step prediction results and the actual value of reservoir outflow during the test year (2021) is shown. The error calculation expression is as follows:

[0116]

[0117] In the formula, The error (in ten thousand m³) between the predicted and actual reservoir outflow on day j within the current time period i. 2 ); The predicted reservoir outflow (in ten thousand m³) for day j within the current time period i. 3 );q i,j This represents the actual reservoir outflow (in ten thousand m³) on day j within the current time period i. 3 ).

[0118] The predicted sediment content and reservoir outflow are input into the predictive control model, which then outputs the proposed strategies for the next ten days. The number of days the reservoir can supply water is calculated based on the reservoir capacity corresponding to the pump start-up action, and risk levels are assigned accordingly. In this embodiment, if the reservoir's water volume can meet water supply needs for more than 10 days while the pumps are off, it is considered safe; 8-10 days is considered Level III risk; 5-8 days is considered Level II risk; and less than 5 days is considered Level I risk. The expression for calculating the number of days the water can supply water is available is as follows:

[0119]

[0120] In the formula, D t R represents the number of days that the reservoir's water volume can meet water supply needs while the pumps are off; t-1 The remaining water volume of the reservoir the previous day (in ten thousand cubic meters) 3 );Q t Daily water inflow (10,000 m³) 3 );O t The daily reservoir outflow (in ten thousand cubic meters) 3 ); E t The daily evaporation from the reservoir (in ten thousand m³) 3 ); The average water outflow from the reservoir during the current period (in ten thousand cubic meters) 3 ).

[0121] S5: Deploy the trained agent strategy and risk warning model into the actual system.

[0122] Compared with existing technologies and artificial water intake strategies, the strategy established by this invention can reduce the unit energy consumption of the water intake system by 8.33% and the annual average sediment content of the water intake by 37.01%, while also having stronger system safety and versatility.

[0123] Figure 6 shows the number of pumps activated daily during the test year under different strategies. A perfect sediment concentration prediction model (assuming the scheduler already has information about future river sediment concentration, and the reinforcement learning model is trained using real sediment concentration data) was used as the upper bound for optimization (Strategy S1). The manual strategy (Strategy S2) and the predictive scheduling strategy (Strategy S3) are compared. Significant differences can be observed between the manual strategy and the control strategies generated by the two deep reinforcement learning training methods. The manual strategy's pump activation time is highly concentrated, mainly in June, August, and November. In contrast, the deep reinforcement learning strategy tends to frequently activate the pumps to extract water during winter when sediment concentration is low, and reduce the number of days for raw water extraction during seasons with higher sediment concentration and greater uncertainty.

[0124] Figure 7 shows the cumulative monthly pump operation status and the daily number of pumps operated under different strategies during the test year. The figure shows that the number of days without pump operation is roughly the same for all three strategies. Compared to the manual strategy, the predictive control strategy and the perfect prediction strategy tend to operate more pumps for water extraction during the winter when sediment content is lower, and reduce the number of pump operation days during the flood season when sediment content is higher and more uncertain. The number of pump operations per month is relatively even, with two pumps typically operating. The manual strategy, on the other hand, is more concentrated in its pump operation timing, continuously operating pumps for water extraction in June, August, and November. The number of pumps operated is more widespread, with one, two, or three pumps operating relatively evenly. In April and September, pump operation is rarely or never initiated.

[0125] like Figure 8 As shown in the figure, this diagram illustrates the reservoir water level changes during the test year under different strategies. It can be seen that the design of the reward function ensures that the predictive control scheme avoids violations such as the reservoir water level exceeding the minimum or maximum capacity, or insufficient water level to meet residential water demand. Simultaneously, it maintains a certain reservoir water level at the end of the year to prepare for the following year's scheduling. It should be noted that data for January and February is missing for the existing manual strategy; therefore, the initial reservoir water level for the predictive control strategy and the perfect prediction strategy is 1000–1500 × 10⁻⁶. 4 m 3 Obtained through random sampling.

[0126] like Figure 9 As shown in the figure, this diagram illustrates the changes in the cumulative average sediment concentration per unit of water intake under different strategies during the test year. It can be observed that the cumulative sediment concentration per unit of water intake is significantly lower under the predictive control strategy than under the artificial strategy, with large amounts of water intake taken when the sediment concentration is low and water intake suspended when the sediment concentration is high. Using predictive control scheduling can reduce the annual sediment concentration per unit of water intake by 37.01%.

[0127] As shown in Figure 10, this figure illustrates the annual energy consumption for water extraction under different strategies. The current artificial strategy consumes 108 kWh / km of electricity per unit of water extracted throughout the year.3 Using predictive control scheduling can reduce annual electricity consumption per unit of water intake by 8.33%.

[0128] Overall, the predictive control model's decisions and results are very close to the upper limit of optimization and significantly outperform human strategies.

[0129] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for scheduling and risk early warning of raw water systems based on deep reinforcement learning, characterized in that, Includes the following steps: S1: Establish a Markov game model for minimizing the total energy consumption of the water intake pumping station and minimizing the sediment content of the water intake. The constraints are water balance and reservoir level within a safe range. Design the corresponding state space, action space and reward function. S2: Acquire historical operational data and hydrological data, and construct a scheduling environment model based on a preset hydraulic model and MLP neural network; S3: By scheduling the environment model and agent PPO algorithm, the deep reinforcement learning agent is trained based on the Markov game model, state space, action space and reward function, and the generalization performance is improved by domain randomization to obtain the agent policy. S4: Construct a seq2seq model, and based on historical operational data, make multi-step predictions on river sediment content and reservoir water supply, and combine the trained agent strategy to generate a risk warning model; S5: Deploy the trained agent strategy and risk warning model into the actual system; The expression for the reward function is: In the formula, Let this be the reward value at time t+1. , , , These are used to control the sediment content of water intake, reservoir level, remaining water volume in the reservoir, and energy consumption, respectively. in, In the formula, The set energy consumption threshold, This refers to the daily energy consumption for water extraction. This refers to the daily water intake. for Average sediment content of water taken before step length For the present Step length river sediment content.

2. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, The state space includes reservoir water volume, cumulative reservoir water volume, historical average water intake sediment content, historical average river sediment content, daily river sediment content, and reservoir outflow. The action space refers to the combination of pumps.

3. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, The scheduling environment model mainly includes a prediction model and a hydraulic model, and it also calculates the reservoir outflow at time t+1 based on the current user water consumption model. The expressions for the prediction model and the hydraulic model are as follows: In the formula, , For the month to which the date to be predicted belongs, This represents the river sediment concentration one day prior to the downstream forecast date. This represents the sediment content of the river downstream over the previous two days. This represents the river flow rate downstream the previous day. This refers to the sediment content of the upstream river. The flow rate of the upstream river. This represents the rainfall from the previous day. This represents the rainfall from the previous two days. To use real historical data to train a tool for predicting the sediment content of rivers on a given day MLP neural network model; The action taken from the t-th step to the (t+1)-th step; For hydraulic models; ;in Let t+1 be the water volume in the reservoir. The cumulative water intake at time t+1 The historical average sediment content of water intake at time t+1. The historical average river sediment concentration at time t+1.

4. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, The hydraulic model includes pump characteristic curves and pipeline system characteristic curves.

5. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, Step S3 includes the following steps: S31: Based on the historical operation data of the water intake pumping station, and by randomizing the information or physical parameters in the simulation environment or introducing disturbances into the simulation environment through domain randomization, the current environmental state is obtained; S32: The agent outputs the action to be taken based on the current environmental state; S33: Apply the action to the simulated environment. The action will affect part of the state at the next moment. The state that is changed due to the action is calculated using a hydraulic model. The state that is not related to the action is predicted using a prediction model. And construct the environmental state and reward at the next moment. S34: Store the current environment state, actions, the state at the next moment, and the reward; S35: Use the PPO algorithm to iterate and obtain the agent's policy.

6. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, The objective function expression of the PPO algorithm is: In the formula, It is the ratio between the new strategy and the old strategy; It is the dominant function; This is a hyperparameter that controls the difference between the old and new strategies, thereby adjusting the magnitude of the strategy update.

7. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, The basic features input to the seq2seq model include sediment content in the downstream river, sediment content in the upstream river, flow rate in the downstream river, flow rate in the upstream river, rainfall, temperature, and monthly data. Evaluation indicators include root mean square error and mean absolute error.

8. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, The calculation expression for the risk warning model is as follows: In the formula, For the current time period The first in The reservoir's estimated flow rate for the day. As an intermediate variable, For the previous period The first in The historical average outflow of the reservoir in Tian For the previous period The first in The actual outflow of water from the reservoir on that day.

9. The method for scheduling and risk early warning of raw water systems based on deep reinforcement learning according to claim 1, characterized in that, Step S4 also includes inputting the predicted sediment content and reservoir outflow into the predictive control model to continuously output the future strategies, calculating the number of days of water availability based on the reservoir capacity corresponding to the pump start-up action, and classifying the risk level accordingly. The expression for calculating the number of days of water availability is as follows: In the formula, This refers to the number of days that the reservoir's water volume can meet the water supply demand when the pumps are shut off. This represents the remaining water level in the reservoir from the previous day. This represents the daily water intake. This represents the reservoir's outflow volume for the day. This represents the daily evaporation from the reservoir. This represents the average water outflow from the reservoir during the current period.

Images