Regional water demand adaptive dynamic prediction method based on deep reinforcement learning framework

Abstract

The invention discloses a regional water demand adaptive dynamic prediction method based on a deep reinforcement learning framework, and the method comprises the following steps: training a time sequence prediction model through historical water supply data, and obtaining an initial prediction value of the water demand at the next moment through the model; a reinforcement learning framework based on depth deterministic strategy gradient (DDPG) is constructed, a DDPG model is trained, the hyper-parameters are adjusted and optimized through Bayesian optimization, and correction of the initial predicted water demand is output; and adjusting the initial prediction value according to the correction value to obtain a final prediction result. According to the method, time sequence prediction and optimization control based on deep learning are effectively combined, the dynamic adaptive capacity is achieved, the regional water demand predicted value can be corrected in real time according to water supply strategy changes and environmental influences, more accurate judgment can be made, and optimal water resource distribution management is helped to be achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of water demand prediction, and in particular to a method for adaptive dynamic prediction of regional water demand based on a deep reinforcement learning framework. Background Art

[0002] In the water supply management of modern cities, accurate prediction of regional water demand (such as living communities, industrial parks, science and technology parks, industrial parks, etc.) is an important part of achieving optimal water resource allocation. Since water demand is affected by many factors, such as meteorological conditions, production operation status and the operating status of the water supply system itself, traditional prediction methods such as statistical time series analysis methods can reflect the law of water supply changes, but it is difficult to dynamically adapt to changes in water supply strategies and environmental conditions, and the prediction granularity is coarse and the prediction accuracy is low.

[0003] With the rapid development of deep learning and reinforcement learning technologies, intelligent water supply forecasting has become possible. Deep learning captures nonlinear features in data by building complex neural network models to improve forecasting accuracy; while reinforcement learning continuously optimizes control strategies and gradually approaches the optimal solution by simulating the decision-making process of intelligent agents in the environment. The combination of these two technologies provides new ideas for regional water demand forecasting. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for adaptive dynamic prediction of regional water demand based on a deep reinforcement learning framework, which can effectively solve the problems raised in the above background technology.

[0005] To solve the above problems, the technical solution adopted by the present invention is: an adaptive dynamic prediction method for regional water demand based on a deep reinforcement learning framework, including obtaining historical water supply data in the region, training a time series prediction model through the historical water supply data, and using the model to obtain an initial prediction value of the water demand at the next moment; constructing a reinforcement learning framework based on a deep deterministic policy gradient framework (DDPG), training the (DDPG) model and using Bayesian optimization to tune the hyperparameters of the model, and outputting a correction to the initial predicted water demand; adjusting the initial prediction value according to the correction value to obtain the final prediction result.

[0006] As a further preferred embodiment of the present invention, the training of the time series prediction model by using historical water supply data comprises the following steps:

[0007] S1. Sort the historical water consumption data by time, with a time granularity of Δt, to form a time series data set X = {x1, x2, ..., x n}, where x1 represents the water consumption vector n≥24 with an interval of Δt before the current time t; the corresponding target value set is y={y1,y2,...,y n}, where yi represents the water consumption at time t; the data set is standardized and divided into training set and test set;

[0008] S2. Build a time series prediction model, where the last layer is a fully connected layer, which is used to output the initial prediction result of water demand at the next moment;

[0009] S3. Train the time series prediction model and select the water consumption prediction value at time t With the actual value y i The mean square error between is used as the loss function;

[0010] S4. Use grid search method to adjust hyperparameters and use test set to verify model performance.

[0011] As a further preferred embodiment of the present invention, the time series prediction model is one of a recurrent neural network, a long short-term memory network, a gated recurrent network or a time convolutional neural network; after the time series prediction model is trained, the model is used to predict the water supply Y at time t+1 t+1 , as the initial prediction value.

[0012] As a further preferred embodiment of the present invention, the reinforcement learning framework based on the deep deterministic policy gradient framework DDPG includes

[0013] State space S: State space S t Contains the initial forecast value Y t+1 and other influencing factors X t , where X t Including water supply status, production operation status and meteorological status; water supply status includes the predicted water supply and actual water supply from t-3 to t, reservoir / water tower water level and pipe network pressure at t; meteorological status includes temperature, humidity, rainfall / snow volume, weather type at t; production operation status includes the switch status of each water pump and valve in the regional water supply network at t;

[0014] Action space A: defined as the initial predicted value Y t+1 The adjustment is δ, and the revised prediction value is Y′ t+1 =Y t+1 +δ, where δ is affected by the current state, that is, δ∝μ(S t );

[0015] Reward function R: Use the absolute error MAE as a metric and select the initial prediction value Y t+1 As the benchmark; if the DDPG model adopts the adjustment strategy δ, the absolute error of the predicted value (AE' t+1 =|Y' t+1 -y t+1|) is lower than the absolute error (AE) of the initial prediction value t+1 =|Y t+1 -y t+1 |), a positive reward is obtained; at the same time, a coefficient k is multiplied to amplify the reward signal and strengthen the difference in benefits caused by the corrective action: R = k(AE-AE′).

[0016] As a further preferred embodiment of the present invention, the deep deterministic policy gradient framework (DDPG) includes two parts, one is a policy network for selecting actions, and the other is a value network for evaluating action quality; the policy network is used to train the policy function, that is, according to the current state S t Output action δ t , δ t =μ(S t |θ μ )+N t , where N t is Gaussian noise, The value network is used to approximate the Q-value function and then evaluate the value of the state-action pair, expressed as Q(S t , δ t |θ Q ).

[0017] As a further preferred embodiment of the present invention, the deep deterministic policy gradient framework (DDPG) uses experience replay and a target network to stabilize the training process, wherein the target network includes a target policy network μ′ (S t |θ μ′ ) and the target value network Q′(S t , δ t |θ Q′ ).

[0018] As a further preferred embodiment of the present invention, the process of establishing the reinforcement learning framework based on the deep deterministic policy gradient framework (DDPG) is specifically as follows:

[0019] (a) Initialization: Randomly initialize the strategy network μ((S t |θ μ ) and value network Q((S t , δ t |θ Q ) and its target network μ′ and Q′, and set the target network parameters to be the same as the main network; initialize the experience replay buffer D; initialize the noise parameters, reward function correction coefficient k and other hyperparameters;

[0020] (b) Strategy selection: At each time step t, an action δ is selected based on the output of the policy network and the addition of noise. t , arrive at S t+1 , calculate the reward rt ; where δ t =μ(S t |θ μ )+N t ;

[0021] (c) Storage experience: Experience (S t , δ t , r t , S t+1 ) is stored in the experience replay buffer;

[0022] (d) Experience replay: A batch of samples (S) are randomly drawn from the experience replay buffer. i , δ i , r i , S i+1 ) for training;

[0023] (e) Value network update: Calculate the target Q value q i And update the value network parameters to

[0024] q i =r i +γQ′(S i+1 ,μ′(S i+1 |θ μ′ )|θ Q′ )

[0025] Minimize the loss function of the value network:

[0026]

[0027] Update the value network parameters θ by gradient descent Q :

[0028]

[0029] (f) Policy network update: The objective function of maximizing the policy network is

[0030] J(θ μ )=E S～D [Q(S,μ(S|θ μ )|θ Q )]

[0031] Update the policy network parameters using the policy gradient method:

[0032]

[0033] (g) Target network soft update: Update the target network parameters through soft update:

[0034] θ Q′ ←τθ Q+(1-r)θ Q′

[0035] θ μ′ ←τ0 μ +(1-τ)θ μ′ .

[0036] As a further preferred embodiment of the present invention, the objective function of the Bayesian optimization is the average reward during DDPG training, and a Gaussian process is used as a probability model of the objective function; the search space of hyperparameters is the learning rate of the policy network and the value network, the size of the experience replay buffer, the soft update parameters of the target network, the sample batch size, and the update frequency of the policy network.

[0037] As a further preferred embodiment of the present invention, the deep reinforcement learning framework outputs a correction value δ for the initial predicted water supply. t , adjust the initial prediction value according to the correction value to obtain the final prediction result Y′ t+1 .

[0038] Compared with the prior art, the present invention provides a method for adaptive dynamic prediction of regional water demand based on a deep reinforcement learning framework, which has the following beneficial effects:

[0039] This method trains the time series prediction network model through historical water supply data, and uses the deep deterministic policy gradient (DDPG) algorithm to optimize and correct the initial prediction value to obtain more accurate prediction results. This method is the first to combine the time series prediction model with deep reinforcement learning to optimize water demand prediction: it not only captures the dynamic changes in the water demand time series, but also selects the optimal adjustment strategy according to changes in the environment and operating conditions and makes real-time corrections, continuously optimizes the prediction results, and significantly improves the controllability, real-time and accuracy of water demand prediction.

[0040] In addition, the present invention uses the Bayesian optimization method to tune the hyperparameters of the model. Bayesian optimization models the objective function through the Gaussian process, finds the optimal hyperparameter combination under limited computing resources, and improves the model performance and training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flowchart of prediction based on a deep reinforcement learning framework in an embodiment;

[0042] Figure 2 This is a schematic diagram of the deep reinforcement learning framework;

[0043] Figure 3 This is a graph showing the change of the DDPG reward function in the embodiment;

[0044] Figure 4 The following is a comparison chart of predictions made by the deep reinforcement learning framework and LSTM predictions for the implementation example; DETAILED DESCRIPTION

[0045] If there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features.

[0046] In addition, if "and / or" or "and / or" appears in the full text, its meaning includes three parallel options. Taking "A and / or B" as an example, it includes option A, or option B, or a option in which both A and B are satisfied.

[0047] In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0048] As a specific embodiment 1 of the present invention: adaptive dynamic prediction of community water demand, this embodiment considers a living community, including residential areas, commercial areas, parks and other areas. The area is mainly supplied with water by a water company, and the collected data includes the total water supply flow, the monitored flow of the water supply network in each area, the switch status of each pump station and valve, and meteorological monitoring data such as temperature, rainfall, humidity, etc. All data are collected and recorded at a frequency of one hour. The prediction process is as follows: Figure 1 As shown, specifically:

[0049] 1. Train the LSTM prediction network model using historical data:

[0050] (1.1) The historical water supply flow data of the most recent year are sorted by time, with a time granularity of 1 hour, to form a time series data set X = {x1, x2, ..., x n}, where x i represents the water consumption vector for 24 hours before the current time t, x i ∈R 1×24 , the corresponding target value set is y = {y1, y2, ..., y n}, where y i represents the water consumption at time t, y i ∈R 1×1 .

[0051] The data set is standardized and the processing formula is: (x i -μ x ) / σ x and (y i -μ y ) / σ y , where μx and σ x are the mean and standard deviation of water consumption data respectively. The data set is divided into training set and test set, the first 80% of the data is used as training set, and the last 20% is used as test set.

[0052] (1.2) Construct an LSTM prediction network model, including genetic gate, input gate, state update, output gate, and the last layer is a fully connected layer to obtain the prediction results

[0053]

[0054] (1.3) Select the predicted water consumption value at time t With the actual value y t The mean square error between is used as the loss function, and the Adam optimizer is selected to update the model parameters using the back propagation algorithm to minimize the loss function. The update rule is: Where η is the learning rate, is the gradient of the loss function with respect to the model parameters.

[0055] (1.4) The grid search method is used to adjust the hyperparameters and the test set is used to verify the model performance.

[0056] 2. After the model training is completed, use the model to predict the water supply Y at time t+1 t+1 , as the initial prediction value.

[0057] 3. Construct a reinforcement learning framework based on the deep deterministic policy gradient framework (DDPG), including the state space, action space and reward function, which are:

[0058] State space S: State space S t Contains the initial forecast value Y t+1 and other influencing factors X t , where X t Including water supply status and regional water supply operation status and meteorological status, where the water supply status includes the predicted water supply volume Y from t-3 to t t-1 , Y t-2 , Y t-3 , actual water supply y t-1 ,y t-2 ,y t-3 , water tower water level H at time t, water supply pressure P w The meteorological state includes the temperature T, humidity M, and precipitation P at time t. The operating state includes the switch states W1, ..., W1 of each water pump in the regional water supply network at time t. n , valve switch status V1, ..., V n The state space is represented by S t =[Y t+1 ,y t-1, ..., V1, ..., V n ]. All data in the state space are standardized.

[0059] The action space A is defined as the initial predicted value Y t+1 The adjustment δ. In this embodiment, it is defined as the predicted value Y t+1 50% up and down, that is, δ∈[-0.5×Y t+1 , 0.5×Y t+1 ], then the corrected predicted value is Y′ t+1 =Y t+1 +δ.

[0060] The reward function R uses MAE as a metric and selects the initial prediction value Y t+1 If the DDPG model adopts the adjustment strategy δ, the absolute error of the predicted value (AE' t+1 =|Y' t+1 -y t+1 |) is lower than the absolute error (AE) of the initial prediction value t+1 =|Y t+1 -y t+1 |), a positive reward is obtained. At the same time, a coefficient k is multiplied to amplify the reward signal and strengthen the difference in benefits caused by the correction action: R = k (AE-AE'), and k is set to 100 in this embodiment.

[0061] 4. Define the Deep Deterministic Policy Gradient (DDPG), which consists of two parts: a policy network for selecting actions and a value network for evaluating the quality of actions. The policy network is used to train the policy function, that is, according to the current state S t Output action δ, δ=μ(S t |θ μ )+N t , where N t is Gaussian noise, The value network is used to approximate the Q-value function and then evaluate the value of the state-action pair, expressed as Q(S t ,δ|θ Q ).

[0062] 5. The DDPG model stabilizes the training process by utilizing experience replay and target network, where the network target policy network μ′(S|θ μ′ ) and the target value network Q′(S,δ|θ Q′ ).

[0063] 6. After DDPG is initialized, a reinforcement learning framework is established, such as Figure 2 As shown, the specific process is:

[0064] (a) Initialization. Randomly initialize the policy network μ(S|θ μ ) and the value network Q(s, a|θ Q ) and its target network μ′ and Q′, and set the target network parameters to be the same as the main network. Initialize the experience replay buffer D, initialize the noise parameters and other hyperparameters.

[0065] (b) Strategy selection. At each time step t, we select action δ based on the output of the policy network and add noise to reach S t+1 , calculate the reward r t , δ t =μ(S t |θ μ )+N t .

[0066] (c) Store experience. t , δ t , r t , S t+1 ) is stored in the experience replay buffer.

[0067] (d) Experience replay. A batch of samples (S i , a i , r i , S i+1 ) for training.

[0068] (e) Value network update. Update the value network parameters θ through gradient descent. Q ,

[0069] (f) Policy network update. Use the policy gradient method to update the policy network parameters:

[0070] (g) Target network soft update. Update the target network parameters by soft update: θ Q′ ←τθ Q +(1-τ)θ Q′ ,θ μ′ ←τθ μ +(1-τ)θ μ′ .

[0071] 7. The reward function during model training is as follows Figure 3 shown.

[0072] 8. Use Bayesian optimization to find the best hyperparameter combination for the DDPG model to maximize the average reward during training. Assume that the objective function is f(x) where x is the hyperparameter vector to be optimized. The Gaussian process assumes that the objective function f(x) obeys a multivariate Gaussian distribution, that is: Among them, μ(x) is the mean function, which is assumed to be a zero function; k(x, x′) is the kernel function, which represents the covariance between any two points x and x′.

[0073] The hyperparameter vector x can be expressed as: x = [η μ , η Q ,D,τ,B,v]

[0074] Where: η μ is the policy network learning rate, η Q is the learning rate of the value network, D is the size of the experience replay buffer, τ is the soft update parameter of the target network, B is the sample batch size, and v is the update frequency of the policy network

[0075] Select the point that maximizes the mean of the objective function (i.e., maximizes the expected improvement), continuously update the Gaussian process model, and find the optimal hyperparameter combination. The formula for calculating expected improvement (EI) is as follows: EI(x) = E[max(0,f(x)-f(x + ))]

[0076] f(x) is the predicted target value at the current point

[0077] f(x + ) is the current optimal target value

[0078] Through Bayesian optimization, find the hyperparameter combination that maximizes the average reward during DDPG training. In this example, it is [0.0003, 0.0001, 4000, 0.005, 64, 5].

[0079] 9. Use the trained deep reinforcement learning framework output to correct the initial predicted water supply according to the current state. The correction value is δ′, and the final prediction value is the sum of the initial prediction value and the deep reinforcement learning prediction correction value Y′ t+1 =Y t+1 +δ′; the final prediction result is as follows Figure 4 As shown, it can be seen that the accuracy of the present invention is better than that of prediction using only the LSTM model.

[0080] As a specific embodiment 2 of the present invention: adaptive dynamic prediction of water demand in industrial parks, this embodiment considers a large chemical industrial park, including public areas and enterprise areas, and 35 enterprises have settled in the enterprise area. The park is mainly supplied with water by a water company, and the collected data includes the total water supply flow, the monitoring flow of the water supply network of each company, the switch status of each pump station and valve, and meteorological monitoring data such as temperature, rainfall, humidity, etc. Industrial parks have higher requirements for water supply and higher monitoring frequency, so all data are collected and recorded at a frequency of every 15 minutes.

[0081] 1. Train the LSTM prediction network model using historical data:

[0082] (1.1) The historical water supply flow data for the most recent year are sorted by time, with a time granularity of 15 minutes, to form a time series data set X = {x1, x2, ..., x n}, where x i represents the vector consisting of the total water consumption in the 24 hours before the current time t and the water consumption of 35 enterprises in a predetermined order, x i ∈R 36×96 .

[0083] The corresponding target value set is y = {y1, y2, ..., y n}, where y i represents the water consumption at time t, y i ∈R 36×1 .

[0084] The data set is normalized and the processing formula is: and The data set is divided into a training set and a test set, with the first 80% of the data as the training set and the last 20% as the test set.

[0085] (1.2) Construct a gated recurrent unit (GRU) model, including an update gate, a reset gate, and a fully connected layer as the last layer to obtain the prediction results

[0086] (1.3) Select the predicted water consumption value at time t With the actual value y t The mean square error between is used as the loss function, and the RMSprop optimizer is selected to update the model parameters using the back propagation algorithm to minimize the loss function. The update rule is: Where η is the learning rate, is the gradient of the loss function with respect to the model parameters.

[0087] (1.4) The grid search method is used to adjust the hyperparameters and the test set is used to verify the model performance.

[0088] 2. After the model training is completed, use the model to predict the water supply Y at time t+1 t+1 ∈R 36×1 , as the initial prediction value.

[0089] 3. Construct a reinforcement learning framework based on the deep deterministic policy gradient framework (DDPG), including the state space, action space and reward function, which are:

[0090] (a) State space S: State space S t Contains the initial forecast value Yt+1 and other influencing factors X t , where X t Including water supply status, park production operation status and meteorological status. Among them, the water supply status includes the predicted water supply volume y from t-3 to t t-1 ,y t-2 ,y t-3 , actual water supply y t-1 ,y t-2 ,y t-3 , Enterprise water supply network pressure P w The meteorological state includes the temperature T, humidity M, and precipitation P at time t. The production operation state of the park includes the switch state W1, ..., W of each water pump in the water supply network at time t. n , valve switch status V1, ..., V n , day of the week D (1-7), hour H (0-23), whether it is a holiday (0 or 1). Then the action space S t =[Y t+1 ,y t-1 ,...,V1,...,V n ],S t ∈R 36×10 ; All parameters in the state space need to be processed according to the normalization formula.

[0091] (b) The action space A is defined as the initial prediction value Y t+1 Adjustment of δ∈[-1,5] 36 , where, δ=[δ1, δ2,..., δ 36 ], each δ i ∈[-1,5], indicating that the minimum adjustment value is -100% and the maximum is 500% of the initial value. The reason for this setting is the high uncertainty of water use by each enterprise in each period. The corrected forecast value is Y′ t+1 =Y t+1 ×(1+δ).

[0092] (c) Reward function R: Use the mean absolute error (MAE) as the metric and select the initial prediction value Y t+1 As the benchmark. If the DDPG model adopts the adjustment strategy δ, the sum of the absolute errors of the predicted values ​​(AE′ t+1 =|Y′ t+1 -y t+1 |) is lower than the sum of absolute errors (AE) of the initial prediction values t+1 =|Y t+1 -y t+1 |), then a positive reward is obtained. At the same time, a coefficient k is multiplied to amplify the reward signal and strengthen the difference in benefits caused by the correction action: R = k (AE-AE'), and k is set to 500 in this embodiment.

[0093] 4. Define the Deep Deterministic Policy Gradient (DDPG), which consists of two parts: a policy network for selecting actions and a value network for evaluating the quality of actions. The policy network is used to train the policy function, that is, according to the current state S t Output action δ, δ=μ(S t |θ μ )+N t , where N t is Gaussian noise, The value network is used to approximate the Q-value function and then evaluate the value of the state-action pair, expressed as Q(S t ,δ|θ Q ).

[0094] 5. The DDpG model stabilizes the training process by utilizing experience replay and the target network, where the network target policy network μ′(S t |θ μ′ ) and the target value network Q′(S t ,δ|θ Q′ ).

[0095] 6. After DDPG is initialized, a reinforcement learning framework is established. The specific process is as in Example 1, including random initialization, strategy selection, experience storage, experience playback, value network update, strategy network update, target network software update and other steps.

[0096] 7. Use Bayesian optimization to find the best hyperparameter combination for the DDPG model. Policy network learning rate η μ and the value network learning rate μ Q It is an important parameter that determines the convergence speed of the optimization process; the size of the experience replay buffer D affects the diversity of samples from past experience; the target network soft update parameter τ is used to smoothly update the target network, thereby improving the stability of training; the sample batch size B determines the number of samples used each time the parameters are updated, and the policy network update frequency v controls the update frequency of the policy network. These parameters jointly affect the performance and effect of the algorithm. The hyperparameter combination that ultimately maximizes the average reward during DDPG training is [0.0005, 0.0001, 20000, 0.005, 128, 5].

[0097] 8. Use the trained deep reinforcement learning framework output to correct the initial predicted water supply according to the current state. The correction value is δ′. The final prediction value is the combination of the initial prediction value and the deep reinforcement learning prediction correction value, i.e., Y′ t+1 =Y t+1 ×(1+δ′).

[0098] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the inventive concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. An adaptive dynamic prediction method for regional water demand based on a deep reinforcement learning framework, characterized in that: It includes obtaining historical water supply data in the area, training a time series prediction model through the historical water supply data, and using the model to obtain the initial prediction value of the water demand at the next moment; building a reinforcement learning framework based on the deep deterministic policy gradient framework DDPG, training the DDPG model and using Bayesian optimization to tune the model's hyperparameters, and outputting a correction to the initial predicted water demand; adjusting the initial prediction value according to the correction value to obtain the prediction result.

2. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: Training a time series forecasting model using historical water supply data includes the following steps: S1. Sort the historical water consumption data by time, with a time granularity of Δt, to form a time series data set X = {x1, x2, ..., x n }, where x1 represents the water consumption vector n≥24 with an interval of Δt before the current time t; the corresponding target value set is y={y1,y2,...,y n }, where y i represents the water consumption at time t; the data set is standardized and divided into training set and test set; S2. Build a time series prediction model, where the last layer is a fully connected layer, which is used to output the initial prediction result of water demand at the next moment; S3. Train the time series prediction model and select the water consumption prediction value at time t With the actual value y i The mean square error between is used as the loss function; S4. Use grid search method to adjust hyperparameters and use test set to verify model performance.

3. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 2 is characterized in that: The time series prediction model is one of a recurrent neural network, a long short-term memory network, a gated recurrent network or a time convolutional neural network; after the time series prediction model is trained, the model is used to predict the water supply Y at time t+1 t+1 , as the initial prediction value.

4. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: The reinforcement learning framework based on the deep deterministic policy gradient framework DDPG includes State space S: State space S t Contains the initial forecast value Y t+1 and other influencing factors X t , where X t Including water supply status, production operation status and meteorological status; water supply status includes the predicted water supply and actual water supply from t-3 to t, reservoir / water tower water level and pipe network pressure at t; meteorological status includes temperature, humidity, rainfall / snow volume, weather type at t; production operation status includes the switch status of each water pump and valve in the regional water supply network at t; Action space A: defined as the initial predicted value Y t+1 The adjustment is δ, and the revised prediction value is Y′ t+1 =Y t+1 +δ, where δ is affected by the current state, that is, δ∝μ(S t ); Reward function R: Use the absolute error MAE as a metric and select the initial prediction value Y t+1 As the benchmark; if the DDPG model adopts the adjustment strategy δ, the absolute error of the predicted value The absolute error (AE) below the initial prediction value t+1 =|Y t+1 -y t+1 |), a positive reward is obtained; at the same time, a coefficient k is multiplied to amplify the reward signal and strengthen the difference in benefits caused by the corrective action: R = k(AE-AE′).

5. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: The deep deterministic policy gradient framework DDPG consists of two parts: one is a policy network for selecting actions, and the other is a value network for evaluating the quality of actions; the policy network is used to train the policy function, that is, according to the current state S t Output action δ t , δ t =μ(S t |θ μ )+N t , where N t is Gaussian noise, The value network is used to approximate the Q-value function and then evaluate the value of the state-action pair, expressed as Q(S t , δ t |θ Q ).

6. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: The deep deterministic policy gradient framework DDPG uses experience replay and target network to stabilize the training process, where the target network includes the target policy network μ′(S t |θ μ′ ) and the target value network Q′(S t , δ t |θ Q′ ).

7. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: The process of establishing the reinforcement learning framework based on the deep deterministic policy gradient framework DDPG is specifically as follows: (a) Initialization: Randomly initialize the policy network μ((S t |θ μ ) and value network Q((S t , δ t |θ Q ) and its target network μ′ and Q′, and set the target network parameters to be the same as the main network; initialize the experience replay buffer D; initialize the noise parameters, reward function correction coefficient k and other hyperparameters; (b) Strategy selection: At each time step t, an action δ is selected based on the output of the policy network and the addition of noise. t , arrive at S t+1 , calculate the reward r t ; where δ t =μ(S t |θ μ )+N t ; (c) Storage experience: Store the experience (S t , δ t , r t , S t+1 ) is stored in the experience replay buffer; (d) Experience replay: A batch of samples (S) are randomly drawn from the experience replay buffer. i , δ i , r i , S i+1 ) for training; (e) Value network update: Calculate the target Q value q i And update the value network parameters to q i =r i +γQ′(S i+1 ,μ′(S i+1 |θ μ′ )|θ Q′ ) Minimize the loss function of the value network: Update the value network parameters θ by gradient descent Q : (f) Policy network update: The objective function of maximizing the policy network is J(θ μ )=E S～D [Q(S,μ(S|θ μ )|θ Q )] Update the policy network parameters using the policy gradient method: (g) Target network soft update: Update the target network parameters through soft update: i Q′ ←tth Q +(1-τ)θ Q′ i μ′ ←tth μ +(1-τ)θ μ′ 。 8. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: The objective function of the Bayesian optimization is the average reward during DDPG training, and a Gaussian process is used as the probability model of the objective function; the search space of hyperparameters is the learning rate of the policy network and the value network, the size of the experience replay buffer, the soft update parameter of the target network, the sample batch size, and the update frequency of the policy network.

9. The method for adaptive dynamic prediction of regional water demand based on deep reinforcement learning framework according to claim 1 is characterized in that: The deep reinforcement learning framework outputs a correction value δ to the initial predicted water supply t , adjust the initial prediction value according to the correction value to obtain the final prediction result Y′ t+1 .

Images