A hydrological runoff prediction model adaptive selection method based on reinforcement learning

By combining multiple data-driven models and reinforcement learning methods, and using the DQN algorithm for adaptive selection, the shortcomings of traditional hydrological runoff prediction under variable climate conditions are solved, achieving higher prediction accuracy and stability.

CN119005390BActive Publication Date: 2026-03-17CHINA YANGTZE POWER
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2026-03-17

AI Technical Summary

Technical Problem

Traditional hydrological runoff prediction methods struggle to maintain optimal prediction performance under variable climate conditions, and single models are ill-suited to complex hydrological data.

Method used

A hydrological runoff prediction model based on reinforcement learning is adopted, which combines multiple data-driven models such as RNN, LSTM and GRU, and adaptively selects them through the DQN algorithm. The root mean square error, relative square error and Nash efficiency coefficient are used as evaluation indicators to optimize the prediction effect.

Benefits of technology

It achieves adaptive model selection in changing environments, improving the accuracy and stability of predictions, and outperforming traditional methods and simple model combinations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005390B_ABST
    Figure CN119005390B_ABST
Patent Text Reader

Abstract

The application discloses a hydrological runoff prediction model adaptive selection method based on reinforcement learning, which comprises the following steps: S1, data preparation: collecting and arranging historical runoff related data of a certain hydrological station, and arranging the data in time sequence; before data analysis, the data is cleaned and pretreated; S2, feature engineering: converting the original data into features suitable for machine learning model; S3, sample set construction and data set division: constructing a sample set according to the size of input and output, and dividing the sample set into a training set and a test set according to a certain proportion; S4, prediction model training: using the training set to train multiple single prediction models driven by data respectively; S5, reinforcement learning model training: using the training set and the DQN algorithm to adaptively select the above trained models, and training the DQN model; S6, evaluation index and performance evaluation: the application can adaptively select the optimal prediction model under different working conditions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of hydrological prediction technology, and in particular to an adaptive selection method for hydrological runoff prediction models based on reinforcement learning. Background Technology

[0002] In the field of hydrology, runoff forecasting is a crucial component, holding significant importance for flood prevention and water resource management. Traditional methods largely rely on physical models, which are computationally complex and struggle to cope with varying climatic conditions. In recent years, data-driven models such as Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and Gated Recurrent Unit Networks (GRUs) have been widely used due to their high prediction accuracy. However, given the complexity of hydrological data and the variability of the environment, a single model often fails to maintain optimal predictive performance under all circumstances. Summary of the Invention

[0003] The purpose of this invention is to overcome the above-mentioned shortcomings and provide an adaptive selection method for hydrological runoff prediction models based on reinforcement learning, so as to solve the problems mentioned in the background art.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an adaptive selection method for hydrological runoff prediction models based on reinforcement learning, which includes the following steps:

[0005] S1. Data Preparation: Collect and organize historical runoff data from a hydrological station and arrange them in chronological order; clean and preprocess the data before data analysis.

[0006] S2, Feature Engineering: Transforming raw data into features suitable for use by machine learning models;

[0007] S3. Sample set construction and dataset partitioning: Construct a sample set according to the input and output sizes, and divide the sample set into a training set and a test set according to a certain ratio;

[0008] S4. Predictive Model Training: Train multiple data-driven individual predictive models using the training set.

[0009] S5. Reinforcement learning model training: Adaptively select the pre-trained model using the training set and the DQN algorithm, and train the DQN model.

[0010] S6. Evaluation Indicators and Performance Assessment: Root mean square error, relative square error, and Nash efficiency coefficient are used as indicators to evaluate the prediction performance. Before evaluation, the normalized prediction results are denormalized to named values. The prediction performance of the proposed method is compared with that of individual models and other ensemble models to verify the superiority of the method.

[0011] If the evaluation metrics are not met, return to S4-S6 for retraining and evaluation until the prediction metrics requirements are met.

[0012] Preferably, in step S1, the process of cleaning and preprocessing the data includes handling missing values, outliers, and data transformation operations to ensure data quality.

[0013] Preferably, in step S2, converting the raw data into features suitable for use by the machine learning model includes feature selection, feature scaling, and feature construction operations.

[0014] Preferably, the feature selection involved in S2 determines the magnitude of the historical input based on the partial autocorrelation function, and the runoff features are linearly normalized.

[0015] Due to the time series x in autocorrelation analysis t The degree of correlation of the values ​​of t = 1, 2, 3, ..., between time t and t+n is expressed as a value not limited to x. t+1 ~x t+n-1 The autocorrelation coefficient is calculated based on the given values, and will be affected by x. t+1 ~x t+n-1 The impact of the chosen value; to calculate the autocorrelation coefficient more realistically, it needs to be calculated under the premise of limiting other values; its definition is as follows:

[0016] pacf n =pf(x t ,x t+n )=rx t x t+n ·x t+1 x t+2 …x t+n-1

[0017] The pf function is used to solve for x. t With x t+n Excluding x t+1 x t+2 …x t+n-1 Partial correlation coefficient under the influence of factors; simultaneously, for x s and x t Under the constraint x k In this case, the partial correlation coefficient is defined as follows:

[0018]

[0019] Where rx s x t The formula for calculating the autocorrelation coefficient is as follows:

[0020]

[0021] Where the function f is the function for calculating the autocorrelation coefficient of the nth lag, Time series data x t and x t+n The average value.

[0022] Preferably, the feature scaling involved in S2 uses a linear normalization method, which allows analysts to obtain the maximum x value x in a certain feature set. max and minimum x value min The difference between them is used to establish a base; the formula is as follows:

[0023]

[0024] Where x' represents a normalized eigenvalue, which ranges from 0 to 1.

[0025] Preferably, in step S4, the training set is used to train the three time-series prediction models: data-driven RNN, LSTM, and GRU.

[0026] Preferably, the Q-Learning update rule and the loss function of the DQN algorithm in S5 are as follows:

[0027] Update rule: Q(s) t ,a t )=Q(s t ,a t )+α(r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t ));

[0028] Loss function: Loss = (r t +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )) 2 ;

[0029] Where s t Let a represent the state at time t. t r represents the action at time t. t Let α represent the reward at time t, α represent the learning rate, and γ represent the reward discount factor.

[0030] The detailed algorithm flow is as follows:

[0031] First, before the algorithm starts, an initial state is randomly selected. Then, based on this state, an action is selected to be executed. A judgment is made, that is, an action is randomly selected from the action space with a certain probability ξ. The action corresponding to the largest Q value is selected through the Q network with a probability of 1-ξ. In the programming, since the relevant parameters in the Q network are random at the beginning.

[0032] After the action selection is completed and the agent performs the operation, the environment will provide a reward and the next state s. t+1 Next, the intelligent system will generate a quadruple (s) consisting of state, action, reward, and next state. t a t r t s t+1 Stored in the experience pool;

[0033] Next, we will move on to the next state s. t+1 Consider the current state s t Repeat the above steps until the experience pool is full;

[0034] Once the experience pool is full, the network in DQN begins to update; that is, it starts randomly sampling from the experience pool and using the sampled reward r. t and the next state s t+1 Estimate the Q-value and compare it with the original Q-value to calculate the loss value, then start updating the Q-network; subsequently, the agent interacts with the environment to generate experience (s). t a t r t s t+1 The experience is then added to the experience pool, and the Q-network is updated by sampling from the experience pool. This process is repeated until the Q-network converges.

[0035] After training is complete, the trained DQN model is tested.

[0036] Preferably, in S6, the following three evaluation metrics are used to measure the predictive performance of the model: Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE), and Nash Efficiency Coefficient (NSE); the calculation formulas for each evaluation metric are as follows:

[0037]

[0038] Where n is the number of samples, y i This is the measured value of runoff. This is the predicted runoff value. This represents the average value of the measured runoff. Before forecasting, the normalized forecast result x' needs to be denormalized to generate forecast data x with actual units. The denormalization formula is as follows:

[0039] x = x min+x'(x max -x min ).

[0040] Beneficial effects of this invention:

[0041] 1. This invention considers the adaptive selection problem of models under changing environments. By combining multiple data-driven models and reinforcement learning strategies, this invention can adaptively select the optimal prediction model under different working conditions.

[0042] 2. The introduced DQN algorithm enhances the dynamic adaptability of model selection and improves the accuracy and stability of prediction;

[0043] 3. The prediction effect is comprehensively evaluated using three different evaluation indicators, and compared with conventional model combination methods (simple averaging and weighted averaging based on least squares), demonstrating the superior performance of the method of this invention. Attached Figure Description

[0044] Figure 1 A diagram illustrating reinforcement learning;

[0045] Figure 2 This is a schematic diagram of the DQN algorithm flow.

[0046] Figure 3 Visualize the partial autocorrelation function (PACF) calculation results of runoff time series at a certain station;

[0047] Figure 4 A bar chart visually compares the results of evaluation indicators for various prediction methods;

[0048] Figure 5 Box plots showing the distribution of prediction errors during the verification period for various prediction methods;

[0049] Figure 6 Visualize all prediction results and reinforcement learning model selection results during the testing period;

[0050] Figure 7 Visualize the prediction results and reinforcement learning model selection results during the testing period. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0052] Example 1: An adaptive selection method for hydrological runoff prediction models based on reinforcement learning. This method integrates multiple data-driven models, specifically RNN, LSTM, and GRU (three common data-driven models) for initial prediction, and introduces reinforcement learning (specifically the DQN algorithm) as the decision-maker. Based on the real-time prediction results, it adaptively selects the most suitable model for runoff prediction to optimize the overall prediction performance.

[0053] The specific implementation steps are as follows:

[0054] S1 Data Preparation: Collect and organize historical runoff data from a hydrological station, arranging them chronologically. Before data analysis, data cleaning and preprocessing are typically required. This includes handling missing values, outliers, and data transformation to ensure data quality.

[0055] S2 Feature Engineering: Feature engineering refers to transforming raw data into features suitable for use by machine learning models. This may include operations such as feature selection, feature scaling, and feature construction.

[0056] S3 Sample Set Construction and Dataset Partitioning: Construct a sample set based on the input and output sizes, and divide the sample set into a training set and a test set according to a certain ratio.

[0057] S4 Prediction Model Training: Multiple data-driven individual prediction models are trained using training sets. This invention initially proposes to use three time-series prediction models: RNN, LSTM, and GRU.

[0058] S5 reinforcement learning model training: Adaptively select from the three pre-trained models mentioned above using the training set and the DQN algorithm, and train the DQN model.

[0059] S6 Evaluation Metrics and Performance Assessment: Root mean square error (RMSE), relative squared error (RSE), and Nash efficiency coefficient (NSE) are used as the main metrics for evaluating prediction performance. Before evaluation, the normalized prediction results need to be denormalized back to named values. The prediction performance of the proposed method is compared with that of individual models and other ensemble models to verify the superiority of the proposed method.

[0060] If the evaluation metrics are not met, return to S4-S6 for retraining and evaluation until the prediction metrics requirements are met.

[0061] Missing values ​​in S1 can be filled in using linear interpolation, while outliers can be directly deleted.

[0062] In S2, the feature selection is based on the partial autocorrelation function (PACF) to determine the magnitude of the historical input. The runoff features are then linearly normalized.

[0063] Due to the time series x in autocorrelation analysis t The degree of correlation of the values ​​of t = 1, 2, 3, ..., between time t and t+n is expressed as a value not limited to x. t+1 ~x t+n-1 The autocorrelation coefficient is calculated based on the given values, and will be more or less affected by x. t+1 ~x t+n-1 The impact of the chosen value. To calculate the autocorrelation coefficient more realistically, it needs to be done under certain constraints. This is known as partial correlation, and its definition is as follows:

[0064] pacf n =pf(x t ,x t+n )=rx t x t+n ·x t+1 x t+2 …x t+n-1

[0065] The pf function is used to solve for x. t With x t+n Excluding x t+1 x t+2 …x t+n-1 The partial correlation coefficient under the influence of factors. Meanwhile, for x... s and x t Under the constraint x k In this case, the partial correlation coefficient is defined as follows:

[0066]

[0067] Where rx s x t The formula for calculating the autocorrelation coefficient is as follows:

[0068]

[0069] Where the function f is the function for calculating the autocorrelation coefficient of the nth lag, Time series data x t and x t+n The average value.

[0070] Feature scaling in S2 uses a linear normalization method, which allows analysts to obtain the maximum x value x in a given feature set. max and minimum x valuemin The difference between them is used to establish a base. The formula is as follows:

[0071]

[0072] Where x' represents a normalized eigenvalue, which ranges from 0 to 1.

[0073] In S3, input-output sample pairs are constructed based on the length of the input and output, and the sample set is constructed by sliding sequentially on the time axis. Then, the sample set is divided into training set and test set according to a certain ratio.

[0074] The structure and principles of the three prediction models involved in S4 are described below:

[0075] 1)RNN (Recurrent Neural Network)

[0076] Model structure: RNN is a type of feedforward neural network. Its network structure contains recurrent connections, which can receive sequential data input and take previous information into account when processing each time step.

[0077] Principle: The key idea of ​​RNNs is to use a shared weight matrix to share parameters across sequential data, thereby passing the contextual information of the sequential data to each time step of the model. However, due to the vanishing or exploding gradient problem, traditional RNNs perform poorly when dealing with long-term dependent data.

[0078] 2)LSTM(Long Short-Term Memory)

[0079] Model structure: LSTM is a special variant of RNN, which includes input gate, forget gate, output gate and memory cell, each gate controlling the transmission and forgetting of information.

[0080] Principle: The gating mechanism in LSTM can effectively solve the long-term dependency problem in traditional RNNs. By controlling the forget gate and input gate, LSTM can selectively remember or forget information, thereby better capturing long-term dependencies in sequence data.

[0081] 3) GRU (Gated Recurrent Unit)

[0082] Model structure: GRU is also an improved RNN model, which is simpler than LSTM and only contains update gate and reset gate.

[0083] Principle: GRU borrows the gating mechanism of LSTM, but reduces the complexity of the model by merging the forget gate and input gate into an update gate, and by using a simplified reset gate. In this way, GRU can capture long-term dependencies in sequence data in a lightweight and efficient manner, making it suitable for many sequence modeling tasks.

[0084] The three models described above each have their unique advantages in processing sequence data. LSTM and GRU address the bottleneck problem of traditional RNNs by introducing gating mechanisms. When choosing a suitable model, one can consider the specific task and the long-term dependency of the data.

[0085] Reinforcement learning in S5 is a feedback-based machine learning technique suitable for sequential decision-making problems. Here, we use it to solve the adaptive prediction model selection problem. Reinforcement learning (RL) aims to study how an agent can maximize its rewards in complex and uncertain environments. Figure 1 As shown, a reinforcement learning system consists of two parts: an agent and an environment, which interact with each other. After encountering the environment, the agent selects an action based on its state, i.e., makes a decision. The environment provides feedback on the agent's actions, including the next state and an immediate reward. The agent's goal is to maximize its chances of obtaining the reward.

[0086] Furthermore, this paper samples the Deep Q-Network (DQN) algorithm, a reinforcement learning algorithm based on neural networks, whose goal is to train an agent to perform optimal behavior in a given environment. DQN is based on the Q-learning algorithm and achieves this by constructing a function that approximates the Q-value.

[0087] The core idea of ​​Q-learning is to learn a Q-value function, which represents the expected reward for a given combination of states and actions. DQN uses a neural network to approximate this Q-value function; the network's input is the current state, and its output is a Q-value estimate for each action. DQN incorporates experience replay to improve training effectiveness and stability.

[0088] Experience replay: During training, the agent's experiences are stored in an experience replay memory. The agent trains by randomly drawing from the experiences stored in the experience replay memory. The advantage of this is that the correlation between training samples is relatively small, allowing for better learning.

[0089] Two important formulas are used in DQN: the Q-Learning update rule and the DQN loss function.

[0090] Update rule: Q(s) t ,a t )=Q(s t ,at )+α(r+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t ))

[0091] Loss function: Loss = (r t +γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )) 2

[0092] Where s t Let a represent the state at time t. t r represents the action at time t. t Let α represent the reward at time t, α represent the learning rate, and γ represent the reward discount factor.

[0093] Detailed algorithm flow is as follows Figure 2 The introduction is as follows:

[0094] First, an initial state is randomly selected before the algorithm begins. Then, based on this state, an action is selected for execution. This requires a judgment: with a certain probability ξ, an action is randomly selected from the action space, and then, with probability 1-ξ, the action corresponding to the largest Q-value is selected through the Q-network. In the programming, since the relevant parameters in the Q-network are initially randomized,...

[0095] After the action selection is completed and the agent performs the operation, the environment will provide a reward and the next state s. t+1 Next, the intelligent system will generate a quadruple (s) consisting of state, action, reward, and next state. t a t r t s t+1 Stored in the experience pool.

[0096] Next, we will move on to the next state s. t+1 Consider the current state s t Repeat the above steps until the experience pool is full.

[0097] Once the experience pool is full, the network in DQN begins to update. This involves randomly sampling from the experience pool and using the sampled reward r. t and the next state s t+1 Estimate the Q-value and compare it with the original Q-value to calculate the loss value, then begin updating the Q-network. Afterwards, the agent interacts with the environment to generate experience (s...). t a t r t s t+1The experience is then placed into the experience pool, and then sampled from the experience pool to update the Q network. This process is repeated until the Q network converges.

[0098] After training is complete, the trained DQN model is tested.

[0099] Furthermore, S6 uses the following three evaluation metrics to measure the model's predictive performance: Mean Absolute Percentage Error (MAPE), Root Mean Square Error (RMSE), and Nash Efficiency Coefficient (NSE). MAPE and RMSE are used to evaluate the model's predictive accuracy and error magnitude; lower values ​​are better. NSE measures the goodness of fit between the model and the observed values; a higher value indicates a better fit. The formulas for each evaluation metric are as follows:

[0100]

[0101] Where n is the number of samples, y i This is the measured value of runoff. This is the predicted runoff value. This represents the average value of the measured runoff. It's worth noting that before forecasting, the normalized forecast result x' needs to be denormalized to obtain forecast data x with actual units. The denormalization formula is as follows:

[0102] x = x min +x'(x max -x min )

[0103] The relevant variables are the same as those in the normalization formula mentioned earlier.

[0104] If the evaluation metric in S6 is satisfactory, the process ends. If not, continue adjusting the model hyperparameters and repeat S4-S6 until the model evaluation metric meets certain conditions, such as (MAPE < 0.2 or NSE > 0.6).

[0105] Example 2:

[0106] Taking a certain hydrological station as an example, a total of 1,458 daily runoff data points were collected from January 1, 2018 to December 28, 2021.

[0107] S1. Arrange the collected data in chronological order. If there are missing or outliers, remove the outliers and then imput them.

[0108] In S2, the `plot_pacf` function from the `statsmodels` library in Python is used to perform partial correlation calculations on the runoff time series data from hydrological stations and the results are visualized. Figure 3 ,from Figure 3The results show that only four historical data points meet the partial autocorrelation test at a confidence interval of 95%. Therefore, the four historical data points are used as input to predict the next data point. Simultaneously, the runoff data are normalized.

[0109] In S3, four consecutive historical runoffs and one future runoff are first used as one input and one output sample, and the sample set is constructed by moving them sequentially. The training sample set and the test sample set are divided in a 7:3 ratio.

[0110] S4 uses the training set to train three neural network models: RNN, LSTM, and GRU. The number of training iterations is set to 200, the learner is Adam, and the learning rate is set to 0.01.

[0111] S5, use the training set to train the reinforcement learning model DQN, set the capacity of the experience pool to 20000, the batch size to 64, and the probability of exploration exploitation ξ to 0.01.

[0112] Furthermore, in this embodiment,

[0113] State S t =q t-4 ,q t-3 ,q t-2 ,q t-1 Here, q represents the historical runoff value;

[0114] award

[0115] Action a t =0, 1, and 2 represent the selection of prediction models RNN, LSTM, and GRU at time t, respectively.

[0116] S5 uses MAPE, RMSE, and NSE metrics and a test set to compare the results of three individual models (RNN, LSTM, and GRU) and two combinations thereof (one using the average of the predictions from the three individual models as the combined prediction result; the other using the least squares method to calculate a weighted average of the predictions from the training set and the three individual models, and using the weighted coefficients to obtain the combined result on the test set) with the results after dynamically selecting the prediction model through reinforcement learning. The final comparison results are shown in Table 1.

[0117] Table 1 Comparison of Evaluation Indicators for Predictive Models

[0118]

[0119] From Table 1 and Figure 4As can be seen, reinforcement learning performs best in all three evaluation metrics, demonstrating the superiority of this invention. The overall prediction performance of the combined model is greater than that of a single model; among the single models, the prediction performance is LSTM > GRU > RNN. In the combined model, reinforcement learning > weighted average > average; the prediction performance of all models from highest to lowest is: reinforcement learning > weighted average > LSTM > average GRU > RNN.

[0120] Figure 5 The box plots show the prediction error distribution of each model during the testing period, revealing the overall prediction error and fluctuation range of each model. Among the individual models, LSTM exhibits the smallest prediction error and fluctuation range. Among the combined models, reinforcement learning shows the smallest prediction error and fluctuation range.

[0121] from Figure 6 As can be seen, the predicted results are quite close to the actual results, and various models were selected when using reinforcement learning for model selection. Due to the high density of data, a portion of the data from the testing period was selected for visualization to further observe the prediction curve, as shown below. Figure 7 As shown. From Figure 7 The results show that reinforcement learning's runoff predictions are closer to the measured runoff values. Furthermore, the number of models selected is generally positively correlated with the prediction performance of each individual model; that is, the proportion of selected individual models, from most to least, is: LSTM > GRU > RNN.

[0122] The above embodiments are merely preferred technical solutions of the present invention and should not be considered as limitations on the present invention. The scope of protection of the present invention should be limited to the technical solutions described in the claims, including equivalent substitutions of the technical features described in the claims. That is, equivalent substitutions and improvements within this scope are also within the scope of protection of the present invention.

Claims

1. A method for adaptive selection of hydrological runoff prediction models based on reinforcement learning, characterized in that: It comprises the following steps: S1, data preparation: collect and organize the historical runoff related data of a hydrological station, and arrange them in chronological order; before data analysis, clean and pretreat the data; S2, feature engineering: convert the original data into features suitable for machine learning models; S3, sample set construction and data set division: construct a sample set according to the size of the input and output, and divide the sample set into a training set and a test set according to a certain proportion; S4, prediction model training: use the training set to train multiple single prediction models driven by data respectively; S5, reinforcement learning model training: use the training set and DQN algorithm to adaptively select the above trained models, and train the DQN model; S6, evaluation index and performance evaluation: use root mean square error, relative square error and Nash efficiency coefficient as evaluation indexes of prediction effect, and denormalize the normalized prediction results to named values before evaluation; compare the prediction effect of the proposed method with that of single model and other integrated models to verify the superiority of the method; If the evaluation index is unqualified, return to S4-S6 for retraining and evaluation until the prediction index requirements are met; The Q-Learning update rule of the DQN algorithm in S5 and the loss function of DQN are as follows: Update rule: ; Loss function: ; where s t represents the state at time t, a t represents the action at time t, represents the reward at time t, a represents the learning rate, and g represents the reward discount factor; The detailed algorithm flow is as follows: First, randomly select an initial state before starting the algorithm, then select an action based on the state, make a judgment, that is, randomly select an action in the action space with a certain probability ξ, and select an action with the maximum Q value with a probability of 1-ξ; in programming, because the related parameters in the Q network are random at the beginning; After the end of action selection, the agent performs the operation, and the environment will feedback the reward and the next state s t+1 ; then, the agent will store the four-tuple (s t , a t , r t , s t+1 ) consisting of state, action, reward and next state into the experience pool; Next state s t+1 Consider the current state s t Repeat the above steps until the experience pool is full; When the experience pool is full, the network in DQN starts to update; that is, it starts to randomly sample from the experience pool, calculates the loss value by comparing the reward r t and the next state s t+1 with the original Q value, and starts to update the Q network; then, the agent interacts with the environment, generates experience (s t , a t , r t , s t+1 ), and puts the experience into the experience pool, and then samples from the experience pool to update the Q network, which is repeated until the Q network converges. After training, test the trained DQN model.

2. The method of claim 1, wherein the method is characterized by: In S1, the process of cleaning and pretreating the data includes processing missing values, outliers and data conversion operations to ensure data quality.

3. The method of claim 1, wherein the method is characterized by: In S2, the original data is converted into features suitable for machine learning models Including feature selection, feature scaling, and feature construction operations.

4. The method of claim 1, wherein the method further comprises: The feature selection involved in S2 determines the size of the historical input according to the partial autocorrelation function, and linearly normalizes the runoff features; Due to the time series of autocorrelation analysis At any time t and t + n The degree of correlation between the values ​​is not limited. The autocorrelation coefficient is calculated based on the given values ​​and will be affected by the given values. The impact of the chosen value; to calculate the autocorrelation coefficient more realistically, it needs to be calculated under the premise of limiting other values; its definition is as follows: ; where the pf function is the solution to with In the case of excluding the influence of factors, the partial correlation coefficient; at the same time, for and In the case of limiting , the definition of the partial correlation coefficient is as follows: ; wherein The formula for calculating the autocorrelation coefficient value is as follows: ; where the function f is a function that calculates the lag n order autocorrelation coefficient, are the mean values of the time series data and respectively.

5. The method of claim 1, wherein the method is characterized by: The feature scaling involved in S2 uses a linear normalization method that allows the analyst to obtain the difference between the maximum x value and the minimum x value in a certain feature set and establish a base; The formula is as follows: ; wherein represents a normalized certain feature value, which ranges between 0-1.

6. The method of claim 1, wherein the method further comprises: In S4, the training set is used to train the three time series prediction models of RNN, LSTM and GRU driven by data respectively.

7. The method of claim 1, wherein the method further comprises: determining a first loss value of the first model; determining a second loss value of the second model; and selecting the model with the smaller loss value as the final model. The following three evaluation indexes are used in S6 to measure the prediction performance of the model: mean absolute percentage error MAPE, root mean square error RMSE, and Nash efficiency coefficient NSE; the calculation formulas of each evaluation index are as follows: ; ; ; where n is the number of samples, is the measured value of the runoff, is the predicted value of the runoff, denotes the average value of the measured values of the runoff; the normalized prediction result needs to be de-normalized before the prediction to the prediction data with actual units The formula for de-normalization is as follows: 。

Citation Information

Patent Citations

  • River water temperature prediction method based on LSTM deep learning

    CN112116147A

  • Photovoltaic power generation power prediction method and system based on multi-data model fusion

    CN113919545A