Lake and reservoir water taking optimization method and system based on LSTM model

Abstract

The invention relates to a lake reservoir water taking optimization method and system based on an LSTM model, and the method comprises the steps: collecting multi-source data including water quality, hydrology and a wind field in real time, carrying out the data preprocessing, inputting a long-short-term memory neural network agent model, generating time sequence feature information, inputting a pre-constructed actor reviewer model, and carrying out the time sequence feature information; an optimal water taking scheme is obtained; the actor commentator model comprises a value network and a strategy network; training the actor reviewer model in combination with a target network mechanism and an experience playback mechanism; and based on the optimal training parameter combination, training the actor reviewer model in a reinforcement learning environment composed of a state space, a reward function and an action space to obtain a trained actor reviewer model. Compared with the prior art, the optimization result and the mechanism model calculation result are high in consistency, the feasible optimal water taking scheme can be rapidly output, and the water supply requirements of lakes and reservoirs in the water taking opening water quality abnormal period are effectively met.

Description

Technical Field

[0001] This invention relates to the field of lake and reservoir scheduling optimization technology, and in particular to a method and system for optimizing water intake in lakes and reservoirs based on an LSTM model, which is applicable to water intake scheduling decisions during periods of abnormal water quality at the water intake of lakes and reservoirs. Background Technology

[0002] Lakes and reservoirs face the dual challenges of exceeding water quality standards and increased water supply pressure during periods of abnormal water quality at their intakes (such as saltwater intrusion or upstream sewage discharge). Traditional methods rely on mechanistic models for simulation, but these are computationally time-consuming and difficult to implement in real-time optimization and scheduling. In recent years, surrogate models based on neural networks have improved computational efficiency, but due to their "black box" nature, they are difficult to use directly for solving multi-objective optimization problems.

[0003] Traditional mathematical optimization algorithms are prone to getting stuck in local optima, slow convergence, and unstable strategies in reservoir management, making them difficult to apply in practical engineering. Reinforcement learning, as a model-free trial-and-error learning algorithm, is suitable for decision-making problems in complex environments. Therefore, it is essential to propose a reinforcement learning method capable of solving multi-objective optimization problems in lakes and reservoirs.

[0004] The invention disclosed in CN118709854A provides a method, device, and medium for predicting water quality in lakes and reservoirs. The method includes the following steps: acquiring online monitoring data; preprocessing the monitoring data to obtain preprocessed data; constructing a two-dimensional hydrodynamic water quality mechanism model using water quality, hydrodynamic, and wind field data as boundary conditions; performing STL time-series decomposition on historical online monitoring data as template data, and generating simulated water quality data based on the decomposition results; combining the current data to construct simulated operating conditions, calculating the water quality indicators at the outlet based on the two-dimensional hydrodynamic water quality mechanism model; training a long short-term memory neural network model using the simulated operating conditions and the outlet water quality indicators to obtain a water quality prediction model; and using the water quality prediction model to predict the water quality indicator values ​​of lakes and reservoirs under gate pump flow operation. Compared with existing technologies, this invention has advantages such as fast calculation speed, low requirement for historical data during the training phase, and high accuracy. However, the model designed in this scheme has weak robustness and cannot effectively cope with the water supply demand of lakes and reservoirs during periods of abnormal water quality at the intake. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a lake and reservoir water intake optimization method based on LSTM model, which can quickly generate the optimal water intake plan that takes into account both water quality and quantity during periods of abnormal water quality at the water intake.

[0006] The objective of this invention can be achieved through the following technical solutions: A method for optimizing water intake in lakes and reservoirs based on an LSTM model, comprising the following steps: Real-time collection of multi-source data, including water quality, hydrology, and wind field, is performed and preprocessed. The preprocessed multi-source data is then input into a pre-constructed long short-term memory neural network surrogate model to generate time-series feature information. The time-series feature information and multi-source data are then input into a pre-constructed actor critic model to obtain the optimal water intake scheme. The actor critic model includes a value network and a policy network; the training process of the actor critic model includes: training the actor critic model by combining a target network mechanism and an experience replay mechanism; constructing a target network with the same network structure as the value network, and periodically using a soft update strategy to update the parameters of the target network according to the parameters of the value network; further optimizing the hyperparameters during the training process to determine the optimal combination of training parameters. Based on the optimal combination of training parameters, the actor critic model is trained in a reinforcement learning environment consisting of a state space, a reward function, and an action space: historical multi-source data, including water quality, hydrology, and wind field, are acquired, preprocessed, and input into a long short-term memory neural network surrogate model to obtain temporal feature information; the policy network selects and outputs water-taking instructions from the action space based on the temporal feature information and historical multi-source data, calculates the reward function, and obtains a reward; the value network evaluates the water-taking instructions, outputs evaluation information, and continuously updates the parameters of the value network based on the reward; the parameters of the policy network are continuously optimized based on the evaluation information; until both the value network and the policy network converge, a well-trained actor critic model is obtained.

[0007] Furthermore, the state space includes environmental states such as water quality, hydrology, and wind field. The water quality state is characterized by the water intake concentration and the water delivery concentration; the hydrological state is characterized by the reservoir water level; the wind field state is characterized by the wind direction and wind speed; the action space includes the water intake actions taken by the policy network according to different states; the reward function includes water level reward and water delivery concentration reward, which is used to provide feedback information after the policy network interacts with the reinforcement learning environment and outputs actions, so as to evaluate the quality of the actions.

[0008] Furthermore, the temporal difference algorithm updates the parameters of the value network, first calculating the temporal difference objective, the calculation formula of which is: Based on the aforementioned temporal difference objective, the temporal difference residuals are calculated, and the parameters of the value network are updated based on these residuals. The calculation formula is as follows: in, Given the current environmental state, Actions output by the value network For value network , The reward obtained by the policy network, for , For the environmental state in the next moment, The action output by the value network for the next moment.

[0009] Furthermore, the calculation formula for the time-series differential residual is updated by setting the target network soft update parameters and update period. The updated calculation formula is as follows: Among them, w target represents the target network model weight parameters, w represents the current value network model weight parameters, and 𝜏 represents the soft update parameters.

[0010] Furthermore, the parameters of the policy network are updated using a policy gradient algorithm. The update formula specifically includes: in, For updating the gradient of the policy network, This represents the environmental state-water-taking action value function. This indicates the parameter under the current environmental conditions. Impact of changes: Choosing water-taking actions The log probability of this action. For environmental conditions, These are the model parameters for the policy network. Represents the action space.

[0011] Furthermore, the experience replay mechanism specifically includes: An experience replay sample database consisting of a queue of four-tuples is established to store four-tuples in the trajectories generated by the interaction between the policy network and the environment. The queue of four-tuples is represented as: (𝑠 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑠 𝑡+1 ); Among them, 𝑠 𝑡 For the current environmental state, 𝑎 𝑡 Actions output by the value network, 𝑟 𝑡 The reward obtained by the policy network, 𝑠 𝑡+1 The environmental state at the next moment; The actor critic model learns from the experience replay sample database based on a pre-set number of single-round training samples during training.

[0012] Furthermore, the training process of the long short-term memory neural network surrogate model includes: Collect multi-source data including historical water quality, hydrology, and wind field and input them into a pre-constructed two-dimensional hydrodynamic water quality mechanism model to obtain water concentration change prediction values. Based on the multi-source data and water concentration prediction values, construct an input-output dataset to train the long short-term memory neural network surrogate model and obtain a trained long short-term memory neural network surrogate model. The construction process of the two-dimensional hydrodynamic water quality mechanism model specifically includes: First, extract the planar coordinates and bottom elevation data of the target lake and reservoir area. After simplifying the terrain, the discrete elevation points are distributed to each node of the computational grid through linear interpolation, thereby obtaining the bottom water depth distribution required by the model. Static geographic data of the target lake / reservoir is acquired, and the substrate roughness, eddy viscosity, and wind drag coefficient are set based on the static geographic data. Historical multi-source data, including water quality, hydrology, and wind field, are acquired, and initial hydrodynamic conditions, boundary conditions, source-sink terms, and wind field are set based on the multi-source data. Using the two-dimensional shallow water equation as the governing equation, under the set substrate roughness, eddy viscosity, and wind drag coefficient, the water level, water velocity, and corresponding water quality concentration at each grid point in the model are solved using the finite volume method with unstructured meshes based on the initial hydrodynamic conditions, boundary conditions, source-sink terms, and wind field, thus completing the coupled simulation of the hydrodynamic and water quality transport process. The simulation results of the coupled simulation were compared with the measured data, and the model parameters were continuously adjusted to obtain the final two-dimensional hydrodynamic water quality mechanism model.

[0013] Furthermore, the training process of the Long Short-Term Memory (LSTM) neural network surrogate model specifically includes: Perform forward propagation and continuously update the model parameters. The calculation formulas include: in, W I , W F , W O , WC These represent the weights of the input gate, forget gate, output gate, and candidate neurons, respectively. b I , b F , b O , b C These are the bias terms for the input gate, forget gate, output gate, and candidate neurons, respectively. For the Sigmoid function, It is the hyperbolic tangent function; C t This is the output of the memory neurons in this round. This indicates the calculation of the Hadamard product of matrices. X t This is the input for the hidden layer. C t-1 Input for the last memory cell. F t Output the value for the forget gate. O t The output value is the output gate value. C (c) t This represents the output value of the candidate neurons in this round. I t The input gate output value, This is the output of the previous hidden layer. H t This is the output of the hidden layer. After obtaining a long short-term memory neural network surrogate model with defined parameters, the root mean square error and Nash efficiency coefficient are used to evaluate the model's predictive performance. The calculation expression is as follows: Where n represents the number of steps in the model's prediction. y and These represent the calculated values ​​of lake and reservoir water quality concentrations from the mechanistic model and the predicted values ​​from the LSTM neural network, respectively, at the corresponding time steps. y This represents the average value calculated by the mechanistic model for the water quality concentration of a single sample across all time steps in the lake / reservoir.

[0014] Furthermore, the hyperparameters used in the training process specifically include experience replay capacity, number of samples per training round, policy network and value network update rate, and reward function parameters; By conducting multiple sets of experiments, the optimization performance of the model under different experience replay capacity, number of samples per training round, policy network and value network update rates, and reward function parameters was compared, and the optimal combination of training parameters was selected.

[0015] The present invention also provides a system for a lake and reservoir water intake optimization method based on an LSTM model, comprising a memory and a processor, wherein the memory stores a computer program, and the processor calls the computer program to execute the steps of any of the methods described above.

[0016] Compared with the prior art, the present invention has the following advantages: (1) This invention achieves rapid prediction of water quality dynamics by: constructing a two-dimensional hydrodynamic water quality mechanism model, generating a high-fidelity simulation dataset; and training an LSTM neural network proxy model on this basis to replace the slow mechanism model with a millisecond-level response speed. Based on the constructed LSTM neural network agent model, a reinforcement learning environment is further built, and the Actor-Critic algorithm is used for decision optimization. Through the interaction between the policy network and the value network and the reinforcement learning environment: the policy network outputs actions, the reward function outputs rewards, and the value network evaluates the quality of the actions and provides evaluation information. The policy network continuously updates its parameters based on the evaluation information to output better actions in the next step. This makes the water intake actions output by the policy network continuously move towards the direction of both meeting water quality standards and ensuring stable water levels after water intake, realizing rapid and scientific decision-making for water intake scheduling schemes in the event of abnormal water quality at the water intake point. By training the LSTM surrogate model using data output from the mechanistic model, the bottleneck of traditional prediction methods being unable to be applied online due to slow inference speed is overcome. Furthermore, the LSTM surrogate model is embedded into the Actor-Critic model, and the Actor-Critic model is reinforced and trained. This enables the finally trained Actor-Critic model to output the optimal water intake scheme that meets the dual requirements of water quality compliance and water level stability based on measured data. A complete intelligent optimization system for lake and reservoir water intake schemes has been constructed, which has good industrial applicability.

[0017] (2) This invention improves the coherence and accuracy of the optimization strategy by designing a chain-like convergent cumulative policy network, taking into account the temporal dependence between multiple actions; it introduces experience replay and target network mechanisms to further stabilize the training process and accelerate the convergence of model training; it determines the optimal training configuration by conducting multiple sets of parameter tuning experiments, ensuring the convergence efficiency and decision reliability of the reinforcement learning model; it has been verified that the optimization results are highly consistent with the calculation results of the mechanism model, and can maintain the stability of lake and reservoir water levels while ensuring water quality meets standards, providing efficient and reliable technical support for emergency scheduling of lakes and reservoirs.

[0018] (3) The method proposed in this invention is applicable to lakes and reservoirs under different geographical and hydrological conditions. It can cope with various abnormal water quality scenarios such as saltwater intrusion and pollution input. The system output results are highly consistent with the calculation results of the mechanism model. While ensuring the safety of water supply quality, it effectively maintains the stability of water level in reservoirs or lakes, providing efficient and reliable technical support for emergency scheduling and normalized refined management of lakes and reservoirs. Attached Figure Description

[0019] Figure 1 The flowchart shows a lake and reservoir water intake optimization method based on an LSTM model provided in an embodiment of the present invention. Figure 2 This is a simplified flowchart of a lake and reservoir water intake optimization method based on an LSTM model provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the actor-critic algorithm structure of a lake and reservoir water intake optimization method based on an LSTM model provided in an embodiment of the present invention. Figure 4 This is a comparison of the optimization performance of a lake and reservoir water intake optimization method based on an LSTM model under different chloride concentration reward functions in an embodiment of the present invention. Figure 5 This is a model optimization effect verification diagram for a lake and reservoir water intake optimization method based on an LSTM model provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0022] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0023] Definitions: The Actor-Critic algorithm is a reinforcement learning framework that integrates policy optimization and value assessment. It comprises two core components: an actor network, acting as the policy model, responsible for generating specific actions based on the current environmental state; and a critic network, acting as the value model, responsible for evaluating the long-term benefits of the actor's chosen actions. Working together, the actor continuously optimizes its policy based on the critic's evaluation, while the critic updates its value judgment based on environmental feedback, forming a closed-loop learning mechanism of "action-evaluation-improvement." This enables stable and efficient optimal decision-making learning in complex and dynamic environments.

[0024] The finite volume method using unstructured meshes is an advanced numerical computation method for solving partial differential equations in fluid mechanics, heat conduction, and other fields. Its core feature lies in the flexible discretization of complex geometric regions using unstructured meshes (such as triangular, quadrilateral, and other polygonal elements), accurately conforming to irregular shoreline boundaries such as lakes and reservoirs. Based on this, the finite volume method is applied to integrate the governing equations over each mesh element, and the Gaussian divergence theorem is used to transform the calculation into flux calculations at the element boundaries, thus strictly guaranteeing the local conservation of physical quantities (such as mass and momentum). This method combines the advantages of strong geometric adaptability and good physical conservation, and is widely used in the simulation of complex processes such as environmental hydrodynamics and water quality migration.

[0025] Two-dimensional shallow water equations are the core governing equations describing large-scale flow and material transport in surface water bodies (such as lakes, reservoirs, estuaries, and nearshore seas). These equations are integrated and averaged vertically, assuming pressure follows a hydrostatic pressure distribution and vertical acceleration is negligible, thus simplifying the three-dimensional flow problem into a two-dimensional horizontal motion model.

[0026] The two-dimensional incompressible Reynolds-averaged Navier-Stokes equations are the core governing equations in environmental fluid mechanics used to describe the average motion and turbulent effects of water bodies. Based on the assumption of incompressibility (constant density), these equations employ the Reynolds-averaged method to decompose instantaneous flow into time-averaged and fluctuating quantities. By introducing turbulent stress terms (Reynolds stress), the equations are closed, thus characterizing the influence of complex turbulence on the time-averaged motion of water flow in a two-dimensional horizontal space. This model can effectively simulate the planar circulation structure, mixing and transport processes, and pollutant diffusion behavior in lakes and reservoirs driven by wind-driven flow, density flow, and bed friction, providing the theoretical foundation for constructing high-precision hydrodynamic-water quality coupled models.

[0027] Temporal Difference (TD) algorithm: It is a core learning method in reinforcement learning. It combines the ideas of Monte Carlo method and dynamic programming. It can update the value function online and incrementally by the difference in value estimation between adjacent states (i.e., temporal difference error) without requiring a complete model of the environment or waiting for the end of the round.

[0028] Policy gradient algorithms are a class of reinforcement learning methods that directly optimize the policy function. Unlike value function-based methods, they parameterize the policy (such as in neural networks) and use the gradient information of the policy parameters with respect to performance metrics to directly update the policy in a gradient ascent manner, making the agent more inclined to choose actions that yield higher cumulative rewards.

[0029] Example 1 like Figure 1 As shown in the figure, this embodiment provides a method for optimizing water intake in lakes and reservoirs based on an LSTM model. The method includes the following steps: S1: Real-time acquisition of multi-source data including water quality, hydrology, and wind field, and data preprocessing. The preprocessed multi-source data is then input into a pre-built long short-term memory neural network surrogate model to generate time-series feature information. The time-series feature information and multi-source data are then input into a pre-built actor critic model to obtain the optimal water intake scheme. Specifically, The training process of the Long Short-Term Memory (LSTM) neural network surrogate model includes: S101: Acquire monitoring data of lakes or reservoirs, including historical water quality, hydrology, and wind field (wind speed, wind direction) data; construct a two-dimensional hydrodynamic and water quality mechanism model of the target lake or reservoir using water quality, hydrology, and wind field monitoring data as boundary conditions; Specifically, The construction process of the two-dimensional hydrodynamic water quality mechanism model specifically includes: Establish a terrain grid. Extract coordinates and elevations to complete terrain generalization, and use linear interpolation to interpolate the water depth data to the grid nodes to obtain the model water depth.

[0030] Parameter settings. Set the substrate roughness, eddy viscosity coefficient, and wind drag coefficient; set the initial hydrodynamic conditions, boundary conditions, source and sink terms, and wind field based on measured data.

[0031] Simulation calculations were performed. The basic equations are two-dimensional shallow water equations in Cartesian coordinates, including the flow continuity equation and the flow momentum equation. Numerical calculations were performed using the finite volume method based on unstructured meshes, based on the solution method of the two-dimensional incompressible Reynolds-averaged Navier-Stokes equations, and satisfying the hydrostatic pressure assumption and the Boussinesq assumption.

[0032] Results calibration and verification. Measured data were used to calibrate and verify the simulation results to ensure reliable simulation accuracy.

[0033] Preferred, First, extract the planar coordinates and bottom elevation data of the target lake and reservoir area. After simplifying the terrain, the discrete elevation points are distributed to each node of the computational grid through linear interpolation, thereby obtaining the bottom water depth distribution required by the model. Static geographic data of the target lake and reservoir were acquired, and the bed roughness, eddy viscosity, and wind drag coefficient were set based on the static geographic data. Historical multi-source data, including water quality, hydrology, and wind field, were acquired, and initial hydrodynamic conditions, boundary conditions, source and sink terms, and wind field were set based on the multi-source data. Using the two-dimensional shallow water equation as the governing equation, under the set bed roughness, eddy viscosity, and wind drag coefficient, the water level, water velocity, and corresponding water quality concentration of each grid point in the model were solved using the finite volume method with unstructured grids based on the initial hydrodynamic conditions, boundary conditions, source and sink terms, and wind field, thus completing the coupled simulation of hydrodynamic and water quality transport processes. The simulation results of the coupled simulation were compared with the measured data, and the model parameters were continuously adjusted to obtain the final two-dimensional hydrodynamic water quality mechanism model.

[0034] S102: Combine water quality and hydrological simulation data with monitoring data such as wind field to construct simulated operating conditions, and calculate the water quality concentration at the lake and reservoir inlet based on the established mechanism model; extract the calculation results of the operating condition scheme of the mechanism model to train the surrogate model of the LSTM neural network; Specifically, Collect multi-source data including historical water quality, hydrology, and wind field and input them into a pre-built two-dimensional hydrodynamic water quality mechanism model to obtain water concentration change prediction values. Based on the multi-source data and water concentration prediction values, construct input and output datasets to train a long short-term memory neural network surrogate model to obtain a trained long short-term memory neural network surrogate model. Preferred, The results of the working condition scheme calculations from the mechanism model are extracted and used to train the surrogate model of the LSTM neural network. The specific steps are as follows: remember C t-1 Input for the last memory cell. X t Using the input of the hidden layer as the current input, calculate the output value of the candidate neurons in this round. C (c) t Input gate output value I t Forget Gate Output Value F t Output gate output value O t The calculation expression is: In the formula, W I , W F , W O , W C These represent the weights of the input gate, forget gate, output gate, and candidate neurons, respectively. b I , b F , b O , b C These are the bias terms for the input gate, forget gate, output gate, and candidate neurons, respectively. For the Sigmoid function, Let be the hyperbolic tangent function, and its calculation expression is: Calculate the output of the memory neuron in this round. C t The calculation expression is: In the formula, This indicates the calculation of the Hadamard product of matrices.

[0035] Calculate the output of this hidden layer. H t The calculation expression is: After the neural network model is trained, the root mean square error (RMSE) and Nash efficiency coefficient (NSE) are used to evaluate the model's prediction performance. The calculation expressions are as follows: In the formula, n represents the number of steps in the model prediction. y and The values ​​represent the calculated values ​​of the lake / reservoir water quality concentration from the mechanistic model and the predicted values ​​from the LSTM neural network at the corresponding time steps, respectively. y This represents the average value calculated by the mechanistic model for the water quality concentration of a single sample across all time steps.

[0036] S2: The actor critic model includes a value network and a policy network; the training process of the actor critic model includes: training the actor critic model by combining the target network mechanism and the experience replay mechanism; constructing a target network with the same network structure as the value network, and periodically using a soft update strategy to update the parameters of the target network according to the parameters of the value network; further optimizing the hyperparameters during the training process to determine the optimal combination of training parameters. Preferred, The parameters of the value network are updated using the temporal difference algorithm. First, the temporal difference objective is calculated, and the calculation formula is as follows: Based on the temporal difference objective, the temporal difference residuals are calculated, and the parameters of the value network are updated based on the temporal difference residuals. The calculation formula is as follows: in, Given the current environmental state, Actions output by the value network For value network , The reward obtained by the policy network, for , For the environmental state in the next moment, The action output by the value network for the next moment.

[0037] Preferred, The formula for calculating the time-series differential residuals is updated by setting the target network soft update parameters and update period. The updated formula is as follows: Among them, w target represents the target network model weight parameters, w represents the current value network model weight parameters, and 𝜏 represents the soft update parameters.

[0038] Preferred, The policy gradient algorithm is used to update the parameters of the policy network. The specific update formula includes: in, For updating the gradient of the policy network, This represents the environmental state-water-taking action value function. This indicates the parameter under the current environmental conditions. Impact of changes: Choosing water-taking actions The log probability of this action. For environmental conditions, These are the model parameters for the policy network. Represents the action space.

[0039] Preferred, The experience replay mechanism is as follows: An experience replay sample database consisting of a queue of quadruples is set up to store quadruples from the trajectories generated by the interaction between the policy network and the environment. The queue of quadruples is represented as: (𝑠 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑠 𝑡+1 ); Among them, 𝑠 𝑡 For the current environmental state, 𝑎 𝑡 Actions output by the value network, 𝑟 𝑡 The reward obtained by the policy network, 𝑠 𝑡+1 The environmental state at the next moment; The actor critic model learns by sampling from an experience replay sample database based on a pre-set number of single-round training samples during training.

[0040] S3: Based on the optimal combination of training parameters, the actor critic model is trained in a reinforcement learning environment consisting of a state space, a reward function, and an action space. Historical multi-source data, including water quality, hydrology, and wind field, are acquired, preprocessed, and input into a long short-term memory neural network surrogate model to obtain temporal feature information. The policy network selects and outputs water-taking instructions from the action space based on the temporal feature information and historical multi-source data, calculates the reward function, and obtains a reward. The value network evaluates the water-taking instructions, outputs evaluation information, and continuously updates the parameters of the value network based on the reward. Based on the evaluation information, the parameters of the policy network are continuously optimized until both the value network and the policy network converge, resulting in a well-trained actor critic model.

[0041] S301: Constructing a reinforcement learning environment; Specifically, The specific steps for building a reinforcement learning environment are as follows: Step 3011: Define the core elements of reinforcement learning, including agent, environment, state, action, and reward; Step 3012: Construct the intelligent agent, the main body in the reinforcement learning algorithm. The intelligent agent can perceive its own state in the environment and choose the action to take based on its observed state in order to interact with the environment, including the water pump at the reservoir intake; Step 3013: Construct the state space, representing the agent's situation after interacting with the environment. The state is a crucial basis for the agent to determine what action to take to further interact with the environment. The set of all possible states of the agent is called the state space, including reservoir water level, water intake concentration, water delivery concentration, LSTM memory elements and hidden states, wind direction, wind speed, etc. Specifically, The state space includes environmental states such as water quality, hydrology, and wind field. Water quality is characterized by water intake concentration and water delivery concentration; hydrological status is characterized by reservoir water level; wind field status is characterized by wind direction and wind speed; the action space contains the water intake actions taken by the policy network according to different states; the reward function includes water level reward and water delivery concentration reward, which are used to provide feedback information after the policy network interacts with the reinforcement learning environment and outputs actions, so as to evaluate the quality of the actions.

[0042] Step 3014: Construct the action space, which contains the specific actions the agent chooses when interacting with the environment. At different times, depending on its state, the agent may choose different actions. The set of all possible actions the agent can take is called the action space, including discretizing the water flow rate levels. Step 3015: Construct the reward function, which contains feedback information after the agent interacts with the environment. The magnitude and positive or negative value of the reward reflect the quality of the agent's "taking a certain action in a certain state," including water level rewards and water concentration rewards. Both are piecewise linear functions, encouraging the water level to approach the target water level and the concentration to be as low as possible.

[0043] S302: Constructing the Actor-Critic algorithm framework; Specifically, like Figure 2 and Figure 3 As shown, the specific steps for constructing the Actor-Critic algorithm framework are as follows: Step 3021: The value network determines the current state based on the value network. 𝑡 Output the action taken by the intelligent agent. 𝑡 .

[0044] Step 3022: The agent outputs action 𝑎 based on the value network. 𝑡 Interact with the environment to earn rewards. 𝑡 .

[0045] Step 3023: Value network based on 𝑠 𝑡 , 𝑎 𝑡 Calculate 𝑄 𝜋 (x, y), and according to x 𝑡 Parameters are updated based on the time-difference algorithm.

[0046] Step 3024: The policy network is based on 𝑄 𝜋 Calculate the gradient ∇(x, y) 𝐽 (𝜃), and update the parameters based on the policy gradient algorithm.

[0047] Step 3025: Repeat steps 1 to 4 until both the value network and the policy network converge.

[0048] In the training process described above, the two most important steps are updating the value network using the Temporal Difference (TD) algorithm and updating the policy network using the Policy Gradient (PGR) algorithm. Let 𝑄(𝑠, 𝑎; w) represent 𝑄 calculated using the value network with parameter w. 𝜋 (x, y), the calculation expression is: The left side of the equation represents the output value of the value network, while the right side is called the Temporal Difference Target (TDTarget). The difference between the two is called the Temporal Difference Residual (TD Error), and its calculation expression is as follows: Since the temporal difference residual should be 0 for a value network that has been trained and converged, the parameters of the value network can be updated through backpropagation by calculating the mean square loss function of the temporal difference error, thereby achieving the training of the value network.

[0049] In the Actor-Critic framework, the value network is used to calculate 𝑄 𝜋 Based on (x, y), the update gradient of the policy network can be calculated. The calculation expression is: S303: Introducing experience playback and target network optimization training process; Specifically, The specific steps for introducing experience replay and target network optimization training are as follows: Step 3031: Set the experience replay capacity. An experience replay is one (k). 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑠 𝑡+1 The quadruple queue acts as a sample database. Quadruples from the trajectories generated by the agent's interaction with the environment are stored in this database, and samples are taken from it for learning during each training session. Step 3032: Set the number of samples per training round; Step 3033: Target network soft update parameters and update period. In the Actor-Critic algorithm using the target network, the expression for calculating the temporal difference residual is: After a certain number of training iterations, the parameters of the current value network are updated into the parameters of the target network. This process can be done by completely assigning the parameters or by using a soft update strategy. The calculation expression is as follows: Among them, w target represents the target network model weight parameters, w represents the current value network model weight parameters, and 𝜏 represents the soft update parameters.

[0050] S304: Determine the optimal training parameters; Specifically, The specific steps for determining the optimal training parameters are as follows: Step 3041: Compare the effects of different experience replay capacity, number of samples per training round, network update rate and reward function parameters on the optimization effect in multiple sets of experiments; Step 3042: Select the optimal parameter combination; Preferred, The Actor-Critic model is trained based on optimal parameters to output the optimal water extraction plan that maximizes the reward value. The specific training process includes the following steps: Step 1: The policy network outputs an action; Step 2: The agent interacts with the environment to receive rewards; Step 3: The value network updates its parameters based on the temporal difference algorithm; Step 4: The policy network updates its parameters based on the policy gradient algorithm; Step 5: Repeat the iteration until the model converges.

[0051] Example 2 This embodiment provides a system for a lake and reservoir water intake optimization method based on an LSTM model, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of any of the methods in Embodiment 1.

[0052] This embodiment also provides specific implementation steps for a lake / reservoir water intake optimization method based on an LSTM model, as shown in Embodiment 1, specifically including: Step 1: Obtain monitoring data for the lake or reservoir, including historical water quality, hydrology, and wind field (wind speed, wind direction) data; This embodiment takes a reservoir in East China as an example. The data source is the online monitoring data of the reservoir from December 24, 2022 to January 7, 2023, including wind direction, wind speed, and chloride concentration in the water intake.

[0053] Step 2: Using online monitoring data of water quality, hydrology, and wind field as boundary conditions, construct a two-dimensional hydrodynamic and water quality mechanism model for the target lake / reservoir; Step 201: Establish a terrain grid. Extract coordinates and elevations to complete terrain generalization, and use linear interpolation to interpolate the water depth data to the grid nodes to obtain the model water depth.

[0054] Step 202: Parameter setting. Set the substrate roughness, eddy viscosity coefficient, and wind drag coefficient; set the initial hydrodynamic conditions, boundary conditions, source and sink terms, and wind field based on the measured data.

[0055] Step 203: Simulation Calculation. The basic equations are two-dimensional shallow water equations in Cartesian coordinates, including the flow continuity equation and the flow momentum equation. Numerical calculations are performed using the finite volume method based on unstructured meshes, based on the solution method of the two-dimensional incompressible Reynolds-averaged Navier-Stokes equations, and satisfying the hydrostatic pressure assumption and the Boussinesq assumption.

[0056] Step 204: Result Calibration and Verification. The simulation results are calibrated and verified using measured hydrodynamic data to ensure the accuracy and reliability of the simulation.

[0057] Step 3: Combine water quality and hydrological simulation data with online monitoring data such as wind field to construct simulated operating conditions, and calculate water quality indicators at the reservoir's water conveyance point based on the mechanism model established in Step 2. In this example, the input parameters for the mechanistic model are wind direction, wind speed, chloride concentration in the intake water, water level in the reservoir, and water intake flow rate. The calculation result is the chloride concentration in the delivered water. Step 4: Extract the calculation results of the working condition scheme from the mechanism model and use them to train the surrogate model of the LSTM neural network; Step 4011: Record C t-1 Input for the last memory cell. X t Using the input of the hidden layer as the current input, calculate the output value of the candidate neurons in this round. C (c) t Input gate output value I t Forget Gate Output Value F t Output gate output value O t The calculation expression is: In the formula, W I , W F , W O , W C These represent the weights of the input gate, forget gate, output gate, and candidate neurons, respectively. b I , b F , b O , b C These are the bias terms for the input gate, forget gate, output gate, and candidate neurons, respectively. For the Sigmoid function, Let be the hyperbolic tangent function, and its calculation expression is: Step 4012: Calculate the output of the memory neurons in this round. C t The calculation expression is: In the formula, This indicates the calculation of the Hadamard product of matrices.

[0058] Step 4013: Calculate the output of the current hidden layer. H t The calculation expression is: Step 4014: After the neural network model training is completed, the root mean square error (RMSE) and Nash efficiency coefficient (NSE) are used to evaluate the model's prediction performance. The calculation expressions are as follows: In the formula, n represents the number of steps in the model prediction. y and These represent the calculated values ​​of the water quality concentration at the reservoir's water conveyance point and the predicted values ​​from the LSTM neural network, respectively, at the corresponding time steps. y This represents the average value calculated by the mechanistic model for the concentration of water quality indicators at the reservoir's water inlet across all time steps for a single sample.

[0059] Step 5: Build a reinforcement learning environment; Step 5011: Define the core elements of reinforcement learning, including agent, environment, state, action, and reward; Step 5012: Construct the intelligent agent, the main body in the reinforcement learning algorithm. The intelligent agent can perceive its own state in the environment and choose the action to take based on its observed state in order to interact with the environment, including the water pump at the reservoir intake; Step 5013: Construct the state space, representing the agent's situation after interacting with the environment. The state is a crucial basis for the agent to determine what action to take to further interact with the environment. The set of all possible states of the agent is called the state space, including reservoir water level, chloride concentration in the intake water, chloride concentration in the transported water, LSTM memory cells and hidden states, wind direction, wind speed, etc. Step 5014: Construct the action space, which contains the specific actions the agent chooses when interacting with the environment. At different times, depending on its state, the agent may choose different actions. The set of all possible actions the agent can take is called the action space, including discretizing the water flow rate levels. Step 5015: Construct a reward function, which contains feedback information after the agent interacts with the environment. The magnitude and positive or negative value of the reward reflect the quality of the agent's "taking a certain action in a certain state," including water level rewards and chloride concentration rewards. Both are piecewise linear functions, encouraging water levels to approach the target water level and chloride concentrations to be as low as possible.

[0060] The LSTM surrogate model takes wind direction, wind speed, chloride concentration in the intake water, water level in the reservoir, and intake flow rate as input parameters, and outputs the chloride concentration in the delivered water. The prediction accuracy has been verified to meet requirements. Core component settings: The agent consists of 5 water pumps with action speeds of 0, 20, 40, 60, 80, and 100 m. 3 / s; water level reward target value 7.0 m, chloride concentration reward range 0~500 mg / L.

[0061] Step 6: Construct the Actor-Critic algorithm framework; Step 6011: The value network determines the current state based on the value network. 𝑡 Output the action taken by the intelligent agent. 𝑡 .

[0062] Step 6012: The agent outputs action 𝑎 based on the value network. 𝑡 Interact with the environment to earn rewards. 𝑡 .

[0063] Step 6013: Value network based on 𝑠 𝑡 , 𝑎𝑡 Calculate 𝑄 𝜋 (x, y), and according to x 𝑡 Parameters are updated based on the time-difference algorithm.

[0064] Step 6014: The policy network is based on 𝑄 𝜋 Calculate the gradient ∇(x, y) 𝐽 (𝜃), and update the parameters based on the policy gradient algorithm.

[0065] Step 6015: Repeat steps 3011 to 3014 until both the value network and the policy network converge.

[0066] In the training process described above, the two most important steps are updating the value network using the Temporal Difference (TD) algorithm and updating the policy network using the Policy Gradient (PGR) algorithm. Let 𝑄(𝑠, 𝑎; w) represent 𝑄 calculated using the value network with parameter w. 𝜋 (x, y), the calculation expression is: The left side of the equation represents the output value of the value network, while the right side is called the Temporal Difference Target (TDTarget). The difference between the two is called the Temporal Difference Residual (TD Error), and its calculation expression is as follows: Since the temporal difference residual should be 0 for a value network that has been trained and converged, the parameters of the value network can be updated through backpropagation by calculating the mean square loss function of the temporal difference error, thereby achieving the training of the value network.

[0067] In the Actor-Critic framework, the value network is used to calculate 𝑄 𝜋 Based on (x, y), the update gradient of the policy network can be calculated. The calculation expression is: In this example, the following network structure is set up: the policy network contains 4 sub-networks, each with 2 hidden layers, and the activation function is tanh; the value network contains 3 hidden layers, and the activation function is ReLU.

[0068] Training parameters: Experience replay capacity 1000, single-round training sampling number 400, target network soft update parameter 0.5, update cycle 1; 5000 training rounds until the reward value converges. Parameter optimization: Through 11 sets of experimental comparisons, the optimal network update rate and sampling number were determined to avoid the model getting trapped in local optima.

[0069] Step 7: Introduce experience replay and target network optimization training process; Step 7011: Set the experience replay capacity. An experience replay is one (k). 𝑡 , 𝑡 , 𝑟 𝑡 , 𝑡+1 The quadruple queue acts as a sample database. Quadruples from the trajectories generated by the agent's interaction with the environment are stored in this database, and samples are taken from it for learning during each training session. Step 7012: Set the number of samples per training round; Step 7013: Target network soft update parameters and update period. In the Actor-Critic algorithm using the target network, the expression for calculating the temporal difference residual is: After a certain number of training iterations, the parameters of the current value network are updated into the parameters of the target network. This process can be done by completely assigning the parameters or by using a soft update strategy. The calculation expression is as follows: Where w target represents the target network model weight parameters, w represents the current value network model weight parameters, and 𝜏 represents the soft update parameters.

[0070] In this example, the experience replay capacity is set to 200~2000, and the number of samples per training round is set to 50~400; the target network soft update parameter τ=0.5, and the update period C=1.

[0071] Step 8: Determine the optimal training parameters; Step 8011: Compare the effects of different experience replay capacity, number of samples per training round, network update rate and reward function parameters on the optimization effect in multiple sets of experiments; Step 8012: Select the optimal parameter combination; In this example, the optimal training parameters are: experience replay capacity V = 1000, number of samples per training round s = 400, and value network update rate γ. value =0.00002, policy network update rate γ policy =0.000002.

[0072] Step 9: Train the Actor-Critic model based on the optimal parameters and output the optimal water extraction scheme with the highest reward value.

[0073] Figure 4The figure shows a comparison of the model's optimization performance under different chloride concentration reward functions for water transfer. As can be seen from the figure, both excessively high and excessively low chloride concentration reward functions are detrimental to the model's ability to correctly solve for the optimal water intake scheme. Figure 5 The effectiveness of the Actor-Critic reinforcement learning-based optimization algorithm in solving the optimal water intake scheme is demonstrated. As shown in the figure, the results calculated by the mechanistic model and the optimal scheme obtained by the Actor-Critic reinforcement learning optimization algorithm generally match the trend of chloride concentration in the water supply, with a peak concentration difference of only 7.34 mg / L. The chloride concentration in the water supply under the optimal water intake scheme obtained by the optimization algorithm is characterized using evaluation parameters, revealing an RMSE of 0.29 mg / L and an NSE of 0.86.

[0074] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for optimizing water intake in lakes and reservoirs based on an LSTM model, characterized in that, include: Real-time acquisition of multi-source data including water quality, hydrology, and wind field is performed and the preprocessed multi-source data is input into a pre-constructed long short-term memory neural network surrogate model to generate time-series feature information. Then, the time-series feature information and multi-source data are input into a pre-built actor critic model to obtain the optimal water intake scheme; The actor critic model includes a value network and a policy network; The training process of the actor critic model includes: training the actor critic model by combining the target network mechanism and the experience replay mechanism; constructing a target network with the same network structure as the value network; periodically using a soft update strategy to update the parameters of the target network according to the parameters of the value network; and further optimizing the hyperparameters during the training process to determine the optimal combination of training parameters. Based on the optimal combination of training parameters, the actor critic model is trained in a reinforcement learning environment consisting of a state space, a reward function, and an action space: historical multi-source data, including water quality, hydrology, and wind field, are acquired, preprocessed, and input into a long short-term memory neural network surrogate model to obtain temporal feature information; the policy network selects and outputs water-taking instructions from the action space based on the temporal feature information and historical multi-source data, calculates the reward function, and obtains a reward; the value network evaluates the water-taking instructions, outputs evaluation information, and continuously updates the parameters of the value network based on the reward; the parameters of the policy network are continuously optimized based on the evaluation information; until both the value network and the policy network converge, a well-trained actor critic model is obtained.

2. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The state space includes environmental states such as water quality, hydrology, and wind field. The water quality state is characterized by the water intake concentration and the water delivery concentration; the hydrological state is characterized by the reservoir water level; the wind field state is characterized by the wind direction and wind speed; the action space includes the water intake actions taken by the policy network according to different states; the reward function includes water level reward and water delivery concentration reward, which is used to provide feedback information after the policy network interacts with the reinforcement learning environment and outputs actions, so as to evaluate the quality of the actions.

3. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The parameters of the value network are updated using a temporal difference algorithm. First, the temporal difference objective is calculated using the following formula: Based on the aforementioned temporal difference objective, the temporal difference residuals are calculated, and the parameters of the value network are updated based on these residuals. The calculation formula is as follows: in, Given the current environmental state, Actions output by the value network For value network , The reward obtained by the policy network, for , For the environmental state in the next moment, Actions that will be output by the value network in the next moment.

4. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 3, characterized in that, The calculation formula for the time-series differential residual is updated by setting the target network soft update parameters and update period. The updated calculation formula is as follows: Among them, w target represents the target network model weight parameters, w represents the current value network model weight parameters, and 𝜏 represents the soft update parameters.

5. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The parameters of the policy network are updated using the policy gradient algorithm. The specific update formula includes: in, For updating the gradient of the policy network, This represents the environmental state-water-taking action value function. This indicates the parameter under the current environmental conditions. Impact of changes: Choosing water-taking actions The log probability of this action. For environmental conditions, These are the model parameters for the policy network. Represents the action space.

6. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The experience replay mechanism is specifically as follows: An experience replay sample database consisting of a queue of four-tuples is established to store four-tuples in the trajectories generated by the interaction between the policy network and the environment. The queue of four-tuples is represented as: (𝑠 𝑡 , 𝑎 𝑡 , 𝑟 𝑡 , 𝑠 𝑡+1 ); Among them, 𝑠 𝑡 For the current environmental state, 𝑎 𝑡 Actions output by the value network, 𝑟 𝑡 The reward obtained by the policy network, 𝑠 𝑡+1 The environmental state at the next moment; The actor critic model learns from the experience replay sample database based on a pre-set number of single-round training samples during training.

7. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The training process of the long short-term memory neural network surrogate model includes: Collect multi-source data including historical water quality, hydrology, and wind field and input them into a pre-constructed two-dimensional hydrodynamic water quality mechanism model to obtain water concentration change prediction values. Based on the multi-source data and water concentration prediction values, construct an input-output dataset to train the long short-term memory neural network surrogate model and obtain a trained long short-term memory neural network surrogate model. The construction process of the two-dimensional hydrodynamic water quality mechanism model specifically includes: First, extract the planar coordinates and bottom elevation data of the target lake and reservoir area. After simplifying the terrain, the discrete elevation points are distributed to each node of the computational grid through linear interpolation, thereby obtaining the bottom water depth distribution required by the model. Static geographic data of the target lake / reservoir is acquired, and the substrate roughness, eddy viscosity, and wind drag coefficient are set based on the static geographic data. Historical multi-source data, including water quality, hydrology, and wind field, are acquired, and initial hydrodynamic conditions, boundary conditions, source-sink terms, and wind field are set based on the multi-source data. Using the two-dimensional shallow water equation as the governing equation, under the set substrate roughness, eddy viscosity, and wind drag coefficient, the water level, water velocity, and corresponding water quality concentration at each grid point in the model are solved using the finite volume method with unstructured meshes based on the initial hydrodynamic conditions, boundary conditions, source-sink terms, and wind field, thus completing the coupled simulation of the hydrodynamic and water quality transport process. The simulation results of the coupled simulation were compared with the measured data, and the model parameters were continuously adjusted to obtain the final two-dimensional hydrodynamic water quality mechanism model.

8. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The training process of the Long Short-Term Memory Neural Network surrogate model specifically includes: Perform forward propagation and continuously update the model parameters. The calculation formulas include: in, W I , W F , W O , W C These represent the weights of the input gate, forget gate, output gate, and candidate neurons, respectively. b I , b F , b O , b C These are the bias terms for the input gate, forget gate, output gate, and candidate neurons, respectively. For the Sigmoid function, It is the hyperbolic tangent function; C t This is the output of the memory neurons in this round. This indicates the calculation of the Hadamard product of matrices. X t This is the input for the hidden layer. C t-1 For the previous memory input, F t Output the value for the forget gate. O t The output value of the output gate. C (c) t This represents the output value of the candidate neurons in this round. I t The input gate output value, This is the output of the previous hidden layer. H t This is the output of the hidden layer. After obtaining a long short-term memory neural network surrogate model with defined parameters, the root mean square error and Nash efficiency coefficient are used to evaluate the model's predictive performance. The calculation expression is as follows: Where n represents the number of steps in the model's prediction. y and These represent the calculated values ​​of lake and reservoir water quality concentrations from the mechanistic model and the predicted values ​​from the LSTM neural network, respectively, at the corresponding time steps. y This represents the average value calculated by the mechanistic model for the water quality concentration of a single sample across all time steps in the lake / reservoir.

9. The method for optimizing water intake in lakes and reservoirs based on an LSTM model according to claim 1, characterized in that, The hyperparameters in the training process specifically include experience replay capacity, number of samples per training round, policy network and value network update rate, and reward function parameters; By conducting multiple sets of experiments, the optimization performance of the model under different experience replay capacity, number of samples per training round, policy network and value network update rates, and reward function parameters was compared, and the optimal combination of training parameters was selected.

10. A system for optimizing water intake in lakes and reservoirs based on an LSTM model, characterized in that, It includes a memory and a processor, the memory storing a computer program, the processor invoking the computer program to perform the steps of the method as described in any one of claims 1 to 9.

Images