A method for optimizing the layout of micro-irrigation unit pipe network based on reinforcement learning algorithm

Abstract

The present invention discloses a micro-irrigation unit pipe network layout optimization design method based on a reinforcement learning algorithm, which relates to the field of agricultural irrigation, including: constructing a proximal strategy optimization algorithm model; setting the state, action and reward of the proximal strategy optimization algorithm model to obtain an optimization model based on the proximal strategy optimization algorithm; obtaining relevant training data sets, and training the optimization model according to the relevant training data sets to obtain a trained optimization model; and using the trained optimization model to predict the micro-irrigation unit pipe network area. The present invention can improve the utilization efficiency of irrigation water in micro-irrigation units.

Description

Technical Field

[0001] The present invention relates to the field of agricultural irrigation, and in particular to a micro-irrigation unit pipe network layout optimization design method based on a reinforcement learning algorithm. Background Art

[0002] China's micro-irrigation technology has gradually shifted from early sporadic demonstrations to large-scale development. In order to reduce the increasing operation and management costs, the control area of ​​a single micro-irrigation system has shown a clear trend of increasing. However, the water head loss and terrain deviation of the micro-irrigation system pipe network increase with the increase of the system control area. How to optimize the system design and operation management mode under the constraint of irrigation uniformity, and then reduce the system investment and operation costs, is a new problem that needs to be solved in the promotion of large-scale micro-irrigation systems. Summary of the invention

[0003] The purpose of the present invention is to provide a micro-irrigation unit pipe network layout optimization design method based on reinforcement learning algorithm, which can improve the irrigation water utilization efficiency of the micro-irrigation unit.

[0004] To achieve the above object, the present invention provides the following solutions:

[0005] A micro-irrigation unit pipe network layout optimization design method based on reinforcement learning algorithm, comprising:

[0006] Construct a proximal strategy optimization algorithm model;

[0007] Setting the state, action and reward of the proximal policy optimization algorithm model to obtain an optimization model based on the proximal policy optimization algorithm;

[0008] Acquire relevant training data sets, and train the optimization model according to the relevant training data sets to obtain a trained optimization model;

[0009] The trained optimization model is used to predict the pipe network area of ​​the micro-irrigation unit.

[0010] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0011] When designing the micro-irrigation unit pipe network, the present invention takes into account the boundary characteristics and actual terrain deviation of the micro-irrigation unit, incorporates all capillary pipes and branch pipes in the micro-irrigation unit into the hydraulic calculation process of the micro-irrigation unit, and optimizes the unit pipe network area based on the calculated flow deviation rate, irrigation uniformity coefficient and economic index, thereby achieving an overall improvement in the irrigation water utilization efficiency and economic benefits of the micro-irrigation unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the design method of micro-irrigation unit pipe network layout based on reinforcement learning algorithm;

[0013] Figure 2 It is the basic information diagram of EPANET pipe network model;

[0014] Figure 3 A schematic diagram of the reinforcement learning neural network layers;

[0015] Figure 4 This is a schematic diagram of the PPO reinforcement learning framework;

[0016] Figure 5 It is a micro-irrigation network control model based on PPO;

[0017] Figure 6 It is a schematic diagram of the structure of Matlab-EPANET collaborative simulation platform based on PPO;

[0018] Figure 7 It is a schematic diagram of the interaction and data exchange of Matlab-EPANET collaborative simulation platform based on PPO;

[0019] Figure 8 It is a micro-irrigation pipe network model. DETAILED DESCRIPTION

[0020] Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without making any creative work shall fall within the scope of protection of the present invention.

[0021] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0022] Embodiment 1

[0023] like Figure 1 As shown, the present invention provides a micro-irrigation unit pipe network layout optimization design method based on a reinforcement learning algorithm, comprising:

[0024] Step S1: Construct the Proximal Policy Optimization (PPO) model. Figure 3 As shown in the figure, the proximal strategy optimization algorithm model includes an input layer, two hidden layers and an output layer. Specifically, input the initialization strategy parameter θ0, initialize the hyperparameter ε, set the number of iterations, and then k =π(θ k ) collects the experience trajectory set D k . Apply the advantage function algorithm to calculate the advantage estimate Calculate the strategy update strategy function The loss function is calculated by taking a small batch of steps And the fitted value function is regressed by mean square error:

[0025] Step S2: Set the state, action and reward of the proximal policy optimization algorithm model to obtain an optimization model based on the proximal policy optimization algorithm. The optimization model is a micro-irrigation system layout optimization model based on PPO.

[0026] In practical applications, such as Figure 4 Figure 5 and Figure 8 As shown in the figure, the optimization goal is to select the unit area with the highest economic benefit as the design reference while meeting the unit pipe network hydraulic design standards. s / D l , mm), nominal outer diameter of the capillary tube (D s / D l , mm), branch pipe diameter roughness coefficient (R s / R l , -), capillary tube diameter roughness coefficient (R s / R l ,-), emitter design flow rate (q d , flow rate under 0.1MPa working pressure, L / h), sprinkler spacing (S e , cm), capillary spacing (S l , m) and capillary arrangement direction are used as physical parameters to describe the system state, and the unit area is used as the control action of RL. Among them, "-" means no unit. Since the unit area and economic benefits are the main optimization objectives of the micro-irrigation system, the pipe network flow deviation rate and irrigation uniformity are used as part of the reward function. In order to further refine the economic benefits, the annual cost of the micro-irrigation unit is introduced into the reward function. Among them, the annual cost of the micro-irrigation unit is calculated in four aspects: (1) system pipe network investment; (2) system maintenance cost; (3) system energy consumption cost; (4) irrigation cost.

[0027] Furthermore, a Matlab-EPANET joint simulation platform based on RL is built. Specifically, Figure 2 As shown in the figure, Matlab is used to set the micro-irrigation network parameters, including the node coordinates of each emitter in the network, the water source coordinates, the water outlet coordinates, the pipe length, the pipe outer diameter and the pipe roughness coefficient, etc., and a program directory is created to couple with EPANET to establish a simulation model. EPANET software is used to track the changes in the system pressure and water flow of the pipeline nodes in different time periods.

[0028] Step S3: Obtain relevant training data sets, and train the optimization model according to the relevant training data sets to obtain a trained optimization model.

[0029] Specifically include:

[0030] 3-1: Initialize the optimization model.

[0031] 3-2: Apply the initialized optimization model to the micro-irrigation unit pipe network simulation model and interact in the micro-irrigation cycle.

[0032] 3-3: Apply the back-propagation algorithm to update the network parameters of the proximal strategy optimization algorithm model.

[0033] 3-4: When the updated proximal strategy optimization algorithm model meets the training requirements of the preset strategy for the micro-irrigation unit pipe network layout, the converged proximal strategy optimization algorithm model is the trained optimization model.

[0034] In practical applications, such as Figure 6 and Figure 7 As shown, the PPO model obtains the system state S output from EPANET t , and control action a t Pass it to EPANET and repeat this process until the algorithm model converges. Specifically, the System module in Matlab can provide a unified interface for model exchange and collaborative simulation between modeling and simulation software, so that EPANET can make full use of the rich data structure in Matlab, define and calculate the variables of the RL algorithm in the Matlab work area, and modify the micro-irrigation pipe network simulation model. The specific joint simulation steps are: 1) The RL agent sends the control action to EPANET and performs a hydraulic calculation in EPANET. Then, the calculation result is input into the RL algorithm for one iteration, the latest control action is output, and the model control action is reset in the EPANET model. Repeat the whole process until the RL algorithm results converge.

[0035] Specifically, the RL policy is usually represented by the symbol π, which is calculated by formula (1) and refers to the mapping from system state to action: (π: S→A). Specifically:

[0036] π(a|s)=P(A=a t |S=S t ) (1)

[0037] Where P is the conditional probability of taking a specific action, A is the control action, and S is the current state.

[0038] Based on (1), the quality of a strategy depends on the size of the cumulative return. The cumulative return can be calculated based on the empirical trajectory, as shown in formula (2):

[0039]

[0040] Among them, G tis the cumulative return, r is the reward value, γ is the discount factor, which is usually a constant between 0 and 1, T is the number of iterations, and its value is the same as t; t is the tth iteration; k is 0-T, with a step size of 1, and r t+k is the t+kth reward value.

[0041] When calculating the cumulative return, the discount factor γ is multiplied, considering that the farther the action is, the less impact it has on the current state. When the value of γ is small, it means that the importance of rewards that are farther away from the current time step is low.

[0042] The state value function is used to evaluate the state S at this time step t The state-action value function is mainly used to evaluate the quality of this state S t Output action a t The quality of is as shown in formulas (3) and (4):

[0043]

[0044]

[0045] Among them, v π (s) is the state value in the S state; Q π (s,a) is state S t Next output action a t The estimated value

[0046] In formula (3), the state value function is to transform the current state S according to the given strategy π t The expected value of the cumulative return obtained. In formula (4), the state-action value function refers to the evaluation strategy π to select action a t The degree of quality.

[0047] The main goal of RL is to find the optimal strategy that maximizes the cumulative return through continuous iteration and updating. In this process, the difference between equation (3) and equation (4) is the advantage function. The advantage function is used to evaluate the size of the current output action relative to the average action value, as shown in equation (5):

[0048] A π (s,a)=Q π (s,a)-v π (s) (5)

[0049] Among them, A π (s,a) is the advantage value at this time step.

[0050] The error value is used as the estimate of the advantage function, and the generalized advantage estimation (GAE) method is used to optimize the advantage function, as shown in equations (6) and (7):

[0051] δ t =r t +γv π (s t+1 )-v π (s t ) (6)

[0052]

[0053] Among them, λ is usually a constant between 0 and 1, which can reduce the impact of variance on the algorithm, δ t The error value at this time step is It is the advantage value after optimization of GAE method.

[0054] Step S4: Use the trained optimization model to predict the pipe network area of ​​the micro-irrigation unit.

[0055] The role of the PPO algorithm is mainly to take the old policy network, interact with the environment to gain specific experience, and use the stored experience data to update the new policy network. Further, the PPO-Clip method is adopted because it can remain stable in the event of network failure.

[0056] The loss function of PPO-Clip is defined in equation (8), and θ is the policy parameter. In addition, the aspect ratio setting value of the micro-irrigation unit requires the algorithm to derive discrete control actions and use Softmax to output the probability of each control action, as shown in equation (9).

[0057]

[0058]

[0059] in, is the feature vector, θ is the policy network parameter, To find the expected value of the sample; is the advantage function; r t (θ) is the relative probability of the new and old strategies; ε is a hyperparameter between 0 and 1; the clip function limits the ratio between the new and old strategies to a smaller range, thereby limiting the update amplitude of the strategy.

[0060] The PPO model contains two main loops, including a control loop and a learning loop. The control loop collects states and outputs control actions, while the learning loop updates the PPO network weight parameters based on historical data stored in the experience pool, as follows:

[0061] 1) The PPO algorithm inputs states, actions, and rewards into the experience pool through continuous interaction with the environment.

[0062] 2) Import the historical values ​​stored in the experience pool into the new and old strategy networks to derive the conditional probabilities under different strategies.

[0063] 3) The stored current state and next state are input into the policy network, and the GAE method described above is used to calculate the estimate of the advantage function.

[0064] 4) Modify the policy network by back-propagation and input the new weights into the old policy network every N iterations.

[0065] The PPO framework can be applied to the unit pipe network in the micro-irrigation system. While meeting the hydraulic design standards of the pipe network, the unit area is optimized to achieve the improvement of the economic benefits of the micro-irrigation system. Generally speaking, for the micro-irrigation pipe network system considering RL, the necessary steps are to design control actions, state parameters, and reward functions. In this study, the state quantity is determined by correlation analysis, and the unit area is used as a reinforcement learning action.

[0066] The emitter flow deviation rate QV in the micro-irrigation unit is shown in formula (10):

[0067]

[0068] Where: q max is the maximum flow rate of the emitter in the unit, L / h; q min is the minimum flow rate of the emitter in the unit, L / h; q d Design flow rate of the emitter, L / h. When QV≤20%, the hydraulic performance meets the design requirements. When QV>20%, the hydraulic performance does not meet the design requirements.

[0069] In order to ensure that all locations in the unit are fully irrigated, the irrigation uniformity coefficient EU is shown in formula (11). When EU ≤ 80%, the hydraulic performance does not meet the design requirements.

[0070]

[0071] Where: CV m is the manufacturing deviation of the sprinkler; N is the number of sprinklers allocated to each crop; is the average flow rate of the emitter in the unit, L / h.

[0072] The economic evaluation of pipeline network layout is shown in formula (12):

[0073] CT=C a +C m +C e +C w (12)

[0074] Among them, C a Micro irrigation network unit year, yuan / (hm2 a); C m is the maintenance cost, yuan / (hm 2 a)); C e is the energy consumption cost, yuan / (hm 2 a); C w is water fee, yuan / (hm 2 a)).

[0075] On this basis, the reward function is divided into three aspects: (1) pipe network flow deviation rate; (2) irrigation uniformity; (3) micro-irrigation unit annual cost. Therefore, the comprehensive reward function consists of three parts, as shown in formula (13). μ1, μ2 and μ3 represent the weights of pipe network flow deviation rate, irrigation uniformity and micro-irrigation unit annual cost, respectively. These three weights indicate the importance of the corresponding rewards of these three parameters under the implemented control strategy.

[0076] Reward=-μ1QV-μ2EU-μ3CT (13)

[0077] By building a Matlab-EPANET co-simulation platform based on RL, the hydraulic performance of the system under different pipe network layouts of micro-irrigation units is simulated, and the optimal unit pipe network area that meets the hydraulic design standards and improves economic benefits is output through continuous interaction and iteration with the environment through RL. The Matlab-EPANET co-simulation platform based on RL can overcome the limitations of the inherent control logic in EPANET. The interaction and data exchange between the two software are dynamic. In the first iteration, the PPO agent obtains the system state S received from EPANET t , and control action a t The optimal unit area of ​​the micro-irrigation network is outputted by EPANET. The duration of each iteration is mainly controlled by the simulation time. In order to make the time of each iteration equal, the simulation time step of EPANET and the control time step of PPO agent are both set to 5 seconds, that is, Δt = 5 seconds.

Claims

1. A micro-irrigation unit pipe network layout optimization design method based on reinforcement learning algorithm, characterized in that: include: Construct a proximal strategy optimization algorithm model; Setting the state, action and reward of the proximal policy optimization algorithm model to obtain an optimization model based on the proximal policy optimization algorithm; Acquire relevant training data sets, and train the optimization model according to the relevant training data sets to obtain a trained optimization model; Using the trained optimization model to predict the pipe network area of ​​the micro-irrigation unit; The proximal strategy optimization algorithm model includes an input layer, a hidden layer and an output layer connected in sequence; specifically: input the initialization strategy parameter θ0, initialize the hyperparameter ε, set the number of iterations, and k =π(θ k ) collects the experience trajectory set D k , using the advantage function algorithm to calculate the advantage estimate Calculate the strategy update strategy function The loss function is calculated by taking a small batch of steps And the fitted value function is regressed by mean square error: Among them, S t is the state of the tth iteration, θ is the policy network parameter, To find the expected value of the sample, r t (θ) is the relative probability of the new and old strategies, is the advantage function, ε is a hyperparameter between 0 and 1, and the clip function limits the ratio between the new and old strategies to a smaller range, limiting the update amplitude of the strategy. t is the cumulative return, T is the number of iterations, Indicates that based on state S t The value function is used to estimate S under the current policy parameters. t The expected estimated return.

2. The micro-irrigation unit pipe network layout optimization design method based on reinforcement learning algorithm according to claim 1 is characterized in that: The states include terrain deviation of micro-irrigation unit, nominal outer diameter of branch pipe, nominal outer diameter of capillary pipe, roughness coefficient of branch pipe diameter, roughness coefficient of capillary pipe diameter, design flow of emitter, spacing of emitters, spacing of capillary pipes and arrangement direction of capillary pipes; the actions are area of ​​micro-irrigation unit; the rewards are flow deviation rate of pipe network and irrigation uniformity.

3. The micro-irrigation unit pipe network layout optimization design method based on reinforcement learning algorithm according to claim 1 is characterized in that: The optimization model is trained according to the relevant training data set to obtain a trained optimization model, which specifically includes: Initializing the optimization model; The initialized optimization model is applied to the micro-irrigation unit pipe network simulation model to interact in the micro-irrigation cycle; Applying a back-propagation algorithm to update the network parameters of the proximal strategy optimization algorithm model; When the updated proximal strategy optimization algorithm model meets the training requirements of the preset strategy for the micro-irrigation unit pipe network layout, the converged proximal strategy optimization algorithm model is the trained optimization model.

Images