Optimal frequency control method for solar-hydro microgrid based on off-policy integral reinforcement learning method

Abstract

The present invention provides a method for optimal frequency control of a photovoltaic-hydro microgrid based on an off-policy integral reinforcement learning method. The present invention establishes mechanism models of a hydro-turbine unit and a photovoltaic generator respectively, thereby obtaining a state space expression of a photovoltaic-hydro microgrid system. At the same time, the frequency optimal control problem is converted into a zero-sum game Nash equilibrium problem according to the power generation characteristics, and the Nash equilibrium point is approximated by a neural network in an off-policy integral reinforcement learning method. The present invention can obtain an approximate optimal control law for frequency control of a photovoltaic-hydro microgrid system in real time based on historical secondary frequency modulation control input and state output data, and when the photovoltaic output fluctuates and the model parameters are uncertain, the frequency deviation is minimized to keep the frequency of the photovoltaic-hydro microgrid system stable, thereby improving the anti-interference ability of the system.

Description

Technical Field

[0001] The present invention belongs to the field of grid-connected control of renewable energy technologies in power systems, and in particular relates to an optimal frequency control method for a light-water microgrid based on an off-strategy integral reinforcement learning method. Background Art

[0002] With the development of the national economy and society, the contradiction between the increase in energy demand and the consumption of energy supply is becoming increasingly obvious. Photovoltaic power generation and hydropower generation, as important components of new energy power generation, are becoming key forces in promoting the development of clean energy.

[0003] As of the end of December 2023, the country's cumulative installed power generation capacity is about 2.92 billion kilowatts, a year-on-year increase of 13.9%. Among them, the installed capacity of solar power generation is about 610 million kilowatts, a year-on-year increase of 55.2%. Photovoltaic power generation has become the main body of new power installations in my country due to its environmentally friendly, renewable, and clean advantages. However, the intermittent and weather-dependent characteristics of photovoltaic power generation cause its output fluctuations to seriously affect the active power balance in the power system, making the frequency fluctuation problem of microgrids particularly prominent. Frequency fluctuations can cause damage to electrical equipment, reduced load performance, and other problems. If they are not effectively controlled, they will seriously threaten the system safety and stable operation of the microgrid.

[0004] In order to solve the above problems, the turbine unit is added to the microgrid as a controllable power source and the photovoltaic power generation system to fill the power shortage of the microgrid, maintain the frequency stability of the microgrid, and ensure the reliability of the overall power system. Therefore, it is necessary to conduct in-depth research on the control method of the controllable power source to obtain high-performance frequency control.

[0005] In current engineering practice, most load frequency controls still use traditional PID control. However, in view of increasingly complex working conditions and uncertain interference, PID control technology cannot meet the actual control requirements of load frequency, and the control effect still needs to be improved. The off-strategy integral reinforcement learning algorithm can solve the optimal control problem of the system when it is difficult or impossible to obtain the dynamic information of the actual system, that is, when the system model is not accurately known. Summary of the invention

[0006] In view of the technical problems raised in the background technology, the purpose of the present invention is: since frequency control plays a very important role in ensuring the power quality of the microgrid and providing high-quality electricity, it is of great significance to study the control method of the microgrid frequency controller, give full play to the regulation advantages of the turbine unit, improve the microgrid's ability to absorb photovoltaic power output, and realize the reliable operation of the microgrid, which is of great significance to promoting the efficient use of renewable energy and the sustainable development of the microgrid system.

[0007] Based on the above invention objectives, the present invention provides a method for optimal frequency control of a photovoltaic-hydro microgrid. The frequency optimal control problem is converted into a zero-sum game problem according to its power generation characteristics, and three neural networks are used to approximate the Nash equilibrium point of the zero-sum game, thereby solving the optimal frequency control problem of the photovoltaic-hydro microgrid and ensuring the power quality of the microgrid.

[0008] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0009] Build models of the turbine and photovoltaic generator respectively;

[0010] A unified frequency control model for solar-hydro microgrids is obtained. The secondary frequency modulation input is used as the control variable, and the load power and solar power of the input system are used as the disturbance variable. The frequency optimal control problem is transformed into a zero-sum game problem according to the power generation characteristics. The established performance index function includes the connection between the system state, control and disturbance.

[0011] The off-strategy integral enhancement method is used to solve the Nash equilibrium point of the zero-sum game problem, and the three neural networks of value function, control law and disturbance law are further used to iteratively approximate the optimal solution, thereby obtaining the frequency optimal control law of the solar-hydro microgrid system.

[0012] Furthermore, a unified frequency control model of solar-hydro microgrid is established, as follows:

[0013]

[0014] Where δ(t) represents the generator power angle, w0 represents the electrical angular velocity reference value w0=2πf0=100π(rad / s), Δf(t), ΔP m (t), ΔP L (t) and Δy(t) represent the relative value of frequency deviation, the relative value of turbine unit output increment, the relative value of load power increment and the relative value of governor relay stroke deviation respectively. ΔP PV and ΔP solar They represent photovoltaic power output power and solar power respectively. D represents the damping coefficient. qh 、r y and r represents the turbine transfer coefficient. j 、T w 、T y and T solar represents the generator time constant, water flow inertia time constant, relay response time constant and photovoltaic power generation time constant, and u(t) represents the secondary frequency modulation input.

[0015] Next, the secondary frequency modulation input u(t) is used as the control variable, [ΔP L (t)ΔP solar (t)]T As a disturbance variable, the established optimal control problem of the frequency of the photovoltaic-hydro microgrid is transformed into a zero-sum game problem. The unified model of the photovoltaic-hydro microgrid can be rewritten as:

[0016]

[0017] Among them, the state variable x = [x1 x2 x3 x4 x5] T =[Δδ(t)Δf(t)ΔP m (t)Δy(t)ΔP PV ] T Control variable u=u(t), disturbance variable v=[ΔP L (t)ΔP solar (t)] T , M, N, and P represent the system coefficient matrices, where x = 0 is the Nash equilibrium point of the solar-hydro microgrid system, and [k] represents the iteration step state variable.

[0018] Furthermore, in order to minimize frequency deviation, reduce frequency fluctuation and improve power quality, the optimal frequency and optimal performance index function is as follows:

[0019]

[0020] Among them, V(x,u,v) and are the performance index function and utility function of the optimal frequency control of the solar-hydro microgrid, respectively. A, B, and C represent positive definite real symmetric matrices.

[0021] Furthermore, the traditional strategy iteration method relies on accurate system information. However, considering the actual complex situation, the accurate dynamics of the solar-hydro microgrid cannot be directly obtained. Therefore, when solving the optimal performance indicator function, the following equation is obtained by derivation to avoid using accurate model information:

[0022]

[0023] Among them, A, B, and C represent real symmetric matrices. The system matrices M, N, and P are all replaced, and it is not necessary to rely on an accurate system model when solving the optimal performance indicator function.

[0024] Furthermore, the frequency optimal solution of the solar-hydro microgrid is solved by using the off-policy integral reinforcement learning method. The steps are as follows:

[0025] Collect data sample set (x n ,u n ,v n ), and the secondary frequency modulation input is used as the control variable, and the load output and solar energy output are used as disturbance variables;

[0026] Given the initial permissible control weights and disturbance weight vectors that can stabilize the system;

[0027] Calculate the estimated total weight γ, where

[0028] Iteration V * (x) = φ V T (x)γ V [k] , Update the evaluation estimation weights, execution estimation weights and disturbance estimation weights respectively, where φ V T (x), and They represent the activation functions of the performance index function, control law, and disturbance law respectively;

[0029] Set i=i+1 and keep updating until the difference ε between the estimated weights of two adjacent iterations is less than or equal to the convergence threshold, and then terminate the iterative calculation;

[0030] The estimated weights are used to further solve the optimal performance indicators, optimal control and disturbance of the photovoltaic-hydro microgrid to design the optimal frequency controller of the photovoltaic-hydro microgrid.

[0031] Furthermore, the optimal controller of the photovoltaic-hydro microgrid is designed by solving the optimal control law, and the optimal control quantity of secondary frequency regulation is obtained to control the coordinated output of hydropower units and photovoltaics, thereby compensating and regulating the power fluctuation of the microgrid, so that the frequency deviation of the microgrid is kept within a reasonable operating range.

[0032] To sum up, the present invention provides an optimal control method for the frequency of the optical-water microgrid, establishing a mechanism model of the water turbine and the photovoltaic generator respectively, and then further obtaining the state space model of the optical-water microgrid system, and converting it into a zero-sum game problem according to its power generation characteristics. By designing the frequency control performance index function, the optimal frequency control scheme of the optical-water microgrid is solved. The off-strategy integral enhancement learning method is used to approximate the Nash equilibrium point of zero-sum game with neural networks, thereby solving the problem of optimal control of optical-water microgrid frequency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1A flow chart of the optimal frequency control steps of a light-water microgrid based on an off-strategy integral reinforcement learning method provided in an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of power generation of a microgrid provided in an embodiment of the present invention;

[0036] Figure 3 It is the light-water microgrid load frequency control model provided in the embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of the off-strategy integral reinforcement learning method provided in the embodiment of the present invention;

[0038] Figure 5 This is a flow chart of a light-water microgrid control method based on an off-strategy integral reinforcement learning method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] This example provides a frequency optimal control method for a photovoltaic-hydro microgrid based on an off-policy integral reinforcement learning method, which mainly includes the establishment of a unified frequency control model for a photovoltaic-hydro microgrid and the frequency optimal control solution process.

[0041] See also Figure 1 ,The establishment of the solar-water microgrid model mainly includes the following steps:

[0042] Step 1: The photovoltaic-hydro microgrid includes a photovoltaic unit and a hydro turbine unit. Figure 2 As shown, a unified frequency control model of solar-hydro microgrid is established:

[0043]

[0044] Step 2: Use the secondary frequency modulation input u(t) as the control variable, [ΔP L (t)ΔP solar (t)] T As a disturbance variable, the optimal frequency control problem of the solar-hydro microgrid is transformed into a zero-sum game problem as follows:

[0045]

[0046] Among them, the state variable x = [x1 x2 x3 x4 x5] T =[δ(t)f(t)ΔP m (t)Δy(t)ΔP PV ] T , control variable u=u(t), disturbance variable v=[ΔP L (t)ΔP solar (t)] T , M, N, P represent the system coefficient matrix.

[0047] Step 3: The performance index function of the optimal frequency control of the solar-hydro microgrid is:

[0048]

[0049] Among them, V(x,u,v) is the performance index function of the optimal frequency control of the solar-hydro microgrid, J(x,u,v) represents the utility function, Matrices A, B, and C represent positive definite real symmetric matrices. The goal of the zero-sum game is to find a series of saddle points u* and v* such that the following inequality holds:

[0050] V(x,u * ,v)≤V(x,u * ,v * )≤V(x,u,v * ). (9)

[0051] Among them, u * and v * Represent the optimal control law and disturbance law respectively. In the zero-sum game problem, u * Expected minimization value function V, v * The expected maximization value function V. The zero-sum game problem has a unique set of solutions that satisfy the following Nash equilibrium conditions:

[0052]

[0053] Step 4: According to the Newton-Leibniz formula, the Bellman equation is derived and expressed in Hamiltonian form as follows:

[0054]

[0055] Where k represents the number of iterations. represents the derivative of the performance index function with respect to x. The traditional policy iteration method is shown as follows. Given the initial admissible control and disturbance, the iterative update equation (11) is given. The iterative calculation of the optimal control law and disturbance law can be written as:

[0056]

[0057] Therefore, it can be seen from equation (12) that the traditional strategy iterative method relies on accurate system dynamic information, but considering that the actual complex situation cannot directly obtain the precise dynamic knowledge of the light-water microgrid. Therefore, the off-strategy integral enhancement learning method is proposed to design the optimal controller for load frequency control of the light-water microgrid. The optimal control law can be solved using system data. The light-water microgrid load frequency control model reference Figure 3 , the schematic diagram of the off-policy integral reinforcement learning method is referenced Figure 4 .

[0058] Step 5: Rewrite equation (7) to obtain:

[0059]

[0060] Where [k] represents the number of iterations.

[0061] Step 6: When solving the optimal performance indicator function, avoid using precise model information by derivation to obtain the following equation:

[0062]

[0063] Where A, B, and C represent real symmetric matrices. The system matrices M, N, and P are all replaced. Compared with equation (12), equation (14) does not require an accurate system model when solving the optimal performance index function.

[0064] Step 7: Define the error function ε(x(t),u,v). When the two iteration values ​​are less than a small positive number, the off-policy integral reinforcement learning method iteration is terminated, as follows:

[0065]

[0066] Where Q represents a positive definite real symmetric matrix, and φ(x(t)) represents an activation function.

[0067] Step 8: With the help of the neural network of the off-strategy integral reinforcement learning method, the performance index function, control law and disturbance law of the optimal frequency control of the solar-hydro microgrid are iteratively approximated to obtain the weights of each part. The total weight is as follows:

[0068]

[0069] Among them, k represents the number of iterations, γ V , and Respectively represent the weights of each part of the performance index function, control law and disturbance law, respectively, and are represented by the optimal equation V * (x) = φ V T (x)γV [k] , and To approximate, where φ V T (x), and They represent the activation functions of the performance index function, control law and disturbance law respectively.

[0070] Step 9: The steps of solving the optimal solution of the solar-hydro microgrid using the off-policy integral reinforcement learning method are as follows Figure 5 As shown, the specific steps are as follows:

[0071] Collect historical data sample sets in the solar-water microgrid (x n ,u n ,v n ), and the secondary frequency modulation input is used as the control variable;

[0072] Given an initial admissible control strategy and a disturbance weight vector that can stabilize the system;

[0073] Calculate the estimated weights, and update the evaluation estimated weights according to equation (16), execute the estimated weights and perturb the estimated weights;

[0074] Set i=i+1 and keep updating;

[0075] As the number of iterations increases, the iterative calculation is terminated when the difference in estimated weights between two adjacent iterations is less than or equal to the convergence threshold ε;

[0076] The estimated weights are used to solve the performance index, optimal control and disturbance of the solar-hydro microgrid. A controller is designed to obtain the optimal control quantity of secondary frequency regulation to control the coordinated output of hydropower units and photovoltaics, thereby compensating and adjusting the power fluctuation of the microgrid so that the frequency deviation of the microgrid is kept within a reasonable operating range.

[0077] It should be noted that the solar-hydro microgrid load frequency controller can still maintain the microgrid frequency stability under the following three working conditions.

[0078] Condition 1: Considering load ΔP L Under the influence of multiple disturbances on the system, the load frequency is set to have step changes at different times. The controller can still ensure that the frequency deviation remains in a very small range under this load fluctuation.

[0079] Condition 2: Considering solar power output ΔP solar Fluctuations are taken as output disturbances to verify the frequency stability of the system when solar energy undergoes continuous multiple fluctuations.

[0080] Working condition 3: The load disturbance and solar power output disturbance are superimposed, and the control effect of the controller is compared to verify that the proposed controller can restore the frequency of the solar-hydro microgrid to near the rated frequency in a short time.

[0081] The present invention can obtain the approximate optimal control law in real time according to the power generation data of the solar-hydro microgrid, maintain the frequency stability of the solar-hydro microgrid system, cope with the frequency fluctuation of the microgrid caused by the fluctuation of photovoltaic output, ensure the frequency quality of the microgrid, improve the microgrid's ability to absorb photovoltaic power, and realize efficient and reliable operation of the microgrid.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimal frequency control of a light-water microgrid based on off-policy integral reinforcement learning method, characterized in that: The following steps are involved: Build models of the turbine and photovoltaic generator respectively; A unified frequency control model of the photovoltaic-hydro microgrid is obtained according to the model of the turbine and the photovoltaic generator, the secondary frequency modulation input is used as the control quantity, the load power and the solar power of the input system are used as the disturbance quantity, the frequency optimal control problem is converted into a zero-sum game problem according to the power generation characteristics, and the established performance index function includes the connection between the state, control and disturbance of the system; The optimal frequency solution of the photovoltaic-hydro microgrid is solved by using the off-policy integral reinforcement learning method: the Nash equilibrium point of the zero-sum game problem is solved by using the off-policy integral reinforcement learning method, and the optimal solution is further approximated by using the three neural networks of value function, control law and disturbance law to obtain the optimal frequency control law of the photovoltaic-hydro microgrid; The unified frequency control model of the solar-hydro microgrid is obtained as follows: Among them, Δδ(t) represents the relative value of the generator power angle deviation, w0 represents the electrical angular velocity reference value w0=2πf0=100π(rad / s), Δf(t), ΔP m (t), ΔP L (t) and Δy(t) represent the relative value of frequency deviation, the relative value of turbine unit output increment, the relative value of load power increment and the relative value of governor relay stroke deviation respectively; ΔP PV and ΔP solar They represent photovoltaic power output power and solar power respectively; D represents the damping coefficient; r qh 、r y and r represent the turbine transfer coefficient; T j , T w , T y and T solar represents the generator time constant, water flow inertia time constant, relay response time constant and photovoltaic power generation time constant, u(t) represents the secondary frequency modulation input; The secondary frequency modulation input u(t) is used as the control variable, [ΔP L (t)ΔP solar (t)] T As a disturbance variable, the established optimal frequency control problem of the photovoltaic-hydro microgrid is transformed into a zero-sum game problem. The unified model of the photovoltaic-hydro microgrid is rewritten as: Among them, the state variable x = [x1 x2 x3 x4 x5] T =[Δδ(t)Δf(t)ΔP m (t)Δy(t)ΔP PV ] T Control variable u=u(t), disturbance variable v=[ΔP L (t)ΔP solar (t)] T , M, N, and P represent the system coefficient matrix, where x = 0 is the Nash equilibrium point of the solar-water microgrid system, and [k] represents the iteration step state variable; In order to minimize frequency deviation, reduce frequency fluctuation and improve power quality, the performance index function of the optimal frequency control of the photovoltaic-hydro microgrid is as follows: Among them, V(x,u,v) and are the performance index function and utility function of the optimal frequency control of the solar-hydro microgrid, respectively. A, B, and C represent positive definite real symmetric matrices.

2. The optimal frequency control method of a light-water microgrid based on off-policy integral reinforcement learning method according to claim 1 is characterized in that: When solving the optimal performance indicator function, the following equation is obtained by derivation to avoid using precise model information: Among them, A, B, and C represent real symmetric matrices; the system matrices M, N, and P are all replaced, and there is no need to rely on an accurate system model when solving the optimal performance indicator function.

3. The optimal frequency control method of a light-water microgrid based on off-policy integral reinforcement learning method according to claim 2 is characterized in that: The frequency optimal solution of the solar-hydro microgrid is solved by using the off-strategy integral reinforcement learning method, and the steps are as follows: Collect data sample set (x n ,u n ,v n ), and the secondary frequency modulation input is used as the control variable, and the load output and solar energy output are used as disturbance variables; Given the initial permissible control weights and disturbance weight vectors that can stabilize the system; Calculate the estimated total weight γ, where γ V , and Respectively represent the weights of each part of the performance index function, control law and disturbance law; iterate V * (x) = φ V T (x)γ V [k] , Update the evaluation estimation weights, execution estimation weights and disturbance estimation weights respectively, where φ V T (x), and They represent the activation functions of the performance index function, control law, and disturbance law respectively; Set i=i+1 and keep updating until the difference ε between the estimated weights of two adjacent iterations is less than or equal to the convergence threshold, and then terminate the iterative calculation; The estimated weights are used to further solve the optimal performance indicators, optimal control and disturbance of the photovoltaic-hydro microgrid to design the optimal frequency controller of the photovoltaic-hydro microgrid.

4. The optimal frequency control method of a light-water microgrid based on off-policy integral reinforcement learning method according to claim 3 is characterized in that: By solving the optimal control law to design the optimal controller of the photovoltaic-hydro microgrid, the optimal control quantity of secondary frequency regulation is obtained to control the coordinated output of hydropower units and photovoltaics, and then compensate and adjust the power fluctuation of the microgrid, so that the frequency deviation of the microgrid is kept within a reasonable operating range.

5. An optimal frequency control device for a light-water microgrid based on a strategy-integrated reinforcement learning method, characterized in that: A method for optimal frequency control of a light-water microgrid based on an off-strategy integral reinforcement learning method as described in any one of claims 1-4.

Images