Power grid voltage control method based on fuzzy control and reinforcement learning

Abstract

The invention relates to a power grid voltage control method based on fuzzy control and reinforcement learning, and the method comprises the following steps: 1, constructing a fuzzy control module, obtaining a current power grid state variable, and carrying out the reasoning according to the current power grid state variable through employing the fuzzy control module at each time step, and obtaining a preliminary reactive power control quantity; 2, constructing a reinforcement learning model, taking the preliminary reactive power control quantity obtained by the fuzzy control module in the step 1 as one input of the reinforcement learning model, sending the preliminary reactive power control quantity and the current power grid state into the reinforcement learning model, and learning, verifying and outputting a final control action by the reinforcement learning model according to the current state and the preliminary reactive power control quantity; and step 3, executing the final control action output in the step 2, and adjusting the output of the generator set or adjusting other reactive power compensation devices so as to maintain the voltage stability of the power grid. According to the invention, the technical problems of poor robustness of a control algorithm and complex system can be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power grid voltage control, and relates to a power grid voltage control method, in particular to a power grid voltage control method based on fuzzy control and reinforcement learning. Background Art

[0002] Supply voltage is one of the important indicators of power quality. Reactive power compensation and reactive power balance are important means to ensure the quality of supply voltage. They also play an important role in the safety, stability and economic operation of the power system. Based on the existing large power grid, reactive power management in various regions generally adopts the principle of "layered and zoned, local balance".

[0003] At present, the voltage regulation of the power grid is mainly adjusted by automatic voltage control. Automatic voltage control refers to the centralized monitoring and analysis and calculation of the reactive voltage state of the whole network, and the coordinated optimization control of the widely dispersed reactive power devices of the power grid from a global perspective. It can not only realize the automatic adjustment of reactive power and voltage, but also has certain optimization functions. It is an important means to maintain the stability of system voltage, improve the quality of power grid voltage and the economic operation level of the whole system. Connecting the controller to the adjustment device is simple to control. The voltage regulator is a device that automatically adjusts the voltage. The grid voltage is transmitted to the device through the sensor, and the grid voltage is continuously adjusted by the automatic control of the voltage regulator. At present, the automatic voltage control system of each provincial company generally consists of a central control subsystem and three decentralized control subsystems. The control model and algorithm generally adopt the optimal power flow and constraint conditions. This type of algorithm has the disadvantages of impractical dynamic partitioning results, complex system, and low robustness of control algorithm.

[0004] Therefore, in order to solve the above problems, the present invention proposes a grid voltage control method based on fuzzy control and reinforcement learning.

[0005] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a grid voltage control method based on fuzzy control and reinforcement learning. The control method can simplify the control system, has strong adaptability and robustness, and has good fault tolerance and system balance, solving the technical problems of poor robustness of previous control algorithms and complex systems.

[0007] The present invention solves the practical problem by adopting the following technical solutions:

[0008] A power grid voltage control method based on fuzzy control and reinforcement learning comprises the following steps:

[0009] Step 1: construct a fuzzy control module to obtain the current grid state variables. At each time step, use the fuzzy control module to infer the current grid state variables and obtain the preliminary reactive power control quantity.

[0010] Step 2: Build a reinforcement learning model. Use the preliminary reactive power control quantity obtained by the fuzzy control module in step 1 as an input of the reinforcement learning module and send it to the reinforcement learning model together with the current power grid state. The reinforcement learning model learns, verifies and outputs the final control action based on the current state and the preliminary reactive power control quantity.

[0011] Step 3: Execute the final control action output from step 2 to adjust the output of the generator set or other reactive compensation devices to maintain the voltage stability of the power grid; observe the new state and reward, and feed it back to the reinforcement learning model to update the strategy.

[0012] Moreover, the specific steps of constructing the fuzzy control module in step 1 include:

[0013] (1) Construct fuzzy subsets:

[0014] Voltage deviation: {NB (negative large), NS (negative small), ZO (zero), PS (positive small), PB (positive large)}

[0015] Voltage change rate: {NB (negative large), NS (negative small), ZO (zero), PS (positive small), PB (positive large)}

[0016] (2) Establish fuzzy reasoning rules:

[0017] Establish fuzzy rule table and perform fuzzy reasoning;

[0018] 1) If the voltage deviation is "NB (negative large)" and the voltage change rate is "NB (negative large)", the fuzzy reasoning is PB (positive large), which means that the reactive power compensation is increased to the maximum extent.

[0019] 2) If the voltage deviation is "NS (negative small)" and the voltage change rate is "NS (negative small)", the fuzzy reasoning is PS (positive small), indicating that reactive power compensation is increased to the minimum extent.

[0020] 3) If the voltage deviation is "ZO (zero)" and the voltage change rate is "ZO (zero)", the fuzzy reasoning is ZO (zero), indicating that the voltage is within the normal range and reactive power compensation is not adjusted for the time being.

[0021] 4) If the voltage deviation is "PS (positive small)" and the voltage change rate is "PS (positive small)", the fuzzy reasoning is NS (negative small), indicating that reactive power compensation is reduced to the minimum.

[0022] 5) If the voltage deviation is "PB (positive large)" and the voltage change rate is "PB (positive large)", the fuzzy reasoning is NB (negative large), which means that reactive power compensation is minimized to the greatest extent.

[0023] (3) Defuzzification is performed to obtain fuzzy output, and the fuzzy output is converted into a specific value;

[0024] Moreover, the specific steps of step (3) of step 1 include:

[0025] ① Taking voltage deviation as an example, the voltage is input into the membership function calculation to obtain one or more membership values ​​belonging to a fuzzy subset corresponding to the voltage deviation;

[0026] ②Convert the actually measured voltage deviation and voltage change rate into the membership function of the fuzzy set;

[0027] ③According to the fuzzy rule table, the fuzzy output is obtained:

[0028] ④Convert the fuzzy output into specific numerical values ​​to obtain the preliminary reactive power control quantity.

[0029] Moreover, the specific steps of step 2 include:

[0030] (1) Define the state space

[0031] Voltage of each node: discretize it into several intervals, such as "low", "medium", and "high".

[0032] Grid frequency: Discretized into "low", "medium" and "high".

[0033] Load demand: The load demand of each node, discretized into "low", "medium" and "high".

[0034] Power output: The active and reactive output of each generator, discretized into "low", "medium" and "high".

[0035] Reactive power compensation: The number or capacity of reactive power compensation equipment put into use, discretized into "low", "medium" and "high".

[0036] (2) Define the action space

[0037] Adjust active output: increase or decrease the active output of the generator.

[0038] Adjust reactive output: Increase or decrease the reactive output of the generator.

[0039] Cut off reactive power compensation equipment: reduce the investment in reactive power compensation equipment.

[0040] Invest in reactive power compensation equipment: Increase investment in reactive power compensation equipment.

[0041] Adjust taps: Change the tap position of a transformer to adjust the voltage.

[0042] (3) Define the reward function

[0043] Voltage stability: When the voltage levels of all nodes are within the allowed range, positive rewards are given; otherwise, negative penalties are given.

[0044] Frequency stability: When the grid frequency is within the permitted range, positive rewards are given; otherwise, negative penalties are given.

[0045] Load tracking accuracy: When the grid input power matches the load demand well, a positive reward is given; otherwise, a negative penalty is given.

[0046] Minimize operating costs: When operating costs are low, give positive rewards; otherwise, give negative penalties.

[0047] (4) Initialize Q table

[0048] Create a Q table, where state s refers to the state set of all nodes, state set S = {s1, s2, s3…s i}, action a refers to all action situations, action set A = {a1, a2, a3…a i}. Initially, the values ​​in the Q table can all be set to 0 or small random numbers.

[0049] (5) Choose learning rate and discount factor

[0050] Learning rate (α): determines the speed at which new information covers old information, and is usually between 0.1 and 0.5.

[0051] Discount factor (γ): reflects the importance of the present value of future rewards relative to immediate rewards, and is usually between 0.9 and 0.99.

[0052] (6) Execute the Q-Learning algorithm and output the final control action;

[0053] 1) Initialization: Start from a certain initial state, for example, the initial state of the power grid is (s0).

[0054] 2) Select an action: Select an action (a0) based on the current Q table and the ε-greedy strategy. For example, if ε=0.1 is set, there is a 90% probability of selecting the action with the highest expected return in the current Q table, and a 10% probability of selecting an action randomly;

[0055] 3) Output the final control action.

[0056] Moreover, the specific steps of step 3 include:

[0057] (1) Take action: Execute the selected action (a0) and observe the next state (s1) and immediate reward (r1).

[0058] (2) Update the Q table: Use the Q-Learning update formula to update the values ​​in the Q table:

[0059] Q(s,a)←Q(s,a)+α[r+γmaxQ(s1,a1)-Q(s0,a0)]

[0060] (3) Iterative optimization: Take (S1) as the new current state and repeat steps (1) and (2) until the Q-table converges, that is, for most state-action pairs, the change in Q-value is very small.

[0061] Advantages and beneficial effects of the present invention:

[0062] The present invention proposes a grid voltage control method based on fuzzy control and reinforcement learning. The control method can simplify the control system, has strong adaptability and robustness, and improves fault tolerance and balance effects, thus solving the technical problems of poor robustness of previous control algorithms and complex systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a process flow chart of the present invention. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0065] A grid voltage control method based on fuzzy control and reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0066] Step 1: construct a fuzzy control module to obtain the current grid state variables. At each time step, use the fuzzy control module to infer the current grid state variables and obtain the preliminary reactive power control quantity.

[0067] The specific steps of constructing the fuzzy control module in step 1 include:

[0068] (1) Construct fuzzy subsets:

[0069] Voltage deviation: {NB (negative large), NS (negative small), ZO (zero), PS (positive small), PB (positive large)}

[0070] Voltage change rate: {NB (negative large), NS (negative small), ZO (zero), PS (positive small), PB (positive large)}

[0071] (2) Establish fuzzy reasoning rules:

[0072] Establish fuzzy rule table and perform fuzzy reasoning;

[0073] In this embodiment, there are 2 fuzzy subsets, each fuzzy subset has 5 states, and the fuzzy rule table should be a two-dimensional table. There are 25 situations in total according to the permutations and combinations. Different fuzzy reasoning outputs can be found by looking up the table.

[0074]

[0075] The above table is displayed as a fuzzy rule table, for example:

[0076] 1) If the voltage deviation is "NB (negative large)" and the voltage change rate is "NB (negative large)", the fuzzy reasoning is PB (positive large), which means that the reactive power compensation is increased to the maximum extent.

[0077] 2) If the voltage deviation is "NS (negative small)" and the voltage change rate is "NS (negative small)", the fuzzy reasoning is PS (positive small), indicating that reactive power compensation is increased to the minimum extent.

[0078] 3) If the voltage deviation is "ZO (zero)" and the voltage change rate is "ZO (zero)", the fuzzy reasoning is ZO (zero), indicating that the voltage is within the normal range and reactive power compensation is not adjusted for the time being.

[0079] 4) If the voltage deviation is "PS (positive small)" and the voltage change rate is "PS (positive small)", the fuzzy reasoning is NS (negative small), indicating that reactive power compensation is reduced to the minimum.

[0080] 5) If the voltage deviation is "PB (positive large)" and the voltage change rate is "PB (positive large)", the fuzzy reasoning is NB (negative large), which means that reactive power compensation is minimized to the greatest extent.

[0081] (3) Defuzzification is performed to obtain fuzzy output, and the fuzzy output is converted into a specific value;

[0082] The specific steps of step (3) of step 1 include:

[0083] The present invention is to construct a control strategy for power grid voltage. This strategy is to fuzzy the voltage deviation and voltage change rate of each node in the power grid. Specifically, the voltage deviation and voltage change rate are input into a fuzzy controller, and the fuzzy controller finds the membership value of each parameter belonging to each fuzzy subset through a membership function.

[0084] ① Taking voltage deviation as an example, the voltage is input into the membership function calculation to obtain one or more membership values ​​belonging to a fuzzy subset corresponding to the voltage deviation;

[0085] The membership function may include but is not limited to the following: a triangular membership function, a Gaussian membership function, a generalized bell-shaped membership function, and an S-shaped membership function.

[0086] The triangle membership function is taken as an example for explanation, where the triangle membership function is shown in formula (1).

[0087]

[0088] Here, x represents a variable, which can be any one of the voltage deviation and the voltage change rate.

[0089] a', b', c' are parameters for calculating the membership value, and A(x) represents the calculated membership value.

[0090] ②Convert the actually measured voltage deviation and voltage change rate into the membership function of the fuzzy set;

[0091] For example, assume that the current voltage deviation is -9% and the voltage change rate is 3% / s.

[0092] Membership function of voltage deviation:

[0093] NB: 0.8; NS: 0.2; ZO: 0.0; PS: 0.0; PB: 0.0

[0094] Membership function of voltage change rate:

[0095] NB: 0; NS: 0; ZO: 0; PS: 0.5; PB: 0.5

[0096] ③According to the fuzzy rule table, the fuzzy output is obtained:

[0097] 1) If the voltage deviation is "NB" and the voltage change rate is "PS", the membership degree is 0.8*0.5=0.4

[0098] 2) If the voltage deviation is "NB" and the voltage change rate is "PB", the membership degree is 0.8*0.5=0.4

[0099] 3) If the voltage deviation is "NS" and the voltage change rate is "PS", the membership degree is 0.2*0.5=0.1

[0100] 4) If the voltage deviation is "NS" and the voltage change rate is "PB", the membership degree is 0.2*0.5=0.1

[0101] In this embodiment, it is assumed that we have the following rules, the fuzzy rule table output is PB, increase reactive compensation by 1000kVAR; the fuzzy rule table output is PS, increase reactive compensation by 500kVAR; the fuzzy rule table output is ZO, increase reactive compensation by 0kVAR (no need to increase reactive input); the fuzzy rule table output is NS, reduce reactive compensation by 500kVAR; the fuzzy rule table output is NB, reduce reactive compensation by 1000kVAR.

[0102] ④Convert the fuzzy output into specific numerical values, and then obtain the preliminary reactive power control quantity;

[0103] Commonly used defuzzification methods include the centroid method and the maximum membership method.

[0104] Using the centroid method formula:

[0105]

[0106] In the present invention, the example calculation is 200:

[0107]

[0108] Step 2: Build a reinforcement learning model. Use the preliminary reactive control quantity obtained by the fuzzy control module in step 1 as an input of the reinforcement learning module and send it into the reinforcement learning model together with the current grid state. The reinforcement learning model learns, verifies and outputs the final control action based on the current state and the preliminary reactive control quantity.

[0109] The specific steps of step 2 include:

[0110] (1) Define the state space

[0111] Voltage at each node: The voltage level at each monitoring node of the power grid can be expressed as a continuous value, but in order to simplify the problem, it can be discretized into several intervals, such as "low", "medium", and "high".

[0112] Grid frequency: The frequency of the grid can also be discretized into "low", "medium" and "high".

[0113] Load demand: The load demand of each node can be discretized into "low", "medium" and "high".

[0114] Power output: The active and reactive output of each generator can be discretized into "low", "medium" and "high". Reactive compensation: The number or capacity of reactive compensation equipment put into use can be discretized into "low", "medium" and "high".

[0115] (2) Define the action space

[0116] Adjust active output: increase or decrease the active output of the generator.

[0117] Adjust reactive output: Increase or decrease the reactive output of the generator.

[0118] Cut off reactive power compensation equipment: reduce the investment in reactive power compensation equipment.

[0119] Invest in reactive power compensation equipment: Increase investment in reactive power compensation equipment.

[0120] Adjust taps: Change the tap position of a transformer to adjust the voltage.

[0121] (3) Define the reward function

[0122] Voltage stability: When the voltage levels of all nodes are within the allowed range, positive rewards are given; otherwise, negative penalties are given.

[0123] Frequency stability: When the grid frequency is within the permitted range, positive rewards are given; otherwise, negative penalties are given.

[0124] Load tracking accuracy: When the grid input power matches the load demand well, a positive reward is given; otherwise, a negative penalty is given.

[0125] Minimize operating costs: When operating costs are low, give positive rewards; otherwise, give negative penalties.

[0126] (4) Initialize Q-table Create a Q-table, where state s refers to the state set of all nodes, state set S = {s1, s2, s3…s i}, action a refers to all action situations, action set A = {a1, a2, a3…a i}. Initially, the values ​​in the Q table can all be set to 0 or small random numbers.

[0127] <![CDATA[a1]]> <![CDATA[a2]]> <![CDATA[a3]]> ... <![CDATA[a i ]]> <![CDATA[s1]]> 0 0 0 0 0 <![CDATA[s2]]> 0 0 0 0 0 <![CDATA[s3]]> 0 0 0 0 0 ... 0 0 0 0 0 <![CDATA[s i ]]> 0 0 0 0 0

[0128] (5) Choose learning rate and discount factor

[0129] Learning rate (α): determines the speed at which new information covers old information, and is usually between 0.1 and 0.5.

[0130] Discount factor (γ): reflects the importance of the present value of future rewards relative to immediate rewards, and is usually between 0.9 and 0.99.

[0131] (6) Execute the Q-Learning algorithm and output the final control action;

[0132] 1) Initialization: Start from a certain initial state, for example, the initial state of the power grid is (s0).

[0133] 2) Select an action: Select an action (a0) based on the current Q table and the ε-greedy strategy. For example, if ε=0.1 is set, there is a 90% probability of selecting the action with the highest expected return in the current Q table, and a 10% probability of selecting an action randomly;

[0134] 3) Output the final control action.

[0135] Step 3: Execute the final control action output from step 2 to adjust the output of the generator set or other reactive compensation devices to maintain the voltage stability of the power grid; observe the new state and reward, and feed it back to the reinforcement learning model to update the strategy.

[0136] The specific steps of step 3 include:

[0137] (1) Take action: Execute the selected action (a0) and observe the next state (s1) and immediate reward (r1).

[0138] (2) Update the Q table: Use the Q-Learning update formula to update the values ​​in the Q table:

[0139] Q(s,a)←Q(s,a)+α[r+γmaxQ(s1,a1)-Q(s0,a0)]

[0140] (3) Iterative optimization: Take (S1) as the new current state and repeat steps (1) and (2) until the Q-table converges, that is, for most state-action pairs, the change in Q-value is very small.

[0141] Embodiment 1:

[0142] Assume we have a simple power grid consisting of a generator G1, a transformer T1 and a load L1. The initial state is:

[0143] The active output of G1 is medium (M)

[0144] · The reactive output of G1 is low (L)

[0145] The tap position of T1 is medium (M)

[0146] The load demand of L1 is high (H)

[0147] Grid frequency is medium (M)

[0148] Each node voltage is medium (M)

[0149] 1. Initialize the Q table: Create a Q table with an initial value of 0.

[0150] 2. Select the initial state: S0 = (P1 = M, Q1 = L, Tap = M, PL = H, Freq = M, V = M)

[0151] 3. Select action: Select an action based on the ε-greedy strategy. Suppose you select "increase the active output of G1".

[0152] 4. Execute action: Execute "Increase the active output of G1", the new state is S1 = (P1 = H, Q1 = L, Tap = M, PL = H, Freq = M, V = M). According to the actual operation of the power grid, calculate the immediate reward R1 brought by this operation.

[0153] 5. Update the Q table: Assuming that the maximum Q value in the new state S1 is 0.8 and the immediate reward R1 is 0.5, the updated Q value is:

[0154] Q((P1=M,Q1=L,Tap=M,PL=H,Freq=M,V=M), "Increase the active power output of G1

[0155] =0+0.1×[0.5+0.95×0.8-0]=0.124

[0156] 6. Iterative optimization: Take S1 as the new current state and repeat the above steps until the Q table converges. Example

[0157] Assume we have a simple power grid consisting of a generator G1, a transformer T1 and a load L1. The initial state is:

[0158] · G1's active output is medium (M)

[0159] · The reactive output of G1 is low (L)

[0160] · The tap position of T1 is medium (M)

[0161] · The load demand of L1 is high (H)

[0162] · The grid frequency is medium (M)

[0163] · The voltage of each node is medium (M)

[0164] 5. Initialize Q table: Create a Q table with an initial value of 0.

[0165] 6. Select the initial state: S0 = (P1 = M, Q1 = L, Tap = M, PL = H, Freq = M, V = M)

[0166] 7. Select action: Select an action based on the ε-greedy strategy. Assume that "increase the active output of G1" is selected.

[0167] 8. Execute action: Execute "Increase the active output of G1", the new state is S1 = (P1 = H, Q1 = L, Tap = M, PL = H, Freq = M, V = M). According to the actual operation of the power grid, calculate the immediate reward R1 brought by this operation.

[0168] 9. Update the Q table: Assuming that the maximum Q value in the new state S1 is 0.8 and the immediate reward R1 is 0.5, the updated Q value is:

[0169] Q((P1=M,Q1=L,Tap=M,PL=H,Freq=M,V=M), "Increase the active power output of G1

[0170] =0+0.1×[0.5+0.95×0.8-0]=0.124

[0171] 10. Iterative optimization: Take S1 as the new current state and repeat the above steps until the Q table converges. Design of reward function:

[0172] Voltage stability

[0173] · Definition: Ensure that the voltage levels of all nodes are within the allowable range.

[0174] · calculate:

[0175] If the voltages of all nodes are within the allowed range (eg, 0.95 pu to 1.05 pu), a positive reward Rv=1 is given.

[0176] If the voltage of any node exceeds the allowable range, a negative penalty Rv=-1 is given or different degrees of penalty are given according to the degree of excess, for example, Rv=-|Vi-Vnom|, where Vi is the voltage of node i and Vnom is the nominal voltage.

[0177] 2. Frequency stability

[0178] · Definition: Ensure that the frequency of the power grid is within the permissible range.

[0179] · calculate:

[0180] If the grid frequency is within the permitted range (e.g., 49.5 Hz to 50.5 Hz), a positive reward Rf = 1 is given. If the grid frequency exceeds the permitted range, a negative penalty Rf = -1 is given or different degrees of penalty are given according to the degree of excess, for example, Rf = -|f-fnom|, where f is the current frequency and fnom is the nominal frequency.

[0181] 3. Load tracking accuracy

[0182] Definition: Ensure that the generator output power is well matched to the load demand.

[0183] calculate:

[0184] Calculate the total generator output power Pgen and the total load demand Pload.

[0185] If Pgen-Pload|<∈ (where ∈ is a small threshold), give a positive reward Rp=1.

[0186] Otherwise, a negative penalty Rp=-|Pgen-Pload| is given.

[0187] 4. Minimize operating costs

[0188] Definition: Minimize the operating cost of the generator.

[0189] calculate:

[0190] The operating cost C of all generators can be calculated using a linear or nonlinear cost function, such as C = ∑iciPi, where ci is the unit cost of generator i and Pi is the output power of generator i. A negative reward Rc = -C is given.

[0191] 5. Comprehensive Reward Function

[0192] Combine the above reward items to form a total reward function R. You can use the weighted sum form, and the weights can be adjusted according to specific needs:

[0193] R=wvRv+wfRf+wpRp+wcRc

[0194] Among them, wv, wf, wp and wc are the weights of each reward item, and their values ​​can be adjusted according to their importance. For example:

[0195] wv=0.4(voltage stability is very important)

[0196] wf=0.3(frequency stability is also important)

[0197] wp=0.2(Load tracking accuracy is important but secondary)

[0198] wc = 0.1 (minimizing operating costs is important but relatively minor)

[0199] Example

[0200] Assume that the current indicators are as follows:

[0201] · Voltage stability: All node voltages are between 0.95pu and 1.05pu, Rv = 1

[0202] · Frequency stability: The grid frequency is 50.2Hz, the allowable range is 49.5Hz to 50.5Hz, Rf = 1

[0203] · Load tracking accuracy: The total generator output power is 100MW, the total load demand is 102MW, Rp = -2

[0204] · Operating cost: The total operating cost is 1000 yuan, Rc = -1000

[0205] Assume the weights are:

[0206] wv=0.4 3wf=0.3wp=0.2wc=0.1

[0207] The total reward is:

[0208] R=0.4×1+0.3×1+0.2×(-2)+0.1×(-1000)=0.4+0.3-0.4-100=-99.7

[0209] The working principle of the present invention is:

[0210] The present invention combines fuzzy control with reinforcement learning algorithm to generate a new grid voltage control strategy, which not only retains the system simplicity and strong robustness characteristics of fuzzy control, but also inherits the characteristic of reinforcement learning that focuses on balance.

[0211] Fuzzy control is a computer intelligent control based on fuzzy set theory, fuzzy language variables and fuzzy logic reasoning. The input of this mechanism is to convert the data of 0 and 1 into a value between 0 and 1 through fuzzification, which is closer to human thinking than the original binary method of either 0 or 1. The data is fuzzy in the process of inference, but through the step of defuzzification, the output is an accurate value. Fuzzy control is often used in intelligent computing, expert system construction and neural network applications. The characteristic of fuzzy control is to simplify the complexity of system design. It is particularly suitable for nonlinear, time-varying, and incomplete model systems. Fuzzy control is a nonlinear control method that is easy to control and master, with good adaptability and robustness, and good fault tolerance.

[0212] Reinforcement learning is a field in machine learning that emphasizes how to act based on the environment to maximize expected benefits. Reinforcement learning does not require labeled input-output pairs, nor does it require precise corrections to non-optimal solutions. Its focus is on the balance between exploring unknown areas and utilizing existing knowledge.

[0213] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A power grid voltage control method based on fuzzy control and reinforcement learning, characterized in that: The following steps are involved: Step 1: construct a fuzzy control module to obtain the current grid state variables. At each time step, use the fuzzy control module to infer the current grid state variables and obtain the preliminary reactive power control quantity. Step 2: Build a reinforcement learning model. Use the preliminary reactive power control quantity obtained by the fuzzy control module in step 1 as an input of the reinforcement learning module and send it to the reinforcement learning model together with the current power grid state. The reinforcement learning model learns, verifies and outputs the final control action based on the current state and the preliminary reactive power control quantity. Step 3: Execute the final control action output from step 2, adjust the output of the generator set or adjust other reactive compensation devices, observe the new state and reward, and feed it back to the reinforcement learning model to update the strategy.

2. The grid voltage control method based on fuzzy control and reinforcement learning according to claim 1, characterized in that: The specific steps of constructing the fuzzy control module in step 1 include: (1) Construct fuzzy subsets: Voltage deviation: {NB (negative large), NS (negative small), ZO (zero), PS (positive small), PB (positive large)} Voltage change rate: {NB (negative large), NS (negative small), ZO (zero), PS (positive small), PB (positive large)} (2) Establish fuzzy reasoning rules: Establish fuzzy rule table and perform fuzzy reasoning; 1) If the voltage deviation is "NB (negatively large)" and the voltage change rate is "NB (negatively large)", the fuzzy reasoning is PB (positively large), which means that reactive power compensation is increased to the maximum extent; 2) If the voltage deviation is "NS (negative small)" and the voltage change rate is "NS (negative small)", the fuzzy reasoning is PS (positive small), which means to increase reactive power compensation to the minimum extent; 3) If the voltage deviation is "ZO (zero)" and the voltage change rate is "ZO (zero)", the fuzzy reasoning is ZO (zero), indicating that the voltage is within the normal range and reactive power compensation is not adjusted for the time being; 4) If the voltage deviation is "PS (positive small)" and the voltage change rate is "PS (positive small)", the fuzzy reasoning is NS (negative small), which means that reactive power compensation is reduced to the minimum; 5) If the voltage deviation is "PB (positive large)" and the voltage change rate is "PB (positive large)", the fuzzy reasoning is NB (negative large), which means that reactive power compensation is minimized; (3) Defuzzification is performed to obtain fuzzy output, and the fuzzy output is converted into a specific value.

3. The grid voltage control method based on fuzzy control and reinforcement learning according to claim 2, characterized in that: The specific steps of step (3) of step 1 include: ① Taking voltage deviation as an example, the voltage is input into the membership function calculation to obtain one or more membership values ​​belonging to a fuzzy subset corresponding to the voltage deviation; ②Convert the actually measured voltage deviation and voltage change rate into the membership function of the fuzzy set; ③According to the fuzzy rule table, the fuzzy output is obtained: ④Convert the fuzzy output into specific numerical values ​​to obtain the preliminary reactive power control quantity.

4. A grid voltage control method based on fuzzy control and reinforcement learning according to claim 1 or 2, characterized in that: The specific steps of step 2 include: (1) Define the state space Voltage of each node: discretize it into several intervals, such as "low", "medium", and "high"; Grid frequency: Discretized into "low", "medium", and "high"; Load demand: The load demand of each node, discretized into "low", "medium", and "high"; Power output: active and reactive output of each generator, discretized into "low", "medium", and "high"; Reactive power compensation: the number or capacity of reactive power compensation equipment put into use, discretized into "low", "medium" and "high"; (2) Define the action space Adjust active output: increase or decrease the active output of the generator; Adjust reactive output: increase or decrease the reactive output of the generator; Removal of reactive power compensation equipment: reduce the investment in reactive power compensation equipment; Invest in reactive power compensation equipment: Increase investment in reactive power compensation equipment; Adjust tap: Change the tap position of the transformer to adjust the voltage; (3) Define the reward function Voltage stability: When the voltage levels of all nodes are within the allowed range, positive rewards are given; otherwise, negative penalties are given; Frequency stability: When the grid frequency is within the permitted range, positive rewards are given; otherwise, negative penalties are given; Load tracking accuracy: When the grid input power matches the load demand well, a positive reward is given; otherwise, a negative penalty is given; Minimize operating costs: When operating costs are low, give positive rewards; otherwise, give negative penalties; (4) Initialize Q table Create a Q table, where state s refers to the state set of all nodes, state set S = {s1, s2, s3…s i }, action a refers to all action situations, action set A = {a1, a2, a3…a i }; Initially, all values ​​in the Q table can be set to 0 or small random numbers; (5) Choose learning rate and discount factor Learning rate (α): determines the speed at which new information covers old information, usually between 0.1 and 0.5; Discount factor (γ): reflects the importance of the present value of future rewards relative to immediate rewards, usually between 0.9 and 0.99; (6) Execute the Q-Learning algorithm and output the final control action; 1) Initialization: starting from a certain initial state, for example, the initial state of the power grid is (s0); 2) Action selection: Select an action (a0) based on the current Q-table and the ε-greedy strategy; for example, setting ε = 0.1 means that there is a 90% probability of selecting the action with the highest expected return in the current Q-table and a 10% probability of selecting an action randomly; 3) Output the final control action.

5. A grid voltage control method based on fuzzy control and reinforcement learning according to claim 1 or 2, characterized in that: The specific steps of step 3 include: (1) Take action: Execute the selected action (a0) and observe the next state (s1) and immediate reward (r1); (2) Update the Q table: Use the Q-Learning update formula to update the values ​​in the Q table: Q(s,a)←Q(s,a)+α[r+γmaxQ(s1,a1)-Q(s0,a0)] (3) Iterative optimization: Take (S1) as the new current state and repeat steps (1) and (2) until the Q-table converges, that is, for most state-action pairs, the change in Q-value is very small.

Images