Power distribution network region voltage control method based on multi-agent deep reinforcement learning

Through the distribution network area voltage control method with deep reinforcement learning of multiple agents, the problem that single agent is difficult to achieve global optimal control is solved, the real-time and robustness of voltage control in the distribution network is improved, and the voltage safety is achieved The control within the scope and the reduction of network losses meet the safety and stability of the distribution network and the economic operation needs.

CN119070315APending Publication Date: 2024-12-03HUAIAN OF JIANGSU ELECTRIC POWER CO POWER SUPPLY +1
View PDF 0 Cites 12 Cited by

Patent Information

Application Number
CN202411047828.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

When the existing technology deals with complex and changeable distribution network environments, it is difficult to achieve global optimal control by single agent deep reinforcement learning, and there are challenges in the stability, real-time and computing complexity of multi-agent systems, which makes it difficult to meet power distribution. The security and stability of the network and the economic operation needs.

Method used

Adopt the distribution network area voltage control method based on multi-agent deep reinforcement learning, and design the multi-agent depth deterministic strategy gradient algorithm by building a reactive voltage optimization control model to realize offline centralized training and online distributed execution, and improve distribution Real-time and robustness of grid voltage control.

Benefits of technology

Effectively control the voltage of each node of the distribution network within the safe range, reduce network losses, improve the overall control performance and robustness of the distribution network, and achieve safer, stable and economical operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119070315A_ABST
    Figure CN119070315A_ABST
Patent Text Reader

Abstract

The invention discloses a power distribution network regional voltage control method and algorithm improvement based on multi-agent deep reinforcement learning, and the method comprises the steps: constructing a power distribution network reactive voltage optimization control model according to a power distribution network regional voltage control target; converting the optimization model into a distributed partially observable Markov decision process; designing a multi-agent deep reinforcement learning improved algorithm based on a multi-agent depth deterministic strategy gradient and using the multi-agent deep reinforcement learning improved algorithm to solve the Markov decision process; the regional voltage cooperative control of the power distribution network is realized in an off-line centralized training and on-line distributed execution mode. According to the method, regional voltage optimization control and algorithm improvement of multi-agent deep reinforcement learning are realized, and the voltage of each node of the power distribution network can be effectively controlled within a safe range.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of new technologies for power system operation and control, and in particular to a distribution network regional voltage control method based on multi-agent deep reinforcement learning. Background Art

[0002] With the proposal of the "dual carbon" goal and the continuous promotion of policies such as "whole-county photovoltaic", various new energy generators represented by photovoltaic and wind power have begun to be connected to the distribution network on a large scale in the form of distributed generators (DG). Due to the randomness, volatility and uncertainty of new energy output, the access of a large number of DGs has caused power flow backflow, resulting in voltage fluctuations, over-limit and increased network losses, which seriously affects the safety, stability and economic operation of the distribution network. Therefore, reactive voltage control has become a current research hotspot.

[0003] From the perspective of modeling the controlled object, traditional active distribution network voltage optimization control is mostly based on accurate distribution network mechanism models to solve the optimal control instructions. However, in the actual operation of the distribution network, accurate physical parameters are difficult to obtain, and factors such as equipment aging and changes in the operating environment will also lead to time-varying and uncertainty in model parameters. This limits the development of optimization methods based on mechanism models. In addition, when dealing with complex and changeable distribution network environments, traditional methods have large computational complexity and poor real-time performance, making it difficult to meet the requirements of rapid response.

[0004] The voltage control strategy based on deep reinforcement learning (DRL) adopts a data-driven approach and does not rely on the mechanism model of the distribution network. It is one of the outstanding representatives of model-free control methods. Based on the massive historical data in the distribution network, DRL can train the agent from historical experience, so there is no need for an accurate mechanism model. It learns interactively with the environment and gives a near-optimal control strategy in a short time. It has strong generalization and adaptability. However, when faced with the complex and changing environment of the distribution network, the single-agent DRL is prone to fall into the local optimal solution and it is difficult to achieve global optimal control.

[0005] In order to solve the above problems, the distribution network voltage control method combined with multi-agent deep reinforcement learning (Multi-Agent DRL, MADRL) has gradually become a research hotspot. MADRL can effectively improve the overall control performance and robustness of the system through the collaborative work of multiple agents, sharing tasks, sharing information and coordinating strategies with each other. However, at this stage, the MADRL method still faces challenges in practical applications, such as the stability and convergence of multi-agent systems, real-time and computational complexity. In order to solve the shortcomings of the prior art, the present invention proposes a distribution network regional voltage control method based on multi-agent deep reinforcement learning, which aims to improve the real-time, stability and robustness of distribution network voltage control by improving the MADRL algorithm, thereby achieving safer, more stable and economical distribution network operation. Summary of the invention

[0006] To solve the above problems, the present invention discloses a distribution network regional voltage control method and algorithm improvement based on multi-agent deep reinforcement learning, comprising the following steps:

[0007] Step 1: According to the voltage control target of the distribution network area, a distribution network reactive voltage optimization control model is constructed;

[0008] Step 2: Convert the optimization model into a distributed partially observable Markov decision process;

[0009] Step 3: Design an improved multi-agent deep reinforcement learning algorithm based on multi-agent deep deterministic policy gradient and use it to solve the Markov decision process;

[0010] Step 4: Realize coordinated voltage control in the distribution network area through offline centralized training and online distributed execution.

[0011] Furthermore, the regional voltage control target is: by adjusting the active and reactive output of distributed power sources, the voltage fluctuation of each node in the system during the scheduling period is limited to an allowable range and the network loss is reduced as much as possible.

[0012] Furthermore, the distributed power source includes: distributed photovoltaic, energy storage system, and wind turbine.

[0013] Furthermore, the expression of the regional voltage control objective function is as follows:

[0014]

[0015] Among them, T is the number of scheduling cycles per day, C u is the voltage over-limit additional cost coefficient, is the average node voltage offset of the system in the tth scheduling period, C loss is the additional cost coefficient of network loss, Pt loss is the system network loss in the tth scheduling period.

[0016] Furthermore, The expression is as follows:

[0017]

[0018] Where n is the number of distribution network nodes, U i,t is the voltage at node i, U ref is the reference voltage.

[0019] Furthermore, P t loss The expression is as follows:

[0020]

[0021] Among them, U j,t is the voltage at node j, G ij and B ij are the conductance and susceptance between nodes, respectively, ij,t is the phase angle difference between the voltage at the node and the voltage across the node.

[0022] Furthermore, the constraints of the reactive voltage optimization control model are as follows:

[0023] Power flow constraints:

[0024]

[0025] Among them, P G,i,t and Q G,i,t are the active power and reactive power injected into node i, respectively, P L,i,t and Q L,i,t are the active load and reactive load of node i respectively.

[0026] Voltage Constraints:

[0027] U i,min ≤U i,t ≤U i,max

[0028] Among them, U i,min and U i,max are the lower and upper limits of the node voltage respectively.

[0029] New energy power constraints:

[0030]

[0031] in: and are the active and reactive power of the node’s new energy, and They are respectively the maximum active and reactive power of the node’s renewable energy, including photovoltaic and wind power.

[0032] Energy storage constraints:

[0033]

[0034] E min ≤E i,t ≤E max

[0035] Where: E i,t is the power level of the energy storage device at node i, η ch and η dis are the charging and discharging efficiency of the energy storage device, is the charging and discharging power of the node energy storage device, E min and E max They are the upper and lower capacity limits of the energy storage device respectively.

[0036] Furthermore, the transformed distributed partially observable Markov decision process mainly includes:

[0037] State space, action space, and reward function.

[0038] Furthermore, the state space is represented as follows:

[0039] S t is the global observation, which is the set of observations obtained by all agents at time t. The local observation of agent k at time t is s k,t , including the voltage amplitude of each node in the area controlled by agent k, the energy level of energy storage, and the active load and reactive load, expressed as:

[0040] s k,t =(U i,t ,E i,t ,P L,i,t ,Q L,i,t )

[0041] Furthermore, the action space is represented as follows:

[0042] A t is composed of the actions of all agents at time, a k,t ∈A t Including the active power output of distributed photovoltaic, wind turbine and energy storage devices in the area controlled by intelligent agent k, expressed as:

[0043]

[0044] Furthermore, the reward function is expressed as follows:

[0045] The reward function is designed under the framework of full cooperation among all agents. The rewards of each agent are consistent, and its expression is as follows:

[0046]

[0047] Among them, each intelligent agent minimizes the voltage deviation and network loss of the entire scheduling cycle through collaborative control.

[0048] Furthermore, the framework principle of the multi-agent deep deterministic policy gradient algorithm is:

[0049] The MATD3 algorithm uses an actor-critic framework and operates in continuous space. The actor is responsible for inputting the state information observed by the agent and outputting actions according to the strategy. The critic will judge the current agent strategy by outputting Q values ​​based on the current state and actions received, and will dynamically adjust the strategy based on the rewards fed back by the environment. The strategy will also be updated by the actor based on the Q value, and finally the optimal strategy will be obtained through continuous adjustment.

[0050] Furthermore, the critic updates the Q value as follows:

[0051] The critic updates the Q value by minimizing the loss function, which is as follows:

[0052]

[0053] Among them, L θ is the loss function, θ is the critic network parameter, and E is the expected return; r t is the reward at time t; γ is the discount factor; is the target Q value at time t; s t is the state at time t; a t is the action at time t; Qθ(s t ,a t ) is the estimated Q value at time t.

[0054] Furthermore, the calculation method of the actor updating the neural network parameter policy gradient is:

[0055]

[0056] in, is the actor network parameter; E s~pπ is the distribution in state p π Expected return under Receives s for the actor t Post-output action.

[0057] Furthermore, the pruned double Q learning of the MATD3 algorithm is expressed as follows:

[0058] The MATD3 algorithm uses two sets of target critics and selects the smaller Q value as the update target, which solves the problem of obtaining a suboptimal strategy due to overestimation of the Q value. The expression is as follows:

[0059]

[0060] Among them, y t is the target value function; r(s t ,a t ) is state s t and action a t The reward under θ i ′ is the target critic network parameter; is the target value function.

[0061] Furthermore, to address the overfitting problem, the principle of MATD3 algorithm smoothing Q value is as follows:

[0062] The MATD3 algorithm smoothes the Q value by introducing a regularization term, that is, adding Gaussian noise to the next action to achieve the average target value and reduce the variance. The expression is as follows:

[0063]

[0064] ε~clip[N(0,σ)-c,c]

[0065] Where: ε is the randomly added noise, which obeys the clipped normal distribution, and c is the clipping boundary.

[0066] Further, the improvement ideas based on the MATD3 algorithm are as follows:

[0067] The PS and PER mechanisms are integrated into the MATD3 algorithm to improve the convergence speed and generalization ability of the algorithm during training by sharing parameters and improving the utilization of empirical data.

[0068] Furthermore, the principle of the PS mechanism is as follows:

[0069] All agents share two sets of critic networks Q(s) on the basis of ensuring that each agent can make different decisions based on different local observations. k,t ,a k,t ) and a set of actor networks π(s k,t), there is no need to build critic and actor networks for individual agents. The shared network is updated based on the experience data collected by all agents, so that the learning behavior of an agent can be influenced by the experience of other agents to achieve better training results.

[0070] Furthermore, the principle of the PER mechanism is as follows:

[0071] The intelligent agent performs importance sampling on the historical experience data stored in the experience buffer pool, distinguishes the importance of the experience data in the experience buffer pool, and solves the instability and overfitting problems caused by the correlation of reinforcement learning sequence data in the neural network training process by breaking the correlation between the data. This enables the intelligent agent to review the historical experience data and mix it with the real-time data, and efficiently mine and utilize important data.

[0072] Furthermore, the calculation process of PER is as follows:

[0073] Sampling the nth empirical data TD deviation δ n The absolute value of is defined as p(n), which is expressed as follows:

[0074]

[0075] Where: y is the target value function.

[0076] Calculate the sampling probability P(n) of the experience data in the experience buffer pool. The expression is as follows:

[0077]

[0078] Where: N is the number of experience data in the experience buffer pool, and α is the priority coefficient.

[0079] Sample part of the experience data in the experience buffer pool according to the sampling probability P(n);

[0080] Calculate the importance sampling weight ω n , the expression is as follows:

[0081] ω n =(NP(n)) -β

[0082] Among them: β is the importance coefficient. After each training, the sampling weight of the empirical data will be updated according to the priority. ω n and δ n It is also used to calculate the loss function and update the critic network.

[0083] Furthermore, the specific process of offline centralized training and online distributed execution is as follows:

[0084] In the offline training phase, the network parameters of all agents are initialized first.

[0085] Based on the collected historical operation data, the intelligent agent learns the optimal voltage control strategy through multiple rounds of training, and each round consists of multiple moments.

[0086] At time t, each agent collects local observation information s of the controlled sub-region k,t , based on local information and shared network output decision action a k,t And add random Gaussian noise ε.

[0087] Through power flow calculation, each agent calculates the reward value r k,t and the observation value s at the next moment k,t+1 , and at the same time all the agent's experience data {s k,t ,a k,t ,r k,t ,s k,t+1}Formulaized and stored in a shared experience pool.

[0088] All agents enter the next decision-making stage.

[0089] All agents sample from the experience buffer pool and update parameters. After training is completed, the optimal control strategy is saved.

[0090] During the online execution phase, each agent observes its own local information and uses trained strategies to make real-time decisions.

[0091] Beneficial effects of the present invention:

[0092] 1. According to the voltage control target of the distribution network area, a reactive voltage optimization control model of the distribution network is constructed; the optimization model is converted into a distributed partially observable Markov decision process; an improved multi-agent deep reinforcement learning algorithm based on multi-agent deep deterministic policy gradient is designed and used to solve the Markov decision process; the coordinated voltage control of the distribution network area is realized through offline centralized training and online distributed execution.

[0093] 2. It can effectively control the voltage of each node in the distribution network within a safe range. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 This is a complete framework diagram of the distribution network voltage optimization control strategy of the present invention;

[0095] Figure 2 This is a framework diagram of the homogeneous intelligent agent of the improved algorithm of the present invention;

[0096] Figure 3IEEE 33-node system topology diagram in an embodiment of the present invention;

[0097] Figure 4 is a voltage distribution diagram of each node in an embodiment of the present invention;

[0098] Figure 5 is a reward curve diagram in an embodiment of the present invention;

[0099] Figure 6 is an active power curve diagram of each photovoltaic output in the embodiment of the present invention;

[0100] Figure 7 It is a charge state diagram of each energy storage device in the embodiment of the present invention. DETAILED DESCRIPTION

[0101] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. It should be noted that the words "front", "rear", "left", "right", "upper" and "lower" used in the following description refer to directions in the accompanying drawings, and the words "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0102] like Figure 1 The present invention provides a method for controlling voltage in a distribution network area based on multi-agent deep reinforcement learning, which specifically includes the following steps:

[0103] Step A: construct a distribution network reactive voltage optimization control model according to the distribution network area voltage control target;

[0104] Step B, transform the optimization model into a distributed partially observable Markov decision process;

[0105] Step C: Design an improved multi-agent deep reinforcement learning algorithm based on multi-agent deep deterministic policy gradient and use it to solve the Markov decision process, such as Figure 2 as stated;

[0106] Step D: Realize the coordinated voltage control of the distribution network area through offline centralized training and online distributed execution.

[0107] In the embodiment of the present invention, the specific operations of step A are as follows:

[0108] Step A01: The regional voltage control target is: by adjusting the active and reactive output of distributed power sources, the voltage fluctuation of each node in the system during the scheduling period is limited to the allowable range and the network loss is reduced as much as possible. This method embodiment is based on the system structure of IEEE 33 node system such as Figure 3 As mentioned above, the voltage distribution in the distribution network structure is as follows: Figure 4 described.

[0109] The distributed power supply comprises:

[0110] Distributed photovoltaics, energy storage systems, and wind turbines.

[0111] The expression of the regional voltage control objective function is as follows:

[0112]

[0113] Among them, T is the number of scheduling cycles per day, C u is the voltage over-limit additional cost coefficient, is the average node voltage offset of the system in the tth scheduling period, C loss is the additional cost coefficient of network loss, P t loss is the system network loss in the tth scheduling period.

[0114] The expression is as follows:

[0115]

[0116] Where n is the number of distribution network nodes, U i,t is the voltage at node i, U ref is the reference voltage.

[0117] P t loss The expression is as follows:

[0118]

[0119] Among them, U j,t is the voltage at node j, G ij and B ij are the conductance and susceptance between nodes, respectively, ij,t is the phase angle difference between the voltage at the node and the voltage across the node.

[0120] Step A02: The constraints of the reactive voltage optimization control model are as follows:

[0121] Power flow constraints:

[0122]

[0123] Among them, P G,i,t and Q G,i,t are the active power and reactive power injected into node i, respectively, P L,i,t and Q L,i,t are respectively the active load and reactive load of node i. The photovoltaic output power of the method embodiment of the present invention is as follows Figure 6 described.

[0124] Voltage Constraints:

[0125] U i,min ≤U i,t ≤U i,max

[0126] Among them, U i,min and U i,max are the lower and upper limits of the node voltage respectively.

[0127] New energy power constraints:

[0128]

[0129] in: and are the active and reactive power of the node’s new energy, and They are respectively the maximum active and reactive power of the node’s renewable energy, including photovoltaic and wind power.

[0130] Energy storage constraints:

[0131]

[0132] E min ≤E i,t ≤E max

[0133] Where: E i,t is the power level of the energy storage device at node i, η ch and η dis are the charging and discharging efficiency of the energy storage device, is the charging and discharging power of the node energy storage device, E min and E max The state of charge of the energy storage device in the method embodiment of the present invention is as follows: Figure 7 described.

[0134] The specific operation of step B of the method of the present invention is as follows:

[0135] Step B01: The converted distributed partially observable Markov decision process mainly includes:

[0136] State space, action space, and reward function.

[0137] The state space representation is as follows:

[0138] S t is the global observation, which is the set of observations obtained by all agents at time t. The local observation of agent k at time t is sk,t , including the voltage amplitude of each node in the area controlled by agent k, the energy level of energy storage, and the active load and reactive load, expressed as:

[0139] s k,t =(U i,t ,E i,t ,P L,i,t ,Q L,i,t )

[0140] The action space is represented as follows:

[0141] A t is composed of the actions of all agents at time, a k,t ∈A t Including the active power output of distributed photovoltaic, wind turbine and energy storage devices in the area controlled by intelligent agent k, expressed as:

[0142]

[0143] The reward function is expressed as follows:

[0144] The reward function is designed under the framework of full cooperation among all agents. The rewards of each agent are consistent, and its expression is as follows:

[0145]

[0146] Among them, each intelligent agent minimizes the voltage deviation and network loss of the entire scheduling cycle through collaborative control. The reward curve of the example of the method of the present invention is as follows: Figure 5 described.

[0147] In the embodiment of the present invention, the specific operations of step C are as follows:

[0148] Step C01: The framework principle of the multi-agent deep deterministic policy gradient algorithm is:

[0149] The MATD3 algorithm uses an actor-critic framework and operates in continuous space. The actor is responsible for inputting the state information observed by the agent and outputting actions according to the strategy. The critic will judge the current agent strategy by outputting Q values ​​based on the current state and actions received, and will dynamically adjust the strategy based on the rewards fed back by the environment. The strategy will also be updated by the actor based on the Q value, and finally the optimal strategy will be obtained through continuous adjustment.

[0150] The critic updates the Q value as follows:

[0151] The critic updates the Q value by minimizing the loss function, which is as follows:

[0152]

[0153] Among them, L θ is the loss function, θ is the critic network parameter, and E is the expected return; r t is the reward at time t; γ is the discount factor; is the target Q value at time t; s t is the state at time t; a t is the action at time t; Q θ (s t ,a t ) is the estimated Q value at time t.

[0154] The calculation method of the actor updating the neural network parameter policy gradient is:

[0155]

[0156] in, is the actor network parameter; E s~pπ is the distribution in state p π Expected return under Receives s for the actor t Post-output action.

[0157] The pruned double Q learning of the MATD3 algorithm is expressed as follows:

[0158] The MATD3 algorithm uses two sets of target critics and selects the smaller Q value as the update target, which solves the problem of obtaining a suboptimal strategy due to overestimation of the Q value. The expression is as follows:

[0159]

[0160] Among them, y t is the target value function; r(s t ,a t ) is state s t and action a t The reward under θ i ′ is the target critic network parameter; is the target value function.

[0161] To address the overfitting problem, the principle of MATD3 algorithm smoothing Q value is as follows:

[0162] The MATD3 algorithm smoothes the Q value by introducing a regularization term, that is, adding Gaussian noise to the next action to achieve the average target value and reduce the variance. The expression is as follows:

[0163]

[0164] ε~clip[N(0,σ)-c,c]

[0165] Where: ε is the randomly added noise, which obeys the clipped normal distribution, and c is the clipping boundary.

[0166] Step C02, the improvement ideas based on the MATD3 algorithm are as follows:

[0167] The PS and PER mechanisms are integrated into the MATD3 algorithm to improve the convergence speed and generalization ability of the algorithm during training by sharing parameters and improving the utilization of empirical data.

[0168] The principle of the PS mechanism is as follows:

[0169] All agents share two sets of critic networks Q(s) on the basis of ensuring that each agent can make different decisions based on different local observations. k,t ,a k,t ) and a set of actor networks π(s k,t ), there is no need to build critic and actor networks for individual agents. The shared network is updated based on the experience data collected by all agents, so that the learning behavior of an agent can be influenced by the experience of other agents to achieve better training results.

[0170] The principle of PER mechanism is as follows:

[0171] The intelligent agent performs importance sampling on the historical experience data stored in the experience buffer pool, distinguishes the importance of the experience data in the experience buffer pool, and solves the instability and overfitting problems caused by the correlation of reinforcement learning sequence data in the neural network training process by breaking the correlation between the data. This enables the intelligent agent to review the historical experience data and mix it with the real-time data, and efficiently mine and utilize important data.

[0172] The calculation process of PER is as follows:

[0173] Sampling the nth empirical data TD deviation δ n The absolute value of is defined as p(n), which is expressed as follows:

[0174]

[0175] Where: y is the target value function.

[0176] Calculate the sampling probability P(n) of the experience data in the experience buffer pool. The expression is as follows:

[0177]

[0178] Where: N is the number of experience data in the experience buffer pool, and α is the priority coefficient.

[0179] Sample part of the experience data in the experience buffer pool according to the sampling probability P(n);

[0180] Calculate the importance sampling weight ω n , the expression is as follows:

[0181] ω n =(NP(n)) -β

[0182] Among them: β is the importance coefficient. After each training, the sampling weight of the empirical data will be updated according to the priority. ω n and δ n It is also used to calculate the loss function and update the critic network.

[0183] In the embodiment of the present invention, the specific operations of step D are as follows:

[0184] The specific process of step D01, offline centralized training, and online distributed execution is as follows:

[0185] In the offline training phase, the network parameters of all agents are initialized first.

[0186] Based on the collected historical operation data, the intelligent agent learns the optimal voltage control strategy through multiple rounds of training, and each round consists of multiple moments.

[0187] At time t, each agent collects local observation information s of the controlled sub-area k,t , based on local information and shared network output decision action a k,t And add random Gaussian noise ε.

[0188] Through power flow calculation, each agent calculates the reward value r k,t and the observation value s at the next moment k,t+1 , and at the same time all the agent's experience data {s k,t ,a k,t ,r k,t ,s k,t+1}Formulaized and stored in a shared experience pool.

[0189] All agents enter the next decision-making stage.

[0190] All agents sample from the experience buffer pool and update parameters. After training is completed, the optimal control strategy is saved.

[0191] During the online execution phase, each agent observes its own local information and uses trained strategies to make real-time decisions.

[0192] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above technical features.

Claims

1. A method for regional voltage control in distribution networks based on multi-agent deep reinforcement learning. It is characterized in that The steps include: Step 1: According to the voltage control target of the distribution network area, a distribution network reactive voltage optimization control model is constructed; Step 2: Convert the optimal control model into a distributed partially observable Markov decision process; Step 3: Design an improved multi-agent deep reinforcement learning method based on multi-agent deep deterministic policy gradient and use it to solve the Markov decision process; Step 4: Realize coordinated voltage control in the distribution network area through offline centralized training and online distributed execution.

2. According to the method for controlling voltage in distribution network area based on multi-agent deep reinforcement learning in claim 1, It is characterized in that In step 1: the distributed power source includes distributed photovoltaic, energy storage system, and wind turbine; wherein the expression of the regional voltage control objective function is as follows: Among them, T is the number of scheduling cycles per day, C u is the voltage over-limit additional cost coefficient, is the average node voltage offset of the system in the tth scheduling period, C loss is the additional cost coefficient of network loss, P t loss is the system network loss in the tth scheduling period; The expression is as follows: Where n is the number of distribution network nodes, U i,t is the voltage at node i, U ref is the reference voltage; P t loss The expression is as follows: Among them, U j,t is the voltage at node j, G ij and B ij are the conductance and susceptance between nodes, respectively, ij,t is the phase angle difference between the voltage at the node and the voltage across the node.

3. According to the method for controlling voltage in distribution network area based on multi-agent deep reinforcement learning in claim 1, It is characterized in that The constraints of the step 1: reactive power voltage optimization control model are as follows: Power flow constraints: Among them, P G,i,t and Q G,i,t are the active power and reactive power injected into node i, respectively, P L,i,t and Q L,i,t are the active load and reactive load of node i respectively; Voltage Constraints: IN i,min ≤U i,t ≤U i,max Among them, U i,min and U i,max are the lower and upper limits of the node voltage respectively; New energy power constraints: in: and are the active and reactive power of the node’s new energy, and are the maximum active and reactive power of the node’s renewable energy, including photovoltaic and wind power; Energy storage constraints: AND min ≤E i,t ≤E max Where: E i,t is the power level of the energy storage device at node i, η ch and η dis are the charging and discharging efficiency of the energy storage device, is the charging and discharging power of the node energy storage device, E min and E max They are the upper and lower capacity limits of the energy storage device respectively.

4. According to the method for controlling voltage in distribution network area based on multi-agent deep reinforcement learning according to claim 1, It is characterized in that The step 2: the converted distributed partially observable Markov decision process mainly includes a state space, an action space and a reward function; the state space is expressed as follows: S t is the global observation, which is the set of observations obtained by all agents at time t. The local observation of agent k at time t is s k,t , including the voltage amplitude of each node in the area controlled by agent k, the energy level of energy storage, and the active load and reactive load, expressed as: s k,t =(U i,t ,E i,t ,P L,i,t ,Q L,i,t ) The action space is represented as follows: A t is composed of the actions of all agents at time, a k,t ∈A t Including the active power output of distributed photovoltaic, wind turbine and energy storage devices in the area controlled by intelligent agent k, expressed as: The reward function is expressed as follows: The reward function is designed under the framework of full cooperation among all agents. The rewards of each agent are consistent, and its expression is as follows: Among them, each intelligent agent minimizes the voltage deviation and network loss of the entire scheduling cycle through collaborative control.

5. According to the method for controlling voltage in distribution network area based on multi-agent deep reinforcement learning in claim 1, It is characterized in that The improved method of multi-agent deep reinforcement learning with multi-agent deep deterministic policy gradient is that the MATD3 algorithm adopts the actor-critic framework and acts on continuous space. The actor is responsible for inputting the state information observed by the agent and outputting actions according to the strategy. The critic will judge the current agent strategy by outputting Q values ​​based on the current state and actions received, and will dynamically adjust the strategy with reference to the rewards fed back by the environment. The strategy will also be updated by the actor according to the Q value, and finally the optimal strategy will be obtained through continuous adjustment.

6. According to the method of voltage control in distribution network area based on multi-agent deep reinforcement learning in claim 5, It is characterized in that The way critic updates Q value is as follows: critic updates Q value by minimizing the loss function, the loss function is as follows: Among them, L θ is the loss function, θ is the critic network parameter, and E is the expected return; r t is the reward at time t; γ is the discount factor; is the target Q value at time t; s t is the state at time t; a t is the action at time t; Q θ (s t ,a t ) is the estimated Q value at time t.

7. According to claim 5, a method for controlling voltage in a distribution network area based on multi-agent deep reinforcement learning, It is characterized in that The calculation method of the actor updating the neural network parameter policy gradient is: in, is the actor network parameter; E s~pπ is the distribution in state p π Expected return under Receives s for the actor t Post-output action.

8. According to claim 5, a method for controlling voltage in a distribution network area based on multi-agent deep reinforcement learning, It is characterized in that The pruned double Q learning of the MATD3 algorithm is expressed as follows: The MATD3 algorithm uses two sets of target critics and selects the smaller Q value as the update target, which solves the problem of obtaining a suboptimal strategy due to overestimation of the Q value. The expression is as follows: Among them, y t is the target value function; r(s t ,a t ) is state s t and action a t The reward under θ i ′ is the target critic network parameter; is the target value function; To address the overfitting problem, the principle of MATD3 algorithm smoothing Q value is as follows: The MATD3 algorithm smoothes the Q value by introducing a regularization term, that is, adding Gaussian noise to the next action to achieve the average target value and reduce the variance. The expression is as follows: ε~clip[N(0,σ)-c,c] Where: ε is the randomly added noise, which obeys the clipped normal distribution, and c is the clipping boundary.

9. According to claim 5, a distribution network area voltage control method based on multi-agent deep reinforcement learning, It is characterized in that The PS and PER mechanisms are integrated into the MATD3 algorithm to improve the convergence speed and generalization ability of the algorithm during training by sharing parameters and improving the utilization of empirical data. The principle of the PS mechanism is as follows: All agents share two sets of critic networks Q(s) on the basis of ensuring that each agent can make different decisions based on different local observations. k,t ,a k,t ) and a set of actor networks π(s k,t ), there is no need to build critic and actor networks for a single agent; the shared network is updated based on the experience data collected by all agents, so that the learning behavior of a certain agent can be influenced by the experience of other agents to achieve better training results; The calculation process of PER is as follows: Sampling the nth empirical data TD deviation δ n The absolute value of is defined as p(n), which is expressed as follows: Where: y is the target value function. Calculate the sampling probability P(n) of the experience data in the experience buffer pool. The expression is as follows: Where: N is the number of experience data in the experience buffer pool, and α is the priority coefficient. Sample part of the experience data in the experience buffer pool according to the sampling probability P(n); Calculate the importance sampling weight ω n , the expression is as follows: ω n =(NP(n)) -β Among them: β is the importance coefficient. After each training, the sampling weight of the empirical data will be updated according to the priority. ω n and δ n It is also used to calculate the loss function and update the critic network.

10. According to claim 5, a method for controlling voltage in a distribution network area based on multi-agent deep reinforcement learning, It is characterized in that The specific process of offline centralized training and online distributed execution in step 4 is as follows: Step 41: In the offline training phase, firstly initialize the network parameters of all agents; Step 42: The intelligent agent learns the optimal voltage control strategy through multiple rounds of training based on the collected historical operation data, each round consisting of multiple moments; Step 43: At time t, each agent collects local observation information s of the controlled sub-region k,t , based on local information and shared network output decision action a k,t And add random Gaussian noise ε; Step 44: Through power flow calculation, each agent calculates the reward value r k,t and the observation value s at the next moment k,t+1 , and at the same time all the agent's experience data {s k,t ,a k,t ,r k,t ,s k,t+1 }Formulaize and store in a shared experience pool; Step 45: All agents enter the next decision-making stage; Step 46: All agents sample from the experience buffer pool and update parameters. After training is completed, the optimal control strategy is saved; Step 47: During the online execution phase, each agent observes its own local information and uses the trained strategy to make real-time decisions.

Citation Information

Cited By

  • Optimized dispatching method for electricity-hydrogen-heat comprehensive energy microgrid based on improved TD3 algorithm

    CN119787352A

  • Optimal Scheduling Method for Integrated Electric-Hydrogen-Thermal Energy Microgrid Based on Improved TD3 Algorithm

    CN119787352B

  • Main distribution cooperative voltage optimization control method and equipment under communication and electrical coupling

    CN119853079A

  • Power distribution network voltage control method based on multi-agent deep reinforcement learning

    CN120582137A

  • A voltage control method for distribution networks based on multi-agent deep reinforcement learning

    CN120582137B