Voltage out-of-limit control system and method based on multi-agent deep reinforcement learning

Abstract

The invention provides a voltage out-of-limit control system based on multi-agent deep reinforcement learning, and the system comprises a power grid region division module which is used for dividing a power grid into a plurality of regions, and each region at least comprises one piece of photovoltaic equipment; the photovoltaic equipment control strategy model building module is used for carrying out modeling by adopting a partially observable Markov decision process aiming at photovoltaic equipment in each region of the power grid to obtain a photovoltaic equipment control strategy model; and the photovoltaic equipment control strategy output module is used for inputting the current operation parameters of the power grid in each region into the photovoltaic equipment control strategy model, and solving the photovoltaic equipment control strategy model by using a double-delay depth deterministic strategy gradient algorithm. And obtaining reactive power of each photovoltaic inverter and outputting the reactive power to a power grid of a corresponding area to control the voltage of each node to be within a preset range. According to the method, the problems of voltage fluctuation and out-of-limit caused by distributed photovoltaic access to the power grid are effectively solved, and the control efficiency and the control effect are improved.

Description

Technical Field

[0001] The present invention relates to the field of power system operation and control technology, and in particular to a voltage over-limit control system and method based on multi-agent deep reinforcement learning. Background Art

[0002] The access of distributed photovoltaic power to the grid changes the power flow, which may lead to voltage over-limit, line loss and other problems. At the same time, photovoltaic power generation is intermittent, random and uncertain, which can easily cause voltage fluctuations, reduce power quality, and fail to guarantee reliable power supply to user loads. Therefore, with the large-scale access of distributed photovoltaic power to the grid, it has brought huge challenges to the safe and stable operation of the grid.

[0003] Preventing grid voltage from exceeding the limit is one of the main means to achieve grid operation risk control. The main goal of grid voltage over-limit control is to maintain the voltage of each node within a feasible range and reduce network losses as much as possible. Voltage control is achieved by adjusting the output of reactive equipment. Grid voltage over-limit control methods mainly include centralized control and distributed control. Centralized control collects information from the entire grid and transmits the collected information to the central processing unit, which calculates the optimal power flow and issues dispatch instructions; distributed control mainly interacts with local and adjacent area information to achieve grid voltage control. However, the above methods are all based on accurate physical models. In practice, the operating parameters of the grid are difficult to obtain accurately, which limits the application of these methods. Reinforcement learning is a model-free data-driven method that does not rely on the precise parameters of the system. It interacts with the environment and continuously tries and fails to learn the optimal control strategy.

[0004] When performing voltage over-limit control based on traditional reinforcement learning, all reactive devices are usually regarded as a unified intelligent agent. However, due to limitations such as communication burden and control delay, treating reactive devices as a unified intelligent agent can easily lead to problems such as low control efficiency and poor control effect when performing grid voltage over-limit control. Summary of the invention

[0005] The purpose of the present invention is to address the defects of the prior art and provide a voltage over-limit control system and method based on multi-agent deep reinforcement learning.

[0006] The present invention provides a voltage over-limit control system based on multi-agent deep reinforcement learning, comprising:

[0007] A power grid area division module, used to divide the power grid into several areas, each area includes at least one photovoltaic device;

[0008] The photovoltaic equipment control strategy model building module is used to model the photovoltaic equipment in each area of ​​the power grid using a partially observable Markov decision process to obtain a photovoltaic equipment control strategy model;

[0009] The photovoltaic equipment control strategy output module is used to input the current operating parameters of the power grid in each area into the photovoltaic equipment control strategy model, and use the double-delay deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model, obtain the reactive power of each photovoltaic inverter and output it to the power grid in the corresponding area to control the voltage of each node within a preset range.

[0010] Furthermore, in the photovoltaic equipment control strategy model building module, a partially observable Markov decision process is used to model the photovoltaic equipment in each area of ​​the power grid, and the specific method for obtaining the photovoltaic equipment control strategy model is as follows:

[0011] First, we build a partially observable Markov decision process: {S,O,A,R,T 1 ,γ}, where S represents the state set of the power grid, O, A, and R represent the observation set, action set, and reward set of the photovoltaic equipment, respectively, and T 1 is the state transfer function, the state transfer function T 1 It represents the probability that the grid will transfer to the next state according to the current state and action. γ is the discount rate. The discount rate γ represents the attention paid by the photovoltaic equipment to future rewards. The larger the γ, the more attention the photovoltaic equipment pays to future rewards.

[0012] Then, the reactive power output by the photovoltaic inverter is used to solve the voltage over-limit control problem. The objective function F of the voltage over-limit control is:

[0013]

[0014] Where, T 总 is the total control period, M is the number of nodes in the power grid, Q PV,n,t is the reactive power of the nth PV inverter at time t, V m,t is the voltage over-limit value of node m at time t, δ is the additional penalty value for voltage over-limit, P loss,t is the line loss at time t, α is the network loss coefficient, and β is the node voltage over-limit penalty coefficient;

[0015] Furthermore, the representation of the variables in the partially observable Markov decision process is as follows:

[0016] The state set of the power grid is expressed as follows:

[0017]

[0018] Where: S is the state set of the power grid at time t, P L,m,tand Q L,m,t are the active load and reactive load at node m at time t, respectively. If there is no load or photovoltaic device at node m, the corresponding power of the load or photovoltaic device is 0. t is the state of the power grid at time t, P PV,m,t is the active power of the photovoltaic inverter at node m at time t, Q PV,m,t is the reactive power of the photovoltaic inverter at node m at time t, V m,t and θ m,t are the voltage amplitude and angle at node m at time t, respectively, and T represents the matrix transpose;

[0019] The set of observations for a photovoltaic device is represented as follows:

[0020]

[0021] Where: O is the observation set of all photovoltaic devices at time t, o n,t Represents the vector composed of the active load and reactive load at node m in the region where the nth photovoltaic device is located and the active power and reactive power of the photovoltaic inverter at node m, n = 1, ..., N is the number of photovoltaic devices, and a single photovoltaic device can only observe the status of the nodes and the status of the photovoltaic inverter in the region to which the photovoltaic device belongs;

[0022] The action set of the photovoltaic device is expressed as follows:

[0023] A={a n,t |n=1,2,…,N}

[0024] Where: A is the action set of all photovoltaic devices at time t, a n,t is the set of all possible actions of the nth photovoltaic device at time t, action a n,t ∈A t And a n,t =Q PV,n,t , Q PV,n,t is the reactive power of PV inverter n at time t;

[0025] The reward set is represented as follows:

[0026] r t =-αP loss,t -β(V m,t +δ)∈R

[0027] In the formula, r t is the reward function, V m,t is the voltage exceeding the limit of node m at time t;

[0028] State transition function T 1 :

[0029] State transition function T 1 Indicates that the photovoltaic device is in state s t Next, perform action a n,t The probability distribution of the state at the next moment after that, the state transition is defined as the photovoltaic equipment controlling the output of the inverter for reactive compensation according to the current environmental observation and its own strategy, so that the grid flow is redistributed, the node voltage and line loss change, the photovoltaic active power and load power fluctuate randomly, and thus transfer to the state at the next moment.

[0030] Furthermore, the photovoltaic device control strategy output module inputs the current operation parameters of the power grid into the photovoltaic device control strategy model, and uses the double-delayed deep deterministic policy gradient algorithm to solve the photovoltaic device control strategy model. The specific method is:

[0031] The double-delayed deep deterministic policy gradient algorithm includes a policy network and a value network, which transforms the current operating parameters of the power grid: the current state of the power grid s t and observation o n,t , as the input of the policy network, the output is action a n,t , define action a n,t is the reactive power of PV inverter n at time t. Each PV device continuously interacts with the grid environment in its respective area as follows: PV inverter n performs action a n,t After that, the state changes from s t Transfer to t+1 , observed by o t Transfer to o t+1 And get the reward r under the current power grid environment t , the state transfer and reward are fed back to the photovoltaic device, and {s t ,o t ,a t ,r t ,s t+1 ,o t+1}; t ,o t ,a t ,r t ,s t+1 ,o t+1}Stored in the experience replay pool of the double-delayed deep deterministic policy gradient algorithm, and randomly extracting states, observations, actions, and rewards from the experience replay pool for training, updating the parameters of the policy network, and the policy network parameters are updated according to maximizing the cumulative expected reward;

[0032] The strategy network is used to output the action of the photovoltaic equipment according to the current state of the power grid. The strategy network is expressed as maximizing the strategy function J(π), and the formula is as follows:

[0033] J(π)=E S,O～Ξ[Q π (s t ,o t ,π(s t ,o t ;θ))]

[0034] In the formula, the maximization strategy function J(π) represents the expected reward that can be obtained, Ξ represents the distribution of strategy π under the current state and observation, and strategy π is defined as the photovoltaic device in state s t Next, perform action a t , E represents mathematical expectation, S, O~Ξ represents that the state and observation follow the distribution Ξ, Q π (s t ,o t ,π(s t ,o t ; θ)) is the value of executing strategy π under the current state and observation, and θ is the parameter of the policy network;

[0035] Use gradient descent to update the parameters θ of the policy function:

[0036]

[0037] In the formula, A = π(S) represents the photovoltaic device in state s t Next, perform action a t ;

[0038] The value network is used to estimate the value of the action predicted by the policy network. The value network is represented by the action value function Q(s t ,a t ), action value function Q(s t ,a t ) is expressed by the Bellman equation:

[0039] Q(s t ,a t )=E[r t +γQ(s t+1 ,a t+1 )]

[0040] The value network is updated based on minimizing the difference between the estimate and the target. The value network is parameterized using a neural network and approximates the action value function Q(s) by minimizing the Bellman residual. t ,a t ), the formula is as follows:

[0041] J Q (ω)=E[r t +γQ(s t+1 ,a t+1 |ω Q' )-Q(s t ,at |ω Q )]

[0042] In the formula, J Q (ω) is the loss function used to minimize the Bellman residual, ω is the parameter of the value network, s t+1 is the state at time t+1, a t+1 is the action taken at time t+1, ω Q is the parameter of the current value network, ω Q' is the parameter of the target value network, and the value network output y at the next moment t It can be expressed as:

[0043]

[0044] In the formula, Refers to the parameters of the target value network, j = 1, 2, j = 1 Represents the parameters of the first target value network, when j = 2 represents the parameters of the second target value network;

[0045] In the interaction process between photovoltaic equipment and power grid, the parameters ω of the current value network are updated by minimizing the loss function Q , the loss function L(ω) is defined as:

[0046]

[0047] In the formula, K is the number of training samples, y' t is the value of the target value network. In the first target value network and the second target value network, the smaller target value network value is selected as the value network output y at the next moment. t , the output is expressed as:

[0048]

[0049] Then use the target action to calculate the value of the target value network to smooth and regularize the target strategy. t+1 The actions are:

[0050] a t+1 =π(s t+1 )+ε

[0051] In the formula, ε is random noise, which is normally distributed, and the sampled noise is truncated ε~clip(N(0,σ),-c,c), σ is the variance, c is the upper and lower limits of the noise, and N represents normal distribution;

[0052] The target strategy network and target value network parameters are updated in the form of soft updates, and the formula is as follows:

[0053]

[0054] θ π' =τθ π +(1-τ)θ π'

[0055] In the formula, τ is the update step size, is the parameter of the current value network, j = 1, 2, j = 1 Represents the parameters of the first current value network, when j = 2 represents the parameters of the second current value network, is the parameter of the target value network, θ π is the parameter of the current policy network, θ π' are the parameters of the target policy network;

[0056] According to the above steps, the action a of each photovoltaic inverter that maximizes the cumulative expected reward J(π) is solved. t , as a control strategy for photovoltaic equipment.

[0057] A voltage over-limit control method based on multi-agent deep reinforcement learning includes the following steps:

[0058] Divide the power grid into several areas, each of which includes at least one photovoltaic device;

[0059] For the photovoltaic equipment in each area of ​​the power grid, a partially observable Markov decision process is used to model the photovoltaic equipment control strategy model;

[0060] The current operating parameters of the power grid in each area are input into the photovoltaic equipment control strategy model, and the photovoltaic equipment control strategy model is solved using a double-delay deep deterministic policy gradient algorithm to obtain the reactive power of each photovoltaic inverter and output it to the power grid in the corresponding area to control the voltage of each node within a preset range.

[0061] Furthermore, the photovoltaic equipment in each area of ​​the power grid is modeled using a partially observable Markov decision process to obtain a specific method for the photovoltaic equipment control strategy model:

[0062] First, we build a partially observable Markov decision process: {S,O,A,R,T 1 ,γ}, where S represents the state set of the power grid, O, A, and R represent the observation set, action set, and reward set of the photovoltaic equipment, respectively, and T 1 is the state transfer function, the state transfer function T 1It represents the probability that the grid will transfer to the next state according to the current state and action. γ is the discount rate. The discount rate γ represents the attention paid by the photovoltaic equipment to future rewards. The larger the γ, the more attention the photovoltaic equipment pays to future rewards.

[0063] Then, the reactive power output by the photovoltaic inverter is used to solve the voltage over-limit control problem. The objective function F of the voltage over-limit control is:

[0064]

[0065] Where, T 总 is the total control period, M is the number of nodes in the power grid, Q PV,n,t is the reactive power of the nth PV inverter at time t, V m,t is the voltage over-limit value of node m at time t, δ is the additional penalty value for voltage over-limit, P loss,t is the line loss at time t, α is the network loss coefficient, and β is the node voltage over-limit penalty coefficient;

[0066] Furthermore, the representation of the variables in the partially observable Markov decision process is as follows:

[0067] The state set of the power grid is expressed as follows:

[0068]

[0069] Where: S is the state set of the power grid at time t, P L,m,t and Q L,m,t are the active load and reactive load at node m at time t, respectively. If there is no load or photovoltaic device at node m, the corresponding power of the load or photovoltaic device is 0. t is the state of the power grid at time t, P PV,m,t is the active power of the photovoltaic inverter at node m at time t, Q PV,m,t is the reactive power of the photovoltaic inverter at node m at time t, V m,t and θ m,t are the voltage amplitude and angle at node m at time t, respectively, and T represents the matrix transpose;

[0070] The set of observations for a photovoltaic device is represented as follows:

[0071]

[0072] Where: O is the observation set of all photovoltaic devices at time t, o n,tRepresents the vector composed of the active load and reactive load at node m in the region where the nth photovoltaic device is located and the active power and reactive power of the photovoltaic inverter at node m, n = 1, ..., N is the number of photovoltaic devices, and a single photovoltaic device can only observe the status of the nodes and the status of the photovoltaic inverter in the region to which the photovoltaic device belongs;

[0073] The action set of the photovoltaic device is expressed as follows:

[0074] A={a n,t |n=1,2,…,N}

[0075] Where: A is the action set of all photovoltaic devices at time t, a n,t is the set of all possible actions of the nth photovoltaic device at time t, action a n,t ∈A t And a n,t =Q PV,n,t , Q PV,n,t is the reactive power of PV inverter n at time t;

[0076] The reward set is represented as follows:

[0077] r t =-αP loss,t -β(V m,t +δ)∈R

[0078] In the formula, r t is the reward function, V m,t is the voltage exceeding the limit of node m at time t;

[0079] State transition function T 1 :

[0080] State transition function T 1 Indicates that the photovoltaic device is in state s t Next, perform action a n,t The probability distribution of the state at the next moment after that, the state transition is defined as the photovoltaic equipment controlling the output of the inverter for reactive compensation according to the current environmental observation and its own strategy, so that the grid flow is redistributed, the node voltage and line loss change, the photovoltaic active power and load power fluctuate randomly, and thus transfer to the state at the next moment.

[0081] Furthermore, the specific method of inputting the current operating parameters of the power grid into the photovoltaic device control strategy model and using the double-delayed deep deterministic policy gradient algorithm to solve the photovoltaic device control strategy model is:

[0082] The double-delayed deep deterministic policy gradient algorithm includes a policy network and a value network, which transforms the current operating parameters of the power grid: the current state of the power grid s t and observation on,t , as the input of the policy network, the output is action a n,t , define action a n,t is the reactive power of PV inverter n at time t. Each PV device continuously interacts with the grid environment in its respective area as follows: PV inverter n performs action a n,t After that, the state changes from s t Transfer to t+1 , observed by o t Transfer to o t+1 And get the reward r under the current power grid environment t , the state transfer and reward are fed back to the photovoltaic device, and {s t ,o t ,a t ,r t ,s t+1 ,o t+1}; t ,o t ,a t ,r t ,s t+1 ,o t+1}Stored in the experience replay pool of the double-delayed deep deterministic policy gradient algorithm, and randomly extracting states, observations, actions, and rewards from the experience replay pool for training, updating the parameters of the policy network, and the policy network parameters are updated according to maximizing the cumulative expected reward;

[0083] The strategy network is used to output the action of the photovoltaic equipment according to the current state of the power grid. The strategy network is expressed as maximizing the strategy function J(π), and the formula is as follows:

[0084] J(π)=E S,O～Ξ [Q π (s t ,o t ,π(s t ,o t ;θ))]

[0085] In the formula, the maximization strategy function J(π) represents the expected reward that can be obtained, Ξ represents the distribution of strategy π under the current state and observation, and strategy π is defined as the photovoltaic device in state s t Next, perform action a t , E represents mathematical expectation, S, O~Ξ represents that the state and observation follow the distribution Ξ, Q π (s t ,o t ,π(s t ,o t ; θ)) is the value of executing strategy π under the current state and observation, and θ is the parameter of the policy network;

[0086] Use gradient descent to update the parameters θ of the policy function:

[0087]

[0088] In the formula, A = π(S) represents the photovoltaic device in state s t Next, perform action a t ;

[0089] The value network is used to estimate the value of the action predicted by the policy network. The value network is represented by the action value function Q(s t ,a t ), action value function Q(s t ,a t ) is expressed by the Bellman equation:

[0090] Q(s t ,a t )=E[r t +γQ(s t+1 ,a t+1 )]

[0091] The value network is updated based on minimizing the difference between the estimate and the target. The value network is parameterized using a neural network and approximates the action value function Q(s) by minimizing the Bellman residual. t ,a t ), the formula is as follows:

[0092] J Q (ω)=E[r t +γQ(s t+1 ,a t+1 |ω Q' )-Q(s t ,a t |ω Q )]

[0093] In the formula, J Q (ω) is the loss function used to minimize the Bellman residual, ω is the parameter of the value network, s t+1 is the state at time t+1, a t+1 is the action taken at time t+1, ω Q is the parameter of the current value network, ω Q' is the parameter of the target value network, and the value network output y at the next moment t It can be expressed as:

[0094]

[0095] In the formula, Refers to the parameters of the target value network, j = 1, 2, j = 1 Represents the parameters of the first target value network, when j = 2 represents the parameters of the second target value network;

[0096] In the interaction process between photovoltaic equipment and power grid, the parameters ω of the current value network are updated by minimizing the loss function Q , the loss function L(ω) is defined as:

[0097]

[0098] In the formula, K is the number of training samples, y' t is the value of the target value network. In the first target value network and the second target value network, the smaller target value network value is selected as the value network output y at the next moment. t , the output is expressed as:

[0099]

[0100] Then use the target action to calculate the value of the target value network to smooth and regularize the target strategy. t+1 The actions are:

[0101] a t+1 =π(s t+1 )+ε

[0102] In the formula, ε is random noise, which is normally distributed, and the sampled noise is truncated ε~clip(N(0,σ),-c,c), σ is the variance, c is the upper and lower limits of the noise, and N represents normal distribution;

[0103] The target strategy network and target value network parameters are updated in the form of soft updates, and the formula is as follows:

[0104]

[0105] θ π' =τθ π +(1-τ)θ π'

[0106] In the formula, τ is the update step size, is the parameter of the current value network, j = 1, 2, j = 1 Represents the parameters of the first current value network, when j = 2 represents the parameters of the second current value network, is the parameter of the target value network, θ π is the parameter of the current policy network, θ π' are the parameters of the target policy network;

[0107] According to the above steps, the action a of each photovoltaic inverter that maximizes the cumulative expected reward J(π) is solved. t , as a control strategy for photovoltaic equipment.

[0108] An electronic device, characterized in that it includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned voltage over-limit control method based on multi-agent deep reinforcement learning by executing the computer instructions.

[0109] A computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the above-mentioned voltage over-limit control method based on multi-agent deep reinforcement learning is implemented.

[0110] The beneficial effects of the present invention are:

[0111] 1. The present invention transforms the voltage over-limit control problem of distributed photovoltaic power grid into a Markov decision process, regards each distributed photovoltaic inverter as a separate intelligent agent, and uses a double-delay deep deterministic policy gradient algorithm to efficiently train the intelligent agent. The intelligent agent interacts with the power grid environment, learns the optimal reactive power output strategy, and realizes the control of the voltage over-limit of the power grid, effectively solving the voltage fluctuation and over-limit problems caused by the access of distributed photovoltaic power grid to the power grid, and improving the control efficiency and control effect.

[0112] 2. The present invention first constructs a power grid model containing distributed photovoltaics, and divides the power grid into several different areas; secondly, the power grid voltage over-limit control problem is converted into a partially observable Markov decision process model, which includes agent state S, observation O, action A, reward R, state transfer function T and discount rate γ; further, the double-delay deep deterministic policy gradient algorithm is used to solve the above-constructed partially observable Markov decision model, each agent selects an action by perceiving the environment, and adjusts the observed result after executing the action. Finally, the fully trained agent can make the optimal action strategy (i.e., the output strategy of the photovoltaic inverter) under the current environment, so as to obtain the maximum reward value. Using this method, the rapid adjustment of the power grid voltage can be achieved, and the occurrence of voltage over-limit behavior can be effectively avoided, thereby achieving the purpose of controlling the risk of power grid operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 This is a block diagram of the voltage over-limit control system based on multi-agent deep reinforcement learning.

[0114] Figure 2 Flowchart of the double-delayed deep deterministic policy gradient algorithm. DETAILED DESCRIPTION

[0115] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0116] Example 1

[0117] refer to Figure 1 , a voltage over-limit control system based on multi-agent deep reinforcement learning, including:

[0118] A power grid area division module, used to divide the power grid into several areas, each area includes at least one photovoltaic device;

[0119] The photovoltaic equipment control strategy model building module is used to model the photovoltaic equipment in each area of ​​the power grid using a partially observable Markov decision process to obtain a photovoltaic equipment control strategy model;

[0120] The photovoltaic equipment control strategy output module is used to input the current operating parameters of the power grid in each area into the photovoltaic equipment control strategy model, and use the double-delay deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model, obtain the reactive power of each photovoltaic inverter and output it to the power grid in the corresponding area to control the voltage of each node within a preset range.

[0121] (1) In the photovoltaic equipment control strategy model building module, a partially observable Markov decision process is used to model the photovoltaic equipment in each area of ​​the power grid. The specific method for obtaining the photovoltaic equipment control strategy model is as follows:

[0122] Due to the large scale of the power grid, in order to reduce the size of the state space and improve the efficiency of the solution, the present invention divides the power grid into several areas. At the same time, due to limitations such as communication delay and privacy protection, distributed photovoltaics can only observe information in local areas. Therefore, the voltage over-limit control problem is modeled as a partially observable Markov decision model. Partially Observable Markov Decision Process (POMDP) ​​is a framework for modeling problems of making decisions in an environment with incomplete information. POMDP is an extension of the Markov decision process (MDP), which assumes that the state of the system is only partially observed. In POMDP, the agent (or decision maker) cannot directly observe all the real states of the environment, but can only infer the state through a series of observations and make decisions based on the belief state of these possible states. Modeling incomplete information: POMDP can explicitly model situations where the agent cannot directly observe all the real states of the environment. This makes POMDP an ideal tool for dealing with complex dynamic environments with uncertainty and incomplete information. Since POMDP takes into account the uncertainty of observations, the agent needs to consider multiple possible states when making decisions, thereby enhancing the robustness of the decision. This robustness enables the agent to better adapt and respond to uncertainty and change. The goal of POMDP is to maximize long-term returns under incomplete information. By considering possible future states and corresponding rewards, the agent can develop a more long-term planning strategy to maximize long-term benefits. The above advantages make POMDP not dependent on precise parameters and very suitable for solving power grid problems.

[0123] First, we build a partially observable Markov decision process: {S,O,A,R,T 1 ,γ}, where S represents the state set of the power grid, O, A, and R represent the observation set, action set, and reward set of the photovoltaic equipment, respectively, and T 1 is the state transfer function, the state transfer function T 1 It represents the probability that the grid will transfer to the next state according to the current state and action. γ is the discount rate. The discount rate γ represents the attention paid by the photovoltaic equipment to future rewards. The larger the γ, the more attention the photovoltaic equipment pays to future rewards.

[0124] Then, the reactive power output by the photovoltaic inverter is used to solve the voltage over-limit control problem. The objective function F of the voltage over-limit control is:

[0125]

[0126] Where, T 总 is the total control period, M is the number of nodes in the power grid, Q PV,n,tis the reactive power of the nth PV inverter at time t, V m,t is the voltage over-limit value of node m at time t, δ is the additional penalty value for voltage over-limit, P loss,t is the line loss at time t, α is the network loss coefficient, and β is the node voltage over-limit penalty coefficient;

[0127] The representation of variables in the partially observable Markov decision process is as follows:

[0128] The state set of the power grid is expressed as follows:

[0129]

[0130] Where: S is the state set of the power grid at time t, P L,m,t and Q L,m,t are the active load and reactive load at node m at time t, respectively. If there is no load or photovoltaic device at node m, the corresponding power of the load or photovoltaic device is 0. t is the state of the power grid at time t, P PV,m,t is the active power of the photovoltaic inverter at node m at time t, Q PV,m,t is the reactive power of the photovoltaic inverter at node m at time t, V m,t and θ m,t are the voltage amplitude and angle at node m at time t, respectively, and T represents the matrix transpose;

[0131] The set of observations for a photovoltaic device is represented as follows:

[0132]

[0133] Where: O is the observation set of all photovoltaic devices at time t, o n,t Represents the vector composed of the active load and reactive load at node m in the region where the nth photovoltaic device is located and the active power and reactive power of the photovoltaic inverter at node m, n = 1, ..., N is the number of photovoltaic devices, and a single photovoltaic device can only observe the status of the nodes and the status of the photovoltaic inverter in the region to which the photovoltaic device belongs;

[0134] The action set of the photovoltaic device is expressed as follows:

[0135] A={a n,t |n=1,2,…,N}

[0136] Where: A is the action set of all photovoltaic devices at time t, a n,t is the set of all possible actions of the nth photovoltaic device at time t, action a n,t ∈A t And a n,t =Q PV,n,t , QPV,n,t is the reactive power of PV inverter n at time t;

[0137] The reward set is represented as follows:

[0138] r t =-αP loss,t -β(V m,t +δ)∈R

[0139] In the formula, r t is the reward function, V m,t The voltage of node m at time t exceeds the limit; each photovoltaic device is in a completely cooperative relationship, and the system voltage deviation and line loss are minimized through mutual coordination, so the rewards are shared among the intelligent agents.

[0140] State transition function T 1 :

[0141] State transition function T 1 Indicates that the photovoltaic device is in state s t Next, perform action a n,t The probability distribution of the state at the next moment after that, the state transition is defined as the photovoltaic equipment controlling the output of the inverter for reactive compensation according to the current environmental observation and its own strategy, so that the grid flow is redistributed, the node voltage and line loss change, the photovoltaic active power and load power fluctuate randomly, and thus transfer to the state at the next moment.

[0142] (2)Reference Figure 2 The photovoltaic equipment control strategy output module inputs the current operation parameters of the power grid into the photovoltaic equipment control strategy model, and uses the double-delayed deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model. The specific method is:

[0143] The twin delayed deep deterministic policy gradient algorithm (TD3) improves the Q-value overestimation problem in the deep deterministic policy gradient algorithm (DDPG). Based on the Actor-Critic framework, TD3 generates two evaluation values ​​by constructing two target Critic (value) networks, learns the two networks at the same time, and selects the Q-value using the minimization method; updates the target network and current network parameters by delaying the Actor (policy) network, and introduces noise to make the estimated value smoother.

[0144] The double-delayed deep deterministic policy gradient algorithm includes a policy network and a value network, which transforms the current operating parameters of the power grid: the current state of the power grid s tand observation o n,t , as the input of the policy network, the output is action a n,t , define action a n,t is the reactive power of PV inverter n at time t. Each PV device continuously interacts with the grid environment in its respective area as follows: PV inverter n performs action a n,t After that, the state changes from s t Transfer to t+1 , observed by o t Transfer to o t+1 And get the reward r under the current power grid environment t , the state transfer and reward are fed back to the photovoltaic device, and {s t ,o t ,a t ,r t ,s t+1 ,o t+1}; t ,o t ,a t ,r t ,s t+1 ,o t+1}Stored in the experience replay pool of the double-delayed deep deterministic policy gradient algorithm, and randomly extracting states, observations, actions, and rewards from the experience replay pool for training, updating the parameters of the policy network, and the policy network parameters are updated according to maximizing the cumulative expected reward;

[0145] The strategy network is used to output the action of the photovoltaic equipment according to the current state of the power grid. The strategy network is expressed as maximizing the strategy function J(π), and the formula is as follows:

[0146] J(π)=E S,o～Ξ [Q π (s t ,o t ,π(s t ,o t ;θ))]

[0147] In the formula, the maximization strategy function J(π) represents the expected reward that can be obtained, Ξ represents the distribution of strategy π under the current state and observation, and strategy π is defined as the photovoltaic device in state s t Next, perform action a t , E represents mathematical expectation, S, O~Ξ represents that the state and observation follow the distribution Ξ, Q π (s t ,o t ,π(s t ,o t ; θ)) is the value of executing strategy π under the current state and observation, and θ is the parameter of the policy network;

[0148] Use gradient descent to update the parameters θ of the policy function:

[0149]

[0150] In the formula, A = π(S) represents the photovoltaic device in state s t Next, perform action a t ;

[0151] The value network is used to estimate the value of the action predicted by the policy network. The value network is represented by the action value function Q(s t ,a t ), action value function Q(s t ,a t ) is expressed by the Bellman equation:

[0152] Q(s t ,a t )=E[r t +γQ(s t+1 ,a t+1 )]

[0153] The value network is updated based on minimizing the difference between the estimate and the target. The value network is parameterized using a neural network and approximates the action value function Q(s) by minimizing the Bellman residual. t ,a t ), the formula is as follows:

[0154] J Q (ω)=E[r t +γQ(s t+1 ,a t+1 |ω Q' )-Q(s t ,a t |ω Q )]

[0155] In the formula, J Q (ω) is the loss function used to minimize the Bellman residual, ω is the parameter of the value network, s t+1 is the state at time t+1, a t+1 is the action taken at time t+1, ω Q is the parameter of the current value network, ω Q' is the parameter of the target value network, and the value network output y at the next moment t It can be expressed as:

[0156]

[0157] In the formula, Refers to the parameters of the target value network, j = 1, 2, j = 1 Represents the parameters of the first target value network, when j = 2 represents the parameters of the second target value network;

[0158] In the interaction process between photovoltaic equipment and power grid, the parameters ω of the current value network are updated by minimizing the loss function Q , the loss function L(ω) is defined as:

[0159]

[0160] In the formula, K is the number of training samples, y' t is the value of the target value network. In the first target value network and the second target value network, the smaller target value network value is selected as the value network output y at the next moment. t , the output is expressed as:

[0161]

[0162] Then use the target action to calculate the value of the target value network to smooth and regularize the target strategy. t+1 The actions are:

[0163] a t+1 =π(s t+1 )+ε

[0164] In the formula, ε is random noise, which is normally distributed, and the sampled noise is truncated ε~clip(N(0,σ),-c,c), σ is the variance, c is the upper and lower limits of the noise, and N represents normal distribution;

[0165] The target strategy network and target value network parameters are updated in the form of soft updates, and the formula is as follows:

[0166]

[0167] θ π' =τθ π +(1-τ)θ π'

[0168] In the formula, τ is the update step size, is the parameter of the current value network, j = 1, 2, j = 1 Represents the parameters of the first current value network, when j = 2 represents the parameters of the second current value network, is the parameter of the target value network, θ π is the parameter of the current policy network, θ π' are the parameters of the target policy network;

[0169] According to the above steps, the action a of each photovoltaic inverter that maximizes the cumulative expected reward J(π) is solved. t , as a control strategy for photovoltaic equipment.

[0170] In the present invention, each agent is in a fully cooperative relationship, and each agent minimizes the objective function through mutual coordination. Although the actions of each agent only act on its own area, each agent shares a global reward function, which includes voltage limit crossing and line loss. Therefore, the actions of all agents will affect the overall reward function, and the global reward function will guide the strategy optimization of each agent.

[0171] The strategy network obtains the size of the action value and sends the action to the photovoltaic equipment. The photovoltaic equipment adjusts its reactive power output, thereby realizing reactive power compensation, effectively suppressing grid voltage fluctuations, avoiding the occurrence of voltage over-limit phenomena, and achieving the purpose of controlling grid operation risks.

[0172] Example 2

[0173] A voltage over-limit control method based on multi-agent deep reinforcement learning, comprising:

[0174] Divide the power grid into several areas, each of which includes at least one photovoltaic device;

[0175] For the photovoltaic equipment in each area of ​​the power grid, a partially observable Markov decision process is used to model the photovoltaic equipment control strategy model;

[0176] The current operating parameters of the power grid in each area are input into the photovoltaic equipment control strategy model, and the photovoltaic equipment control strategy model is solved using a double-delay deep deterministic policy gradient algorithm to obtain the reactive power of each photovoltaic inverter and output it to the power grid in the corresponding area to control the voltage of each node within a preset range.

[0177] (1) The photovoltaic equipment in each area of ​​the power grid is modeled using a partially observable Markov decision process to obtain a specific method for the photovoltaic equipment control strategy model:

[0178] Due to the large scale of the power grid, in order to reduce the size of the state space and improve the efficiency of the solution, the present invention divides the power grid into several areas. At the same time, due to limitations such as communication delay and privacy protection, distributed photovoltaics can only observe information in local areas. Therefore, the voltage over-limit control problem is modeled as a partially observable Markov decision model. Partially Observable Markov Decision Process (POMDP) ​​is a framework for modeling problems of making decisions in an environment with incomplete information. POMDP is an extension of the Markov decision process (MDP), which assumes that the state of the system is only partially observed. In POMDP, the agent (or decision maker) cannot directly observe all the real states of the environment, but can only infer the state through a series of observations and make decisions based on the belief state of these possible states. Modeling incomplete information: POMDP can explicitly model situations where the agent cannot directly observe all the real states of the environment. This makes POMDP an ideal tool for dealing with complex dynamic environments with uncertainty and incomplete information. Since POMDP takes into account the uncertainty of observations, the agent needs to consider multiple possible states when making decisions, thereby enhancing the robustness of the decision. This robustness enables the agent to better adapt and respond to uncertainty and change. The goal of POMDP is to maximize long-term returns under incomplete information. By considering possible future states and corresponding rewards, the agent can develop a more long-term planning strategy to maximize long-term benefits. The above advantages make POMDP not dependent on precise parameters and very suitable for solving power grid problems.

[0179] First, we build a partially observable Markov decision process: {S,O,A,R,T 1 ,γ}, where S represents the state set of the power grid, O, A, and R represent the observation set, action set, and reward set of the photovoltaic equipment, respectively, and T 1 is the state transfer function, the state transfer function T 1 It represents the probability that the grid will transfer to the next state according to the current state and action. γ is the discount rate. The discount rate γ represents the attention paid by the photovoltaic equipment to future rewards. The larger the γ, the more attention the photovoltaic equipment pays to future rewards.

[0180] Then, the reactive power output by the photovoltaic inverter is used to solve the voltage over-limit control problem. The objective function F of the voltage over-limit control is:

[0181]

[0182] Where, T 总 is the total control period, M is the number of nodes in the power grid, Q PV,n,tis the reactive power of the nth PV inverter at time t, V m,t is the voltage over-limit value of node m at time t, δ is the additional penalty value for voltage over-limit, P loss,t is the line loss at time t, α is the network loss coefficient, and β is the node voltage over-limit penalty coefficient;

[0183] The representation of variables in the partially observable Markov decision process is as follows:

[0184] The state set of the power grid is expressed as follows:

[0185]

[0186] Where: S is the state set of the power grid at time t, P L,m,t and Q L,m,t are the active load and reactive load at node m at time t, respectively. If there is no load or photovoltaic device at node m, the corresponding power of the load or photovoltaic device is 0. t is the state of the power grid at time t, P PV,m,t is the active power of the photovoltaic inverter at node m at time t, Q PV,m,t is the reactive power of the photovoltaic inverter at node m at time t, V m,t and θ m,t are the voltage amplitude and angle at node m at time t, respectively, and T represents the matrix transpose;

[0187] The set of observations for a photovoltaic device is represented as follows:

[0188]

[0189] Where: O is the observation set of all photovoltaic devices at time t, o n,t Represents the vector composed of the active load and reactive load at node m in the region where the nth photovoltaic device is located and the active power and reactive power of the photovoltaic inverter at node m, n = 1, ..., N is the number of photovoltaic devices, and a single photovoltaic device can only observe the status of the nodes and the status of the photovoltaic inverter in the region to which the photovoltaic device belongs;

[0190] The action set of the photovoltaic device is expressed as follows:

[0191] A={a n,t |n=1,2,…,N}

[0192] Where: A is the action set of all photovoltaic devices at time t, a n,t is the set of all possible actions of the nth photovoltaic device at time t, action a n,t ∈A t And a n,t =Q PV,n,t , QPV,n,t is the reactive power of PV inverter n at time t;

[0193] The reward set is represented as follows:

[0194] r t =-αP loss,t -β(V m,t +δ)∈R

[0195] In the formula, r t is the reward function, V m,t The voltage of node m at time t exceeds the limit; each photovoltaic device is in a completely cooperative relationship, and the system voltage deviation and line loss are minimized through mutual coordination, so the rewards are shared among the intelligent agents.

[0196] State transition function T 1 :

[0197] State transition function T 1 Indicates that the photovoltaic device is in state s t Next, perform action a n,t The probability distribution of the state at the next moment after that, the state transition is defined as the photovoltaic equipment controlling the output of the inverter for reactive compensation according to the current environmental observation and its own strategy, so that the grid flow is redistributed, the node voltage and line loss change, the photovoltaic active power and load power fluctuate randomly, and thus transfer to the state at the next moment.

[0198] (2)Reference Figure 2 The specific method of inputting the current operation parameters of the power grid into the photovoltaic equipment control strategy model and using the double-delayed deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model is:

[0199] The twin delayed deep deterministic policy gradient algorithm (TD3) improves the Q-value overestimation problem in the deep deterministic policy gradient algorithm (DDPG). Based on the Actor-Critic framework, TD3 generates two evaluation values ​​by constructing two target Critic (value) networks, learns the two networks at the same time, and selects the Q-value using the minimization method; updates the target network and current network parameters by delaying the Actor (policy) network, and introduces noise to make the estimated value smoother.

[0200] The double-delayed deep deterministic policy gradient algorithm includes a policy network and a value network, which transforms the current operating parameters of the power grid: the current state of the power grid s t and observation o n,t, as the input of the policy network, the output is action a n,t , define action a n,t is the reactive power of PV inverter n at time t. Each PV device continuously interacts with the grid environment in its respective area as follows: PV inverter n performs action a n,t After that, the state changes from s t Transfer to t+1 , observed by o t Transfer to o t+1 And get the reward r under the current power grid environment t , the state transfer and reward are fed back to the photovoltaic device, and {s t ,o t ,a t ,r t ,s t+1 ,o t+1}; t ,o t ,a t ,r t ,s t+1 ,o t+1}Stored in the experience replay pool of the double-delayed deep deterministic policy gradient algorithm, and randomly extracting states, observations, actions, and rewards from the experience replay pool for training, updating the parameters of the policy network, and the policy network parameters are updated according to maximizing the cumulative expected reward;

[0201] The strategy network is used to output the action of the photovoltaic equipment according to the current state of the power grid. The strategy network is expressed as maximizing the strategy function J(π), and the formula is as follows:

[0202] J(π)=E S,o～Ξ [Q π (s t ,o t ,π(s t ,o t ;θ))]

[0203] In the formula, the maximization strategy function J(π) represents the expected reward that can be obtained, Ξ represents the distribution of strategy π under the current state and observation, and strategy π is defined as the photovoltaic device in state s t Next, perform action a t , E represents mathematical expectation, S, O~Ξ represents that the state and observation follow the distribution Ξ, Q π (s t ,o t ,π(s t ,o t ; θ)) is the value of executing strategy π under the current state and observation, and θ is the parameter of the policy network;

[0204] Use gradient descent to update the parameters θ of the policy function:

[0205]

[0206] In the formula, A = π(S) represents the photovoltaic device in state s t Next, perform action a t ;

[0207] The value network is used to estimate the value of the action predicted by the policy network. The value network is represented by the action value function Q(s t ,a t ), action value function Q(s t ,a t ) is expressed by the Bellman equation:

[0208] Q(s t ,a t )=E[r t +γQ(s t+1 ,a t+1 )]

[0209] The value network is updated based on minimizing the difference between the estimate and the target. The value network is parameterized using a neural network and approximates the action value function Q(s) by minimizing the Bellman residual. t ,a t ), the formula is as follows:

[0210] J Q (ω)=E[r t +γQ(s t+1 ,a t+1 |ω Q' )-Q(s t ,a t |ω Q )]

[0211] In the formula, J Q (ω) is the loss function used to minimize the Bellman residual, ω is the parameter of the value network, s t+1 is the state at time t+1, a t+1 is the action taken at time t+1, ω Q is the parameter of the current value network, ω Q' is the parameter of the target value network, and the value network output y at the next moment t It can be expressed as:

[0212]

[0213] In the formula, Refers to the parameters of the target value network, j = 1, 2, j = 1 Represents the parameters of the first target value network, when j = 2 represents the parameters of the second target value network;

[0214] In the interaction process between photovoltaic equipment and power grid, the parameters ω of the current value network are updated by minimizing the loss function Q , the loss function L(ω) is defined as:

[0215]

[0216] In the formula, K is the number of training samples, y' t is the value of the target value network. In the first target value network and the second target value network, the smaller target value network value is selected as the value network output y at the next moment. t , the output is expressed as:

[0217]

[0218] Then use the target action to calculate the value of the target value network to smooth and regularize the target strategy. t+1 The actions are:

[0219] a t+1 =π(s t+1 )+ε

[0220] In the formula, ε is random noise, which is normally distributed, and the sampled noise is truncated ε~clip(N(0,σ),-c,c), σ is the variance, c is the upper and lower limits of the noise, and N represents normal distribution;

[0221] The target strategy network and target value network parameters are updated in the form of soft updates, and the formula is as follows:

[0222]

[0223] θ π' =τθ π +(1-τ)θ π'

[0224] In the formula, τ is the update step size, is the parameter of the current value network, j = 1, 2, j = 1 Represents the parameters of the first current value network, when j = 2 represents the parameters of the second current value network, is the parameter of the target value network, θ π is the parameter of the current policy network, θ π' are the parameters of the target policy network;

[0225] According to the above steps, the action a of each photovoltaic inverter that maximizes the cumulative expected reward J(π) is solved. t , as a control strategy for photovoltaic equipment.

[0226] In the present invention, each agent is in a fully cooperative relationship, and each agent minimizes the objective function through mutual coordination. Although the actions of each agent only act on its own area, each agent shares a global reward function, which includes voltage limit crossing and line loss. Therefore, the actions of all agents will affect the overall reward function, and the global reward function will guide the strategy optimization of each agent.

[0227] The strategy network obtains the size of the action value and sends the action to the photovoltaic equipment. The photovoltaic equipment adjusts its reactive power output, thereby realizing reactive power compensation, effectively suppressing grid voltage fluctuations, avoiding the occurrence of voltage over-limit phenomena, and achieving the purpose of controlling grid operation risks.

[0228] Example 3

[0229] An electronic device, characterized in that it includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the voltage over-limit control method based on multi-agent deep reinforcement learning in Example 2 by executing the computer instructions.

[0230] Example 4

[0231] A computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the voltage over-limit control method based on multi-agent deep reinforcement learning or embodiment 2 is implemented.

[0232] The contents not described in detail in this specification belong to the prior art known to those skilled in the art. It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0233] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0234] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0235] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0236] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit its protection scope. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that after reading the present invention, those skilled in the art can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the pending claims of the invention.

Claims

1. A voltage over-limit control system based on multi-agent deep reinforcement learning, characterized in that: include: A power grid area division module, used to divide the power grid into several areas, each area includes at least one photovoltaic device; The photovoltaic equipment control strategy model building module is used to model the photovoltaic equipment in each area of ​​the power grid using a partially observable Markov decision process to obtain a photovoltaic equipment control strategy model; The photovoltaic equipment control strategy output module is used to input the current operating parameters of the power grid in each area into the photovoltaic equipment control strategy model, and use the double-delay deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model, obtain the reactive power of each photovoltaic inverter and output it to the power grid in the corresponding area to control the voltage of each node within a preset range.

2. The voltage over-limit control system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In the photovoltaic equipment control strategy model building module, for photovoltaic equipment in each area of ​​the power grid, a partially observable Markov decision process is used to model the photovoltaic equipment. The specific method for obtaining the photovoltaic equipment control strategy model is as follows: First, we build a partially observable Markov decision process: {S,O,A,R,T 1 ,γ}, where S represents the state set of the power grid, O, A, and R represent the observation set, action set, and reward set of the photovoltaic equipment, respectively, and T 1 is the state transfer function, the state transfer function T 1 It represents the probability that the power grid transfers to the next state according to the current state and action. γ is the discount rate. The discount rate γ represents the attention paid by the photovoltaic equipment to future rewards. The larger the γ, the more attention the photovoltaic equipment pays to future rewards. Then, the reactive power output by the photovoltaic inverter is used to solve the voltage over-limit control problem. The objective function F of the voltage over-limit control is: Where, T 总 is the total control period, M is the number of nodes in the power grid, Q PV,n,t is the reactive power of the nth PV inverter at time t, V m,t is the voltage over-limit value of node m at time t, δ is the additional penalty value for voltage over-limit, P loss,t is the line loss at time t, α is the network loss coefficient, and β is the node voltage over-limit penalty coefficient.

3. The voltage over-limit control system based on multi-agent deep reinforcement learning according to claim 2 is characterized in that: The representation of variables in the partially observable Markov decision process is as follows: The state set of the power grid is expressed as follows: Where: S is the state set of the power grid at time t, P L,m,t and Q L,m,t are the active load and reactive load at node m at time t, respectively. If there is no load or photovoltaic device at node m, the corresponding power of the load or photovoltaic device is 0. t is the state of the power grid at time t, P PV,m,t is the active power of the photovoltaic inverter at node m at time t, Q PV,m,t is the reactive power of the photovoltaic inverter at node m at time t, V m,t and θ m,t are the voltage amplitude and angle at node m at time t, respectively, and T represents the matrix transpose; The set of observations for a photovoltaic device is represented as follows: Where: O is the observation set of all photovoltaic devices at time t, o n,t Represents the vector composed of the active load and reactive load at node m in the region where the nth photovoltaic device is located and the active power and reactive power of the photovoltaic inverter at node m, n = 1, ..., N is the number of photovoltaic devices, and a single photovoltaic device can only observe the status of the nodes and the status of the photovoltaic inverter in the region to which the photovoltaic device belongs; The action set of the photovoltaic device is expressed as follows: A={a n,t |n=1,2,…,N} Where: A is the action set of all photovoltaic devices at time t, a n,t is the set of all possible actions of the nth photovoltaic device at time t, action a n,t ∈A t And a n,t =Q PV,n,t , Q PV,n,t is the reactive power of PV inverter n at time t; The reward set is represented as follows: r t =-αP loss,t -β(V m,t +δ)∈R In the formula, r t is the reward function, V m,t is the voltage exceeding the limit of node m at time t; State transition function T 1 : State transition function T 1 Indicates that the photovoltaic device is in state s t Next, perform action a n,t The probability distribution of the state at the next moment after that, the state transition is defined as the photovoltaic equipment controlling the output of the inverter for reactive compensation according to the current environmental observation and its own strategy, so that the grid flow is redistributed, the node voltage and line loss change, the photovoltaic active power and load power fluctuate randomly, and thus transfer to the state at the next moment.

4. The voltage over-limit control system based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: The photovoltaic equipment control strategy output module inputs the current operation parameters of the power grid into the photovoltaic equipment control strategy model, and uses the double-delayed deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model. The specific method is: The double-delayed deep deterministic policy gradient algorithm includes a policy network and a value network, which transforms the current operating parameters of the power grid: the current state of the power grid s t and observation o n,t , as the input of the policy network, the output is action a n,t , define action a n,t is the reactive power of PV inverter n at time t. Each PV device continuously interacts with the grid environment in its respective area as follows: PV inverter n performs action a n,t After that, the state changes from s t Transfer to t+1 , observed by o t Transfer to o t+1 And get the reward r under the current power grid environment t , the state transfer and reward are fed back to the photovoltaic device, and {s t ,o t ,a t ,r t ,s t+1 ,o t+1 }; t ,o t ,a t ,r t ,s t+1 ,o t+1 }Stored in the experience replay pool of the double-delayed deep deterministic policy gradient algorithm, and randomly extracting states, observations, actions, and rewards from the experience replay pool for training, updating the parameters of the policy network, and the policy network parameters are updated according to maximizing the cumulative expected reward; The strategy network is used to output the action of the photovoltaic equipment according to the current state of the power grid. The strategy network is expressed as maximizing the strategy function J(π), and the formula is as follows: J(π)=E S,O～Ξ [Q π (s t ,o t ,π(s t ,o t (i))] In the formula, the maximization strategy function J(π) represents the expected reward that can be obtained, Ξ represents the distribution of strategy π under the current state and observation, and strategy π is defined as the photovoltaic device in state s t Next, perform action a t , E represents mathematical expectation, S, O~Ξ represents that the state and observation follow the distribution Ξ, Q π (s t ,o t ,π(s t ,o t ; θ)) is the value of executing strategy π under the current state and observation, and θ is the parameter of the policy network; Use gradient descent to update the parameters θ of the policy function: In the formula, A = π(S) represents the photovoltaic device in state s t Next, perform action a t ; The value network is used to estimate the value of the action predicted by the policy network. The value network is represented by the action value function Q(s t ,a t ), action value function Q(s t ,a t ) is expressed by the Bellman equation: Q(s t ,a t )=E[r t +γQ(s t+1 ,a t+1 )] The value network is updated based on minimizing the difference between the estimate and the target. The value network is parameterized using a neural network and approximates the action value function Q(s) by minimizing the Bellman residual. t ,a t ), the formula is as follows: J Q (ω)=E[r t +γQ(s t+1 ,a t+1 |ω Q’ )-Q(s t ,a t |ω Q )] In the formula, J Q (ω) is the loss function used to minimize the Bellman residual, ω is the parameter of the value network, s t+1 is the state at time t+1, a t+1 is the action taken at time t+1, ω Q is the parameter of the current value network, ω Q’ is the parameter of the target value network, and the value network output y at the next moment t It can be expressed as: In the formula, Refers to the parameters of the target value network, j = 1, 2, j = 1 Represents the parameters of the first target value network, when j = 2 represents the parameters of the second target value network; In the interaction process between photovoltaic equipment and power grid, the parameters ω of the current value network are updated by minimizing the loss function Q , the loss function L(ω) is defined as: In the formula, K is the number of training samples, y' t is the value of the target value network. In the first target value network and the second target value network, the smaller target value network value is selected as the value network output y at the next moment. t , the output is expressed as: Then use the target action to calculate the value of the target value network to smooth and regularize the target strategy. t+1 The actions are: a t+1 =π(s t+1 )+e In the formula, ε is random noise, which is normally distributed, and the sampled noise is truncated ε~clip(N(0,σ),-c,c), σ is the variance, c is the upper and lower limits of the noise, and N represents normal distribution; The target strategy network and target value network parameters are updated in the form of soft updates, and the formula is as follows: i π’ =tθ π +(1-τ)θ π’ In the formula, τ is the update step size, is the parameter of the current value network, j = 1, 2, j = 1 Represents the parameters of the first current value network, when j = 2 represents the parameters of the second current value network, is the parameter of the target value network, θ π is the parameter of the current policy network, θ π’ are the parameters of the target policy network; According to the above steps, the action a of each photovoltaic inverter that maximizes the cumulative expected reward J(π) is solved. t , as a control strategy for photovoltaic equipment.

5. A voltage over-limit control method based on multi-agent deep reinforcement learning, characterized in that: The following steps are involved: Divide the power grid into several areas, each of which includes at least one photovoltaic device; For the photovoltaic equipment in each area of ​​the power grid, a partially observable Markov decision process is used to model the photovoltaic equipment control strategy model; The current operating parameters of the power grid in each area are input into the photovoltaic equipment control strategy model, and the photovoltaic equipment control strategy model is solved using a double-delay deep deterministic policy gradient algorithm to obtain the reactive power of each photovoltaic inverter and output it to the power grid in the corresponding area to control the voltage of each node within a preset range.

6. The voltage over-limit control method based on multi-agent deep reinforcement learning according to claim 5 is characterized in that: The photovoltaic equipment in each area of ​​the power grid is modeled using a partially observable Markov decision process to obtain a specific method for the photovoltaic equipment control strategy model: First, we build a partially observable Markov decision process: {S,O,A,R,T 1 ,γ}, where S represents the state set of the power grid, O, A, and R represent the observation set, action set, and reward set of the photovoltaic equipment, respectively, and T 1 is the state transfer function, the state transfer function T 1 It represents the probability that the power grid transfers to the next state according to the current state and action. γ is the discount rate. The discount rate γ represents the attention paid by the photovoltaic equipment to future rewards. The larger the γ, the more attention the photovoltaic equipment pays to future rewards. Then, the reactive power output by the photovoltaic inverter is used to solve the voltage over-limit control problem. The objective function F of the voltage over-limit control is: Where, T 总 is the total control period, M is the number of nodes in the power grid, Q PV,n,t is the reactive power of the nth PV inverter at time t, V m,t is the voltage over-limit value of node m at time t, δ is the additional penalty value for voltage over-limit, P loss,t is the line loss at time t, α is the network loss coefficient, and β is the node voltage over-limit penalty coefficient.

7. The voltage over-limit control method based on multi-agent deep reinforcement learning according to claim 6 is characterized in that: The representation of variables in the partially observable Markov decision process is as follows: The state set of the power grid is expressed as follows: Where: S is the state set of the power grid at time t, P L,m,t and Q L,m,t are the active load and reactive load at node m at time t, respectively. If there is no load or photovoltaic device at node m, the corresponding power of the load or photovoltaic device is 0. t is the state of the power grid at time t, P PV,m,t is the active power of the photovoltaic inverter at node m at time t, Q PV,m,t is the reactive power of the photovoltaic inverter at node m at time t, V m,t and θ m,t are the voltage amplitude and angle at node m at time t, respectively, and T represents the matrix transpose; The set of observations for a photovoltaic device is represented as follows: Where: O is the observation set of all photovoltaic devices at time t, o n,t Represents the vector composed of the active load and reactive load at node m in the region where the nth photovoltaic device is located and the active power and reactive power of the photovoltaic inverter at node m, n = 1, ..., N is the number of photovoltaic devices, and a single photovoltaic device can only observe the status of the nodes and the status of the photovoltaic inverter in the region to which the photovoltaic device belongs; The action set of the photovoltaic device is expressed as follows: A={a n,t |n=1,2,…,N} Where: A is the action set of all photovoltaic devices at time t, a n,t is the set of all possible actions of the nth photovoltaic device at time t, action a n,t ∈A t And a n,t =Q PV,n,t , Q PV,n,t is the reactive power of PV inverter n at time t; The reward set is represented as follows: r t =-αP loss,t -β(V m,t +δ)∈R In the formula, r t is the reward function, V m,t is the voltage exceeding the limit of node m at time t; State transition function T 1 : State transition function T 1 Indicates that the photovoltaic device is in state s t Next, perform action a n,t The probability distribution of the state at the next moment after that, the state transition is defined as the photovoltaic equipment controlling the output of the inverter for reactive compensation according to the current environmental observation and its own strategy, so that the grid flow is redistributed, the node voltage and line loss change, the photovoltaic active power and load power fluctuate randomly, and thus transfer to the state at the next moment.

8. The voltage over-limit control method based on multi-agent deep reinforcement learning according to claim 7 is characterized in that: The specific method of inputting the current operation parameters of the power grid into the photovoltaic equipment control strategy model and using the double-delayed deep deterministic policy gradient algorithm to solve the photovoltaic equipment control strategy model is: The double-delayed deep deterministic policy gradient algorithm includes a policy network and a value network, which transforms the current operating parameters of the power grid: the current state of the power grid s t and observation o n,t , as the input of the policy network, the output is action a n,t , define action a n,t is the reactive power of PV inverter n at time t. Each PV device continuously interacts with the grid environment in its respective area as follows: PV inverter n performs action a n,t After that, the state changes from s t Transfer to t+1 , observed by o t Transfer to o t+1 And get the reward r under the current power grid environment t , the state transfer and reward are fed back to the photovoltaic device, and {s t ,o t ,a t ,r t ,s t+1 ,o t+1 }; t ,o t ,a t ,r t ,s t+1 ,o t+1 }Stored in the experience replay pool of the double-delayed deep deterministic policy gradient algorithm, and randomly extracting states, observations, actions, and rewards from the experience replay pool for training, updating the parameters of the policy network, and the policy network parameters are updated according to maximizing the cumulative expected reward; The strategy network is used to output the action of the photovoltaic equipment according to the current state of the power grid. The strategy network is expressed as maximizing the strategy function J(π), and the formula is as follows: J(π)=E S,O～Ξ [Q π (s t ,o t ,π(s t ,o t (i))] In the formula, the maximization strategy function J(π) represents the expected reward that can be obtained, Ξ represents the distribution of strategy π under the current state and observation, and strategy π is defined as the photovoltaic device in state s t Next, perform action a t , E represents mathematical expectation, S, O~Ξ represents that the state and observation follow the distribution Ξ, Q π (s t ,o t ,π(s t ,o t ; θ)) is the value of executing strategy π under the current state and observation, and θ is the parameter of the policy network; Use gradient descent to update the parameters θ of the policy function: In the formula, A = π(S) represents the photovoltaic device in state s t Next, perform action a t ; The value network is used to estimate the value of the action predicted by the policy network. The value network is represented by the action value function Q(s t ,a t ), action value function Q(s t ,a t ) is expressed by the Bellman equation: Q(s t ,a t )=E[r t +γQ(s t+1 ,a t+1 )] The value network is updated based on minimizing the difference between the estimate and the target. The value network is parameterized using a neural network and approximates the action value function Q(s) by minimizing the Bellman residual. t ,a t ), the formula is as follows: J Q (ω)=E[r t +γQ(s t+1 ,a t+1 |ω Q’ )-Q(s t ,a t |ω Q )] In the formula, J Q (ω) is the loss function used to minimize the Bellman residual, ω is the parameter of the value network, s t+1 is the state at time t+1, a t+1 is the action taken at time t+1, ω Q is the parameter of the current value network, ω Q’ is the parameter of the target value network, and the value network output y at the next moment t It can be expressed as: In the formula, Refers to the parameters of the target value network, j = 1, 2, j = 1 Represents the parameters of the first target value network, when j = 2 represents the parameters of the second target value network; In the interaction process between photovoltaic equipment and power grid, the parameters ω of the current value network are updated by minimizing the loss function Q , the loss function L(ω) is defined as: In the formula, K is the number of training samples, y' t is the value of the target value network. In the first target value network and the second target value network, the smaller target value network value is selected as the value network output y at the next moment. t , the output is expressed as: Then use the target action to calculate the value of the target value network to smooth and regularize the target strategy. t+1 The actions are: a t+1 =π(s t+1 )+e In the formula, ε is random noise, which is normally distributed, and the sampled noise is truncated ε~clip(N(0,σ),-c,c), σ is the variance, c is the upper and lower limits of the noise, and N represents normal distribution; The target strategy network and target value network parameters are updated in the form of soft updates, and the formula is as follows: i π’ =tθ π +(1-τ)θ π’ In the formula, τ is the update step size, is the parameter of the current value network, j = 1, 2, j = 1 Represents the parameters of the first current value network, when j = 2 represents the parameters of the second current value network, is the parameter of the target value network, θ π is the parameter of the current policy network, θ π’ are the parameters of the target policy network; According to the above steps, the action a of each photovoltaic inverter that maximizes the cumulative expected reward J(π) is solved. t , as a control strategy for photovoltaic equipment.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the voltage over-limit control method based on multi-agent deep reinforcement learning as described in any one of claims 5 to 8 by executing the computer instructions.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the voltage over-limit control method based on multi-agent deep reinforcement learning as described in any one of claims 5-8 is implemented.

Images