A distribution network voltage control method and system
By abstracting the energy storage device, photovoltaic inverter and microgrid as nodes, building a Markov game model and using reinforcement learning algorithm to train the agent, the problem of traditional voltage regulation devices not responding to fast voltage fluctuations in time is solved, and the problem of traditional voltage regulation devices not responding to rapid voltage fluctuations is realized. Stable control of distribution network voltage and comprehensiveness of information utilization.
Patent Information
- Application Number
- CN202411162083.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-23
AI Technical Summary
In the prior art, traditional voltage regulating devices respond slowly to fast voltage fluctuations, have high capital costs, and limited discrete operations, so they cannot effectively deal with voltage fluctuations in the distribution network, and incomplete information utilization between nodes, resulting in limited voltage regulation capabilities and ineffective Maintain the stability of the power grid.
Abstract the energy storage device, photovoltaic inverter and microgrid as nodes, build a voltage control mathematical model based on the distribution network structure, transform it into a Markov game model, train the agent through reinforcement learning algorithm, calculate the virtual electrical distance, and use Topology diagram and multi-agents perform voltage regulation control.
Through the integration and management of various energy sources, we can effectively provide voltage regulation services for the distribution network, improve voltage quality, reduce network losses, and achieve stable control of the power grid voltage.
Smart Images

Figure CN118693836B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power distribution networks, and in particular to a power distribution network voltage control method and system. Background Art
[0002] With the increasing popularity of distributed renewable energy, the voltage of the distribution network fluctuates. The distribution network faces more frequent voltage violations and higher losses. These problems are mainly caused by the intermittent characteristics of distributed renewable energy. Existing technologies often use reinforcement learning to control controllable active and reactive resources in the distribution network, such as energy storage systems and reactive compensators. These technologies have a certain effect on alleviating voltage fluctuations. As in the prior art, the invention patent with the patent publication number CN113363998A discloses a distribution network voltage control method based on multi-agent deep reinforcement learning, which involves the fields of electrical engineering technology and computer science technology, including: step 10) constructing a distribution network multi-time scale voltage control model for multiple voltage regulating devices such as capacitor banks, photovoltaic inverters, and energy storage devices; step 20) assigning control variables to multiple agents, and converting the voltage control problem into a Markov decision process; step 30) using a multi-agent deep reinforcement learning algorithm based on multi-agent deep deterministic policy gradient to solve the MDP process, wherein the algorithm is improved for the characteristics of discrete voltage regulating devices; step 40) training and executing multi-agents to implement a multi-time scale voltage control method. Prior art devices contain traditional voltage regulating devices, such as capacitor banks, which are largely limited in their voltage regulation capabilities due to their slow response to rapid voltage fluctuations, high capital costs, limited number of discrete actions, independent control methods, and inability to cope with rapid changes.
[0003] In the prior art, the invention patent with the patent publication number CN117913842A discloses a distribution network voltage control method based on multi-agent deep reinforcement learning, which models the voltage control problem as a distributed partially observable Markov decision process, introduces a self-attention encoder, and improves the strategy network and value network. j,t After linear projection, it is sent to L A In the masked self-attention encoder, the encoder mask is the adjacency matrix D corresponding to the region where agent j is located. j , thereby introducing topological information and the network node output of the distribution network, only for the feature area weights of adjacent nodes, but not for non-adjacent nodes. In the prior art, the adjacency matrix only determines whether the nodes are directly connected, either connected or disconnected, which may cause each node to not fully utilize the information of other nodes, so the trained voltage control model is not accurate and cannot effectively maintain the voltage stability of the power grid. Summary of the invention
[0004] The technical problem to be solved by the present invention is to solve the problem that the conventional technology uses a traditional voltage regulating device, the voltage regulating capability is limited, and the information utilization rate of each node on the distribution network is incomplete.
[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0006] A distribution network voltage control method, defined as follows: a plurality of voltage control devices are provided in the distribution network, including an energy storage device ES, a photovoltaic inverter PV and a microgrid MG; the method comprises the following steps:
[0007] S10, constructing a voltage control mathematical model based on the distribution network structure, including mathematical models of the energy storage device ES, the photovoltaic inverter PV and the microgrid MG;
[0008] S20, taking each voltage control device as an intelligent agent, assigning each variable to the intelligent agent, so that the voltage control mathematical model is transformed into a Markov game model;
[0009] S30, abstract each energy storage device ES, photovoltaic inverter PV and microgrid MG in the distribution network as a node, abstract the connection line between each two nodes as an edge, and obtain the voltage control graph; based on the voltage control graph and the reinforcement learning algorithm, continue to train all agents under the Markov game model, and obtain the optimal agent, so as to transmit the output actions of the optimal energy storage device ES agent, the optimal photovoltaic inverter PV agent, and the optimal microgrid MG agent to the energy storage device ES, the photovoltaic inverter PV, and the microgrid MG accordingly.
[0010] In one embodiment of the present invention, the energy storage device ES mathematical model is constructed by the following formula:
[0011] ;
[0012] In the formula, express Moment bus The real number control variable of the energy storage device ES; express Moment bus The charging and discharging power of the energy storage device ES; express Moment bus The maximum charge and discharge power of the energy storage device ES; and Respectively Moment bus Reactive power and rated capacity of the energy storage device ES; express Moment bus The energy storage device ES is in operation state. express +1 Moment Bus The energy storage device ES is in operating state; and Respectively represent busbar The minimum and maximum values of the operating state of the energy storage device ES; and They represent the charging efficiency and discharging efficiency of the energy storage device ES respectively; Indicates the electric capacity of the energy storage device ES; express +1 moment with The interval between moments.
[0013] In one embodiment of the present invention, a photovoltaic inverter PV mathematical model is constructed by the following formula:
[0014] ;
[0015] In the formula, express Moment bus The real control variable of the photovoltaic inverter PV; express Moment bus The rated capacity of the photovoltaic inverter PV; express Moment bus The active output of the photovoltaic inverter PV; express Moment bus The maximum reactive power output of the photovoltaic inverter PV; express Moment bus The reactive output of the photovoltaic inverter PV injected into the distribution grid.
[0016] In one embodiment of the present invention, the mathematical model of the microgrid MG is constructed by the following formula:
[0017] ;
[0018] In the formula, express Moment bus is the real number control variable of the energy storage device ES inside the microgrid MG; express Moment bus is the real number control variable of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The maximum reactive power of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The maximum charging and discharging power of the energy storage device ES inside the microgrid MG; and Respectively Moment bus The active power and reactive power of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The charging and discharging power of the energy storage device ES inside the microgrid MG; express Moment bus The reactive power of the energy storage device ES inside the microgrid MG; and Respectively Moment bus Internal load of microgrid MG Active power and reactive power; and Respectively Moment bus The active power and reactive power exchanged between the microgrid MG and the distribution network; and Respectively Moment bus The net injected active power and reactive power at and Respectively Moment bus The minimum active power and maximum active power exchanged between the microgrid MG and the distribution network; and Respectively Moment bus The minimum reactive power and maximum reactive power exchanged between the microgrid MG and the distribution network; express Moment bus The rated capacity of the microgrid MG.
[0019] In one embodiment of the present invention, a mathematical model of the power flow of each branch in the distribution network is established, and all voltage control mathematical models are subject to rule constraints; wherein the mathematical model of the power flow of each branch is constructed by the following formula:
[0020] ;
[0021] In the formula, and Respectively Moment bus Load Active and reactive power; Indicates busbar To busbar The resistance of the branch between the points; Indicates busbar To busbar The reactance of the branch between the points; express Moment bus The voltage at express Moment bus The voltage at and Indicates busbar The upper and lower limits of the allowable voltage range; express Moment bus To busbar The current in the branch between the points; Indicates busbar The set of all buses with other buses as the first endpoints corresponding to the branch with the end point at Busbar Set A first terminal bus in; and Respectively Time first end bus To busbar The active power and reactive power of the branches between the locations; Indicates busbar The set of all buses corresponding to the branch with the first endpoint and other buses as the end points, Busbar Set One end point of the generatrix; and Respectively Moment bus To the end point busbar The active power and reactive power of the branches between the locations; express Moment bus The active output of the photovoltaic inverter PV; express Moment bus The charging and discharging power of the energy storage device ES; and Respectively Moment bus The active power and reactive power exchanged between the microgrid MG and the distribution network; express Moment bus The reactive output of the photovoltaic inverter PV injected into the distribution grid.
[0022] In one embodiment of the present invention, the voltage control mathematical model is transformed into a Markov game model, comprising the following steps:
[0023] S21, establishing the action space and state space of each energy storage device ES agent, photovoltaic inverter PV agent and microgrid MG agent;
[0024] S22, Establishment Rewards for the ES agent of the moment energy storage device , Rewards for PV Agents 、Rewards for microgrid MG intelligent body ;
[0025] S23, obtaining the maximum discounted return of each agent according to the action space, state space and reward of each agent;
[0026] S24, all the agent's action space, state space, and reward , Maximizing the discounted return combination to form a Markov game model ;in represents the total number of all agents, Represents the global state, Indicates The state of an agent, Indicates The actions of an agent, Indicates The reward for an agent, represents the state transition function, Represents the reward discount factor.
[0027] In one embodiment of the present invention, Rewards for the ES agent of the moment energy storage device , Rewards for PV Agents 、Rewards for microgrid MG intelligent body for:
[0028] ;
[0029] In the formula, express Moment bus To busbar The loss of the branch between the locations; express Total loss of the branch at timeLoss ; represents the loss coefficient of the branch; express The voltage exceeds the limit at the moment; Indicates the voltage over-limit coefficient; represents a busbar set; Indicates the operation of taking a larger value; and They represent the prices of buying and selling active power in the distribution network respectively; and They represent the prices of buying and selling reactive power in the distribution network respectively; Indicates that the active power of the microgrid flows to the distribution network; Indicates that the active power of the distribution network flows to the microgrid; It indicates that the reactive power of the microgrid flows to the distribution network; Indicates that the reactive power of the distribution network flows to the microgrid; represents the weight factor; Indicates busbar To busbar The resistance of the branch between the points; express Moment bus To busbar The current in the branch between the points; express Moment bus The voltage at and Indicates busbar The upper and lower limits of the allowable voltage range; Represents the cost of buying and selling active power and reactive power in the distribution network.
[0030] In one embodiment of the present invention, the maximum discounted reward of each agent is:
[0031] ;
[0032] In the formula, Indicates The strategy of an agent, are the parameters of the strategy; express The reward discount factor at the moment; express Moment Rewards for each agent; express Moment The actions of an agent; express Moment The state of an agent, express Moment The state of an agent; Represents the state transition function of each agent; express expectations; Indicated in strategy The maximum value under Indicates the total number of steps.
[0033] In one embodiment of the present invention, obtaining the optimal agent includes the following steps:
[0034] S31, according to the line connection relationship of each node in the voltage control diagram, obtain the first The node and Voltage interaction coefficient of each node ; Based on the voltage interaction coefficient , get the The node and Virtual electrical distance of nodes , the connection relationship of multiple voltage control devices in the distribution network is represented by virtual electrical distance Indicates that we can get the adjacency matrix D ;
[0035] S32, establish Actor network of agents and the target actor network ; and are the parameters of the actor network and the target actor network respectively; Indicates The actions of an agent; Indicates The state of an agent;
[0036] S33, for Actor network of agents Add Gaussian-noise to the output ;
[0037] S34, set up the attention network, combined with the adjacency matrix D, Establish the A critic network with two agents and target critic network ; and are the parameters of the critic network and the target critic network respectively; and Represent the global state and joint action respectively;
[0038] S35, from The experience pool of an agent Randomly select strip , for A critic network with two agents and actor networks To update; among them, Indicates The reward for an agent, Indicates Agents take A The next state after the action;
[0039] S36, using loss function For A critic network with two agents Make updates;
[0040] S37, using the gradient descent algorithm Actor network of agents Update the parameters of
[0041] S38, using reinforcement learning soft update to the target actor network and target critic network Update the parameters of
[0042] S39, repeat the process from step S35 to step S38, Parameters of the actor network of agents and the parameters of the critic network Perform iterative training to obtain The optimal parameters of an agent The corresponding optimal actor network and optimal parameters The corresponding optimal critic network constitutes the The optimal agent is obtained, thus all the optimal agents are obtained.
[0043] In one embodiment of the present invention, The node and Voltage interaction coefficient of each node , obtained by the following formula:
[0044] ;
[0045] In the formula, Indicates The node voltage is affected by Sensitivity coefficient of active power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of reactive power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of active power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of reactive power injection of each node;
[0046] Get the The node and Virtual electrical distance of nodes Obtained by the following formula:
[0047] ;
[0048] In the formula, Indicates The nodes and Voltage interaction metric coefficient of the node.
[0049] In one embodiment of the present invention, the Agent-based critic network , through the following formula:
[0050] ;
[0051] In the formula, Represents matrix concatenation; express and The concatenated matrix; The weight matrix representing the input features; Represents the characteristic matrix after linear transformation; Represents the new feature matrix; , Respectively represent nodes and The characteristic matrix after linear transformation; Represents the weight vector for calculating attention; , Respectively represent nodes For Node and nodes The attention coefficient; Represents the normalized node For Node The attention coefficient; Indicates The node where the agent is located The set of neighbor nodes of Indicates A fully connected layer of the agent-critic network; Indicates The attention network in the agent critic network; and Represent two different activation functions respectively; Represents an exponential function.
[0052] The present invention also provides a system using the above-mentioned distribution network voltage control method, which is defined as follows: a plurality of voltage control devices are provided in the distribution network, including an energy storage device ES, a photovoltaic inverter PV and a microgrid MG; including:
[0053] The voltage control device module builds a voltage control mathematical model based on the distribution network structure, including the mathematical models of the energy storage device ES, the photovoltaic inverter PV and the microgrid MG;
[0054] The Markov game model module treats each voltage control device as an intelligent agent, assigns each variable to the intelligent agent, and transforms the voltage control mathematical model into a Markov game model;
[0055] The voltage control module abstracts each energy storage device ES, photovoltaic inverter PV and microgrid MG in the distribution network as a node, and abstracts the connection line between each two nodes as an edge to obtain the voltage control graph; based on the voltage control graph and the reinforcement learning algorithm, all agents under the Markov game model are continuously trained to obtain the optimal agent, so that the output actions of the optimal energy storage device ES agent, the optimal photovoltaic inverter PV agent, and the optimal microgrid MG agent are correspondingly transmitted to the energy storage device ES, the photovoltaic inverter PV, and the microgrid MG.
[0056] Compared with the prior art, the beneficial effects of the present invention are: using the microgrid as a voltage regulating device, effectively providing voltage regulation services for the distribution network by integrating and managing various energy sources, calculating the virtual electrical distance between each node, using the virtual electrical distance to form an adjacency matrix, and then using topology graphs, multi-agents, and reinforcement learning to control the voltage regulating device, thereby improving voltage quality and reducing network losses, and effectively maintaining the voltage stability of the power grid. Each energy storage device ES, photovoltaic inverter PV, and microgrid MG in the distribution network are abstracted as nodes, and the adjacency matrix is used to establish connections between all nodes, and all node information on the distribution network is applied to perform voltage stabilization and regulation from the macro perspective of the entire distribution network node.
[0057] In order to solve the voltage fluctuation and voltage instability problems caused by the popularization of new energy systems in distribution networks, the present invention uses microgrids as a voltage regulation device, which integrates and manages various energy sources inside the microgrid to effectively provide voltage regulation services for the distribution network.
[0058] The present invention transforms the mathematical model of distribution network voltage control into a Markov game model, abstracts each device in the distribution network into a topological node, constructs a graph, calculates the virtual electrical distance between each node, uses the virtual electrical distance to form an adjacency matrix, and then uses the topological graph, multi-agent, and reinforcement learning to control the voltage regulating device to achieve control of the distribution network voltage. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 The present invention is a flow chart of a method for controlling voltage in a power distribution network according to an embodiment of the present invention.
[0060] Figure 2 This is a schematic diagram of a 33-node distribution network according to an embodiment of the present invention.
[0061] Figure 3 The present invention is a block diagram of a distribution network voltage control system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0062] In order to facilitate those skilled in the art to understand the technical solution of the present invention, the technical solution of the present invention is further described in conjunction with the accompanying drawings of the specification.
[0063] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0064] See also Figure 1 As shown, the present invention provides a distribution network voltage control method, which first defines: a plurality of voltage control devices are provided in the distribution network, including an energy storage device ES, a photovoltaic inverter PV and a microgrid MG. Specifically, the method includes the following steps:
[0065] S10, constructing a voltage control mathematical model based on the distribution network structure, including mathematical models of the energy storage device ES, the photovoltaic inverter PV and the microgrid MG.
[0066] See also Figure 2 As shown, in one embodiment of the present invention, a voltage control mathematical model based on the distribution network structure is constructed for a 33-node distribution network. The 33-node distribution network is only one embodiment of the present invention, and specifically also includes distribution networks with other number of nodes.
[0067] In this embodiment, the energy storage device ES mathematical model is constructed by the following formula:
[0068] ;
[0069] In the formula, express Moment bus The real number control variable of the energy storage device ES; express Moment bus The charging and discharging power of the energy storage device ES; express Moment bus The maximum charge and discharge power of the energy storage device ES; and Respectively Moment bus Reactive power and rated capacity of the energy storage device ES; express Moment bus The energy storage device ES is in operation state. express +1 Moment Bus The energy storage device ES is in operating state; and Respectively represent busbar The minimum and maximum values of the operating state of the energy storage device ES; and They represent the charging efficiency and discharging efficiency of the energy storage device ES respectively; Indicates the electric capacity of the energy storage device ES; express +1 moment with The interval between moments.
[0070] The photovoltaic inverter PV mathematical model is constructed through the following formula:
[0071] ;
[0072] In the formula, express Moment bus The real number control variable of the photovoltaic inverter PV; express Moment bus The rated capacity of the photovoltaic inverter PV; express Moment bus The active output of the photovoltaic inverter PV; express Moment bus The maximum reactive power output of the photovoltaic inverter PV; express Moment bus The reactive output of the photovoltaic inverter PV injected into the distribution grid.
[0073] The mathematical model of the microgrid MG is constructed by the following formula:
[0074] ;
[0075] In the formula, express Moment bus is the real number control variable of the energy storage device ES inside the microgrid MG; express Moment bus is the real number control variable of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The maximum reactive power of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The maximum charging and discharging power of the energy storage device ES inside the microgrid MG; and Respectively Moment bus The active power and reactive power of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The charging and discharging power of the energy storage device ES inside the microgrid MG; express Moment bus The reactive power of the energy storage device ES inside the microgrid MG; and Respectively Moment bus Internal load of microgrid MG Active power and reactive power; and Respectively Moment bus The active power and reactive power exchanged between the microgrid MG and the distribution network; and Respectively Moment bus The net injected active power and reactive power at and Respectively Moment bus The minimum active power and maximum active power exchanged between the microgrid MG and the distribution network; and Respectively Moment bus The minimum reactive power and maximum reactive power exchanged between the microgrid MG and the distribution network; express Moment bus The rated capacity of the microgrid MG.
[0076] The mathematical model of the power flow of each branch in the distribution network is established, and all voltage control mathematical models are subject to rule constraints. The mathematical model of the power flow of each branch is constructed through the following formula:
[0077] ;
[0078] In the formula, and Respectively Moment bus Load Active and reactive power; Indicates busbar To busbar The resistance of the branch between the points; Indicates busbar To busbar The reactance of the branch between the points; express Moment bus The voltage at express Moment bus The voltage at and Indicates busbar The upper and lower limits of the allowable voltage range; express Moment bus To busbar The current in the branch between the points; Indicates busbar The set of all buses with other buses as the first endpoints corresponding to the branch with the end point at Busbar Set A first terminal bus in; and Respectively Time first end bus To busbar The active power and reactive power of the branches between the locations; Indicates busbar The set of all buses corresponding to the branch with the first endpoint and other buses as the end points, Busbar Set One end point of the generatrix; and Respectively Moment bus To the end point busbar The active power and reactive power of the branches between the locations; express Moment bus The active output of the photovoltaic inverter PV; express Moment bus The charging and discharging power of the energy storage device ES; and Respectively Moment bus The active power and reactive power exchanged between the microgrid MG and the distribution network; express Moment bus The reactive output of the photovoltaic inverter PV injected into the distribution grid.
[0079] S20, taking each voltage control device as an intelligent agent, allocating each variable to the intelligent agent, so that the voltage control mathematical model is transformed into a Markov game model.
[0080] In one embodiment of the present invention, based on the voltage control mathematical model of the distribution network, each device is regarded as an intelligent agent, and each variable is assigned to the intelligent agent, so that the voltage control mathematical model is transformed into a Markov game model. ,in represents the total number of all agents, Represents the global state, Indicates The state of an agent, Indicates The actions of an agent, Indicates The reward for an agent, represents the state transition function, Represents the reward discount factor. Specifically, it includes:
[0081] S21, establish the action space and state space of each energy storage device ES agent, photovoltaic inverter PV agent and microgrid MG agent.
[0082] In one embodiment of the present invention, the action space of each device agent is established using the following formula:
[0083] ;
[0084] In the formula, , , Respectively The action space of the energy storage device ES agent, the action space of the photovoltaic inverter PV agent, and the action space of the microgrid MG agent; express Moment The actions of the energy storage device ES agent; express Moment The actions of a photovoltaic inverter PV agent; express Moment The actions of the microgrid MG agent; , , They represent the number of energy storage device ES agent, photovoltaic inverter PV agent, and microgrid MG agent respectively.
[0085] In one embodiment of the present invention, the state space of each device agent is established using the following formula:
[0086] ;
[0087] In the formula, , , Respectively The state space of the energy storage device ES agent, the state space of the photovoltaic inverter PV agent, and the state space of the microgrid MG agent; Respectively The voltage amplitude of the busbar at each location at the moment, the injected active power, and the injected reactive power; and Respectively Reactive power and active power of the energy storage device ES agent at all times; and Respectively Reactive power and active power of photovoltaic inverter PV agent at all times; and Respectively The reactive power and active power exchanged between the microgrid MG agent and the distribution network at the moment. Among them, the state transfer function Status S Take Action A Transfer to state probability.
[0088] S22, Establishment Rewards for the ES agent of the moment energy storage device , Rewards for PV Agents 、Rewards for microgrid MG intelligent body .
[0089] In one embodiment of the present invention, Rewards for the ES agent of the moment energy storage device , Rewards for PV Agents 、Rewards for microgrid MG intelligent body for:
[0090] ;
[0091] In the formula, express Moment bus To busbar The loss of the branch between the locations; express Total loss of the branch at time Loss ; represents the loss coefficient of the branch; express The voltage exceeds the limit at the moment; Indicates the voltage over-limit coefficient; represents a busbar set; Indicates the operation of taking a larger value; and They represent the prices of buying and selling active power in the distribution network respectively; and They represent the prices of buying and selling reactive power in the distribution network respectively; Indicates that the active power of the microgrid flows to the distribution network; Indicates that the active power of the distribution network flows to the microgrid; It indicates that the reactive power of the microgrid flows to the distribution network; Indicates that the reactive power of the distribution network flows to the microgrid; represents the weight factor; Indicates busbar To busbar The resistance of the branch between the points; express Moment bus To busbar The current in the branch between the points; express Moment bus The voltage at and Indicates busbar The upper and lower limits of the allowable voltage range; Represents the cost of buying and selling active power and reactive power in the distribution network.
[0092] Taking into account the operating costs of the microgrid MG, including the buying and selling prices of active power and reactive power, The smaller the value, the lower the operating cost of the microgrid MG, but the greater the voltage fluctuation. The larger the value, the smaller the corresponding voltage fluctuation, but the operating cost of the microgrid MG may be higher. The line loss, voltage limit violation, and microgrid MG operating cost in the distribution network are all taken into account by setting the reward function.
[0093] S23, obtaining the maximum discounted return of each agent according to the action space, state space and reward of each agent.
[0094] In one embodiment of the present invention, the maximum discounted reward of each agent is:
[0095] ;
[0096] In the formula, Indicates The strategy of an agent, are the parameters of the strategy; express The reward discount factor at the moment; express Moment Rewards for each agent; express Moment The actions of an agent; express Moment The state of an agent, express Moment The state of an agent; Represents the state transition function of each agent; express expectations; Indicated in strategy The maximum value under Indicates the total number of steps.
[0097] S24, all the agent's action space, state space, and reward , Maximizing the discounted return combination to form a Markov game model ;in represents the total number of all agents, Represents the global state, Indicates The state of an agent, Indicates The actions of an agent, Indicates The reward for an agent, represents the state transition function, Represents the reward discount factor.
[0098] S30, abstract each energy storage device ES, photovoltaic inverter PV and microgrid MG in the distribution network as a node, abstract the connection line between each two nodes as an edge, and obtain the voltage control graph; based on the voltage control graph and the reinforcement learning algorithm, continue to train all agents under the Markov game model, and obtain the optimal agent, so as to transmit the output actions of the optimal energy storage device ES agent, the optimal photovoltaic inverter PV agent, and the optimal microgrid MG agent to the energy storage device ES, the photovoltaic inverter PV, and the microgrid MG accordingly.
[0099] In one embodiment of the present invention, the voltage control diagram ;in, Represents the set of device nodes in the distribution network, Represents the set of lines between each device node; voltage control diagram It can intuitively and naturally express the complex relationship between nodes. To obtain the optimal agent, the following steps are included:
[0100] S31, according to the line connection relationship of each node in the voltage control diagram, obtain the first The node and Voltage interaction coefficient of each node ; Based on the voltage interaction coefficient , get the The node and Virtual electrical distance of nodes , the connection relationship of multiple voltage control devices in the distribution network is represented by virtual electrical distance Indicates that we can get the adjacency matrix D .
[0101] In one embodiment of the present invention, The node and Voltage interaction coefficient of each node , obtained by the following formula:
[0102] ;
[0103] In the formula, Indicates The node voltage is affected by Sensitivity coefficient of active power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of reactive power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of active power injection of each node; Indicates The node voltage is affected by The sensitivity coefficient of reactive power injection of each node. The nodes and Voltage interaction coefficient of the node The method of obtaining the voltage interaction coefficient The method is the same as that of Replace with , Replace with , Replace with , in the denominator Replace with , Replace with , Replace with .
[0104] And, get the The node and Virtual electrical distance of nodes Obtained by the following formula:
[0105] .
[0106] As an adjacency matrix Middle Line Column elements, adjacency matrix For the voltage control problem we are studying, it can determine the connection relationship between each device in the distribution network.
[0107] S32, establish Actor network of agents and the target actor network ; and are the parameters of the actor network and the target actor network respectively; Indicates The actions of an agent; Indicates The state of an agent.
[0108] S33, for Actor network of agents Add Gaussian-noise to the output .
[0109] Among them, the actor network Add Gaussian-noise to the output Then:
[0110] ;
[0111] In the formula, noise Obey Gaussian distribution , is the mean value of Gaussian distribution, is the standard deviation of the Gaussian distribution.
[0112] S34, set up the attention network, combined with the adjacency matrix D, Establish the A critic network with two agents and target critic network ; and are the parameters of the critic network and the target critic network respectively; and Represent the global state and joint action respectively.
[0113] In one embodiment of the present invention, the attention network GAT can make full use of the topological graph structure information. Through the attention mechanism, the node can pay attention to the neighboring nodes related to it, thereby propagating information on the graph, capturing the interaction between nodes, and establishing the first A critic network with two agents , through the following formula:
[0114] ;
[0115] In the formula, Represents matrix concatenation; express and The concatenated matrix; The weight matrix representing the input features; Represents the characteristic matrix after linear transformation; Represents the new feature matrix; , Respectively represent nodes and The characteristic matrix after linear transformation; Represents the weight vector for calculating attention; , Respectively represent nodes For Node and nodes The attention coefficient; Represents the normalized node For Node The attention coefficient; Indicates The node where the agent is located The set of neighbor nodes of Indicates A fully connected layer of the agent-critic network; Indicates The attention network in the agent critic network; and Represent two different activation functions respectively; Represents an exponential function.
[0116] S35, from The experience pool of an agent Randomly select strip , for Agent-based critic network and actor networks to update.
[0117] S36, using loss function For Agent-based critic network to update.
[0118] In one embodiment of the present invention, the loss function For Agent-based critic network Update as follows:
[0119] ;
[0120] In the formula, Indicates The critic network of each agent is in the global state Take joint action The action-value function of Represents the updated global state, represents the updated joint action; express The updated local state of each agent is express The local actions of each agent after update; Indicates The target critic network of each agent is in the updated global state Updated joint actions The action-value function of Indicates Agents in local state Take local action Rewards received; is the discount factor; is the temperature parameter that determines the balance between maximum entropy and reward; Represents the logarithmic function; Expressing the Accurate estimation of the action-value function of an agent.
[0121] S37, using the gradient descent algorithm Actor network of agents The parameters are updated.
[0122] In one embodiment of the present invention, the gradient algorithm is used for Actor network of agents The parameters are updated as follows:
[0123] ;
[0124] In the formula, Indicates that except The actions of other agents outside the agent; Indicates The critic network of each agent is in the global state Fix other agent actions , and take local action The action-value function of express expectations; Express Find the gradient; Represents the objective function.
[0125] S38, using reinforcement learning soft update to the target actor network and target critic network The parameters are updated.
[0126] In one embodiment of the present invention, the reinforcement learning soft update method is used to update the target actor network. and target critic network The parameters are updated as follows:
[0127] ;
[0128] In the formula, represents weight; Indicates assignment.
[0129] S39, repeat the process from step S35 to step S38, Parameters of the actor network of agents and the parameters of the critic network Perform iterative training to obtain The optimal parameters of an agent The corresponding optimal actor network and optimal parameters The corresponding optimal critic network constitutes the The optimal agent is obtained, thus all the optimal agents are obtained.
[0130] See also Figure 3 As shown, the present invention also provides a system using the above-mentioned distribution network voltage control method, characterized in that: a plurality of voltage control devices are provided in the distribution network, including an energy storage device ES, a photovoltaic inverter PV and a microgrid MG; including:
[0131] The voltage control device module constructs a voltage control mathematical model based on the distribution network structure, including the mathematical models of the energy storage device ES, the photovoltaic inverter PV and the microgrid MG.
[0132] The Markov game model module regards each voltage control device as an intelligent agent and assigns various variables to the intelligent agent, so that the voltage control mathematical model is transformed into a Markov game model.
[0133] The voltage control module abstracts each energy storage device ES, photovoltaic inverter PV and microgrid MG in the distribution network as a node, and abstracts the connection line between each two nodes as an edge to obtain the voltage control graph; based on the voltage control graph and the reinforcement learning algorithm, all agents under the Markov game model are continuously trained to obtain the optimal agent, so that the output actions of the optimal energy storage device ES agent, the optimal photovoltaic inverter PV agent, and the optimal microgrid MG agent are correspondingly transmitted to the energy storage device ES, the photovoltaic inverter PV, and the microgrid MG.
[0134] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting from any point of view, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.
[0135] The above-described embodiments merely represent implementation methods of the invention. The protection scope of the present invention is not limited to the above-described embodiments. For those skilled in the art, several modifications and improvements may be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A distribution network voltage control method, characterized in that: Definition: There are multiple voltage control devices in the distribution network, including energy storage device ES, photovoltaic inverter PV and microgrid MG; The method comprises the following steps: S10, constructing a voltage control mathematical model based on the distribution network structure, including mathematical models of the energy storage device ES, the photovoltaic inverter PV and the microgrid MG; S20, taking each voltage control device as an intelligent agent, assigning each variable to the intelligent agent, so that the voltage control mathematical model is transformed into a Markov game model; S30, abstract each energy storage device ES, photovoltaic inverter PV and microgrid MG in the distribution network as a node, abstract the connection line between each two nodes as an edge, and obtain the voltage control graph; based on the voltage control graph and the reinforcement learning algorithm, continue to train all agents under the Markov game model, and obtain the optimal agent, so as to transmit the output actions of the optimal energy storage device ES agent, the optimal photovoltaic inverter PV agent, and the optimal microgrid MG agent to the energy storage device ES, the photovoltaic inverter PV, and the microgrid MG accordingly; Among them, the mathematical model of the microgrid MG is constructed by the following formula: ; In the formula, express Moment bus is the real number control variable of the energy storage device ES inside the microgrid MG; express Moment bus is the real number control variable of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The maximum reactive power of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The maximum charging and discharging power of the energy storage device ES inside the microgrid MG; and Respectively Moment bus The active power and reactive power of the photovoltaic inverter PV inside the microgrid MG; express Moment bus The charging and discharging power of the energy storage device ES inside the microgrid MG; express Moment bus The reactive power of the energy storage device ES inside the microgrid MG; and Respectively Moment bus Internal load of microgrid MG Active power and reactive power; and Respectively Moment bus The active power and reactive power exchanged between the microgrid MG and the distribution network; and Respectively Moment bus The net injected active power and reactive power at and Respectively Moment bus The minimum active power and maximum active power exchanged between the microgrid MG and the distribution network; and Respectively Moment bus The minimum reactive power and maximum reactive power exchanged between the microgrid MG and the distribution network; express Moment bus The rated capacity of the microgrid MG; Obtaining the optimal agent includes the following steps: S31, according to the line connection relationship of each node in the voltage control diagram, obtain the first The node and Voltage interaction coefficient of each node ; Based on the voltage interaction coefficient , get the The node and Virtual electrical distance of nodes , the connection relationship of multiple voltage control devices in the distribution network is represented by virtual electrical distance Indicates that we can get the adjacency matrix D ; S32, establish Actor network of agents and the target actor network ; and are the parameters of the actor network and the target actor network respectively; Indicates The actions of an agent; Indicates The state of an agent; S33, for Actor network of agents Add Gaussian-noise to the output ; S34, set up the attention network, combined with the adjacency matrix D, Establish the A critic network with two agents and target critic network ; and are the parameters of the critic network and the target critic network respectively; and Represent the global state and joint action respectively; S35, from The experience pool of an agent Randomly select strip , for A critic network with two agents and actor networks To update; among them, Indicates The reward for an agent, Indicates Agents take A The next state after the action; S36, using loss function For A critic network with two agents Make updates; S37, using the gradient descent algorithm Actor network of agents Update the parameters of S38, using reinforcement learning soft update to the target actor network and target critic network Update the parameters of S39, repeat the process from step S35 to step S38, Parameters of the actor network of agents and the parameters of the critic network Perform iterative training to obtain The optimal parameters of an agent The corresponding optimal actor network and optimal parameters The corresponding optimal critic network constitutes the The optimal agent is obtained, thus all the optimal agents are obtained.
2. The distribution network voltage control method according to claim 1, characterized in that: The ES mathematical model of the energy storage device is constructed through the following formula: ; In the formula, express Moment bus The real number control variable of the energy storage device ES; express Moment bus The charging and discharging power of the energy storage device ES; express Moment bus The maximum charge and discharge power of the energy storage device ES; and Respectively Moment bus Reactive power and rated capacity of the energy storage device ES; express Moment bus The energy storage device ES is in operation state. express +1 Moment Bus The energy storage device ES is in operating state; and Respectively represent busbar The minimum and maximum values of the operating state of the energy storage device ES; and They represent the charging efficiency and discharging efficiency of the energy storage device ES respectively; Indicates the electric capacity of the energy storage device ES; express +1 moment with The interval between moments.
3. The distribution network voltage control method according to claim 1, characterized in that: The photovoltaic inverter PV mathematical model is constructed through the following formula: ; In the formula, express Moment bus The real number control variable of the photovoltaic inverter PV; express Moment bus The rated capacity of the photovoltaic inverter PV; express Moment bus The active output of the photovoltaic inverter PV; express Moment bus The maximum reactive power output of the photovoltaic inverter PV; express Moment bus The reactive output of the photovoltaic inverter PV injected into the distribution grid.
4. The distribution network voltage control method according to claim 1, characterized in that: The mathematical model of the power flow of each branch in the distribution network is established, and all voltage control mathematical models are subject to rule constraints; the mathematical model of the power flow of each branch is constructed by the following formula: ; In the formula, and Respectively Moment bus Load Active and reactive power; Indicates busbar To busbar The resistance of the branch between the points; Indicates busbar To busbar The reactance of the branch between the points; express Moment bus The voltage at express Moment bus The voltage at and Indicates busbar The upper and lower limits of the allowable voltage range; express Moment bus To busbar The current in the branch between the points; Indicates busbar The set of all buses with other buses as the first endpoints corresponding to the branch with the end point at Busbar Set A first terminal bus in; and Respectively Time first end bus To busbar The active power and reactive power of the branches between the locations; Indicates busbar The set of all buses corresponding to the branch with the first endpoint and other buses as the end points, Busbar Set One end point of the generatrix; and Respectively Moment bus To the end point busbar The active power and reactive power of the branches between the locations; express Moment bus The active output of the photovoltaic inverter PV; express Moment bus The charging and discharging power of the energy storage device ES; and Respectively Moment bus The active power and reactive power exchanged between the microgrid MG and the distribution network; express Moment bus The reactive output of the photovoltaic inverter PV injected into the distribution grid.
5. The method for controlling voltage in a power distribution network according to claim 1, characterized in that: Transforming the voltage control mathematical model into a Markov game model includes the following steps: S21, establishing the action space and state space of each energy storage device ES agent, photovoltaic inverter PV agent and microgrid MG agent; S22, Establishment Rewards for the ES agent of the moment energy storage device , Rewards for PV Agents 、Rewards for microgrid MG intelligent body ; S23, obtaining the maximum discounted return of each agent according to the action space, state space and reward of each agent; S24, all the agent's action space, state space, and reward , Maximizing the discounted return combination to form a Markov game model ;in represents the total number of all agents, Represents the global state, Indicates The state of an agent, Indicates The actions of an agent, Indicates The reward for an agent, represents the state transition function, Represents the reward discount factor.
6. The method for controlling the voltage of the distribution network according to claim 5, characterized in that: Rewards for the ES agent of the moment energy storage device , Rewards for PV Agents 、Rewards for microgrid MG intelligent body for: ; In the formula, express Moment bus To busbar The loss of the branch between the locations; express Total loss of the branch at time Loss ; represents the loss coefficient of the branch; express The voltage exceeds the limit at the moment; Indicates the voltage over-limit coefficient; represents a busbar set; Indicates the operation of taking a larger value; and They represent the prices of buying and selling active power in the distribution network respectively; and They represent the prices of buying and selling reactive power in the distribution network respectively; Indicates that the active power of the microgrid flows to the distribution network; Indicates that the active power of the distribution network flows to the microgrid; It indicates that the reactive power of the microgrid flows to the distribution network; Indicates that the reactive power of the distribution network flows to the microgrid; represents the weight factor; Indicates busbar To busbar The resistance of the branch between the points; express Moment bus To busbar The current in the branch between the points; express Moment bus The voltage at and Indicates busbar The upper and lower limits of the allowable voltage range; Represents the cost of buying and selling active power and reactive power in the distribution network.
7. The method for controlling voltage in a power distribution network according to claim 6, characterized in that: The maximum discounted return of each agent is: ; In the formula, Indicates The strategy of an agent, are the parameters of the strategy; express The reward discount factor at the moment; express Moment Rewards for each agent; express Moment The actions of an agent; express Moment The state of an agent, express Moment The state of an agent; Represents the state transition function of each agent; express expectations; Indicated in strategy The maximum value under Indicates the total number of steps.
8. The method for controlling voltage in a power distribution network according to claim 1, characterized in that: No. The node and Voltage interaction coefficient of each node , obtained by the following formula: ; In the formula, Indicates The node voltage is affected by Sensitivity coefficient of active power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of reactive power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of active power injection of each node; Indicates The node voltage is affected by Sensitivity coefficient of reactive power injection of each node; Get the The node and Virtual electrical distance of nodes Obtained by the following formula: ; In the formula, Indicates The nodes and Voltage interaction metric coefficient of the node.
9. The method for controlling voltage in a power distribution network according to claim 1, characterized in that: Establish the A critic network with two agents , through the following formula: ; In the formula, Represents matrix concatenation; express and The concatenated matrix; The weight matrix representing the input features; Represents the characteristic matrix after linear transformation; Represents the new feature matrix; , Respectively represent nodes and The characteristic matrix after linear transformation; Represents the weight vector for calculating attention; , Respectively represent nodes For Node and nodes The attention coefficient; Represents the normalized node For Node The attention coefficient; Indicates The node where the agent is located The set of neighbor nodes of Indicates A fully connected layer of the agent-critic network; Indicates The attention network in the agent critic network; and Represent two different activation functions respectively; Represents an exponential function.
10. A system using the distribution network voltage control method according to any one of claims 1 to 9, characterized in that: Definition: There are multiple voltage control devices in the distribution network, including energy storage device ES, photovoltaic inverter PV and microgrid MG; including: The voltage control device module builds a voltage control mathematical model based on the distribution network structure, including the mathematical models of the energy storage device ES, the photovoltaic inverter PV and the microgrid MG; The Markov game model module treats each voltage control device as an intelligent agent, assigns each variable to the intelligent agent, and transforms the voltage control mathematical model into a Markov game model; The voltage control module abstracts each energy storage device ES, photovoltaic inverter PV and microgrid MG in the distribution network as a node, and abstracts the connection line between each two nodes as an edge to obtain the voltage control graph; based on the voltage control graph and the reinforcement learning algorithm, all agents under the Markov game model are continuously trained to obtain the optimal agent, so that the output actions of the optimal energy storage device ES agent, the optimal photovoltaic inverter PV agent, and the optimal microgrid MG agent are correspondingly transmitted to the energy storage device ES, the photovoltaic inverter PV, and the microgrid MG.
Citation Information
Patent Citations
Power distribution network voltage control method based on multi-agent deep reinforcement learning
CN113363998A
Power distribution network voltage control method based on multi-agent deep reinforcement learning
CN117913842A
Power distribution network voltage autonomous optimization control method and device
CN113872213A
Voltage control method and device, terminal equipment and storage medium
CN116154788A
Distributed photovoltaic and energy storage combined distribution network low-voltage treatment method
CN117411007A
