A voltage control method for distribution networks based on multi-agent deep reinforcement learning

Abstract

The present invention provides a distribution network voltage control method based on multi-agent deep reinforcement learning, which relates to the technical field of distribution networks. The method includes dividing the distribution network into multiple areas and centrally managing and optimizing the voltage of each area; converting the collaborative voltage control problem between areas into a Markov game to achieve effective collaboration between agents; adopting a centralized training and decentralized execution framework so that the agents can fully utilize global information in the training phase and make independent decisions based on local observations in the execution phase; integrating a self-attention mechanism so that the agents are more focused on information related to their own rewards, thereby combining multi-agent reinforcement learning technology with the self-attention mechanism to effectively deal with the voltage fluctuation problem in the distribution network, thereby achieving stable control and optimized management of the voltage, improving the voltage regulation accuracy and response speed of the distribution network, and ensuring the stability and reliability of the power system.

Description

Technical Field

[0001] The present invention relates to the field of power distribution network technology, and in particular to a power distribution network voltage control method based on multi-agent deep reinforcement learning. Background Art

[0002] The widespread deployment of solar photovoltaic (PV) systems in power distribution networks has brought about numerous technical challenges. Voltage regulation has become increasingly problematic due to the uncertainty and volatility of renewable energy generation.

[0003] Traditional technologies maintain voltage stability by adjusting on-load tap-changing transformers and capacitor banks, but these methods struggle to cope with rapid voltage fluctuations. While photovoltaic systems equipped with inverters can regulate voltage through reactive power generation, voltage control is often formulated as an optimal power flow (OPF) problem for optimizing distribution network operation, requiring optimization algorithms. Centralized control schemes, such as those based on the model predictive control (MPC) framework, which utilize real-time measurements to regulate voltage, often rely on extensive communication and computational resources. While multi-period OPF models and local control strategies have developed, these strategies can lead to suboptimal voltage regulation. Intelligent optimization algorithms typically employ heuristic search strategies, such as particle swarm optimization, ramp system optimization, and gravity search algorithms. However, due to the non-convexity and NP-hard nature of the problem, these algorithms struggle to converge to the global optimum. The fusion of reinforcement learning (RL) and deep learning has led to deep reinforcement learning (DRL), which has potential for application in power grid voltage control. However, single-agent RL methods can face challenges in handling the dynamic characteristics of distribution networks. Although multi-agent deep reinforcement learning (MADRL) has become an effective paradigm for solving complex tasks in power grids, existing algorithms, such as multi-agent deep deterministic policy gradient (MADDPG), often have problems such as high hyperparameter sensitivity, insufficient agent collaboration, and limited exploration.

[0004] Therefore, it is necessary to provide a distribution network voltage control method based on multi-agent deep reinforcement learning to solve the above technical problems. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention provides a distribution network voltage control method based on multi-agent deep reinforcement learning, which is used to solve the voltage control problem in the distribution network that the existing technology cannot effectively deal with.

[0006] The present invention provides a distribution network voltage control method based on multi-agent deep reinforcement learning, the method comprising:

[0007] The distribution network is constructed based on graph theory, and the power flow balance equation and voltage constraint formula are used to describe the dynamic behavior of the distribution network;

[0008] Optimize voltage control and power flow, analyze the impact of photovoltaic power generation on node voltage, photovoltaic inverter controls node voltage by adjusting photovoltaic active power, define the optimal power flow problem, optimize the control device parameters of the power system, and formulate the objective function to minimize voltage deviation and power loss;

[0009] A Markov game model is established to transform the active voltage control problem into a Markov game. The agent is the photovoltaic inverter, the environment is the distribution network, the state contains various power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward.

[0010] Introducing a multi-agent soft actor-critic algorithm that enhances collaboration, defining a centralized training and decentralized execution framework, and integrating a self-attention mechanism;

[0011] Implement a voltage control strategy. At the beginning of each round, store the photovoltaic and load configuration profiles. At each time step, fuse the photovoltaic and load curves and states to determine the subsequent state and reward. Then, split and sort the states according to the corresponding areas of the agent. The agent makes a decision after receiving the state and reward of the corresponding area.

[0012] Preferably, the construction of the distribution network based on graph theory and the use of power flow balance equations and voltage constraint formulas to describe the dynamic behavior of the distribution network specifically include:

[0013] Establish the topology of the power distribution network and model the power distribution network as a radial network. The radial network contains nodes, Represents the set of all nodes in the distribution network. The transformer is located at node 0. For node , , the node voltage and phase angle are and ;

[0014] Define the voltage and power characteristics of the node. , the input node active power is ,in, Indicates the photovoltaic active power, represents the active load power, and the input node reactive power is ,in, Represents photovoltaic reactive power, Indicates reactive load power;

[0015] The power flow balance equation is used to control each node to be in a power balance state. The expression of the power flow balance equation is as follows:

[0016] in, and Represents nodes respectively and Admittance and reactance of the line between them; Representation node and The phase angle difference between ; Representation node Complex power injection; Representation node and The impedance of the line between By resistor and reactance Composition, that is ; Represents a slave node Flow Node Current; Representation node The voltage, ;

[0017] Apply the voltage constraint formula and set the voltage standard to , the allowable voltage deviation is 5%, then the voltage The following conditions must be met:

[0018] .

[0019] Preferably, the optimizing voltage control and power flow, analyzing the influence of photovoltaic power generation on node voltage, and the photovoltaic inverter controlling the node voltage by adjusting photovoltaic active power, specifically includes:

[0020] Setting Node The impedance is ,in, and Represents nodes respectively resistance and reactance, and set the node The reference voltage is ;

[0021] In a double busbar network, voltage drops The expression is:

[0022] When the photovoltaic active power When the photovoltaic inverter increases, it adjusts the photovoltaic active power Voltage management: When the node voltage is higher than the upper limit, the photovoltaic inverter reduces the photovoltaic active power , reduce the node voltage; when the node voltage is lower than the lower limit, the photovoltaic inverter increases the photovoltaic active power , increase the node voltage, and the expression of the regulated photovoltaic active power is:

[0023] in, Indicates the regulated photovoltaic active power; Indicates the regulation coefficient of the node voltage;

[0024] And the active power loss in the distribution network The expression is:

[0025] in, and They represent the regulation coefficients of active load power and reactive load power, respectively, and are adjusted according to the operating status and control objectives of the power system; Represents the weight coefficient of the reference voltage;

[0026] If the active power loss is minimized , then it is necessary to meet .

[0027] Preferably, the definition of the optimal power flow problem, the optimization of the control device parameters of the power system, and the formulation of the objective function to minimize the voltage deviation and power loss specifically include:

[0028] The optimal power flow problem is defined. The corresponding objective function is to minimize the active power of the balanced bus. The corresponding constraints are node power balance and various safety constraints. The expressions of the objective function and constraints are as follows:

[0029] in, Indicates the active power of the balancing bus; Indicates the bus index set equipped with the photovoltaic inverter; and Represent the lower and upper limits of the node voltage respectively; Indicates the maximum photovoltaic reactive power of the photovoltaic inverter;

[0030] Assuming that the capacity of each PV inverter is expanded to , Indicates the maximum node active power, then the maximum photovoltaic reactive power The expression is:

[0031] The objective function of voltage control is formulated, that is, by adjusting the photovoltaic reactive power, minimizing the voltage deviation, active power loss and current deviation. The expression of the objective function of voltage control is as follows:

[0032] in, Represents the voltage deviation of all nodes in the distribution network; represents the active power loss of all nodes; represents the current deviation of all nodes; Indicates the current standard; Indicates the balance weight coefficient corresponding to the voltage deviation; Indicates the balancing weight coefficient corresponding to the active power loss; Indicates the balance weight coefficient corresponding to the current deviation; The value range is 0 to 1.

[0033] Preferably, the Markov game model is established to convert the active voltage control problem into a Markov game, and the intelligent agent is determined to be a photovoltaic inverter, the environment is a power distribution network, the state includes multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward, specifically including:

[0034] The distribution network is divided into different areas, each area is equipped with multiple photovoltaic modules, each photovoltaic module contains a photovoltaic inverter, each photovoltaic inverter is an intelligent agent, and each intelligent agent is located at a node in the distribution network;

[0035] Model the power distribution network as an environment and divide the environment into Regions, the environment is represented by ,The agent observes the node information in the corresponding area and makes a decision, estimating the voltage of each node, and determining the reward obtained by the agent;

[0036] The state space is defined as , the state space includes the states of all agents, a single agent The status of composition;

[0037] Set the continuous action set to , the continuous action set contains all possible actions of the agent. For a single agent , continuous action represents the proportion of the maximum photovoltaic reactive power generated by the photovoltaic inverter, and ,and and The following relationship exists:

[0038] in, Represents a connection node The apparent power of the photovoltaic inverter; when When the photovoltaic inverter sends power to the node Injecting photovoltaic reactive power, when When , the photovoltaic inverter absorbs photovoltaic reactive power; Indicates power factor The reciprocal of Indicates the power factor. When the power factor When it decreases, the photovoltaic reactive power increases; Indicates the loss coefficient of photovoltaic reactive power. When , there is loss of photovoltaic reactive power;

[0039] Constructing time steps The reward function , the corresponding expression is as follows:

[0040] in, represents the time weight factor; when When the power system is in the initial stage of startup, the voltage fluctuation is large, and the setting ,when When the power system is in the transition stage, the power system tends to be stable, with a small amount of voltage fluctuation. ,when When the power system is in a stable operation stage, the voltage fluctuation is small, and the setting ; represents the weight parameter used to balance the voltage deviation and photovoltaic reactive power loss; represents the voltage potential function; Represents photovoltaic reactive power generation loss; Indicates photovoltaic reactive power loss.

[0041] Preferably, the voltage potential function The expression is as follows:

[0042] in, represents a hyperparameter used to tune the configuration of the voltage potential function;

[0043] Photovoltaic reactive power generation losses The expression is as follows:

[0044] The objective function of establishing the Markov game model is:

[0045] in, Represents the discount factor.

[0046] Preferably, the multi-agent soft Actor-Critic algorithm for enhanced collaboration is introduced, and a centralized training and decentralized execution framework is defined, integrating a self-attention mechanism, specifically including:

[0047] It adopts a centralized training and decentralized execution framework. During the training phase, each agent's critic evaluation network receives collective data from all agents, including state and action. During the execution phase, each agent makes independent decisions through the corresponding actor policy network.

[0048] Introducing the self-attention mechanism, based on the agent collaboration mechanism, initializes the states and actions of all agents, the critic evaluation network, and the parameters of the self-attention model;

[0049] At time step , agent The input of the Critic evaluation network The state of all agents and actions The corresponding expression for the aggregation is:

[0050] in, Representing an agent The encoding function of Representing an agent At time step Status; Representing an agent At time step Actions; represents the neural network used to fit the evaluation function; Represents the self-attention input, that is, except for the agent The weighted sum of the encoded values ​​of all agents except The expression is:

[0051] in, Representing an agent At time step For intelligent agents The self-attention weight of Representing an agent At time step The encoded values ​​of the states and actions; Representing an agent The encoding function of Representing an agent At time step Status; Representing an agent At time step Actions; Represents the activation function, that is, the RELU function; represents the linear transformation matrix; and represents the coding transformation matrix; Expressed as a natural constant e An exponential function with base .

[0052] Preferably, the parameters of the Actor strategy network and the Critic evaluation network, the experience replay buffer, and the agent-related parameters are randomly initialized;

[0053] For the range from 1 to For each round, get the initial state, for the range from 1 to At each time step, determine the action of the agent, perform the action and calculate the reward, determine the new state of the next time step, for the time from 1 to Each agent stores the transition information including the initial state, action, reward, and new state of the next time step in the experience replay buffer;

[0054] When the experience replay buffer reaches the preset area capacity, a group of agents are randomly selected for learning. , the parameters to be updated are expressed as ,in, and Represent the parameters of the Actor strategy network and the target Actor strategy network respectively, and Represent the parameters of the critic evaluation network and the target critic evaluation network, respectively. By minimizing the loss function L To update:

[0055] in, represents the batch size of randomly selected agents; represents the regularization coefficient; represents the action-value function; Represents the target value, including the time step The reward and the next time step Action value estimates; The temperature coefficient of entropy, used to balance reward and entropy; Represents the learning rate of the Critic evaluation network; Representing an agent At the next time step Actions; Representing an agent At the next time step Status; Represents the agent in the Critic evaluation network A collection of actions; represents the mathematical expectation operator;

[0056] Parameters of the Actor Policy Network Through the gradient descent function To update:

[0057] in, Represents the agent in the Actor strategy network A collection of actions; Represents the learning rate of the Actor strategy network;

[0058] The target actor policy network and the target critic evaluation network are updated with parameters through the soft update method and synchronized with the online network:

[0059] in, Represents the tracking parameters, which are used to control the parameter update rate of the target Actor policy network and the target Critic evaluation network;

[0060] After each agent completes the parameter update, the parameters of the Actor strategy network of each agent are fixed, and each agent makes independent decisions based on the corresponding state based on the Actor strategy network.

[0061] Preferably, the voltage control strategy is implemented by storing the photovoltaic and load configuration profiles at the beginning of each round, fusing the photovoltaic and load curves and states at each time step, determining the subsequent state and reward, and dividing and sorting the states according to the corresponding areas of the agent. The agent makes a decision after receiving the states and rewards of the corresponding areas, specifically including:

[0062] At the beginning of each round, the PV and load configuration profiles are stored in the experience replay buffer;

[0063] At each time step, the corresponding PV and load curves are retrieved, the state is calculated and the reward is determined by Pandapower, the PV and load curves and the state are integrated and the subsequent state is established;

[0064] Before sending rewards to agents, split the states of all agents into multiple state batches and sort the state batches according to the corresponding regions of the agents;

[0065] The agent makes a decision after receiving the status and reward of the corresponding area, and repeats the above process until the end of the entire round.

[0066] A distribution network voltage control system based on multi-agent deep reinforcement learning, the system comprising:

[0067] A construction module for constructing distribution networks based on graph theory and describing their dynamic behavior using power flow balance equations and voltage constraint formulas;

[0068] The optimization module is used to optimize voltage control and power flow, analyze the impact of photovoltaic power generation on node voltage, and control the node voltage by adjusting the photovoltaic active power of the photovoltaic inverter. It also defines the optimal power flow problem, optimizes the control device parameters of the power system, and formulates the objective function to minimize voltage deviation and power loss.

[0069] The conversion module is used to establish a Markov game model, converting the active voltage control problem into a Markov game. The intelligent agent is the photovoltaic inverter, the environment is the distribution network, the state contains multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward.

[0070] Introducing a module for introducing a collaborative multi-agent soft actor-critic algorithm, defining a centralized training and decentralized execution framework, and integrating a self-attention mechanism;

[0071] The implementation module is used to implement the voltage control strategy. At the beginning of each round, the photovoltaic and load configuration profiles are stored. At each time step, the photovoltaic and load curves and states are integrated to determine the subsequent state and reward. The states are then divided and sorted according to the corresponding areas of the agent. The agent makes a decision after receiving the state and reward of the corresponding area.

[0072] Compared with related technologies, the distribution network voltage control method based on multi-agent deep reinforcement learning provided by the present invention has the following beneficial effects:

[0073] The present invention can combine multi-agent reinforcement learning technology with a self-attention mechanism to effectively address voltage fluctuations in the distribution network caused by photovoltaic system integration, thereby achieving stable voltage control and optimized management. The present invention can divide the distribution network into multiple regions, and then centrally manage and optimize the voltage in each region; then, the collaborative voltage control problem between regions can be transformed into a Markov game to facilitate effective collaboration between agents; further, a centralized training and decentralized execution framework can be adopted, allowing agents to fully utilize global information during the training phase and independently make real-time decisions based on local observations during the execution phase; finally, a self-attention mechanism can be integrated, allowing agents to focus more on information related to their own rewards, thereby improving the voltage control effect of the distribution network.

[0074] This method overcomes the limitations of traditional voltage control methods under rapidly fluctuating conditions, improving the voltage regulation accuracy and response speed of distribution networks while ensuring the stability and reliability of the power system. It also offers significant advantages in computational efficiency and cost, making it suitable for distribution networks with high photovoltaic penetration rates, as well as for smart grids requiring efficient voltage management. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A flow chart of a distribution network voltage control method based on multi-agent deep reinforcement learning provided by an embodiment of the present invention;

[0076] Figure 2 A flowchart of a multi-agent soft Actor-Critic algorithm for enhanced collaboration provided by an embodiment of the present invention;

[0077] Figure 3 A flow chart of the agent collaboration mechanism provided by an embodiment of the present invention;

[0078] Figure 4 A flowchart of the implementation process of the voltage control strategy provided by an embodiment of the present invention;

[0079] Figure 5 A system block diagram of a distribution network voltage control system based on multi-agent deep reinforcement learning provided by an embodiment of the present invention;

[0080] Figure 6 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0081] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0082] like Figure 1 FIG. 1 is a flow chart of a distribution network voltage control method based on multi-agent deep reinforcement learning provided by an embodiment of the present invention. Figure 1The execution subject of the method shown may be a software and / or hardware device. The execution subject of the present application may include but is not limited to at least one of the following: user equipment, network equipment, etc. Among them, user equipment may include but is not limited to computers, smart phones, personal digital assistants (PDAs) and the electronic devices mentioned above. Network equipment may include but is not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computers. This embodiment does not limit this. It includes steps S1 to S5, as follows:

[0083] S1, constructs the distribution network based on graph theory, and uses power flow balance equations and voltage constraint formulas to describe the dynamic behavior of the distribution network;

[0084] S2, optimizes voltage control and power flow, analyzes the impact of photovoltaic power generation on node voltage, and controls node voltage by adjusting photovoltaic active power through photovoltaic inverter. It also defines the optimal power flow problem, optimizes the control device parameters of the power system, and formulates the objective function to minimize voltage deviation and power loss.

[0085] S3: Establish a Markov game model to convert the active voltage control problem into a Markov game. The agent is the photovoltaic inverter, the environment is the distribution network, the state contains multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward.

[0086] S4 introduces a multi-agent soft actor-critic algorithm with enhanced collaboration, defines a centralized training and decentralized execution framework, and integrates a self-attention mechanism;

[0087] S5 implements the voltage control strategy. At the beginning of each round, the photovoltaic and load configuration profiles are stored. At each time step, the photovoltaic and load curves and states are integrated to determine the subsequent state and reward. The states are divided and sorted according to the corresponding areas of the agent. The agent makes a decision after receiving the state and reward of the corresponding area.

[0088] At the distribution network modeling level, graph theory is used to abstract the distribution network into a topological structure of interconnected nodes and branches, and the voltage and power characteristics of the nodes are clearly defined. The power flow balance equation ensures that each node in the distribution network adheres to the principle of power conservation during energy transfer, ensuring stable power transmission. The voltage constraint formula is used to limit the fluctuation range of node voltages in the distribution network, maintaining it within a safe operating range. These two equations together construct a mathematical description of the dynamic behavior of the distribution network, ensuring the effectiveness of voltage control within the distribution network and laying a solid foundation for the subsequent design of voltage control strategies.

[0089] In terms of voltage control and power flow optimization, it is necessary to analyze the impact of changes in photovoltaic power generation on node voltages. As a key control unit, the photovoltaic inverter flexibly adjusts the photovoltaic active power output to regulate node voltages in real time. Based on this, the optimal power flow problem is defined, incorporating the parameters of various control devices in the power system into the optimization scope. At the same time, an objective function is set to minimize voltage deviations, bringing node voltages closer to their ideal values ​​and reducing the adverse effects of voltage fluctuations. This also reduces power losses, improves power transmission efficiency, and achieves economical and efficient operation of the power system.

[0090] To more effectively address the complex decision-making issues in voltage control, a Markov game model is introduced, transforming the active voltage control problem into a Markov game. The PV inverter is defined as the agent, and the distribution network as the agent's environment. The state space encompasses key power information, including active and reactive power, comprehensively reflecting the system's operating status. The action is set as the reactive power output ratio of the PV inverter to achieve precise reactive power regulation. The reward function comprehensively considers factors such as voltage status and power loss. The agent aims to maximize the cumulative expected reward, improving voltage control effectiveness through continuous learning and decision-making.

[0091] At the algorithmic level, a collaborative multi-agent soft actor-critic algorithm is employed, designing a centralized training and decentralized execution framework. During centralized training, information from each agent is processed centrally, and a self-attention mechanism is used to explore inter-agent connections and collaborative relationships. This enhances the agents' ability to perceive and utilize global information, thereby strengthening inter-agent collaboration. During the decentralized execution phase, agents make independent decisions based on locally acquired information, ensuring the real-time and reliability of the power system.

[0092] When implementing the voltage control strategy, at the beginning of each control cycle, the PV and load configuration profiles are stored to determine the initial state of the distribution network. Within each time step, the PV output curve, load variation curve, and current distribution network status information are integrated to infer the subsequent distribution network state and corresponding reward value. This state information is then segmented and sorted according to the area the agent is responsible for. After receiving the status and reward for the corresponding area, the agent makes decisions based on the trained strategy, achieving dynamic and precise control of the distribution network voltage and ensuring its safe and stable operation.

[0093] In the specific implementation process, the distribution network is constructed based on graph theory, and the power flow balance equation and voltage constraint formula are used to describe the dynamic behavior of the distribution network, which specifically includes:

[0094] Establish the topology of the power distribution network and model the power distribution network as a radial network. The radial network contains nodes, Represents the set of all nodes in the distribution network. The transformer is located at node 0. For node , , the node voltage and phase angle are and ;

[0095] Define the voltage and power characteristics of the node. , the input node active power is ,in, Indicates the photovoltaic active power, represents the active load power, and the input node reactive power is ,in, Represents photovoltaic reactive power, that is, the reactive power generated by photovoltaic at this node. Indicates reactive load power;

[0096] The power flow balance equation is used to control each node to be in a power balance state, that is, the input power is equal to the output power. The power flow balance equation of the distribution network is expressed as follows:

[0097] in, and Represents nodes respectively and Admittance and reactance of the line between them; Representation node and The phase angle difference between ; Representation node Complex power injection; Representation node and The impedance of the line between By resistor and reactance Composition, that is ; Represents a slave node Flow Node Current; Representation node The voltage, ;

[0098] In order to ensure the safe and stable operation of the distribution network, the voltage constraint formula is applied and the voltage standard is set as , the allowable voltage deviation is 5%, then the voltage The following conditions are met:

[0099] .

[0100] As you can understand, we can first construct the topology of the distribution network, modeling it as a radial network. This network structure consists of numerous nodes, which together form the node set of the distribution network. The transformer is located at node 0. Each of the other nodes corresponds to a specific voltage amplitude and phase angle, which are key parameters reflecting the node's voltage status.

[0101] Secondly, the voltage and power characteristics of each node can be defined. Each node's active power input consists of both PV active power and active load power, while its reactive power input consists of both PV reactive power and reactive load power. These power parameters comprehensively reflect the power inflow and outflow at the node and are crucial for analyzing power balance in the distribution network.

[0102] The power flow balance equation then ensures that each node is in a state of power balance. This equation, based on the electrical parameters of the lines between nodes, such as admittance, reactance, and phase angle differences between nodes, establishes a relationship between the complex power injected into a node and the line current and voltage. This constraint ensures that the power input and output of each node in the distribution network remain balanced, ensuring stable power transmission within the network. The line impedance involved in the equation, consisting of resistance and reactance, determines the power loss and voltage drop during power transmission, while the current and voltage between nodes provide a direct reflection of the power flow.

[0103] Finally, voltage constraint formulas can be applied to ensure the safe operation of the distribution network. A standard voltage is set as a reference value, and a certain percentage of voltage deviation is allowed. This constraint limits node voltage fluctuations to a reasonable range, ensuring that the distribution network operates at a safe and reliable voltage level. This prevents problems such as equipment damage and power quality degradation caused by excessively high or low voltage, thereby maintaining the safe and stable operation of the entire distribution network.

[0104] In the distribution network, photovoltaic power generation has a significant impact on node voltage, especially under strong sunlight conditions, where the increase in photovoltaic power generation output may lead to an increase in node voltage.

[0105] The optimization of voltage control and power flow, analysis of the impact of photovoltaic power generation on node voltage, and photovoltaic inverter control of node voltage by adjusting photovoltaic active power specifically include:

[0106] Setting Node The impedance is ,in, and Represents nodes respectively resistance and reactance, and set the node The reference voltage is ;

[0107] In a double busbar network, voltage drops The expression is:

[0108] When the photovoltaic active power When the voltage at the node increases, the node voltage may increase due to the power flow, which may cause the node voltage to exceed the safe range. In order to achieve effective voltage control, the photovoltaic inverter adjusts the photovoltaic active power. Voltage management: When the node voltage is higher than the upper limit, the photovoltaic inverter reduces the photovoltaic active power , reduce the node voltage; when the node voltage is lower than the lower limit, the photovoltaic inverter increases the photovoltaic active power , increase the node voltage, and the expression of the regulated photovoltaic active power is:

[0109] in, Indicates the regulated photovoltaic active power; Indicates the regulation coefficient of the node voltage;

[0110] By changing the active power output of the photovoltaic inverter, the power balance of the node can be affected, thereby changing the node voltage. When the active power output is reduced, the power injection of the node is reduced, and the voltage will drop accordingly; conversely, when the active power output is increased, the voltage will rise. And the active power loss in the distribution network The expression is:

[0111] in, and They represent the regulation coefficients of active load power and reactive load power, respectively, and are adjusted according to the operating status and control objectives of the power system; Represents the weight coefficient of the reference voltage, based on the sensitivity of the power system to voltage changes Adjust if sensitivity ,but ,on the contrary, ;

[0112] If the active power loss is minimized , then it is necessary to meet , which contradicts the goal of voltage control, therefore, it is necessary to balance the relationship between voltage safety and power loss.

[0113] In practical applications, the network parameters are first set to assign a specific impedance value to each node, which is composed of resistance and reactance. At the same time, the reference voltage of the node is determined as a benchmark for measuring voltage levels. In the dual-bus network scenario, the formula Describing the voltage drop phenomenon, this formula covers key factors such as line impedance and transmission power, and clearly presents the intrinsic connection between voltage changes and network electrical parameters.

[0114] During fluctuations in photovoltaic power generation, increases in photovoltaic active power can cause node voltages to deviate from their normal range. In this case, the photovoltaic inverter implements dynamic regulation based on the actual node voltage. If the node voltage exceeds the upper limit, the inverter proactively reduces photovoltaic active power output to mitigate the voltage increase. If the node voltage falls below the lower limit, the inverter promptly increases photovoltaic active power to bring the voltage back to a reasonable range. By introducing a node voltage regulation coefficient, the relationship between voltage fluctuations and photovoltaic active power regulation is quantified, ensuring a scientific, rational, accurate, and effective voltage regulation process.

[0115] The active power loss expression comprehensively considers factors such as active load power, reactive load power regulation coefficient, and reference voltage weight coefficient. The load power regulation coefficient is flexibly adjusted based on the power system's real-time operating status and control objectives to optimize power distribution. The reference voltage weight coefficient is closely related to the power system's sensitivity to voltage variations, thereby balancing voltage stability with power loss.

[0116] The optimal power flow problem is defined, the control device parameters of the power system are optimized, and the objective function is formulated to minimize the voltage deviation and power loss. Specifically, the following are involved:

[0117] Optimal power flow (OPF) generally refers to the process of adjusting and optimizing the parameters of various control devices in a power system. The corresponding objective function is to minimize the active power of the balancing bus, and the corresponding constraints are node power balance and various safety constraints. The expressions of the objective function and constraints are as follows:

[0118] in, Indicates the active power of the balancing bus; Indicates the bus index set equipped with the photovoltaic inverter; and Represent the lower and upper limits of the node voltage respectively; Indicates the maximum photovoltaic reactive power of the photovoltaic inverter;

[0119] Assuming that the capacity of each PV inverter is expanded to , Indicates the maximum node active power, then the maximum photovoltaic reactive power The expression is:

[0120] Regarding the voltage control process of the distribution network, special attention is paid to scenarios where voltage fluctuations caused by active power, reactive power, and photovoltaic power generation at the node exceed safety limits. The objective function of voltage control is formulated to minimize voltage deviation, active power loss, and current deviation by adjusting photovoltaic reactive power. The expression of the voltage control objective function is as follows:

[0121] in, Represents the voltage deviation of all nodes in the distribution network; represents the active power loss of all nodes; represents the current deviation of all nodes; Indicates the current standard; Indicates the balance weight coefficient corresponding to the voltage deviation; Indicates the balancing weight coefficient corresponding to the active power loss; Indicates the balance weight coefficient corresponding to the current deviation; The value range is 0 to 1.

[0122] It is understandable that the optimal power flow problem is defined first. This problem takes minimizing the active power of the balancing bus as its core goal, and this optimization goal is achieved by constructing an objective function. At the same time, in order to ensure the safety and reliability of system operation, a series of constraints are set. Among them, the node power balance constraint ensures the rational distribution of power among the nodes in the network; various safety constraints ensure that the system operates within a safe range from multiple dimensions such as voltage and power. Within the framework of this problem, the active power of the balancing bus is the focus of the objective function, and factors such as the bus where the photovoltaic inverter is located, the node voltage range, and the reactive power upper limit of the inverter are all key elements of the constraints, which together define the system operation state and ensure that the optimization process is carried out within the safety boundary.

[0123] When optimizing the control device parameters of the power system, it is necessary to consider the expansion of PV inverter capacity. As the capacity of each PV inverter increases to a certain level, the maximum node active power changes accordingly, which in turn affects the maximum PV reactive power.

[0124] Finally, a voltage control objective function can be formulated, aiming to comprehensively optimize the distribution network's operating indicators by regulating photovoltaic reactive power. This objective function takes voltage deviation, active power loss, and current deviation into account, minimizing these key indicators to improve the system's power supply quality and operational efficiency. Voltage deviation reflects the degree to which the actual node voltage deviates from its ideal value, active power loss reflects energy loss during power transmission, and current deviation measures the difference between the actual current value and the standard value. To balance the weights of various indicators during the optimization process, corresponding balancing weight coefficients are introduced. These coefficients range from 0 to 1 and can be flexibly adjusted according to actual operational requirements, making the objective function better adapted to different system operating scenarios. It seeks the optimal balance between reducing voltage deviation, minimizing power loss, and stabilizing current levels, thereby achieving efficient and stable operation of the distribution network.

[0125] The Markov game model is established to convert the active voltage control problem into a Markov game. The intelligent agent is determined to be a photovoltaic inverter, the environment is a distribution network, the state contains multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward, specifically including:

[0126] To improve the operating efficiency of the distribution network, the distribution network is divided into different areas. Each area is equipped with multiple photovoltaic modules. Each photovoltaic module contains a photovoltaic inverter. Each photovoltaic inverter serves as an intelligent agent, and each intelligent agent is located at a node in the distribution network. To enhance the security of the distribution network, photovoltaic modules within a region can share data observations related to that region.

[0127] Model the power distribution network as an environment and divide the environment into Regions, the environment is represented by ,The agent observes the node information in the corresponding area and makes a decision, estimating the voltage of each node, and determining the reward obtained by the agent;

[0128] The state space is defined as , the state space includes the states of all agents, a single agent The status of composition;

[0129] Set the continuous action set to , the continuous action set contains all possible actions of the agent. For a single agent , continuous action represents the proportion of the maximum photovoltaic reactive power generated by the photovoltaic inverter, and ,and and The following relationship exists:

[0130] in, Represents a connection node The apparent power of the photovoltaic inverter; when When the photovoltaic inverter sends power to the node Injecting photovoltaic reactive power, when When , the photovoltaic inverter absorbs photovoltaic reactive power; Indicates power factor The reciprocal of Indicates the power factor. When the power factor When it decreases, the photovoltaic reactive power increases; Indicates the loss coefficient of photovoltaic reactive power. When the photovoltaic reactive power is lost, more reactive power needs to be generated to compensate;

[0131] Constructing time steps The reward function , the corresponding expression is as follows:

[0132] in, represents the time weight factor; when When the power system is in the initial stage of startup, the voltage fluctuates greatly and the agent needs to adjust the voltage quickly. ,when When the power system is in the transition stage, the power system tends to be stable, with a small amount of voltage fluctuation. ,when When the power system is in a stable operation stage, the voltage fluctuation is small, and the setting ; represents the weight parameter used to balance the voltage deviation and photovoltaic reactive power loss; represents the voltage potential function; Represents photovoltaic reactive power generation loss; Indicates photovoltaic reactive power loss.

[0133] The voltage potential function The expression is as follows:

[0134] in, represents a hyperparameter used to tune the configuration of the voltage potential function;

[0135] The traditional potential function is generally or However, the former will lead to excessive generation of reactive power, while the latter, while avoiding waste, will react too slowly when the voltage deviation exceeds the safety limit. The defined potential function combines the advantages of both, enabling the agent to achieve the voltage control target more accurately.

[0136] Photovoltaic reactive power generation losses It shows that less reactive power generation leads to lower power loss, and the corresponding expression is as follows:

[0137] The objective function of establishing the Markov game model is:

[0138] in, Represents the discount factor.

[0139] First, agents can be defined and divided into regions. The distribution network can be divided into several regions based on their structural characteristics. Each region is equipped with multiple photovoltaic modules, and the photovoltaic inverter within each module is deployed as an independent agent on a network node. Agents within a region can share data observations, improving decision-making security and efficiency through collaborative mechanisms, forming a distributed network of intelligent control units.

[0140] Secondly, an environmental model can be constructed, modeling the entire power distribution network as a game environment and dividing it into multiple zones. Agents make decisions based on node information in their zones, such as voltage and power. After a decision is made, the model estimates the voltage changes at each node through power flow analysis and uses this information to determine the agent's reward, thereby providing quantitative feedback on the environmental state and decision effectiveness.

[0141] The definition of the state space encompasses key system operating parameters. The global state space is formed by aggregating the local states of all agents. The state of a single agent includes information such as active load, reactive load, photovoltaic active power output, and reactive power output from the previous step. This fully reflects the node power balance state and historical control behavior, providing a basis for current decision-making.

[0142] The action set is designed as a continuous control variable. The action of a single agent is expressed as a ratio of the inverter's maximum reactive power. The actual reactive power output is calculated based on the product of this ratio and the apparent power. A positive action value indicates reactive power injection, while a negative value indicates reactive power absorption. The effects of the inverse power factor and reactive loss coefficient are also considered, making the action definition more realistic and enabling refined reactive power regulation.

[0143] The reward function is constructed to balance voltage stability and power loss optimization. By introducing a time-weighted factor, the voltage control priority is dynamically adjusted according to the system's operating stage. A high weight is assigned during the initial startup phase to quickly suppress voltage fluctuations, while a lower weight is assigned during the stabilization phase to balance losses.

[0144] The voltage potential function combines the advantages of traditional potential functions, avoids the problems of excessive reactive power generation or slow response, and adapts to different scenarios through hyperparameter adjustment; the reactive power generation loss term can quantify the loss caused by reactive output, guiding the intelligent agent to take into account economy in voltage control.

[0145] The ultimate objective function is to maximize the cumulative expected reward. By using a discount factor to balance near-term and long-term rewards, the agent seeks the optimal control strategy over the long term, achieving coordinated optimization of voltage stability and system losses in the distribution network. This model, through the formalization of a Markov game, transforms the complex multi-agent collaborative control problem into a mathematical optimization problem, providing theoretical support and an algorithmic framework for voltage control in distribution networks.

[0146] like Figure 2 As shown in the figure, the multi-agent soft Actor-Critic algorithm with enhanced collaboration is introduced, and a centralized training and distributed execution framework is defined, integrating the self-attention mechanism, specifically including:

[0147] It adopts a centralized training and decentralized execution framework. During the training phase, each agent's critic evaluation network receives collective data from all agents, including state and action. During the execution phase, each agent makes independent decisions through the corresponding actor policy network.

[0148] Introducing the self-attention mechanism, based on the agent collaboration mechanism, initializes the states and actions of all agents, the critic evaluation network, and the parameters of the self-attention model;

[0149] At time step , agent The input of the Critic evaluation network The state of all agents and actions The corresponding expression for the aggregation is:

[0150] in, Representing an agent The encoding function of Representing an agent At time step Status; Representing an agent At time step Action; represents the neural network used to fit the evaluation function; Represents the self-attention input, that is, except for the agent The weighted sum of the encoded values ​​of all agents except The expression is:

[0151] in, Representing an agent At time step For intelligent agents The self-attention weight of Representing an agent At time step The encoded values ​​of the states and actions; Representing an agent The encoding function of Representing an agent At time step Status; Representing an agent At time step Action; Represents the activation function, that is, the RELU function; represents the linear transformation matrix; and represents the coding transformation matrix; Expressed as a natural constant e An exponential function with base .

[0152] like Figure 3 As shown, for each time step , for each agent , first initialize an empty self-attention input , then calculate the encoding value , for every not equal to , calculates the corresponding self-attention input, and finally calculates the input of the Critic evaluation network.

[0153] The parameters of the actor policy network and the critic evaluation network, the experience replay buffer, and the agent-related parameters are randomly initialized. When initializing the agent-related parameters, each photovoltaic inverter is modeled as a SoftActor-Critic (SAC) agent. The actor network is used to generate the agent's policy, and the critic network is used to evaluate the agent's action value, as well as entropy regularization.

[0154] For the range from 1 to For each round, get the initial state, for the range from 1 to At each time step, determine the action of the agent, perform the action and calculate the reward, determine the new state of the next time step, for the time from 1 to Each agent stores the transition information including the initial state, action, reward, and new state of the next time step in the experience replay buffer;

[0155] When the experience replay buffer reaches the preset area capacity, a group of agents are randomly selected for learning. , the parameters to be updated are expressed as ,in, and Represent the parameters of the Actor strategy network and the target Actor strategy network respectively, and Represent the parameters of the critic evaluation network and the target critic evaluation network, respectively. By minimizing the loss function L Updates are made to close the gap between the action-value function and the target value:

[0156] in, represents the batch size of randomly selected agents; represents the regularization coefficient; represents the action-value function; Represents the target value, including the time step The reward and the next time step Action value estimates; The temperature coefficient of entropy, used to balance reward and entropy; Represents the learning rate of the Critic evaluation network; Representing an agent At the next time step Action; Representing an agent At the next time step Status; Represents the agent in the Critic evaluation network A collection of actions; represents the mathematical expectation operator;

[0157] Parameters of the Actor Policy Network Through the gradient descent function To update:

[0158] in, Represents the agent in the Actor strategy network A collection of actions; Represents the learning rate of the Actor strategy network;

[0159] The target actor policy network and the target critic evaluation network are updated with parameters through the soft update method and synchronized with the online network:

[0160] in, Represents the tracking parameters, which are used to control the parameter update rate of the target Actor policy network and the target Critic evaluation network;

[0161] After the training phase, each agent completes its parameter update, fixing the parameters of its actor policy network. During the testing phase, each agent makes independent decisions based on its state, without requiring real-time data from other agents. This decentralized execution eliminates the need for agents to access the global state or actions of other agents. Each agent makes decisions based solely on its state.

[0162] Specifically, we can first build a centralized training and decentralized execution framework. During the training phase, the Critic evaluation network of each intelligent agent receives the status and action data of all intelligent agents to achieve global information fusion; during the execution phase, the intelligent agent only makes independent decisions through its own Actor strategy network to ensure the real-time performance of the power system.

[0163] When introducing the self-attention mechanism, we first initialize the states and actions of all agents, the critic network, and the self-attention model parameters. The input to the agent critic network is the aggregated state and actions of all agents, while the self-attention input is the weighted sum of the encoding values ​​of other agents. By comparing the encoding values ​​between agents, a softmax function is used to generate self-attention weights, allowing agents to focus on information that significantly impacts their own rewards, thereby enhancing the efficiency of collaboration between agents.

[0164] Then, the actor policy network, critic evaluation network parameters, experience replay buffer, and agent-related parameters are randomly initialized. Each photovoltaic inverter is modeled as a SAC agent. The actor network generates a policy, and the critic network evaluates the value of actions. Entropy regularization is combined to enhance policy exploratory power. During training, the initial state is obtained for each round. At each time step, the agent action is determined, the reward is calculated, and the state is updated. The transition information is stored in the experience replay buffer.

[0165] After the buffer is full, batches of experience are randomly sampled for learning. The critic evaluation network updates its parameters by minimizing the loss function, which includes the current reward and the estimated value of the action at the next moment, thereby taking into account both immediate feedback and long-term benefits. The actor policy network uses gradient descent optimization to improve the generation quality of the voltage control policy.

[0166] The target actor policy network and target critic evaluation network utilize a soft update mechanism, tracking parameters and synchronizing them with the online network to avoid drastic parameter fluctuations during training. After training, the actor policy network parameters are fixed. During the testing phase, the agent makes independent decisions based solely on local state, eliminating the need for global information exchange. This reduces communication costs while ensuring the real-time and reliable nature of the control strategy. This algorithm integrates global knowledge through centralized training, adapts to distributed scenarios through decentralized execution, and strengthens collaborative relationships between agents through a self-attention mechanism. This effectively addresses the voltage collaborative control challenges associated with photovoltaic integration in distribution networks, improving the voltage regulation accuracy and stability of the distribution network.

[0167] like Figure 4 As shown, the voltage control strategy is implemented. At the beginning of each round, the photovoltaic and load configuration profiles are stored. At each time step, the photovoltaic and load curves and states are integrated to determine the subsequent state and reward. The states are divided and sorted according to the corresponding areas of the agent. After receiving the states and rewards of the corresponding areas, the agent makes a decision, which specifically includes:

[0168] At the beginning of each round, the PV and load configuration profiles are stored in the experience replay buffer;

[0169] At each time step, the corresponding PV and load curves are retrieved, the state is calculated and the reward is determined by Pandapower, the PV and load curves and the state are integrated and the subsequent state is established;

[0170] Before sending rewards to agents, split the states of all agents into multiple state batches and sort the state batches according to the corresponding regions of the agents;

[0171] The agent makes a decision after receiving the status and reward of the corresponding area, and repeats the above process until the end of the entire round.

[0172] When implementing the voltage control strategy, at the beginning of each round, the continuous photovoltaic and load configuration profiles are stored in the experience replay buffer, providing basic data support for subsequent control. At each time step, the corresponding photovoltaic output curve and load change curve are retrieved. Simultaneously, the current system state is calculated using the Pandapower tool and the corresponding reward value is determined. Subsequently, the photovoltaic and load curves are integrated with the current state information to construct the subsequent state, providing complete system operation information for the agent's decision-making.

[0173] Before sending rewards to agents, the states of all agents are preprocessed and divided into multiple state batches. These batches are then sorted by the agent's region to ensure that the state information accurately corresponds to the agent's control region. After this processing, the agent receives the state information for its corresponding region and the global reward. Based on its trained strategy, it makes decisions to adjust the reactive power output of the photovoltaic inverter to achieve voltage control in its region. This cycle repeats until the end of the round, completing a full cycle of voltage control. This strategy ensures efficient and accurate voltage control through orderly data management and regionalized decision-making by agents.

[0174] like Figure 5 FIG. 1 is a system block diagram of a distribution network voltage control system based on multi-agent deep reinforcement learning provided by an embodiment of the present invention. The system includes:

[0175] A construction module for constructing distribution networks based on graph theory and describing their dynamic behavior using power flow balance equations and voltage constraint formulas;

[0176] The optimization module is used to optimize voltage control and power flow, analyze the impact of photovoltaic power generation on node voltage, and control the node voltage by adjusting the photovoltaic active power of the photovoltaic inverter. It also defines the optimal power flow problem, optimizes the control device parameters of the power system, and formulates the objective function to minimize voltage deviation and power loss.

[0177] The conversion module is used to establish a Markov game model, converting the active voltage control problem into a Markov game. The intelligent agent is the photovoltaic inverter, the environment is the distribution network, the state contains multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward.

[0178] Introducing a module for introducing a collaborative multi-agent soft actor-critic algorithm, defining a centralized training and decentralized execution framework, and integrating a self-attention mechanism;

[0179] The implementation module is used to implement the voltage control strategy. At the beginning of each round, the photovoltaic and load configuration profiles are stored. At each time step, the photovoltaic and load curves and states are integrated to determine the subsequent state and reward. The states are then divided and sorted according to the corresponding areas of the agent. The agent makes a decision after receiving the state and reward of the corresponding area.

[0180] Figure 5 The apparatus of the embodiment shown can be used to perform Figure 1 The implementation principles and technical effects of the steps in the method embodiment shown are similar and will not be repeated here.

[0181] An electronic device comprises a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the steps of a distribution network voltage control method based on multi-agent deep reinforcement learning as described above.

[0182] like Figure 6 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. The electronic device 60 includes: a processor 61, a memory 62 and a computer program;

[0183] The memory 62 is used to store the computer program, which may also be a flash memory. The computer program is, for example, an application program or a functional module for implementing the above method.

[0184] The processor 61 is configured to execute the computer program stored in the memory to implement the various steps performed by the device in the above method. For details, please refer to the relevant description in the above method embodiment.

[0185] Optionally, the memory 62 may be independent or integrated with the processor 61 .

[0186] When the memory 62 is a device independent of the processor 61, the device may further include:

[0187] The bus 63 is used to connect the memory 62 and the processor 61 .

[0188] A readable storage medium having a computer program stored therein, wherein the computer program, when executed by a processor, is used to implement the steps of a distribution network voltage control method based on multi-agent deep reinforcement learning as described in any one of the above.

[0189] The readable storage medium may be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transfer of computer programs from one location to another. Computer storage media may be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium may also be an integral part of the processor. The processor and the readable storage medium may be located in an application-specific integrated circuit (ASIC). In addition, the ASIC may be located in a user device. Of course, the processor and the readable storage medium may also exist as discrete components in a communication device. The readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0190] The present invention also provides a program product, which includes execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the methods provided in the various embodiments described above.

[0191] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the present invention may be directly executed by a hardware processor or by a combination of hardware and software modules within the processor.

[0192] Through the introduction of the above embodiments, the present invention can combine multi-agent reinforcement learning technology with the self-attention mechanism through a distribution network voltage control method based on multi-agent deep reinforcement learning, effectively deal with the voltage fluctuation problem caused by the integration of photovoltaic systems in the distribution network, and thus achieve stable control and optimized management of voltage. The present invention can divide the distribution network into multiple regions, and then centrally manage and optimize the voltage of each region; then, the collaborative voltage control problem between regions can be transformed into a Markov game to facilitate effective collaboration between agents; further, a centralized training and decentralized execution framework can be adopted, so that the agent can fully utilize global information in the training phase, and can independently make real-time decisions based on local observations in the execution phase; finally, the self-attention mechanism can be integrated so that the agent can focus more on information related to its own rewards, thereby improving the voltage control effect of the distribution network.

[0193] This method overcomes the limitations of traditional voltage control methods under rapidly fluctuating conditions, improving the voltage regulation accuracy and response speed of distribution networks while ensuring the stability and reliability of the power system. It also offers significant advantages in computational efficiency and cost, making it suitable for distribution networks with high photovoltaic penetration rates, as well as for smart grids requiring efficient voltage management.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distribution network voltage control method based on multi-agent deep reinforcement learning, characterized in that: The method comprises: The distribution network is constructed based on graph theory, and the power flow balance equation and voltage constraint formula are used to describe the dynamic behavior of the distribution network; Optimize voltage control and power flow, analyze the impact of photovoltaic power generation on node voltage, photovoltaic inverter controls node voltage by adjusting photovoltaic active power, define the optimal power flow problem, optimize the control device parameters of the power system, and formulate the objective function to minimize voltage deviation and power loss; A Markov game model is established to transform the active voltage control problem into a Markov game. The agent is the photovoltaic inverter, the environment is the distribution network, the state contains various power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward. Introducing a multi-agent soft actor-critic algorithm that enhances collaboration, defining a centralized training and decentralized execution framework, and integrating a self-attention mechanism; Implement a voltage control strategy. At the beginning of each round, store the PV and load configuration profiles. At each time step, integrate the PV and load curves and states to determine the subsequent state and reward. Then, split and sort the states according to the corresponding areas of the agent. The agent makes a decision after receiving the state and reward of the corresponding area. The optimal power flow problem is defined, the control device parameters of the power system are optimized, and the objective function is formulated to minimize the voltage deviation and power loss. Specifically, the following are involved: The optimal power flow problem is defined. The corresponding objective function is to minimize the active power of the balanced bus. The corresponding constraints are node power balance and various safety constraints. The expressions of the objective function and constraints are as follows: Where P0 represents the active power of the balancing bus; V PV Represents the bus index set equipped by the photovoltaic inverter; V min and V max Represent the lower and upper limits of the node voltage respectively; Indicates the maximum photovoltaic reactive power of the photovoltaic inverter; Indicates photovoltaic active power; Indicates active load power; V i represents the node voltage of node i, i∈N, N=(0,1,...,n) represents the set of all nodes in the distribution network; V j represents the voltage of node j, j∈N; G ij and A ij denote the admittance and reactance of the line between nodes i and j respectively; θ ij represents the phase angle difference between nodes i and j, θ ij =θ i -θ j ; Represents photovoltaic reactive power, Indicates reactive load power; V0 indicates voltage standard; Assume that the capacity of each PV inverter is expanded to F i =1.2P i,max , P i,max Indicates the maximum node active power, then the maximum photovoltaic reactive power The expression is: The objective function of voltage control is formulated, that is, by adjusting the photovoltaic reactive power, minimizing the voltage deviation, active power loss and current deviation. The expression of the objective function of voltage control is as follows: in, Represents the voltage deviation of all nodes in the distribution network; represents the active power loss of all nodes; represents the current deviation of all nodes; I0 represents the current standard; τ represents the balancing weight coefficient corresponding to the voltage deviation; δ represents the balancing weight coefficient corresponding to the active power loss; Indicates the balance weight coefficient corresponding to the current deviation; τ, δ, The value range is 0 to 1.

2. A distribution network voltage control method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The distribution network is constructed based on graph theory, and the dynamic behavior of the distribution network is described using power flow balance equations and voltage constraint formulas, specifically including: The topology of the distribution network is established and the distribution network is modeled as a radial network. The radial network contains n+1 nodes, where N=(0,1,...,n) represents the set of all nodes in the distribution network. The transformer is located at node 0. For node i, i∈N, the node voltage and phase angle are V i and θ i ; Define the voltage and power characteristics of the node. For node i, the input node active power is in, Indicates the photovoltaic active power, represents the active load power, and the input node reactive power is in, Represents photovoltaic reactive power, Indicates reactive load power; The power flow balance equation is used to control each node to be in a power balance state. The expression of the power flow balance equation is as follows: B i =P i +jQ i V i -V j =Z ij I ij Among them, G ij and A ij denote the admittance and reactance of the line between nodes i and j respectively; θ ij represents the phase angle difference between nodes i and j, θ ij =θ i -θ j ; B i represents the complex power injection of node i; Z ij The impedance Z represents the impedance of the line between node i and node j. ij By the resistor R ij and reactance A ij Composition, namely Z ij =R ij +jA ij ;I ij represents the current flowing from node i to node j; V j represents the voltage of node j, j∈N; Apply the voltage constraint formula, set the voltage standard to V0 = 1.0pu, and the allowed voltage deviation to 5%, then the voltage V i The following conditions are met: 0.95pu <Vi<1.05p.u.。 3. The method for controlling voltage in a power distribution network based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The optimization of voltage control and power flow, analysis of the impact of photovoltaic power generation on node voltage, and photovoltaic inverter control of node voltage by adjusting photovoltaic active power specifically include: Assume the impedance of node i is Z i =R i +jA i , where R i and A i Represent the resistance and reactance of node i respectively, and set the reference voltage of node i to V ip ; In a double busbar network, the voltage drop ΔV i The expression is: When the photovoltaic active power When the photovoltaic inverter increases, it adjusts the photovoltaic active power Voltage management: When the node voltage is higher than the upper limit, the photovoltaic inverter reduces the photovoltaic active power Reduce the node voltage; when the node voltage is lower than the lower limit, the photovoltaic inverter increases the photovoltaic active power By increasing the node voltage, the expression of the regulated photovoltaic active power is: in, represents the regulated PV active power; κ represents the regulation coefficient of the node voltage; And the active power loss P in the distribution network l The expression is: Wherein, k1 and k2 represent the adjustment coefficients of active load power and reactive load power, respectively, which are adjusted according to the operating status and control objectives of the power system; η represents the weight coefficient of the reference voltage; If the active power loss P is minimized l , then it is necessary to meet 4. The method for controlling voltage in a power distribution network based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The Markov game model is established to convert the active voltage control problem into a Markov game. The intelligent agent is determined to be a photovoltaic inverter, the environment is a distribution network, the state contains multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward, specifically including: The distribution network is divided into different areas, each area is equipped with multiple photovoltaic modules, each photovoltaic module contains a photovoltaic inverter, each photovoltaic inverter is an intelligent agent, and each intelligent agent is located at a node in the distribution network; The distribution network is modeled as an environment and divided into M regions, represented by E = {0, 1, ..., M}. The agent observes the node information in the corresponding region and makes decisions, estimating the voltage of each node and determining the reward received by the agent. The state space is defined as S = {P L ,Q L ,P PV ,Q PV }, the state space includes the states of all agents, and the state of a single agent i is represented by composition; Set the continuous action set to A={a1,...,a i }, the continuous action set contains all possible actions of the agent, for a single agent i, the continuous action a i represents the proportion of the maximum photovoltaic reactive power generated by the photovoltaic inverter, and - 1<a i <1, and and The following relationship exists: in, Represents the apparent power of the photovoltaic inverter connected to node i; when a i > 0, the photovoltaic inverter injects photovoltaic reactive power to node i. i When <0, the photovoltaic inverter absorbs photovoltaic reactive power; Indicates power factor The reciprocal of Indicates the power factor. When the power factor When it decreases, the photovoltaic reactive power increases; k Q Indicates the loss coefficient of photovoltaic reactive power. When k Q When >0, there is loss of photovoltaic reactive power; Construct the reward function r at time step t t , the corresponding expression is as follows: Among them, ω t Represents the time weight factor; when 0<t<t1, the power system is in the initial stage of startup and the voltage fluctuation is large, set ω t =1.5, when t1<t<t2, the power system is in the transition stage, the power system tends to be stable, and there is a small amount of voltage fluctuation. Set ω t =1.2, when t>t2, the power system is in a stable operation stage, the voltage fluctuation is small, and ω is set t =1.2; Φ represents the weight parameter used to balance voltage deviation and photovoltaic reactive power loss; x v (V i ) represents the voltage potential function; Represents photovoltaic reactive power generation loss; Indicates photovoltaic reactive power loss.

5. A distribution network voltage control method based on multi-agent deep reinforcement learning according to claim 4, characterized in that: The voltage potential function x v (V i ) is as follows: Where a, b, c, and d represent hyperparameters used to adjust the configuration of the voltage potential function; Photovoltaic reactive power generation losses The expression is as follows: The objective function of establishing the Markov game model is: Here, γ represents the discount factor.

6. The method for controlling voltage in a power distribution network based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The proposed method introduces a multi-agent soft actor-critic algorithm with enhanced collaboration, defines a centralized training and distributed execution framework, and integrates a self-attention mechanism. Specifically, it includes: It adopts a centralized training and decentralized execution framework. During the training phase, each agent's critic evaluation network receives collective data from all agents, including state and action. During the execution phase, each agent makes independent decisions through the corresponding actor policy network. Introducing the self-attention mechanism, based on the agent collaboration mechanism, initializes the states and actions of all agents, the critic evaluation network, and the parameters of the self-attention model; At time step t, the input O of the Critic evaluation network of agent i is i is the state S of all agents t and action A t The corresponding expression for the aggregation is: Among them, f i () represents the encoding function of agent i; represents the state of agent i at time step t; represents the action of agent i at time step t; g i () represents the neural network used to fit the evaluation function; represents the self-attention input, which is the weighted sum of the encoding values ​​of all agents except agent i. The expression is: in, represents the self-attention weight of agent i on agent j at time step t; represents the encoded value of the state and action of agent j at time step t; f j () represents the encoding function of agent j; represents the state of agent j at time step t; Represents the action of agent j at time step t; σ() represents the activation function, that is, the RELU function; T represents the linear transformation matrix; W1 and W2 represent the encoding transformation matrix; exp() represents the exponential function with the natural constant e as the base.

7. A distribution network voltage control method based on multi-agent deep reinforcement learning according to claim 6, characterized in that: Randomly initialize the parameters of the Actor strategy network and the Critic evaluation network, the experience replay buffer, and agent-related parameters; For each round from 1 to K, get the initial state, for each time step from 1 to T, determine the agent's action, perform the action and calculate the reward, determine the new state for the next time step, and for each agent from 1 to N, store the transition information containing the initial state, action, reward, and new state for the next time step in the experience replay buffer; When the experience replay buffer reaches the preset area capacity, a batch of agents are randomly selected for learning. For agent i, the parameters to be updated are expressed as θ i ={θ Actor,i ,θ' Actor,i ,θ Critic,i ,θ' Critic,i }, where θ Actor,i and θ' Actor,i Denote the parameters of the Actor strategy network and the target Actor strategy network, θ Critic,i and θ' Critic,i Represent the parameters of the critic evaluation network and the target critic evaluation network, respectively. The parameters of the critic evaluation network θ Critic,i Update by minimizing the loss function L: Where H is the batch size of randomly drawn agents; ξ is the regularization coefficient; U is the action-value function; y is the target value, which includes the reward at time step t and the action-value estimate at the next time step t+1; β is the temperature coefficient of entropy, which is used to balance reward and entropy. Represents the learning rate of the Critic evaluation network; represents the action of agent i at the next time step t+1; represents the state of agent i at the next time step t+1; represents the action set of agent i in the Critic evaluation network; E represents the mathematical expectation operator; Parameters θ of the Actor policy network Actor,i Update by gradient descent function J: in, represents the action set of agent i in the Actor strategy network; Represents the learning rate of the Actor strategy network; The target actor policy network and the target critic evaluation network are updated with parameters through the soft update method and synchronized with the online network: I will Actor,i =tθ Actor,i +(1-τ)θ' Actor,i I will Critic,i =tθ Critic,i +(1-τ)θ' Critic,i Among them, τ represents the tracking parameter, which is used to control the parameter update rate of the target actor policy network and the target critic evaluation network; After each agent completes the parameter update, the parameters of the Actor strategy network of each agent are fixed, and each agent makes independent decisions based on the corresponding state based on the Actor strategy network.

8. The method for controlling voltage in a power distribution network based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The voltage control strategy is implemented by storing the photovoltaic and load configuration profiles at the beginning of each round. At each time step, the photovoltaic and load curves and states are integrated to determine the subsequent state and reward. The states are then divided and sorted according to the corresponding areas of the agent. After receiving the states and rewards of the corresponding areas, the agent makes a decision, specifically including: At the beginning of each round, the PV and load configuration profiles are stored in the experience replay buffer; At each time step, the corresponding PV and load curves are retrieved, the state is calculated and the reward is determined by Pandapower, the PV and load curves and the state are integrated and the subsequent state is established; Before sending rewards to agents, split the states of all agents into multiple state batches and sort the state batches according to the corresponding regions of the agents; The agent makes a decision after receiving the status and reward of the corresponding area, and repeats the above process until the end of the entire round.

9. A distribution network voltage control system based on multi-agent deep reinforcement learning, applied to a distribution network voltage control method based on multi-agent deep reinforcement learning as claimed in any one of claims 1 to 8, characterized in that: The system comprises: A construction module for constructing distribution networks based on graph theory and describing their dynamic behavior using power flow balance equations and voltage constraint formulas; The optimization module is used to optimize voltage control and power flow, analyze the impact of photovoltaic power generation on node voltage, and control the node voltage by adjusting the photovoltaic active power of the photovoltaic inverter. It also defines the optimal power flow problem, optimizes the control device parameters of the power system, and formulates the objective function to minimize voltage deviation and power loss. The conversion module is used to establish a Markov game model, converting the active voltage control problem into a Markov game. The intelligent agent is the photovoltaic inverter, the environment is the distribution network, the state contains multiple power information, the action is the reactive power ratio of the photovoltaic inverter, the reward includes voltage and power loss, and the goal is to maximize the cumulative expected reward. Introducing a module for introducing a collaborative multi-agent soft actor-critic algorithm, defining a centralized training and decentralized execution framework, and integrating a self-attention mechanism; The implementation module is used to implement the voltage control strategy. At the beginning of each round, the photovoltaic and load configuration profiles are stored. At each time step, the photovoltaic and load curves and states are integrated to determine the subsequent state and reward. The states are then divided and sorted according to the corresponding areas of the agent. The agent makes a decision after receiving the state and reward of the corresponding area.

Images