Voltage reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning

Abstract

The invention discloses a voltage reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning, which is suitable for voltage stability and reactive power optimization control in a power distribution network. According to the method, through a multi-agent reinforcement learning algorithm and in combination with heterogeneous equipment such as a voltage regulator, a switchable capacitor and a battery, dynamic adjustment of the voltage of the power distribution network and cooperative control of reactive power are achieved. The multi-agent continuously optimizes the control strategy through the interaction with the dynamic environment, so as to maximize the voltage stability of the power distribution network, minimize the reactive power loss and reduce the equipment operation cost. During actual operation, the multiple agents make decisions autonomously according to the system state monitored in real time, coordinated control over the voltage regulator, the switchable capacitor and the battery is achieved, and the stability of the voltage and reactive power of the power distribution network in the complex operation environment is guaranteed. The method has good adaptivity and expandability, can effectively cope with challenges such as load change and equipment state fluctuation in the power grid, and improves the operation efficiency and safety of the power distribution network.

Description

Technical Field

[0001] The present invention relates to the technical field of power system control, and in particular to a voltage reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning. Background Art

[0002] Modern power grids face challenges due to the wide range of power infrastructure components and the variety of equipment types. These challenges are exacerbated by the increasing integration of renewable energy sources, which can introduce unstable voltage fluctuations. In addition, as the grid expands, it becomes increasingly important to maintain more stable voltage levels and minimize power losses. To address these issues, researchers proposed a reinforcement learning-based voltage and reactive power control method that leverages the adaptive learning capabilities of intelligent agents to coordinate the operation of heterogeneous devices.

[0003] However, existing reinforcement learning methods usually assume that all agents are homogeneous, which severely limits their applicability in real power grids. There are many different types of equipment in the power grid, such as voltage regulators, capacitor banks, and batteries, which have different action spaces and physical constraints and require coordinated control strategies. In addition, existing algorithms usually cannot guarantee monotonic performance improvement during training, which may lead to uncertainty in system performance and potential safety hazards in real-time operation. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to propose a voltage and reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning, which can effectively improve the performance of voltage and reactive power control, enhance the adaptability of distribution network to heterogeneous equipment, and provide an effective solution for the optimized operation of distribution network.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A voltage and reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning is proposed. The heterogeneous devices in the distribution network are controlled by a multi-agent reinforcement learning algorithm to achieve dynamic optimization control of voltage and reactive power. The heterogeneous devices include voltage regulators, switchable capacitors and batteries. The specific steps are as follows:

[0007] Modeling of heterogeneous multi-agent state space and action space: A heterogeneous device is modeled as an agent, and a collection of multiple different heterogeneous devices is modeled as a heterogeneous multi-agent. The voltage and reactive power control problem of the distribution network is modeled as a heterogeneous multi-agent reinforcement learning problem. The state space of each node and the action space of the heterogeneous device are defined. The state space of each node includes the node voltage, the operating status of the heterogeneous device, and the load demand. The action space of the heterogeneous device specifically includes the gear state of the voltage regulator, the switch state of the switchable capacitor, and the charge and discharge state of the battery.

[0008] Design reward function according to control objectives: The overall goal of voltage and reactive power control is to maintain voltage stability and reduce reactive power loss in the distribution network by adjusting voltage regulators, switchable capacitors and batteries, while meeting the operation constraints of heterogeneous equipment; design reward function based on the overall goal of voltage and reactive power control, comprehensively consider the penalties for voltage violations, reactive power loss and control errors, and introduce Lagrange multipliers to handle constraints to ensure that the agent can effectively optimize the voltage and reactive power control strategy during training and ensure that the distribution network meets physical and operational constraints;

[0009] Voltage-reactive power sensitivity analysis: Based on the voltage-reactive power sensitivity matrix, calculate the impact of each node on the voltage and reactive power of the distribution network, and dynamically adjust the control priority of each node;

[0010] Determine the update order of multiple agents: Based on voltage and reactive power sensitivity, select nodes with greater impact on the distribution network and update their control strategies first to improve the response speed and control accuracy of the distribution network;

[0011] Equipment collaborative control: During the operation of the distribution network, the voltage regulator controls the node voltage by adjusting the gear position; the switchable capacitor performs reactive power compensation according to demand; the battery adjusts reactive power by charging and discharging to achieve reactive power balance and voltage stability in the distribution network;

[0012] Strategy update and optimization: Continuously calculate the update order of multi-agents, and update the multi-agent strategy network according to the update order through iterative training of the multi-agent reinforcement learning algorithm, so as to achieve optimal control of heterogeneous devices under different operating conditions and improve the voltage and reactive power control effect of the distribution network;

[0013] Real-time control and feedback adjustment: In actual operation, each intelligent agent makes autonomous decisions based on the real-time observed state of the distribution network, realizes dynamic control of voltage and reactive power, and adjusts future control strategies based on the actual control effect.

[0014] Preferably, the multi-agent reinforcement learning algorithm adopts a proximal strategy optimization algorithm with monotonic improvement characteristics, ensuring that the voltage and reactive power control effect of the distribution network is continuously optimized after each strategy update, avoiding the instability of the distribution network caused by immature strategies. The collaborative control strategy of heterogeneous devices is optimized through centralized training and distributed execution.

[0015] Preferably, the voltage-reactive power sensitivity matrix is ​​used to evaluate the sensitivity of each node to voltage and reactive power, and the voltage-reactive power sensitivity matrix is ​​dynamically updated according to the voltage fluctuation and load demand of the node to ensure the flexibility and accuracy of the control strategy.

[0016] Preferably, the voltage regulator controls the node voltage by adjusting the gear position, so that the distribution network voltage is maintained within a predetermined range and voltage fluctuations are reduced.

[0017] Preferably, the switchable capacitor performs switching operation according to reactive power demand of the distribution network, and provides reactive power compensation to maintain reactive power balance of the distribution network.

[0018] Preferably, the battery adjusts reactive power output by controlling the charging and discharging state to assist the voltage regulator and the switchable capacitor in achieving reactive balance of the distribution network.

[0019] The present invention proposes a voltage reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning, which has the following beneficial effects:

[0020] Modeling the voltage and reactive power regulation problem as a heterogeneous multi-agent reinforcement learning problem can effectively describe the dynamic characteristics of various heterogeneous devices in the distribution network and improve the control performance.

[0021] The voltage-reactive power sensitivity matrix is ​​introduced to give priority to updating the intelligent agents with higher sensitivity at the nodes, thus enhancing the learning ability of the dynamic characteristics of the distribution network.

[0022] A policy-based multi-agent reinforcement learning algorithm is used to train the agents, enabling them to simultaneously control multiple heterogeneous device types, thereby improving the adaptability and robustness of the distribution network.

[0023] During the training process, the performance indicators of the multi-agent reinforcement learning algorithm can be improved monotonically and eventually converge to the optimal level, achieving autonomous optimization.

[0024] This method can effectively improve the performance of voltage and reactive power control, reduce power loss, and provide an effective solution for the optimized operation of distribution networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is the workflow diagram of the voltage and reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning. DETAILED DESCRIPTION

[0026] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] like Figure 1 As shown, the present invention provides a voltage reactive power control method based on constraint heterogeneous ordered multi-agent reinforcement learning, which is mainly used in distribution networks to optimize voltage reactive power through coordinated control of heterogeneous devices such as voltage regulators, switchable capacitors and batteries. The following are the specific implementation steps:

[0028] Step S1: Modeling heterogeneous multi-agent state space and action space

[0029] When modeling the voltage and reactive power regulation problem as a heterogeneous multi-agent reinforcement learning problem, it is necessary to define a state space containing various observations. These observations include node bus voltage, voltage regulator tap state, switchable capacitor state, battery pack load state, and battery pack discharge power. Each state can be represented as a binary vector or a continuous state normalized to a bounded range depending on the device type. The node bus voltage is represented using the reference voltage unit on the node bus. Therefore, the state space can be represented as a set containing the above various observations as follows:

[0030]

[0031] s=(s1,s2,s3,s4,s5)

[0032] in, represents the bus voltage of the i-th node, and N represents the number of nodes; Indicates the tap state of the R i voltage regulator, N reg Indicates the number of voltage regulator taps; represents the state of the Ci-th switchable capacitor, N cap represents the number of switchable capacitors; Indicates the load status of the Bi-th battery pack, Indicates the discharge power of the Bith battery pack, N bat Represents the number of battery packs; Concats means concatenating the state vectors.

[0033] The action space of heterogeneous devices is defined as the set of control variables that can affect the state of the distribution network within certain limits. In the distribution network, the action space is the state of each node and the control variables of the heterogeneous devices. Specifically, the control of each switchable capacitor is determined by a switch variable a C ∈{0,1} represents. For a voltage regulator, each tap has a controllable on / off state, so the action space of the voltage regulator is a binary vector a R , whose dimensions range from 0 to N tap Where N tap is the number of taps, the default value is 19. The battery action is discretized into a bat A binary vector of dimension Similar to a voltage regulator, N bat The default value of is also set to 19, where the first 10 dimensions represent charging and the rest represent discharging.

[0034] Step S2: Design reward function according to control objectives

[0035] The control goal of this method is to maintain the voltage within the specified range and avoid voltage violations. It also optimizes reactive power compensation and reduces power loss in the distribution network system.

[0036] The present invention adopts a multi-agent reinforcement learning algorithm. Each heterogeneous device, including voltage regulators, switchable capacitors, and batteries, is controlled by an independent agent. The goal of each agent is to optimize its own control strategy so that it can maximize the overall effect of voltage and reactive power control while cooperating. The proximal strategy optimization algorithm is used for centralized training, and the distributed execution framework enables each agent to make decisions under local observation.

[0037] The reward function is set in combination with the overall goal of voltage and reactive power control, aiming to reduce the combined impact of voltage violations, control errors, and power losses. The reward function is as follows:

[0038]

[0039] Among them, r(s t ,a t ) means in state s t Take action a t Rewards, V viol is the voltage violation penalty term, P loss is the power loss penalty term, C error is the control error penalty term, α, β and γ are the weight parameters corresponding to each penalty term, which are used to balance the influence of each part in the total reward, constraint k Indicates the degree of violation of the kth constraint.

[0040] Voltage violation penalty V viol It is used to measure whether the node voltage exceeds the allowable range. Specifically, it calculates each node voltage v i Exceeding the maximum voltage V max or below the minimum voltage V min To determine the extent of:

[0041]

[0042] Where N is the set of all nodes in the network.

[0043] Power loss penalty term P loss It is used to evaluate the power loss in the distribution network. The calculation method is the power loss on all edges (i, j) and the total injected power P ij Ratio of:

[0044]

[0045] Among them, G ij represents the conductivity of edge (i,j), vi and v j are the voltages at nodes i and j, respectively, and θ ij is the phase angle between node i and node j, P ij is the total power injected into the distribution network, and E is the set of all edges in the network.

[0046] Control error penalty term C error It is used to measure the error between the control device action and the expected state. It is determined by calculating the absolute difference between the states of each controllable node f of the heterogeneous device at time t and t+1, and taking the weighted sum:

[0047]

[0048] Among them, N a represents a set of nodes with action space, s f (t) and s f (t+1) are the states of the controllable node f of the heterogeneous device at time t and t+1, respectively, f is the weight parameter for controlling the error.

[0049] The constraint penalty part is used to ensure that the distribution network meets various physical and operational constraints in actual operation. k , each constraint k The degree of violation is incorporated into the reward function to increase the penalty for violating the constraint. The mathematical expression of the constraint penalty is as follows:

[0050]

[0051] Among them, λ k Represents the weight parameter associated with the kth constraint, which is used to adjust the penalty for violations of different constraints.

[0052] Step S3: Voltage-reactive power sensitivity analysis

[0053] After the distribution network is initialized, the impact of each node and heterogeneous device on the voltage and reactive power of the distribution network is calculated. The specific steps are as follows:

[0054] 1. Calculate the voltage-reactive sensitivity of the node

[0055] First, the relationship between the active P and reactive Q power of the node and the voltage V and phase angle θ is calculated by the following equations:

[0056]

[0057] in, is the Jacobian matrix calculated from the impedance between nodes, where J QθIt represents the partial derivative of reactive power Q with respect to phase angle θ, that is, The same applies to other cases. VQ is the voltage-reactive sensitivity matrix, which is used to evaluate the voltage stability of the node.

[0058] Next, the voltage-reactive sensitivity of each node during the iteration is calculated:

[0059]

[0060] Among them, S VQ,t (i) represents the voltage-reactive sensitivity of node i at time t, and N is the total number of nodes in the network.

[0061] 2. Calculate the weight w VQ,t

[0062] Calculate the weight w based on the ratio of each step reward to the total reward VQ,t :

[0063]

[0064] Among them, w VQ,t represents the weight at time t, r t is the reward at step t, r H is the total reward and H is the number of iterations.

[0065] 3. Calculate agent priority ranking

[0066]

[0067] Among them, O(i) represents the priority ranking of node i, It is used to shield non-agent nodes. Sort indicates the sorting operation. VQ,t is the weight, S VQ,t (i) is the voltage-reactive sensitivity of node i at time t.

[0068] Step S4: Multi-agent strategy training and updating

[0069] 1. Initialization of the agent's policy network: Each agent is given an initial policy network, which is built based on a predefined state space and action space. The initial policy may be generated based on historical data or random policies to provide a starting point for subsequent training.

[0070] 2. Environment interaction and data collection: Each heterogeneous device executes its strategy and collects state transition, reward and action data.

[0071] 3. Calculate the update order: Determine the update priority of the agent based on the results of the voltage-reactive power sensitivity analysis.

[0072] 4. Iterative update of policy network: A reinforcement learning algorithm with proximal policy optimization is used to iteratively update the policy network of the multi-agent step by step based on the collected data and the calculated update sequence. In each iteration, the multi-agent updates its policy network to optimize future control decisions based on the feedback obtained from the environment.

[0073] Step S5: Real-time control and feedback adjustment

[0074] When the distribution network is actually running, heterogeneous devices execute the trained strategies in a distributed manner, and the intelligent agent monitors the voltage, reactive power demand, and operating status of each node in the distribution network in real time through sensors. According to the observed current environmental status, the intelligent agent makes autonomous decisions based on the trained strategy. The voltage regulator adjusts the gear to control the voltage level, the switchable capacitor performs switching operations according to the reactive power demand, and the battery performs charging and discharging operations according to the changes in the reactive power demand of the system, thereby achieving a dynamic balance of reactive power. The intelligent agent ensures the optimal regulation of voltage and reactive power by collaboratively controlling the three devices.

[0075] After completing the operation, the agent continues to monitor the feedback information in the environment and evaluates the voltage and reactive power status changes of the distribution network in real time. If the distribution network status does not achieve the expected effect, the agent will adjust the control strategy based on the feedback and continue to perform further decision-making operations until the optimization goal is achieved. As the load of the power system changes and the status of the distribution network fluctuates, the agent can continue to adapt and continuously optimize the control strategy through reinforcement learning algorithms to ensure the long-term stability and reliability of voltage and reactive power control.

[0076] The method of the present invention is applicable to voltage stability and reactive power optimization control in distribution networks. The method realizes dynamic adjustment of distribution network voltage and coordinated control of reactive power through multi-agent reinforcement learning algorithm, combined with heterogeneous devices such as voltage regulators, switchable capacitors and batteries. Multi-agents continuously optimize their control strategies through interaction with dynamic environments to maximize distribution network voltage stability, minimize reactive losses and reduce equipment operation costs. The algorithm ensures the optimization of the global system under the framework of centralized training and distributed execution. During actual operation, multi-agents make autonomous decisions based on the real-time monitored system status to achieve coordinated control of voltage regulators, switchable capacitors and batteries, and ensure the stability of voltage and reactive power of the distribution network in a complex operating environment. The method of the present invention has good adaptability and scalability, can effectively cope with challenges such as load changes and equipment status fluctuations in the power grid, and improve the operating efficiency and safety of the distribution network.

Claims

1. A voltage and reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning, characterized in that: The method controls heterogeneous devices in a distribution network through a multi-agent reinforcement learning algorithm to achieve dynamic optimization control of voltage and reactive power. The heterogeneous devices include a voltage regulator, a switchable capacitor, and a battery. The specific steps are as follows: Modeling of heterogeneous multi-agent state space and action space: A heterogeneous device is modeled as an agent, and a collection of multiple different heterogeneous devices is modeled as a heterogeneous multi-agent. The voltage and reactive power control problem of the distribution network is modeled as a heterogeneous multi-agent reinforcement learning problem. The state space of each node and the action space of the heterogeneous device are defined. The state space of each node includes the node voltage, the operating status of the heterogeneous device, and the load demand. The action space of the heterogeneous device specifically includes the gear state of the voltage regulator, the switch state of the switchable capacitor, and the charge and discharge state of the battery. Design reward function according to control objectives: The overall goal of voltage and reactive power control is to maintain voltage stability and reduce reactive power loss in the distribution network by adjusting voltage regulators, switchable capacitors and batteries, while meeting the operation constraints of heterogeneous equipment; design reward function based on the overall goal of voltage and reactive power control, comprehensively consider the penalties for voltage violations, reactive power loss and control errors, and introduce Lagrange multipliers to handle constraints to ensure that the agent can effectively optimize the voltage and reactive power control strategy during training and ensure that the distribution network meets physical and operational constraints; Voltage-reactive sensitivity analysis: Based on the voltage-reactive sensitivity matrix, calculate the impact of each node on the voltage and reactive power of the distribution network, and dynamically adjust the control priority of each node; Determine the update order of multiple agents: Based on voltage and reactive power sensitivity, select nodes with great influence on the distribution network and update their control strategies first to improve the response speed and control accuracy of the distribution network; Equipment collaborative control: During the operation of the distribution network, the voltage regulator controls the node voltage by adjusting the gear position; the switchable capacitor performs reactive power compensation according to demand; the battery adjusts reactive power by charging and discharging to achieve reactive power balance and voltage stability in the distribution network; Strategy update and optimization: Continuously calculate the update order of multi-agents, and update the multi-agent strategy network according to the update order through iterative training of the multi-agent reinforcement learning algorithm, so as to achieve optimal control of heterogeneous devices under different operating conditions and improve the voltage and reactive power control effect of the distribution network; Real-time control and feedback adjustment: In actual operation, each intelligent agent makes autonomous decisions based on the real-time observed state of the distribution network, realizes dynamic control of voltage and reactive power, and adjusts future control strategies based on the actual control effect.

2. The voltage reactive power control method based on constrained heterogeneous ordered multi-agent reinforcement learning according to claim 1 is characterized in that: The multi-agent reinforcement learning algorithm adopts a proximal strategy optimization algorithm with a monotonic improvement characteristic, ensuring that the voltage and reactive power control effect of the distribution network is continuously optimized after each strategy update, avoiding the instability of the distribution network caused by immature strategies; and optimizing the collaborative control strategy of heterogeneous devices through centralized training and distributed execution.

3. The voltage reactive power control method based on constraint heterogeneous ordered multi-agent reinforcement learning according to claim 1 is characterized in that: The voltage-reactive power sensitivity matrix is ​​used to evaluate the sensitivity of each node to voltage and reactive power. The voltage-reactive power sensitivity matrix is ​​dynamically updated according to the voltage fluctuation and load demand of the node to ensure the flexibility and accuracy of the control strategy.

4. The voltage reactive power control method based on constraint heterogeneous ordered multi-agent reinforcement learning according to claim 1 is characterized in that: The voltage regulator controls the node voltage by adjusting the gear position, so that the distribution network voltage is maintained within a predetermined range and the voltage fluctuation is reduced.

5. The voltage reactive power control method based on constraint heterogeneous ordered multi-agent reinforcement learning according to claim 1 is characterized in that: The switchable capacitor performs switching operation according to reactive power demand of the distribution network, and provides reactive power compensation to maintain reactive power balance of the distribution network.

6. The voltage reactive power control method based on constraint heterogeneous ordered multi-agent reinforcement learning according to claim 1 is characterized in that: The battery adjusts reactive power output by controlling the charging and discharging state, so as to assist the voltage regulator and the switchable capacitor in achieving reactive power balance of the distribution network.

Images