Graph reinforcement learning reactive voltage control method considering domain knowledge

Abstract

The invention discloses a graph reinforcement learning reactive voltage control method considering domain knowledge, and the method comprises the steps: (I) enabling a reactive voltage control problem to be built into a partially observable Markov decision process, and enabling an intelligent agent corresponding to each inverter to only observe the local state quantity of a limited number of nodes around; (2) constructing a space-time diagram in which domain knowledge is embedded; (3) utilizing a graph attention neural network to carry out domain knowledge-based feature reinforcement learning of the multi-agent; and (IV) performing multi-inverter action decision making based on a GAMARL method and the like. According to the invention, by integrating domain knowledge and an advanced graph reinforcement learning technology, the voltage regulation capability of the novel power distribution system under the condition of high-proportion distributed photovoltaic access is remarkably improved, and the intelligent and sustainable development of a power distribution network is promoted.

Description

Technical Field

[0001] The present invention belongs to the field of voltage control methods, and in particular relates to a graph reinforcement learning reactive voltage control method taking domain knowledge into account. Background Art

[0002] With the large-scale integration of distributed photovoltaics, the active nature of distribution systems is becoming increasingly prominent. In some areas, voltage regulation needs are gradually shifting from one-way regulation at night to two-way regulation throughout the day and at multiple times. How to standardize the acceptance and regulation of distributed energy has become a key issue that needs to be addressed in the construction of new distribution systems.

[0003] The Volt-Var control (VVC) problem is characterized by its multifaceted nature and requires rapid reactive power support. Traditional voltage control methods, such as optimal power flow and the alternating direction multiplier method, are limited by poor real-time performance and demanding communication requirements, making them inadequate for current needs. Deep reinforcement learning, a control method that does not rely on precise models, offers new insights into the control of time-varying nonlinear systems. However, existing multi-agent reinforcement learning methods often fail to fully consider the potential differences in coupling between agents during training, simply stacking all agent features, resulting in excessive interactions between agents with low actual correlations. Furthermore, the control models constructed by these methods are generally general models of continuous variables, lacking the domain knowledge of distribution network reactive power regulation to effectively guide inverter action strategies. This makes it difficult to ensure model interpretability and comprehensiveness of strategies. Although graph neural networks have demonstrated significant advantages in tasks such as distribution network transient assessment, load forecasting, and electricity price prediction, providing a new path for mining differentiated features from massive data sets and diversely expressing domain knowledge, the increasing penetration of distributed photovoltaics significantly increases the feasible domain of agent control actions and the strong spatiotemporal uncertainty of distribution network states. Graph convolutional networks based on pre-fixed graph structures have limitations in understanding the complex coupling mechanism when multiple inverters participate in VVC, and their ability to learn domain knowledge is restricted.

[0004] In summary, while existing deep learning methods can effectively address the shortcomings of traditional physics-based optimization methods, such as their strong reliance on accurate modeling and slow solution speed, they also generate control strategies from a data-driven perspective and lack action guidance based on cross-integration with domain knowledge, which affects the overall effectiveness of the strategies. Therefore, it is urgent to develop effective strategies to implement real-time inverter-based VVC. Summary of the Invention

[0005] The present invention is proposed to overcome the shortcomings of the prior art, and its purpose is to provide a graph reinforcement learning reactive power voltage control method taking into account domain knowledge.

[0006] The present invention is achieved through the following technical solutions:

[0007] A domain knowledge-based graph reinforcement learning method for reactive power and voltage control, comprising the following steps:

[0008] (I) The reactive power and voltage control problem is modeled as a partially observable Markov decision process, with each inverter considered as an intelligent agent;

[0009] (II) Constructing a spatiotemporal graph that embeds domain knowledge, treating agents as nodes in the spatiotemporal graph, and characterizing the relationships between nodes based on the electrical distances between multiple agents;

[0010] (III) Using graph attention neural networks for multi-agent domain knowledge-based feature enhancement learning;

[0011] (IV) Optimize the multi-inverter action decision based on the GAMARL method by introducing the space-time graph and GAT module.

[0012] In the above technical solution, the Markov decision process is composed of a set N of intelligent agents, a state space S, a multi-agent joint observation space O, a multi-agent joint action space A, a state transition function TF, a shared reward function R, and a discount factor γ that balances the importance of long-term and short-term returns.

[0013] In the above technical solution, the set N of intelligent agents is a set of m intelligent agents, where N={1, 2, ..., m}; each inverter in the distribution network is regarded as an intelligent agent i.

[0014] In the above technical solution, the expression of the state space S is:

[0015] S=L×P×Ω×V

[0016] Where: Indicates the active power p of the load L and reactive power q L A collection of

[0017] Represents the active power p generated by the photovoltaic system PV A collection of

[0018] Represents the reactive power q generated by the photovoltaic inverter PV A collection of

[0019] Represents a set of voltage amplitude v and phase angle θ.

[0020] In the above technical solution, the multi-agent joint observation space O is composed of the observation space o of each agent. i The observation space of each agent is selected based on the local measurement results of the entire power grid;

[0021] The observation space o of the agent i The expression is:

[0022]

[0023] In the formula: In the formula: p L,k is the active power of the load in the kth area, in kW; q L,k is the reactive power of the load in the kth region, in kvar; p PV,k is the active power generated by the photovoltaic system in the kth region, in kW; q PV,k is the reactive power generated by the photovoltaic system in the kth region, in kvar; v k is the voltage amplitude of the kth region, in V; θ k is the voltage phase angle of the kth region, in degrees.

[0024] In the above technical solution, the multi-agent joint action space A is composed of the action space A of each agent i. i The action space of agent i is a continuous interval, A i The expression is:

[0025] A i ={a i ∣-a min ≤a i ≤a max}

[0026] Where: a i is the maximum reactive power generation ratio, dimensionless; a min The minimum value of the maximum reactive power generation ratio, dimensionless; a max It is the maximum value of the maximum reactive power generation ratio, dimensionless;

[0027] By the maximum reactive power generation ratio a i Get the reactive power generated by the inverter The reactive power generated by the i-th photovoltaic inverter is The calculation formula is:

[0028]

[0029] Where: a i is the maximum reactive power generation ratio, dimensionless; is the maximum apparent power of the ith node, in kVA, which depends on the physical capacity of the PV inverter; P i max is the maximum active power of the i-th node, in kW;

[0030] when a i >0, reactive power is injected into the distribution network; when a i When <0, reactive power is absorbed; min with a max The value of is selected according to the load capacity of the active distribution network.

[0031] In the above technical solution, the shared reward function R is composed of reducing voltage deviation and reducing inverter reactive power. The expression of the shared reward function R is:

[0032]

[0033] Where: EG(v i ) represents the composite voltage control function, evaluating the node voltage v i Deviation from the reference value; g q (q PV ) represents the reactive power generation loss, that is, the sum of the reactive power generated by all PV inverters; β represents the weight coefficient, which is used to balance the importance between voltage control and reactive power loss and is dimensionless;

[0034] The expression for the reactive power generation loss is:

[0035]

[0036] Where: D is the number of photovoltaic nodes, in units; q PV sum The sum of reactive power generated by each photovoltaic node, in kVar;

[0037] Composite voltage control function EG(v i ) is:

[0038]

[0039] Where: α, β, c, d are the shape control parameters of the composite voltage control function; μ is the mean of the Gaussian distribution; σ 2 is the standard deviation of Gaussian distribution; v ref It is the boundary value of the reasonable voltage range.

[0040] In the above technical solution, the expression of the spatiotemporal graph embedded with domain knowledge is:

[0041] G t =(X t ,A t )

[0042] Where: X t is the set of voltage regulating agent nodes, X tContains observation-action joint features, X∈R N×D , D is the feature dimension; X t The expression is:

[0043] A is a 0 / 1 variable sparse adjacency matrix that represents the topological structure of the graph data. Based on whether the electrical distance is close, a sparse adjacency matrix embedded with domain knowledge is constructed.

[0044] In the above technical solution, the specific method of step (III) using the graph attention neural network to perform domain knowledge-based feature enhancement learning of multiple agents is:

[0045] (III-i) Convert the observation-action joint features of the agent nodes and calculate the attention coefficient between the nodes. The calculation formula of the attention coefficient between the nodes is:

[0046]

[0047] Where: α ic represents the attention weight between node i and node c; x i with x c are the observation-action joint features of nodes i and c respectively; W is the feature transformation matrix, which is used to convert the feature vector to the new representation space; N(i) represents the set of neighboring pressure regulating agent nodes of node i, and the neighboring pressure regulating agent nodes are obtained through the adjacency matrix A. Represents vector connection, a T is the weight vector in the attention mechanism, and LeakyReLU is the activation function used to calculate the association mechanism between nodes;

[0048] (III-ii) Perform weighted aggregation on the information of each node according to the attention weight to generate a new feature expression to achieve feature enhancement based on domain knowledge. The new feature expression is:

[0049]

[0050] Where: H represents the number of attention heads, is the attention weight between node i and node c calculated by the hth attention head, W h is the weight matrix corresponding to the h-th attention head, and σ is the activation function ReLU.

[0051] In the above technical solution, the step (IV) optimizes the multi-inverter action decision based on the GAMARL method by introducing the space-time graph and the GAT module, and specifically includes the following steps:

[0052] (IV-i) Training and updating of the critic network embedded in GAT

[0053] A GAT module is embedded in the Critic network. Through the GAT module, the Critic network dynamically adjusts and optimizes the relationship weights between nodes while considering the local state of each inverter;

[0054] The critic network parameter update formula is:

[0055]

[0056] Q=FC(GAT(FC(X),A)) (6)

[0057] Where: E o,a,r,o′～D represents the expected value of the action a, reward r, observations o and o' sampled from the experience replay buffer D; is the value network; r is the immediate reward; γ is the discount factor; FC is fully connected for feature processing; specifically, in GAT, the local observation o1 and action a1 are processed in pairs as x1 = {(o1, a1)}, and X = {x1,…,x n};

[0058] (IV-II) Optimization of Actor Network

[0059] The Actor network is trained using the policy gradient method, optimizing the action decisions of each inverter based on the feedback from the Critic network.

[0060] The Actor network outputs the control action of the inverter based on the current state and updates the strategy using the following formula:

[0061]

[0062] Where: E o,a～D represents the expected value of observation o and action a sampled from the experience replay buffer D; Indicated in μ i Given an observation o i Output action a i For action network parameters gradient; Output of the Critic network that introduces domain knowledge For the action gradient a of the i-th agent i ;

[0063] (IV-III) Application of Soft Update Mechanism

[0064] During the training process, the parameters of the target network are updated using soft updates;

[0065] The formula for the soft update is:

[0066] Qμ′ ←τQ μ +(1-τ)Q μ′ (8)

[0067]

[0068] Where: τ is the soft update coefficient;

[0069] (IV-IV) Inverter collaborative decision-making

[0070] The Actor is trained using the above method. In the decentralized execution phase after training, the Actor network outputs the inverter action strategy as follows:

[0071]

[0072] Where: a i The inverter action for output, Represents the process of mapping the policy network from observations to actions.

[0073] The beneficial effects of the present invention are:

[0074] The present invention provides a graph reinforcement learning reactive voltage control method that takes domain knowledge into account, which effectively addresses the voltage regulation challenges brought about by high-penetration distributed photovoltaic access; the present invention designs an efficient reactive voltage control strategy that adapts to all-weather, multi-period bidirectional regulation needs, uses graph neural networks to enhance the in-depth understanding of the complex coupling mechanism of the distribution network, optimizes the multi-agent reinforcement learning algorithm to enhance the interpretability of the model and the comprehensiveness of the strategy, and realizes real-time reactive compensation through the inverter, thereby improving the voltage stability and overall reliability of the distribution system; by integrating domain knowledge with advanced graph reinforcement learning technology, the present invention significantly improves the voltage regulation capability of the new distribution system under the condition of high-proportion distributed photovoltaic access, and promotes the intelligent and sustainable development of the distribution network. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 This is the traditional MADDPG algorithm framework diagram;

[0076] Figure 2 This is a schematic diagram of the differentiated enhancement process of embedded features in the global voltage regulation intelligent body based on GAT;

[0077] Figure 3 This is a framework diagram of the multi-agent graph reinforcement learning model with centralized training and decentralized execution proposed by the present invention;

[0078] Figure 4 This is the IEEE 33 node and area division diagram of Example 1 of the present invention.

[0079] For ordinary technicians in this field, other relevant drawings can be obtained based on the above drawings without any creative work. DETAILED DESCRIPTION

[0080] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution of the present invention will be further described below with reference to the accompanying drawings and through specific implementation methods.

[0081] A domain knowledge-based graph reinforcement learning method for reactive power and voltage control, comprising the following steps:

[0082] (I) The reactive power and voltage control problem is formulated as a partially observable Markov decision process (Decentralized Partially Observable Markov Decision Process, Dec-POMDP). Each inverter is considered as an agent, and the agent corresponding to each inverter can only observe the local state of a limited number of surrounding nodes.

[0083] The Markov decision process consists of a set N of agents, a state space S, a multi-agent joint observation space O, a multi-agent joint action space A, a state transition function TF, a shared reward function R, and a discount factor γ that balances the importance of long-term and short-term returns;

[0084] At each time step t, each agent i obtains a local observation space o according to the environment i,t ∈O i , and according to the strategy π i,t Select action a i,t ∈A i ; These actions cause the system to transition to a new state with a certain probability and generate a joint reward r t ∈R; the goal of the multi-agent is to find a set of joint strategies π={π i} i∈N To maximize the cumulative expected reward Where T is the time span.

[0085] The set N of intelligent agents is a set of m intelligent agents, where N={1, 2, ..., m}; each inverter in the distribution network is regarded as an intelligent agent i.

[0086] The expression of the state space S is:

[0087] S=L×P×Ω×V

[0088] Where: Indicates the active power p of the load L and reactive power q LThe superscript V represents the dimension of the node set, that is, how many nodes there are, R v represents a real vector space of |V| dimensions;

[0089] Represents the active power p generated by the photovoltaic system PV The superscript I represents the dimension of the photovoltaic node set, that is, how many photovoltaic nodes there are;

[0090] Represents the reactive power q generated by the photovoltaic inverter PV A collection of

[0091] Represents a set of voltage amplitude v and phase angle θ.

[0092] The multi-agent joint observation space O is composed of the observation space o of each agent. i The set of (o i ∈O), O=m i∈N O i ,The observation space of each agent is selected based on the local ,measurements of the entire power grid;

[0093] The observation space o of the agent i The expression is:

[0094] o i ={p L,k ,q L,k ,p PV,k ,q PV,k ,v k ,θ k}

[0095] Where: p L,k is the active power of the load in the kth area, in kW; q L,k is the reactive power of the load in the kth region, in kvar; p PV,k is the active power generated by the photovoltaic system in the kth region, in kW; q PV,k is the reactive power generated by the photovoltaic system in the kth region, in kvar; v k is the voltage amplitude of the kth region, in V; θ k is the voltage phase angle of the kth region, in degrees;

[0096] The multi-agent joint action space A is the action space A of each agent i. i The action space of agent i is a continuous interval, A i The expression is:

[0097] Ai ={a i ∣-a min ≤a i ≤a max}

[0098] Where: a i is the maximum reactive power generation ratio, dimensionless; a min The minimum value of the maximum reactive power generation ratio, dimensionless; a max It is the maximum value of the maximum reactive power generation ratio, dimensionless;

[0099] By the maximum reactive power generation ratio a i Get the reactive power generated by the inverter The reactive power generated by the i-th photovoltaic inverter is The calculation formula is:

[0100]

[0101] Where: a i is the maximum reactive power generation ratio, dimensionless; is the maximum apparent power of the ith node, in kVA, which depends on the physical capacity of the PV inverter; P i max is the maximum active power of the i-th node, in kW;

[0102] when a i >0, reactive power is injected into the distribution network; when a i When <0, reactive power is absorbed;

[0103] a min with a max The value of is usually selected according to the load capacity of the active distribution network for safe operation.

[0104] The expression of the state transfer function TF is:

[0105] TF(s′∣s,ac)=Pr(s′∣s,ac)

[0106] Where: s represents the current state, ac represents the joint action, s' represents the state at the next moment, and Pr(s'|s,a) represents the probability of transitioning to the next state s' under the current state s and the joint action ac;

[0107] In the present invention, the shared reward function R is composed of reducing voltage deviation and reducing inverter reactive power. The expression of the shared reward function R is:

[0108]

[0109] Where: EG(v i ) represents the composite voltage control function, evaluating the node voltage v i Deviation from the reference value; g q (q PV ) represents the reactive power generation loss, that is, the sum of the reactive power generated by all PV inverters; β is the weight coefficient, dimensionless, used to balance the importance between voltage control and reactive power loss;

[0110] The expression for the reactive power generation loss is:

[0111]

[0112] Where: D is the number of photovoltaic nodes, in units; q PV sum The sum of reactive power generated by each photovoltaic node, in kVar;

[0113] In the present invention, the composite voltage control function EG(v i ) Introducing a fusion Gaussian regulation and exponential surge penalty mechanism, which uses an exponential surge penalty mechanism when the voltage seriously exceeds the limit;

[0114] Composite voltage control function EG(v i ) is:

[0115]

[0116] Where: α, β, c, d are dimensionless shape control parameters of the composite voltage control function; μ is the mean of the Gaussian distribution, dimensionless; σ 2 is the standard deviation of the Gaussian distribution, dimensionless; v ref It is the boundary value of the reasonable voltage range, in V.

[0117] The discount factor for balancing the importance of long-term and short-term returns is γ = 0.99;

[0118] (II) Constructing a spatiotemporal graph that embeds domain knowledge, treating agents as nodes in the spatiotemporal graph, and characterizing the relationships between nodes based on the electrical distances between multiple agents;

[0119] The VCC problem in the distribution network is characterized by being multi-faceted and wide-ranging. The degree of over-limit and risk of voltages at each node vary significantly. Therefore, scheduling decisions require differentiated modeling and regulation of inverters. The Multi-agent Deep Deterministic Policy Gradient (MADDPG) algorithm provides a framework for centralized training and decentralized execution. Specifically, in traditional MADDPG-based VVC, during the centralized training phase, each voltage regulating agent (inverter) has its own actor network that decides actions based on observations, and sends the observations and actions to the critic network to evaluate the action strategy. This traditional architecture allows the critic network to obtain global information from all agents. During the decentralized execution phase, each agent uses the optimized actor network to independently decide on the inverter action.

[0120] However, the above traditional methods simply concatenate the observation-action features of all agents in the centralized training and pass them to the evaluation network, see Figure 1 . During the voltage regulation process, there are significant differentiated correlations between different reactive devices. For example, reactive devices that are far away may not have significant correlations. However, traditional methods mainly make decisions on inverter actions from a data-driven perspective, which results in some intelligent agents having low actual correlations despite frequent interactions during training. The above methods only generate control strategies from a data-driven perspective, ignoring the importance of domain knowledge and lacking an in-depth understanding of the differentiated coupling mechanism between voltage regulation agents in the VVC process; and when the distribution network is large, the increase in nodes leads to excessively long feature vector dimensions and limited feature extraction capabilities.

[0121] Graph neural networks have a natural and efficient ability to express domain knowledge and extract differentiated features. Each voltage regulating agent can be abstracted as a voltage regulating agent node in a time-space graph. The differentiated potential coupling relationships between voltage regulating agents in non-Euclidean space can be mapped into the time-space graph as domain knowledge. Graph neural networks can then be used to extract complex potential interaction features, facilitating the generation of multi-voltage regulating agent control strategies.

[0122] The expression of the spatiotemporal graph embedded with domain knowledge is:

[0123] G t =(X t ,A)

[0124] Where: X t is the set of voltage regulating agent nodes, X t Contains observation-action joint features, X∈R N×D , N is the number of nodes; D is the feature dimension; X t The expression is:

[0125] A is a sparse adjacency matrix of 0 / 1 variables, representing the topological structure of graph data. During the voltage regulation process of reactive equipment, there is a potential correlation between the correlation between reactive equipment and the electrical distance. Based on whether the electrical distance is close (the 1-2 nodes closest to each node are considered to be adjacent nodes), a sparse adjacency matrix embedded with domain knowledge is constructed.

[0126] The constructed spatiotemporal graph embedded with domain knowledge considers the potential impact of distribution network topology information on the decision-making of the voltage regulation agent and models the potential coupling relationships between multiple inverters from a non-Euclidean perspective. This provides a data foundation for subsequent differentiated feature reinforcement learning that takes domain knowledge into account, while also enhancing model interpretability.

[0127] (III) Using Graph Attention Neural Networks for Domain-Based Feature Enhancement Learning of Multi-Agents

[0128] To capture the dynamic correlation between the degree of coupling between agents caused by changes in source and load power, a graph attention mechanism layer is built into the critic network to optimize the neighbor node weights that are most suitable for the VVC task and generate spatial aggregation features for each agent. This provides a more accurate value function for policy evaluation and realizes feature-enhanced learning based on domain knowledge.

[0129] In the feature extraction of spatiotemporal graphs, traditional methods usually rely on the static edge features between nodes in the pre-defined spatiotemporal graph as the node weights of neighboring voltage regulation agents. However, the operating characteristics of the distribution network under strong random source and load access are extremely complex, and there are temporal differences in the bidirectional voltage regulation requirements between nodes. The degree of correlation between the corresponding control strategies of the corresponding voltage regulation agents is becoming increasingly complex and presents a temporal dynamic change characteristic.

[0130] When combining feature extraction with domain knowledge, it is difficult to accurately reflect the dynamic coupling relationship between multi-agent nodes by using fixed static edge features. Therefore, the present invention only uses the static adjacency matrix in the expression of the spatiotemporal graph to reflect the potential coupling characteristics between agents to construct the spatiotemporal graph, but does not rely on the static edge weights in the expression of the spatiotemporal graph during the feature extraction process. Instead, it dynamically learns and adjusts the weights of neighboring voltage-regulating agent nodes with potential associations, thereby realizing spatial aggregation feature generation based on domain knowledge. Specifically, the Graph Attention Network (GAT) is used to dynamically update the weights of neighboring voltage-regulating agent nodes in each iteration, and adaptively and dynamically generates spatial aggregation features for each voltage-regulating agent, thereby realizing feature enhancement learning based on domain knowledge.

[0131] GAT realizes the dynamic coupling relationship between intelligent agents through node feature transformation and information aggregation. Figure 2This figure shows the process of feature enhancement learning based on domain knowledge through GAT. The data structure shown in this figure is non-Euclidean data with embedded domain knowledge, that is, the number of neighbor nodes of each node is different, and its structure is determined by A. Taking node ① as an example, Figure 2 It can be seen that this node has an electrical proximity relationship with neighboring voltage regulator agent nodes ②, ③, ④, and ⑤. A multi-head attention mechanism (K heads) is used to extract differentiated features between K groups of neighboring voltage regulator agent nodes. These K groups of features are then integrated to achieve feature-enhanced learning based on domain knowledge.

[0132] The details are as follows: First, the observation-action joint features of the agent node are converted to calculate the attention coefficient between the nodes. Take node i and its neighbor node c as an example:

[0133]

[0134] Where: α ic represents the attention weight between node i and node c; x i with x c are the observation-action joint features of nodes i and c respectively; W is the feature transformation matrix, which is used to convert the feature vector to the new representation space; N(i) represents the set of neighboring pressure regulating agent nodes of node i, and the neighboring pressure regulating agent nodes are obtained through the adjacency matrix A. Represents vector connection, a T is the weight vector in the attention mechanism, and LeakyReLU is the activation function used to calculate the association mechanism between nodes;

[0135] Then, the information of each node is weighted and aggregated according to the attention weight to generate a new feature representation to achieve feature enhancement based on domain knowledge. Taking node i as an example, its new feature representation is:

[0136]

[0137] Where: H represents the number of attention heads, is the attention weight between node i and node c calculated by the hth attention head, W h is the weight matrix corresponding to the h-th attention head, and σ is the activation function ReLU.

[0138] (IV) Multi-inverter action decision-making based on GAMARL (Graph Adaptive Multi-Agent Reinforcement Learning) method, by introducing spatiotemporal graph and GAT module, the action decision strategy of multi-inverter is optimized; the action decision strategy of multi-inverter is the reactive voltage control method;

[0139] The Critic network provides optimization direction for the Actor network through value evaluation, enabling the Actor network to generate action strategies guided by domain knowledge.

[0140] The GAMARL method aims to solve the complex decision-making problems in inverter cooperative control, especially how to effectively capture the coupling relationships between multiple inverters and dynamically adjust the control strategy of each inverter based on these relationships.

[0141] In order to fully consider the potential role of distribution network topology information on intelligent agent decision-making, accurately capture the potential coupling associations between multiple inverters in non-Euclidean domains, and realize the effective targeted guidance of inverter action strategies by domain knowledge, the present invention proposes a GAMARL multi-inverter action decision-making method. Based on MADDPG, the method of the present invention constructs a time-space graph with embedded domain knowledge to reflect the potential coupling associations between multiple inverters, and embeds the GAT network in the Critic network to dynamically learn and capture the dynamic correlation degree of each intelligent agent voltage regulation node in the time-space graph, adaptively generate differentiated spatial aggregation features of each voltage regulation agent, and realize feature enhancement learning based on domain knowledge.

[0142] The specific steps include:

[0143] (IV-i) Training and updating of the critic network embedded in GAT

[0144] In the GAMARL approach, the core task of the Critic network is to evaluate the actions of each inverter, thereby helping the Actor network update its strategy and optimize its overall decision-making. To effectively capture the coupling relationships between inverters, a GAT module is embedded in the Critic network. Through the GAT module, the Critic network can dynamically adjust and optimize the relationship weights between nodes while considering the local state of each inverter, so that the decision it ultimately generates depends not only on its own state but also on its relationships with other inverters. The parameter update formula of the Critic network is:

[0145]

[0146] Q=FC(GAT(FC(X),A)) (6)

[0147] Where: E o,a,r,o′～D represents the expected value of the action a, reward r, observations o and o' sampled from the experience replay buffer D; is the value network; r is the immediate reward; γ is the discount factor; FC is fully connected for feature processing; specifically, in GAT, the local observation o1 and action a1 are processed in pairs as x1 = {(o1, a1)}, and X = {x1,…,x n};

[0148] (IV-II) Optimization of Actor Network

[0149] The gradient feedback provided by the Critic network guides the Actor network to adjust its strategy, enabling the inverter to dynamically optimize its reactive power regulation strategy based on the Critic's evaluation results, ultimately ensuring the optimization of the overall voltage regulation effect;

[0150] Actor training is performed using a policy gradient method. Each agent updates its own action network parameters based on the gradient ascent method. In the centralized training phase, each agent continuously updates its action strategy in the direction where the voltage is closer to the expected value. The policy gradient expression is as follows:

[0151]

[0152] Where: represents the expected value of observation o and action a sampled from the experience replay buffer D; Indicated in μ i Given an observation o i Output action a i For action network parameters gradient; Output of the Critic network that introduces domain knowledge For the action gradient a of the i-th agent i ;

[0153] (IV-III) Application of Soft Update Mechanism

[0154] In order to improve the stability and convergence speed of the model, a soft update mechanism is applied to the target network update. During the training process, the parameters of the target network are not directly updated using traditional hard updates. Instead, soft updates are used to make the changes of the target network smoother, thereby avoiding instability during training. The formula for soft update is:

[0155] Q μ′ ←τQ μ +(1-τ)Q μ′ (8)

[0156]

[0157] Where: τ is the soft update coefficient;

[0158] (IV-IV) Inverter collaborative decision-making

[0159] The Actor is trained using the above method. In the decentralized execution phase after training, the Actor network outputs the inverter action strategy as follows:

[0160]

[0161] Where: a i The inverter action for output, Represents the process of policy network mapping from observation to action;

[0162] To effectively guide the action strategy of the voltage regulating agent based on domain knowledge, this method maps the differentiated potential associations between agents as domain knowledge to a spatiotemporal graph during training. Through GAT, the coupling degree of adjacent nodes in the spatiotemporal graph is dynamically quantified and updated, and the spatial aggregation features of each voltage regulating agent are adaptively generated to achieve feature-enhanced learning based on domain knowledge. This process can effectively explore the complex coupling mechanism when multiple inverters participate in VVC and guide the actor network to complete strategy updates. During the decentralized execution phase, each actor makes on-site decisions on the inverter action through partial observations.

[0163] Example 1

[0164] In this embodiment 1 Figure 4 The improved IEEE33 node shown in the figure verifies the performance of the proposed method. The distribution network is divided into four regions and connected to six photovoltaic (corresponding to six voltage regulating agents). Each voltage regulating agent can only observe information in its own region. The validation is performed using actual collected photovoltaic active power, load active power, and reactive power data. The data set spans three years and has a resolution of three minutes. Twenty days of data were randomly selected, totaling 9,600 data sets for testing. The remaining key parameter settings are shown in Table 1:

[0165] Table 1: Key parameters

[0166]

[0167]

[0168] Comparison of distribution network voltage control performance:

[0169] The proposed method was validated through simulation tests using an NVIDIA GeForce RTX 4060Ti 16GB GPU, a 4.5GHz 16-core AMD R9 7950X CPU, 64GB of RAM, and the PyTorch deep learning framework. The proposed method demonstrated a single-time-section decision time of 0.001 seconds, meeting real-time voltage regulation requirements. The primary computational burden is concentrated in the centralized training phase, which can be performed remotely.

[0170] Traditional physical model-based voltage control methods, such as optimal power flow, despite their excellent performance, rely heavily on system physical models and require precise modeling of the distribution network's line parameters, global state, and topology. However, weak distribution network communication capabilities and data acquisition difficulties limit the practical application of these methods. Therefore, this study did not select these traditional methods for simulation. Instead, it selected advanced MARL methods with their own unique characteristics, while using the original method ④ without any regulation as a reference. Among them, ① is the method proposed in this paper:

[0171] ①GAMARL-EG: Adopts the MADDPG multi-agent reinforcement learning framework, embeds GAT learning domain knowledge (dynamic learning expression of edge weights), and adopts EG voltage control function.

[0172] ②GCMARL-EG: Adopts the MADDPG multi-agent reinforcement learning framework, embeds GCN (fixed edge features) to learn domain knowledge, and adopts EG voltage control function.

[0173] ③MADDPG-Bowl: Based on MADDPG multi-agent reinforcement learning, it uses the Bowl voltage control function.

[0174] ④ Original method: No regulation is performed, that is, the inverter does not take any action.

[0175] Table 2 Statistics of voltage limit violations for each method over 20 days

[0176]

[0177] Table 2 compares the voltage control results of the proposed method with other methods. Table 3 shows that, without any intervention before optimization, 73.2% of the 9600 data sets showed no voltage violations at any node in 73.2% of the time sections. The proposed GAMARL algorithm exhibits significant advantages over other methods. After the voltage regulation agent's actions, all nodes remained within the normal voltage fluctuation range in 99.2% of the time sections, an improvement of 35.52%. The other methods achieved improvements ranging from 1.74% to 35.20%.

[0178] Table 3 Statistics of voltage regulation effects of various methods over 20 days

[0179]

[0180] In addition, the present invention also conducts a comprehensive comparative analysis from the perspectives of voltage average deviation, voltage maximum deviation and network loss, see Table 3. The voltage average deviation evaluates the ability of each method to track the target voltage. Compared with other methods, the method proposed in the present invention has improved by 29.41% to 47.83% in terms of voltage average deviation, showing that it has significant advantages in the accuracy and stability of voltage regulation. The maximum voltage deviation reflects the system's ability to suppress voltage fluctuations under extreme conditions, which is one of the key indicators in the operation of the distribution network. The method of the present invention has improved by 4.92% to 43.14% in terms of voltage maximum deviation compared with other methods. In terms of network loss optimization, the method proposed in the present invention has improved by 4.67% to 12.49% compared with other methods.

[0181] The proposed method achieves these results through optimizations in two dimensions. First, during centralized training, it accurately captures the dynamic, differentiated coupling between various voltage-regulating agents. Using GAT for convolution in a non-Euclidean domain, it adaptively and dynamically generates spatially aggregated features for each voltage-regulating agent, achieving differentiated enhancement of embedded features across the entire voltage-regulating agent. This enables domain-knowledge-based feature-enhanced learning and more precise feature mining across multiple agents within a spatiotemporal graph. Second, compared to MADDPG-Bowl, the proposed method introduces a composite voltage penalty function that combines Gaussian regulation with an exponential surge penalty mechanism. This applies an exponential surge penalty mechanism when voltage exceeds a certain threshold, forcing the voltage to return to a normal range. Furthermore, despite outperforming other methods across 9,600 time slices, the proposed method still cannot eliminate voltage violations in certain scenarios. When the reactive power support capability of a photovoltaic inverter is limited by capacity and cannot provide any more reactive power, further voltage regulation should be combined with OLTC, CB, and electric vehicles.

[0182] The applicant declares that the above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art should understand that any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention fall within the scope of protection and disclosure of the present invention.

Claims

1. A domain knowledge-based graph reinforcement learning method for reactive power and voltage control, characterized by: The following steps are involved: (I) The reactive power and voltage control problem is modeled as a partially observable Markov decision process, with each inverter considered as an intelligent agent; (II) Constructing a spatiotemporal graph that embeds domain knowledge, treating agents as nodes in the spatiotemporal graph, and characterizing the relationships between nodes based on the electrical distances between multiple agents; (III) Using graph attention neural networks for multi-agent domain knowledge-based feature enhancement learning; (IV) Optimize the multi-inverter action decision based on the GAMARL method by introducing the space-time graph and GAT module.

2. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 1, characterized in that: The Markov decision process consists of a set N of agents, a state space S, a multi-agent joint observation space O, a multi-agent joint action space A, a state transition function TF, a shared reward function R, and a discount factor γ that balances the importance of long-term and short-term returns.

3. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 2, characterized in that: The set N of intelligent agents is a set of m intelligent agents, where N={1, 2, ..., m}; each inverter in the distribution network is regarded as an intelligent agent i.

4. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 2, wherein: The expression of the state space S is: S=L×P×Ω×V Where: Indicates the active power p of the load L and reactive power q L A collection of Indicates the active power p generated by the photovoltaic system PV A collection of Represents the reactive power q generated by the photovoltaic inverter PV A collection of Represents a set of voltage amplitude v and phase angle θ.

5. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 2, characterized in that: The multi-agent joint observation space O is composed of the observation space o of each agent. i The observation space of each agent is selected based on the local measurement results of the entire power grid; The observation space o of the agent i The expression is: o i ={p L,k ,q L,k ,p PV,k ,q PV,k ,v k ,θ k } Where: p L,k is the active power of the load in the kth area, in kW; q L,k is the reactive power of the load in the kth region, in kvar; p PV,k is the active power generated by the photovoltaic system in the kth region, in kW; q PV,k is the reactive power generated by the photovoltaic system in the kth region, in kvar; v k is the voltage amplitude of the kth region, in V; θ k is the voltage phase angle of the kth region, in degrees.

6. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 2, characterized in that: The multi-agent joint action space A is the action space A of each agent i. i The action space of agent i is a continuous interval, A i The expression is: A i ={a i ∣-a min ≤a i ≤a max } Where: a i is the maximum reactive power generation ratio, dimensionless; a min The minimum value of the maximum reactive power generation ratio, dimensionless; a max It is the maximum value of the maximum reactive power generation ratio, dimensionless; By the maximum reactive power generation ratio a i Get the reactive power generated by the inverter The reactive power generated by the i-th photovoltaic inverter is The calculation formula is: Where: a i is the maximum reactive power generation ratio, dimensionless; is the maximum apparent power of the ith node, in kVA, which depends on the physical capacity of the PV inverter; is the maximum active power of the i-th node, in kW; when a i >0, reactive power is injected into the distribution network; when a i When <0, reactive power is absorbed; min with a max The value of is selected according to the load capacity of the active distribution network.

7. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 2, characterized in that: The shared reward function R consists of reducing voltage deviation and reducing inverter reactive power. The expression of the shared reward function R is: Where: EG(v i ) represents the composite voltage control function, evaluating the node voltage v i Deviation from the reference value; g q (q PV ) represents the reactive power generation loss, that is, the sum of the reactive power generated by all PV inverters; β represents the weight coefficient, which is used to balance the importance between voltage control and reactive power loss and is dimensionless; The expression for the reactive power generation loss is: Where: D is the number of photovoltaic nodes, in units; q PV sum The sum of reactive power generated by each photovoltaic node, in kVar; Composite voltage control function EG(v i ) is: Where: α, β, c, d are the shape control parameters of the composite voltage control function; μ is the mean of the Gaussian distribution; σ 2 is the standard deviation of Gaussian distribution; v ref It is the boundary value of the reasonable voltage range.

8. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 1, characterized in that: The expression of the spatiotemporal graph embedded with domain knowledge is: G t =(X t ,A t ) Where: X t is the set of voltage regulating agent nodes, X t Contains observation-action joint features, X∈R N×D , D is the feature dimension; X t The expression is: A is a 0 / 1 variable sparse adjacency matrix that represents the topological structure of the graph data. Based on whether the electrical distance is close, a sparse adjacency matrix embedded with domain knowledge is constructed.

9. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 1, characterized in that: The specific method of step (III) using graph attention neural network to perform domain knowledge-based feature enhancement learning of multiple agents is: (III-i) Convert the observation-action joint features of the agent nodes and calculate the attention coefficient between the nodes. The calculation formula of the attention coefficient between the nodes is: Where: α ic represents the attention weight between node i and node c; x i with x c are the observation-action joint features of nodes i and c, respectively; W is the feature transformation matrix, which is used to convert the feature vector to the new representation space; N(i) represents the set of neighboring voltage regulating agent nodes of node i, and the neighboring voltage regulating agent nodes are obtained through the adjacency matrix A. Represents vector connection, a T is the weight vector in the attention mechanism, and LeakyReLU is the activation function used to calculate the association mechanism between nodes; (III-ii) Perform weighted aggregation on the information of each node according to the attention weight to generate a new feature expression to achieve feature enhancement based on domain knowledge. The new feature expression is: Where: H represents the number of attention heads, is the attention weight between node i and node c calculated by the hth attention head, W h is the weight matrix corresponding to the h-th attention head, and σ is the activation function ReLU.

10. The domain knowledge-based graph reinforcement learning method for reactive power and voltage control according to claim 1, characterized in that: The step (IV) optimizes the multi-inverter action decision based on the GAMARL method by introducing the space-time graph and the GAT module, and specifically includes the following steps: (IV-i) Training and updating of the critic network embedded in GAT A GAT module is embedded in the Critic network. Through the GAT module, the Critic network dynamically adjusts and optimizes the relationship weights between nodes while considering the local state of each inverter; The critic network parameter update formula is: Q=FC(GAT(FC(X),A))(6) Where: E o,a,r,o′～D represents the expected value of the action a, reward r, observations o and o' sampled from the experience replay buffer D; is the value network; r is the immediate reward; γ is the discount factor; FC is fully connected for feature processing; specifically, in GAT, the local observation o1 and action a1 are processed in pairs as x1 = {(o1, a1)}, and X = {x1,…,x n }; (IV-II) Optimization of Actor Network The Actor network is trained using the policy gradient method, optimizing the action decisions of each inverter based on the feedback from the Critic network. The Actor network outputs the control action of the inverter based on the current state and updates the strategy using the following formula: Where: E o,a～D represents the expected value of observation o and action a sampled from the experience replay buffer D; Indicated in μ i Given an observation o i Output action a i For action network parameters gradient; Output of the Critic network that introduces domain knowledge For the action gradient a of the i-th agent i ; (IV-III) Application of Soft Update Mechanism During the training process, the parameters of the target network are updated using soft updates; The formula for the soft update is: Q μ′ ←τQ μ +(1-τ)Q μ′ (8) Where: τ is the soft update coefficient; (IV-IV) Inverter collaborative decision-making The Actor is trained using the above method. In the decentralized execution phase after training, the Actor network outputs the inverter action strategy as follows: Where: a i The inverter action for output, Represents the process of mapping the policy network from observations to actions.

Images