A Method and System for Power Distribution Network Load Balancing Security Control Based on Multi-Agent Reinforcement Learning

By adopting a two-stage training architecture of generative adversarial imitation learning and graph attention mechanism, combined with prototype matching decision and safety constraint verification, the problems of low training efficiency, imprecise collaborative decision and insufficient adaptive capability of agent model in power distribution network source-load balance control are solved, thus achieving efficient and safe source-load balance control.

CN123121569APending Publication Date: 2026-11-17WEIHAI POWER SUPPLY COMPANY OF STATE GRID SHANDONG ELECTRIC POWER COMPANY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611203800.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-10
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing power grid source-load balance control methods face problems such as insufficient training efficiency and generalization ability of agent models, imprecise multi-agent collaborative decision-making, insufficient security of control strategies, and limited adaptive capabilities after a high proportion of distributed photovoltaic access.

Method used

A two-stage training architecture based on generative adversarial imitation learning is adopted to pre-train the agent model. Combined with graph attention mechanism and prototype matching decision, multi-agent collaborative decision-making is carried out through local state observation and neighborhood information interaction. A security constraint verification module is introduced to realize the security control of centralized training and distributed execution.

Benefits of technology

It significantly improves the training efficiency and generalization ability of the agent model, realizes refined multi-agent collaborative decision-making, ensures the security of the control strategy, enhances the adaptability to unknown scenarios, and balances the computational advantages of centralized training with the real-time requirements of distributed execution.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123121569A_ABST
    Figure CN123121569A_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for power distribution network source-load balance safety control based on multi-agent reinforcement learning, belonging to the field of power distribution network control technology. It includes configuring independent agent models, obtaining a policy network through a two-stage architecture combining offline pre-training via generative adversarial learning and online fine-tuning via multi-agent reinforcement learning; each agent makes forward decisions based on the local state observation space of its local and neighboring nodes, and uses a graph attention mechanism to dynamically calculate the influence weights of the neighborhood to achieve multi-agent collaborative correction; a centralized training and distributed execution framework balances global optimization and real-time response; and online verification of safety constraints and constraint projection correction strictly ensure that control actions meet physical operating constraints. This invention effectively improves the training efficiency, decision-making collaboration, safety assurance capability, and scenario adaptability of power distribution network source-load balance control.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a method and system for source-load balance security control of distribution networks based on multi-agent reinforcement learning, belonging to the field of distribution network control technology. Background Technology

[0002] With the deepening implementation of the "dual carbon" goals, the penetration rate of distributed photovoltaic (PV) power generation in distribution networks continues to rise. The high proportion of distributed PV integration has profoundly changed the operating characteristics of distribution networks: on the one hand, the intermittency and volatility of PV output have led to a significant increase in the power fluctuation amplitude of the distribution network; on the other hand, the traditional unidirectional power flow distribution network is gradually evolving into an active distribution network with bidirectional power flow, and the uncertainty of power flow direction is becoming increasingly prominent. These changes pose a severe challenge to the real-time source-load balance control of the distribution network.

[0003] Currently, the source-load balance control of power distribution networks mainly relies on the following technical approaches: (I) Control methods based on traditional physical models. These methods are based on accurate physical modeling of the distribution network and formulate control strategies by solving the Optimal Power Flow (OPF) problem. However, with the integration of a high proportion of distributed photovoltaic power, the distribution network topology is highly variable and the operating scenarios are complex. Traditional physical models are difficult to accurately describe the real-time dynamic characteristics of the system, and the computational burden of solving the optimal power flow problem online is too heavy, making it difficult to meet the requirements of millisecond-level real-time control.

[0004] (II) Control Methods Based on Single-Agent Reinforcement Learning. In recent years, Deep Reinforcement Learning (DRL) has been gradually introduced into the field of distribution network control. These methods learn control strategies through trial and error between the agent and the environment, without requiring a precise physical model. However, the distribution network is a typical complex system with multiple nodes, multiple devices, and strong coupling. The single-agent method maps the control problem of the entire distribution network to the decision-making problem of a single agent, which faces scalability bottlenecks such as excessively high state space dimensionality and explosive action space, making it difficult to adapt to the distributed control requirements of large-scale distribution networks.

[0005] (III) Control Methods Based on Multi-Agent Reinforcement Learning. To address the scalability issues of single-agent methods, Multi-Agent Reinforcement Learning (MARL) has been introduced into the field of distribution network control. Existing technologies disclose load and power supply collaborative control methods based on MARL, and active distribution network collaborative voltage regulation methods based on multi-agent deep reinforcement learning. The Centralized Training with Decentralized Execution (CTDE) framework has been applied in multi-agent distribution network control. The application of Transformer architecture and self-attention mechanism in multi-agent distribution network control has also been disclosed.

[0006] However, existing technologies still have the following shortcomings: First, the training efficiency and generalization ability of agent models are insufficient. Existing methods typically train multi-agent models from scratch, requiring a large number of online interaction samples, resulting in long training times and slow convergence. Especially when the operating scenarios of distribution networks change frequently, the generalization ability of the models is difficult to guarantee. Although some studies have explored the application of imitation learning in distribution network optimization, a systematic training paradigm of "pre-training + online fine-tuning" has not yet been formed.

[0007] Second, the collaborative decision-making mechanism among multiple agents is not sophisticated enough. Existing methods often employ simple information sharing or global state splicing in multi-agent collaborative decision-making, failing to fully consider the topological characteristics of the distribution network and the spatial coupling relationships between agents. Although graph attention mechanisms have been explored for distribution network voltage control, their deep integration with online decision-making among multiple agents remains insufficient.

[0008] Third, the security mechanisms for control strategies are insufficient. Reinforcement learning agents may output control actions that violate the physical constraints of the distribution network during the exploration process (such as exceeding node voltage limits or branch current limits). Most existing methods indirectly constrain the agent's behavior by introducing penalty terms into the reward function, which cannot strictly guarantee the security of the control strategy. Although some research has explored secure projection verification mechanisms, a framework for online verification and projection correction of security constraints that is organically integrated with multi-agent collaborative decision-making has not yet been established.

[0009] Fourth, the agent's ability to adapt to unknown scenarios is limited. Distribution network operation scenarios are highly time-varying and uncertain. Random output of distributed photovoltaic power, load fluctuations, and topology changes may all cause the agent to encounter states not seen during the training phase. Existing methods lack effective mechanisms to quickly adapt to new scenarios using historical experience.

[0010] In summary, there is an urgent need for a power distribution network source-load balance safety control method and system that can balance control accuracy, computational efficiency, decision security, and scenario adaptability. Summary of the Invention

[0011] Based on the problems described in the background, the problem to be solved by the present invention is to provide a method and system for power distribution network source-load balance security control based on multi-agent reinforcement learning, so as to solve the problems mentioned above.

[0012] To achieve the above objectives, the present invention provides the following technical solution: a method for source-load balance security control of distribution networks based on multi-agent reinforcement learning, comprising the following steps: S1. Construct a multi-agent system: Configure an independent agent model for each controlled device in the power distribution network, and obtain each agent model using a two-stage training architecture; The controlled equipment includes distributed power supply equipment, energy storage equipment, controllable load equipment, and reactive power compensation equipment. The two-stage training architecture includes: the first stage, which uses historical operating data of the distribution network to pre-train the policy network of the agent model offline based on generative adversarial imitation learning, so that the policy network learns the distribution characteristics of expert control actions; the second stage, which deploys the pre-trained policy network to the distribution network environment and performs online fine-tuning and optimization based on multi-agent reinforcement learning algorithm. S2. Obtain the local state observation space: Each controlled device collects the electrical state information of its local node and the electrical state information of its physically connected neighbor nodes, splices and normalizes the collected information, and constructs the local state observation space at the current moment. The electrical status information includes node voltage amplitude, node injected active power, node injected reactive power, branch current amplitude, and power factor. S3. Agent forward decision-making: Each agent model inputs the local state observation space into its policy network for forward propagation calculation, and the policy network outputs the candidate action decision for the corresponding controlled device. S4. Multi-agent collaborative decision-making: Each agent model interacts with neighboring agents through a preset communication topology to obtain candidate action decision information and state information of the neighboring agents. Based on its own local state observation space and the obtained neighborhood information, each agent model uses an attention mechanism to calculate the influence weight of the neighboring agents on the current agent's decision. Based on the influence weight, it collaboratively corrects its own candidate action decision to generate a collaborative action decision. S5. Joint Action Optimization and Safety Verification: The action decisions of each intelligent agent model after collaboration are aggregated into a joint control action vector. The joint control action vector is globally evaluated through a centralized evaluation network. The evaluated joint control action vector is input into the safety constraint verification module. Verification and constraint projection correction are performed according to the physical operation constraints of the distribution network to generate the final control action vector that satisfies all physical operation constraints. S6. Control command execution: Convert the final control action vector into control commands for each controlled device and issue them for execution to achieve real-time balance control of source, grid, load and storage.

[0013] Preferably, the policy network in step S1 integrates an encoder based on the Transformer architecture and a decision generator based on prototype matching. The encoder of the Transformer architecture uses a multi-head self-attention mechanism to jointly encode the temporal features and spatial topological features in the local state observation space to generate a state feature vector. The prototype matching-based decision maker has a prototype library pre-installed. The prototype library is constructed in the following way: collecting state-action pair data of the power distribution network under various typical operating scenarios, using a clustering algorithm to cluster the state-action pair data, storing each cluster center as a prototype in the prototype library, and each prototype contains a typical state feature vector and its corresponding typical action feature vector. The decision-maker performs similarity matching between the state feature vector and each prototype in the prototype library. Based on the similarity calculation result, it activates at least one active prototype that best matches the current state from the prototype library. The decision-maker then performs feature fusion between the state feature vector and the typical action feature vector of the active prototype to generate the candidate action decision.

[0014] Preferably, the prototype library is dynamically updated during the operation of the distribution network, and the dynamic update includes: During the operation of the power distribution network, the actual control effect data of each controlled device is collected in real time, and the similarity between the current state-action pair and each prototype in the prototype library is calculated. If the similarity between the current state-action pair and all prototypes is lower than a preset threshold, then the current state-action pair will be added to the prototype library as a new prototype. If the similarity between the current state-action pair and a certain prototype is higher than a preset threshold, but the deviation between the actual control effect and the typical control effect corresponding to the prototype exceeds a preset deviation threshold, then the corresponding prototype is updated with weight based on the current state-action pair. The weights for the weighted update are dynamically determined based on the current state and the confidence level of the action pair.

[0015] Preferably, the offline pre-training based on generative adversarial imitation learning in step S1 includes: Collect historical operating data of the distribution network. The historical operating data includes the state sequence of each controlled device under various source and load scenarios and the corresponding expert control action sequence. The expert control action sequence is obtained based on the optimal power flow calculation of the distribution network or human expert experience. A generative adversarial imitation learning framework is constructed, comprising a generator and a discriminator; the generator adopts the same network structure as the policy network and is used to generate imitation actions based on the current state; the discriminator is used to determine the degree of similarity between the generated actions and the expert-controlled actions; The generator and discriminator are trained adversarially based on the historical operating data. During the training process, the generator aims to maximize the probability of misjudgment by the discriminator, while the discriminator aims to maximize the accuracy of distinguishing between generated actions and expert-controlled actions. Training stops when the discrimination accuracy of the discriminator converges to a preset threshold range. The network parameters of the generator after training convergence are used as the initialization parameters of the policy network to complete offline pre-training.

[0016] Preferably, the attention mechanism in step S4 is a graph attention mechanism. Each agent model calculates the influence weights of neighboring agents on the current agent's decision through the graph attention mechanism, specifically including: Each agent model uses its own local state observation space features and the state information features of each neighboring agent as query vector and key vector, respectively. The attention coefficients between the current agent and each neighboring agent are calculated through a graph attention layer. These attention coefficients are calculated based on the similarity between the query vector and the key vector. The attention coefficients are normalized to obtain the influence weights of each neighboring agent on the current agent's decision. Each agent model performs weighted aggregation of the candidate action decisions of each neighboring agent according to the influence weight, and performs residual connection and fusion with its own candidate action decisions to generate collaborative action decisions. The network parameters of the graph attention mechanism are jointly optimized with the policy network parameters during the online fine-tuning process of multi-agent reinforcement learning in the second stage.

[0017] Preferably, the constraint projection correction of the safety constraint verification module in step S5 includes: The physical operation constraints include upper and lower limits of node voltage amplitude, thermal stability limit of branch current, upper and lower limits of active power output of distributed power sources, upper and lower limits of state of charge of energy storage devices, and node power balance constraints. The safety constraint verification module first determines whether the joint control action vector satisfies all physical operation constraints. If it does, it directly outputs the joint control action vector as the final control action vector. If the joint control action vector does not satisfy any physical operation constraint, the safety constraint verification module constructs a constraint projection optimization problem with the objective function of minimizing the control action adjustment and with all physical operation constraints as constraints. Solve the constraint projection optimization problem by projecting the joint control action vector to the nearest point on the feasible region boundary to obtain a modified control action vector that satisfies all physical operation constraints, which is then output as the final control action vector.

[0018] Preferably, the second stage of multi-agent reinforcement learning online fine-tuning optimization in step S1 adopts a centralized training and distributed execution framework, specifically including: During the training phase, each agent model sends its local state observation space, candidate action decisions, and collaborative action decisions to a centralized evaluation network in the cloud. The centralized evaluation network acquires the global state information of the distribution network, calculates the Q value of the joint action based on the global state information and the joint action of each agent, and feeds back the Q value as a reward signal to each agent model. Each agent model updates its own policy network parameters based on the reward signal using a policy gradient algorithm. During the execution phase, each intelligent agent model runs independently on the side of each controlled device, making distributed decisions based only on its own local state observation space and limited neighborhood information obtained through communication interaction. It does not need to obtain global state information, nor does it need to rely on the centralized evaluation network. When a change in the distribution network topology or a change in the access status of distributed power sources is detected, rapid retraining of each agent model is triggered. The rapid retraining is completed by a small number of iterations and fine-tuning based on the network parameters obtained from offline pre-training.

[0019] Preferably, in step S6, the real-time balance control of the source, grid, load, and storage includes active power balance control and reactive power balance control: The active power balance control includes: achieving real-time balance of active power in the distribution network by adjusting the active power output setpoint of the distributed power inverter, the charging and discharging power setpoint of the energy storage device, and the switching status of the controllable load; The reactive power balance control includes: achieving real-time stability of the voltage at distribution network nodes by adjusting the reactive power output setpoint of the distributed power inverter and the switching capacity of the reactive power compensation equipment; The active power balance control and reactive power balance control are jointly decided by the intelligent agent model of the corresponding controlled equipment based on a unified local state observation space. The joint control action vector contains both active power control commands and reactive power control commands.

[0020] This invention also provides a power distribution network source-load balance safety control system based on multi-agent reinforcement learning, comprising: Multiple agent models are configured on the controlled equipment side of the power distribution network. Each agent model includes a state perception module, a policy decision module, and a communication interaction module. The state perception module is used to collect electrical state information of local nodes and physically connected neighboring nodes to construct a local state observation space. The policy decision module integrates a policy network obtained using a two-stage training architecture to generate candidate action decisions based on the local state observation space. The communication interaction module is used to interact with neighboring agents through a preset communication topology to obtain candidate action decision information and state information of neighboring agents. The multi-agent collaborative decision engine is connected to the communication and interaction modules of each agent model. It is used to receive the candidate action decisions and state information of each agent model, calculate the influence weights between neighboring agents based on the attention mechanism and perform collaborative corrections to generate the collaborative action decisions of each controlled device. A centralized evaluation network, deployed on a cloud server, communicates with the multi-agent collaborative decision engine. It is used to acquire global state information of the distribution network and the joint actions of each agent during the training phase, calculate the Q value of the joint actions, and feed them back to each agent model for policy network parameter updates. The safety constraint verification module is connected to the multi-agent collaborative decision engine and is used to perform physical operation constraint verification and constraint projection correction on the joint control action vector formed by the aggregation of collaborative action decisions. The control command execution module is connected to the safety constraint verification module and each controlled device, and is used to convert the final control action vector after verification or correction into control commands for each controlled device and issue them for execution.

[0021] Preferably, the system adopts a centralized training and distributed execution architecture: During the training phase, the centralized evaluation network acquires the global state information of the distribution network and the joint actions of each agent model. Based on the global state information, it calculates the global Q value of the joint actions and distributes the global Q value as a training signal to each agent model through the multi-agent collaborative decision engine. Each agent model updates the policy network parameters synchronously based on the training signal using the policy gradient algorithm. During the execution phase, each agent model is decoupled from the centralized evaluation network. The state perception module, strategy decision-making module, and communication interaction module of each agent model run independently on the edge computing unit of each controlled device. Each agent model makes distributed decisions based only on its own local state observation space and the limited neighborhood information obtained through the communication interaction module. The safety constraint verification module and the control command execution module run on the local controller of the distribution network. All modules work together to complete the real-time balance control of source-grid-load-storage. When the operating status of the distribution network changes significantly, the centralized evaluation network and each agent model are triggered to undergo rapid retraining. The rapid retraining is completed by a small number of iterations and fine-tuning based on the network parameters obtained by offline pre-training. The retrained network parameters are then updated to the agent models of each controlled device through cloud delivery.

[0022] The beneficial effects of this invention are: 1. Significantly improves the training efficiency and generalization ability of intelligent agent models.

[0023] This invention employs a two-stage training architecture: "Generative Adversarial Imitation Learning (GAP) pre-training + multi-agent reinforcement learning online fine-tuning." In the first stage, GAP utilizes historical power grid operation data for offline pre-training of the policy network, enabling agents to possess strong initial decision-making capabilities before deployment, thus avoiding the "cold start" problem of traditional reinforcement learning. In the second stage, policy fine-tuning and optimization require only minimal online interaction, significantly shortening training time and improving the model's generalization ability across different operating scenarios. Compared to existing methods of training multi-agent models from scratch, this invention improves training efficiency by over 50%.

[0024] 2. Achieve refined multi-agent collaborative decision-making.

[0025] This invention constructs a multi-agent collaborative decision-making framework based on graph attention mechanisms, consisting of "local observation, graph attention collaboration, and residual fusion." Each agent adaptively learns the influence weights of neighboring agents on its own decisions through the graph attention mechanism, enabling dynamic adjustment of collaborative strategies according to the real-time topology and operating status of the distribution network. Compared to traditional global state splicing or simple information sharing methods, the collaborative decision-making mechanism of this invention is more refined and targeted, effectively handling the complex spatial coupling relationships between agents in the distribution network.

[0026] 3. Strictly ensure the security of control strategies.

[0027] This invention adds an online verification and constraint projection correction stage for safety constraints after the reinforcement learning action output. Unlike existing technologies that indirectly constrain agent behavior by introducing a penalty term into the reward function, this invention solves a constraint projection optimization problem, directly projecting control actions that do not meet physical constraints to the nearest point on the feasible region boundary. This mathematically guarantees that the final executed control action satisfies all physical operational constraints. This mechanism effectively avoids the risk of the reinforcement learning agent outputting dangerous actions due to exploration or model errors, enabling the method of this invention to be truly applied to real-time control scenarios in power distribution networks with extremely high safety requirements.

[0028] 4. Enhance the agent's ability to adapt to unknown scenarios.

[0029] This invention introduces a prototype matching and decision-making mechanism into the policy network, storing state-action patterns under typical operating scenarios through a pre-built prototype library. When the agent encounters an unknown scenario, prototype matching can quickly activate the most similar historical experience to assist in generating reasonable action decisions. Simultaneously, the prototype library is dynamically updated during the operation of the distribution network, continuously absorbing new operating patterns, thus enhancing the agent's decision-making ability with the accumulation of operating experience. This mechanism significantly improves the agent's adaptability when facing uncertainties such as stochastic output of distributed photovoltaic power, load fluctuations, and topology changes.

[0030] 5. It balances the computational advantages of centralized training with the real-time requirements of distributed execution.

[0031] This invention employs a centralized training and distributed execution (CTDE) framework. During the training phase, a cloud-based centralized evaluation network is used to acquire global information for joint optimization, overcoming the coordination difficulties caused by incomplete information in distributed execution. During the execution phase, each agent makes independent decisions based solely on local observations and limited neighborhood information, without needing to acquire global state information, significantly reducing communication bandwidth requirements and computational latency. This architectural design allows the method of this invention to possess both the theoretical advantages of centralized optimization for global optimality and meet the engineering requirements of low latency and high reliability for real-time control of power distribution networks.

[0032] 6. Achieve joint optimization control of active and reactive power.

[0033] This invention achieves joint decision-making and coordinated execution of active and reactive power balance control in distribution networks through a unified intelligent agent model and joint control action vectors. Compared with traditional methods where active and reactive power control are independent and lack coordination, the joint optimization method of this invention can fully utilize the active / reactive power regulation capabilities of distributed power inverters, as well as various flexible resources such as energy storage devices, reactive power compensation devices, and controllable loads, to achieve global optimal coordination among power sources, grids, loads, and storage. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Detailed Implementation

[0035] The embodiments of the present invention will be further described below with reference to the accompanying drawings: Example

[0036] like Figure 1 As shown, this invention provides a power distribution network source-load balance security control method based on multi-agent reinforcement learning, comprising the following steps: S1. Construct a multi-agent system: Configure an independent agent model for each controlled device in the power distribution network, and obtain each agent model using a two-stage training architecture; The controlled equipment includes distributed power supply equipment, energy storage equipment, controllable load equipment, and reactive power compensation equipment. The two-stage training architecture includes: the first stage, which uses historical operating data of the distribution network to pre-train the policy network of the agent model offline based on generative adversarial imitation learning, so that the policy network learns the distribution characteristics of expert control actions; the second stage, which deploys the pre-trained policy network to the distribution network environment and performs online fine-tuning and optimization based on multi-agent reinforcement learning algorithm. The second stage of multi-agent reinforcement learning online fine-tuning optimization in step S1 adopts a centralized training and distributed execution framework, specifically including: During the training phase, each agent model sends its local state observation space, candidate action decisions, and collaborative action decisions to a centralized evaluation network in the cloud. The centralized evaluation network acquires the global state information of the distribution network, calculates the Q value of the joint action based on the global state information and the joint action of each agent, and feeds back the Q value as a reward signal to each agent model. Each agent model updates its own policy network parameters based on the reward signal using a policy gradient algorithm. During the execution phase, each intelligent agent model runs independently on the side of each controlled device, making distributed decisions based only on its own local state observation space and limited neighborhood information obtained through communication interaction. It does not need to obtain global state information, nor does it need to rely on the centralized evaluation network. When a change in the distribution network topology or a change in the access status of distributed power sources is detected, rapid retraining of each agent model is triggered. The rapid retraining is completed by a small number of iterations and fine-tuning based on the network parameters obtained from offline pre-training.

[0037] In the first stage, the core of this invention lies in achieving imitation learning by utilizing the adversarial training mechanism of Generative Adversarial Networks (GANs). Specifically, two neural network modules, a generator and a discriminator, are constructed. The network structure of the generator is completely consistent with the policy network structure of the subsequent online reinforcement learning stage. Its input is the local state observation space at the current moment, and its output is the control action to be imitated. The input of the discriminator is a "state-action" pair, and its output is the probability that the action is an expert action or a generated action.

[0038] During training, the generator attempts to produce imitation actions that are as similar as possible to the expert's actions to "deceive" the discriminator, while the discriminator tries to accurately distinguish between the generated actions and the expert's actions. Both continuously optimize through iterative adversarial training: the discriminator gradually learns to differentiate between real expert actions and imitation actions with greater finer detail, while the generator gradually adjusts its policy network parameters to make its generated actions increasingly closer to the distribution characteristics of the expert's actions. When the discriminator's accuracy converges to near the level of random guessing (i.e., 50%), it indicates that the generator has successfully learned the distribution characteristics of the expert's control policy.

[0039] At this point, the network parameters of the generator are extracted and used as the initialization parameters for the policy networks of each agent model. The advantage of this approach is that, through offline pre-training, the agents have already mastered the basic knowledge of expert control policies before deployment, avoiding the blind exploration during the "cold start" of traditional reinforcement learning, and significantly improving the starting point and efficiency of subsequent online learning.

[0040] In the second stage, the pre-trained policy network is deployed to the power distribution network environment, and online fine-tuning and optimization are performed using a multi-agent reinforcement learning algorithm. Since the agents already possess reasonable initial decision-making capabilities, the online fine-tuning stage only requires minimal interaction with the real environment to complete the fine-tuning of the policy, resulting in a significant improvement in sample efficiency compared to training from scratch.

[0041] The online fine-tuning specifically employs a Centralized Training and Distributed Execution (CTDE) framework, whose core feature lies in the separation of the training and execution phases: during the training phase, each agent can acquire global state information for joint training; during the execution phase, each agent relies only on local observation information for distributed decision-making. A detailed implementation of the CTDE framework will be further explained in subsequent embodiments.

[0042] The two-stage training architecture effectively solves the technical problems of low training efficiency, difficulty in cold start, and insufficient generalization ability in multi-agent reinforcement learning control of power distribution networks by organically combining the sample efficiency advantage of offline pre-training with the environment adaptation capability of online fine-tuning. It achieves the beneficial technical effects of significantly improved training efficiency, reasonable decision-making ability after deployment, and scene adaptation with small samples.

[0043] This invention adopts a centralized training and distributed execution (CTDE) framework. Its core idea lies in the separation of the training phase and the execution phase, enabling the system to simultaneously achieve the global optimality of centralized optimization and the real-time response capability of distributed execution.

[0044] During the training phase, each agent sends its local information to the centralized evaluation network. The centralized evaluation network can acquire global state information of the distribution network—a global perspective unavailable to individual agents in distributed execution. Based on the global state and the joint actions of all agents, the centralized evaluation network calculates the Q-value of the joint actions, which reflects the quality of the current joint control strategy on a global scale. This Q-value is fed back to each agent as a training signal, and each agent updates its own network parameters using a policy gradient algorithm based on this signal. In this stage, the presence of the centralized evaluation network allows each agent to consider the impact of other agents' decisions on the global effect during training, thereby solving the credit allocation problem among multiple agents and achieving effective collaborative learning.

[0045] During the execution phase, each agent model is completely decoupled from the centralized evaluation network. Each agent operates independently on the edge computing unit of its respective controlled device, making decisions solely based on its local state observation space and limited neighborhood information acquired through communication interaction. The engineering significance of this design is threefold: First, it eliminates reliance on global communication, reducing communication bandwidth requirements and avoiding system paralysis caused by single-point communication failures; second, it eliminates reliance on centralized computing, significantly reducing decision latency and meeting millisecond-level real-time control requirements; and third, each agent can operate independently, providing excellent system scalability.

[0046] During the operation of a distribution network, topology changes (such as line maintenance, fault reconfiguration, etc.) or changes in the access status of distributed power sources (such as the addition of new photovoltaic grid connections) may occur. Such changes can lead to significant alterations in the dynamic characteristics of the system, and previously trained strategies may no longer be applicable.

[0047] To address the above issues, this invention designs a rapid retraining mechanism. When the system detects a significant change in the distribution network state, it triggers the rapid retraining process. The core technical feature of this process is that retraining does not start from scratch, but rather uses the model parameters obtained in the previous stage (offline pre-training stage) as initial values, collecting a small amount of operational data under the new scenario for iterative fine-tuning.

[0048] The technical advantage of this mechanism lies in the fact that the offline pre-trained model already contains the basic knowledge and general strategy patterns of distribution network control. When the scenario changes, the agent does not need to "relearn" all the knowledge, but only needs to make "adaptive adjustments" on the new scenario data. This rapid retraining based on the idea of ​​transfer learning greatly reduces the sample size and time required for model updates, enabling the agent to quickly recover high-performance decision-making capabilities after changes in the distribution network scenario.

[0049] In summary, this implementation method achieves beneficial technical effects such as global optimization during the training phase, real-time reliability during the execution phase, and rapid recovery from scene changes.

[0050] The policy network in step S1 integrates an encoder based on the Transformer architecture and a decision maker based on prototype matching. The encoder of the Transformer architecture uses a multi-head self-attention mechanism to jointly encode the temporal features and spatial topological features in the local state observation space to generate a state feature vector. The prototype matching-based decision maker has a prototype library pre-installed. The prototype library is constructed in the following way: collecting state-action pair data of the power distribution network under various typical operating scenarios, using a clustering algorithm to cluster the state-action pair data, storing each cluster center as a prototype in the prototype library, and each prototype contains a typical state feature vector and its corresponding typical action feature vector. The decision-maker performs similarity matching between the state feature vector and each prototype in the prototype library. Based on the similarity calculation result, it activates at least one active prototype that best matches the current state from the prototype library. The decision-maker then performs feature fusion between the state feature vector and the typical action feature vector of the active prototype to generate the candidate action decision.

[0051] The local state observation space of a distribution network has two main characteristics: first, it contains time-series information from multiple consecutive moments, reflecting dynamic processes such as power fluctuations and voltage changes; second, it contains spatial topology information of local nodes and neighboring nodes, reflecting the power flow coupling relationship of the distribution network. How to effectively extract meaningful features from high-dimensional, heterogeneous, time-series, and spatially coupled observation data is the core challenge of policy network design.

[0052] This invention employs a Transformer-based encoder to address the aforementioned problems. The core of the Transformer encoder is a self-attention mechanism, the basic principle of which is: for each position in the input feature sequence, global context information is adaptively aggregated by calculating the attention weights between that position and all other positions in the sequence. Specifically, in this invention, the encoder input is a temporally stacked sequence of local state observations. The self-attention mechanism can adaptively focus on relevant information from other times while calculating the feature representation at each time step, thereby effectively capturing the temporal dynamics of power fluctuations. Simultaneously, since the observation vector contains the topological connections between nodes, the self-attention mechanism can also implicitly model the spatial coupling relationships between nodes.

[0053] Compared to traditional recurrent neural networks (RNNs) or convolutional neural networks (CNNs), the Transformer encoder has the following advantages: First, the self-attention mechanism can directly model the long-range dependency between any two positions in a sequence, without being limited by the gradient vanishing problem in RNNs; second, self-attention computation can be highly parallelized, with computational efficiency superior to the stepwise recursive method of RNNs; third, the multi-head self-attention mechanism can simultaneously capture different feature patterns of input data from multiple different subspaces, enhancing the richness and robustness of feature representation.

[0054] Through the encoding process of the Transformer encoder, the original local state observation space is transformed into a highly expressive state feature vector, providing a high-quality input representation for subsequent prototype matching and decision-making.

[0055] Despite the powerful feature extraction capabilities of the Transformer encoder, neural networks that rely solely on data-driven approaches still have the following shortcomings: when the agent encounters a scenario that has not been seen during training, the network may produce unpredictable outputs, and there is a lack of effective mechanisms to use historical experience to guide current decisions.

[0056] To address the aforementioned issues, this invention innovatively introduces a prototype-matching-based decision-maker into the policy network. The prototype library is a pre-built knowledge base that stores "state-action" patterns of the distribution network under various typical operating scenarios. Each prototype comprises two parts: a typical state feature vector and a corresponding typical action feature vector. The prototype library is constructed based on historical operating data, using a clustering algorithm to extract the most representative state-action patterns as prototypes.

[0057] The prototype matching decision-maker works as follows: First, it calculates the similarity between the state feature vector output by the Transformer encoder and the typical state features of each prototype in the prototype library; then, it activates one or more prototypes that best match the current state based on the similarity ranking; finally, it fuses the current state features with the typical action features of the activated prototypes to generate the final action decision.

[0058] The core technological advantage of this mechanism lies in the following: when the agent faces a familiar scene, prototype matching can quickly retrieve the most similar historical experiences to assist in generating reasonable action decisions; when the agent faces a completely unfamiliar new scene, although there is no perfectly matching prototype, it can still make inference decisions based on existing knowledge by activating several of the closest prototypes and fusing their features, rather than blindly trying "from scratch". This "experience-based" decision-making method significantly improves the agent's adaptability and decision reliability in unknown scenarios.

[0059] In addition, the prototype library has the advantage of "knowledge readability"—by checking which prototypes are activated, the decision-making basis of the agent can be intuitively understood, which enhances the interpretability of the model and makes it easier for operation and maintenance personnel to review and trust the control strategy.

[0060] In summary, the "Transformer encoder + prototype matching decision-maker" architecture described in this embodiment organically combines data-driven feature extraction with experience-based pattern matching. This not only leverages the expressive power of a large-capacity neural network but also utilizes historical experience knowledge to enhance generalization ability and decision reliability, resulting in beneficial technical effects such as high decision quality, strong generalization ability, and good interpretability.

[0061] The prototype library is dynamically updated during the operation of the power distribution network. This dynamic updating includes: During the operation of the power distribution network, the actual control effect data of each controlled device is collected in real time, and the similarity between the current state-action pair and each prototype in the prototype library is calculated. If the similarity between the current state-action pair and all prototypes is lower than a preset threshold, then the current state-action pair will be added to the prototype library as a new prototype. If the similarity between the current state-action pair and a certain prototype is higher than a preset threshold, but the deviation between the actual control effect and the typical control effect corresponding to the prototype exceeds a preset deviation threshold, then the corresponding prototype is updated with weight based on the current state-action pair. The weights for the weighted update are dynamically determined based on the current state and the confidence level of the action pair.

[0062] The operation of power distribution networks is highly time-varying and uncertain: distributed photovoltaic output varies with weather conditions, load levels fluctuate with time and season, and the topology may change due to faults or maintenance. These changes mean that even if the prototype library covers as many typical scenarios as possible during its construction, new operating modes that were not included in the training data may still emerge during actual operation. If the prototype library is static and unchanging, the accuracy of prototype matching will decrease when new scenarios appear, thus affecting the quality of decision-making.

[0063] To address the aforementioned issues, this invention designs an online dynamic update mechanism for the prototype library. This mechanism operates continuously during the normal operation of the power distribution network, monitoring and evaluating each control decision made by the intelligent agent and its actual effects.

[0064] Specifically, for the state-action pair at the current runtime, the dynamic update mechanism performs the following judgments and operations: First, calculate the overall similarity between the current state-action pair and all prototypes in the prototype library. If the similarity with all prototypes is lower than a preset first threshold, it means that the current state-action pattern has never appeared in historical experience and belongs to a new operating mode. Therefore, the current state-action pair is added to the prototype library as a new prototype to achieve "learning and memorizing" unknown scenarios.

[0065] Second, if the similarity between the current state-action pair and a certain prototype is higher than a preset second threshold (the second threshold is higher than the first threshold), but the actual control effect deviates significantly from the typical control effect corresponding to that prototype, it indicates that although the current scenario is similar to a certain historical prototype at the state level, the action strategy corresponding to the original prototype is no longer optimal due to potential changes in the operating conditions of the distribution network (such as changes in line parameters, equipment aging, etc.). In this case, the corresponding prototype is updated with weights based on the current state-action pair, enabling the prototype to adapt to the slow changes in system characteristics.

[0066] Third, a confidence weighting mechanism is introduced during the update process, which dynamically adjusts the update step size based on the reliability of the current control effect. When the uncertainty of the control effect is high (such as when it is affected by a large disturbance), a smaller update weight is assigned to avoid abnormal data from polluting the prototype library; when the control effect is reliable, a larger update weight is assigned to accelerate the prototype's ability to track system changes.

[0067] Fourth, to prevent the prototype library from growing indefinitely and causing a decrease in retrieval efficiency, the dynamic update mechanism also includes regular prototype simplification operations, deleting "outdated" prototypes that have not been activated for a long time, and controlling the size of the prototype library within a reasonable range.

[0068] Through the aforementioned dynamic update mechanism, the prototype library evolves from a static knowledge base into a continuously evolving and self-improving knowledge system. This mechanism enables the agent's decision-making ability to continuously enhance with the accumulation of operational experience, allowing it to adapt to the dynamic changes in power distribution network operation scenarios and achieving beneficial technical effects such as long-term performance without degradation, rapid adaptation to new scenarios, and continuous knowledge accumulation.

[0069] The offline pre-training based on generative adversarial imitation learning in step S1 includes: Collect historical operating data of the distribution network. The historical operating data includes the state sequence of each controlled device under various source and load scenarios and the corresponding expert control action sequence. The expert control action sequence is obtained based on the optimal power flow calculation of the distribution network or human expert experience. A generative adversarial imitation learning framework is constructed, comprising a generator and a discriminator; the generator adopts the same network structure as the policy network and is used to generate imitation actions based on the current state; the discriminator is used to determine the degree of similarity between the generated actions and the expert-controlled actions; The generator and discriminator are trained adversarially based on the historical operating data. During the training process, the generator aims to maximize the probability of misjudgment by the discriminator, while the discriminator aims to maximize the accuracy of distinguishing between generated actions and expert-controlled actions. Training stops when the discrimination accuracy of the discriminator converges to a preset threshold range. The network parameters of the generator after training convergence are used as the initialization parameters of the policy network to complete offline pre-training.

[0070] The core of generative adversarial learning lies in using "expert data" to guide the agent's learning. In this invention, expert data comes from the historical operation data of the distribution network. Specifically, it involves collecting the operating state sequences of the distribution network under various typical source-load scenarios, as well as the corresponding control action sequences. The "expert control actions" can be obtained through several methods: firstly, by offline solving the optimal power flow (OPF) problem in the distribution network. OPF can calculate the globally optimal control strategy given known system parameters and load forecasts. Although online OPF solving is computationally intensive and difficult to apply in real time, its results can serve as high-quality offline training labels; secondly, manual control strategies formulated by experienced dispatchers based on operational experience; and thirdly, existing historical control records of safe operation. Multiple data sources can be integrated to enrich the coverage of expert strategies.

[0071] After constructing the generative adversarial imitation learning framework, adversarial training is performed. The core mechanism of the training lies in the "game" between the generator and the discriminator: the generator aims to generate imitation actions that are difficult to distinguish from expert actions, thereby maximizing the discriminator's misclassification probability; the discriminator aims to determine as accurately as possible whether an action is an expert action or a generated action. The two mutually promote each other through alternating optimization—the discriminator continuously improves its discrimination ability, forcing the generator to continuously improve its generation quality; the generator's continuous improvement in generation quality, in turn, forces the discriminator to continuously improve its discrimination accuracy.

[0072] The mathematical essence of this adversarial training process is that the generator continuously "simulates" the expert policy distribution, while the discriminator acts as a "distribution difference measurer," providing gradient feedback signals to the generator. Compared to traditional behavioral cloning, generative adversarial learning has the following advantages: behavioral cloning is easily affected by data distribution shifts—when the agent's state distribution deviates from the state distribution in the training data, the error gradually accumulates and amplifies; while generative adversarial learning, through the adversarial feedback of the discriminator, can better learn the distribution characteristics of the expert policy, rather than just a simple state-action mapping, thus exhibiting stronger robustness to distribution shifts.

[0073] When the discriminator's accuracy converges to a preset threshold, it indicates that the generator is able to generate imitation actions that are highly similar to the expert's action distribution, and training is stopped at this point. The parameters of the generator network after training convergence are output as the initialization parameters of the policy network, completing offline pre-training.

[0074] It is worth emphasizing that this pre-training process is completed entirely offline, without any online interaction with the real power distribution network environment, and therefore poses no risk to the operational safety of the actual power distribution network. The policy network obtained after pre-training already possesses good initial decision-making capabilities, providing a high-quality starting point for subsequent online fine-tuning.

[0075] The technical solution described in this embodiment achieves the beneficial technical effects of significantly improving the starting point of online training, possessing safe decision-making capabilities from the early stages of training, and not consuming online resources during offline training by fully utilizing historical data to learn expert strategy knowledge during the offline phase.

[0076] S2. Obtain the local state observation space: Each controlled device collects the electrical state information of its local node and the electrical state information of its physically connected neighbor nodes, splices and normalizes the collected information, and constructs the local state observation space at the current moment. The electrical status information includes node voltage amplitude, node injected active power, node injected reactive power, branch current amplitude, and power factor. S3. Agent forward decision-making: Each agent model inputs the local state observation space into its policy network for forward propagation calculation, and the policy network outputs the candidate action decision for the corresponding controlled device. S4. Multi-agent collaborative decision-making: Each agent model interacts with neighboring agents through a preset communication topology to obtain candidate action decision information and state information of the neighboring agents. Based on its own local state observation space and the obtained neighborhood information, each agent model uses an attention mechanism to calculate the influence weight of the neighboring agents on the current agent's decision. Based on the influence weight, it collaboratively corrects its own candidate action decision to generate a collaborative action decision. The attention mechanism in step S4 is a graph attention mechanism. Each agent model calculates the influence weights of neighboring agents on the current agent's decision through the graph attention mechanism, specifically including: Each agent model uses its own local state observation space features and the state information features of each neighboring agent as query vector and key vector, respectively. The attention coefficients between the current agent and each neighboring agent are calculated through a graph attention layer. These attention coefficients are calculated based on the similarity between the query vector and the key vector. The attention coefficients are normalized to obtain the influence weights of each neighboring agent on the current agent's decision. Each agent model performs weighted aggregation of the candidate action decisions of each neighboring agent according to the influence weight, and performs residual connection and fusion with its own candidate action decisions to generate collaborative action decisions. The network parameters of the graph attention mechanism are jointly optimized with the policy network parameters during the online fine-tuning process of multi-agent reinforcement learning in the second stage.

[0077] In multi-agent control of power distribution networks, the coupling relationships between agents (i.e., controlled devices) are not equal. Physically connected nodes have strong electrical coupling, and changes in power injection directly affect the voltage and power flow of neighboring nodes; while the coupling between non-neighboring nodes is relatively weak. Therefore, in collaborative decision-making, agents should give different levels of attention to different neighboring agents—for strongly coupled neighbors, their action decisions have a greater impact on the current agent and should be given higher weights; for weakly coupled neighbors, the impact is smaller, and the weights are correspondingly lower.

[0078] Traditional multi-agent cooperative methods typically employ two approaches: one is to simply concatenate the information from all agents and input it into the network, allowing the network to learn the correlations between them. This approach ignores prior knowledge of the distribution network's topology, resulting in low learning efficiency. The other approach uses fixed weights or rule-based weighting, which cannot adapt to the dynamic changes in the distribution network's operating state. To address these issues, this invention introduces a graph attention mechanism.

[0079] The core idea of ​​the graph attention mechanism is to dynamically calculate the attention coefficient between the agent and its neighboring agents at each moment based on the agent's real-time state information, thereby measuring the importance of each neighbor to the agent's current decision.

[0080] The specific calculation process is as follows: The current agent uses its own state features as a query vector and the state features of each neighboring agent as key vectors. A learnable attention network is used to calculate the relevance score between the query vector and each key vector. Then, the relevance score is normalized (usually using the Softmax function) to obtain a normalized attention coefficient. This attention coefficient reflects the influence weight of each neighboring agent on the current agent's decision—the higher the attention coefficient, the stronger the relevance between the neighbor's state and the current agent, and the more important its action decision is to the current agent.

[0081] After obtaining the attention coefficients, the current agent performs weighted aggregation of the candidate action decisions of each neighboring agent to obtain a comprehensive representation of the neighborhood actions. Then, it performs residual connection and fusion of its own candidate action decisions with the weighted aggregation results to generate a collaborative action decision.

[0082] In the residual connection, a learnable gating coefficient is introduced to adaptively balance the relative importance of the decision itself and the neighborhood information. When the neighborhood information is helpful to the current decision, the gating coefficient increases, and more collaborative information is adopted. When the neighborhood information is less relevant or unreliable, the gating coefficient decreases, and more reliance is placed on the decision itself.

[0083] The technical advantages of the graph attention mechanism described in this embodiment are reflected in the following aspects: First, the attention coefficient is dynamically calculated according to the state, which can adaptively adjust the collaborative weight according to the real-time operating state of the distribution network, making it more flexible and accurate than the fixed weight scheme; Second, the network parameters of the graph attention mechanism are jointly optimized with the policy network parameters during the multi-agent reinforcement learning process, so that the calculation of the attention coefficient continuously tends to the optimal, and the quality of collaborative decision-making is continuously improved; Third, by limiting the attention calculation to the range of physical neighbors, the distributed nature of the decision is maintained, and the dependence on global information is avoided.

[0084] In summary, this implementation method achieves beneficial technical effects such as refined multi-agent collaborative decision-making, adaptive topology structure, dynamic weight optimization, and feasible distributed execution.

[0085] S5. Joint Action Optimization and Safety Verification: The action decisions of each intelligent agent model after collaboration are aggregated into a joint control action vector. The joint control action vector is globally evaluated through a centralized evaluation network. The evaluated joint control action vector is input into the safety constraint verification module. Verification and constraint projection correction are performed according to the physical operation constraints of the distribution network to generate the final control action vector that satisfies all physical operation constraints. The constraint projection correction of the safety constraint verification module in step S5 includes: The physical operation constraints include upper and lower limits of node voltage amplitude, thermal stability limit of branch current, upper and lower limits of active power output of distributed power sources, upper and lower limits of state of charge of energy storage devices, and node power balance constraints. The safety constraint verification module first determines whether the joint control action vector satisfies all physical operation constraints. If it does, it directly outputs the joint control action vector as the final control action vector. If the joint control action vector does not satisfy any physical operation constraint, the safety constraint verification module constructs a constraint projection optimization problem with the objective function of minimizing the control action adjustment and with all physical operation constraints as constraints. Solve the constraint projection optimization problem by projecting the joint control action vector to the nearest point on the feasible region boundary to obtain a modified control action vector that satisfies all physical operation constraints, which is then output as the final control action vector.

[0086] In the process of exploration and decision-making, the action strategies output by reinforcement learning agents are not mathematically guaranteed to satisfy the physical constraints of the system. In power distribution network control scenarios, even a seemingly minor control action exceeding the limit (such as voltage setpoints exceeding allowable ranges or energy storage charging / discharging power exceeding equipment ratings) can trigger electrical safety accidents. How to fully leverage the flexibility of reinforcement learning decision-making while strictly ensuring the safety of control strategies is a key technical problem that this invention must solve.

[0087] A common approach to this problem in existing technologies is to introduce a penalty term for constraint violation into the reward function, indirectly guiding the agent to learn a policy that satisfies the constraints through the penalty mechanism. However, this approach has a fundamental flaw: the penalty mechanism can only "guide" rather than "force," and the agent may still output actions that violate the constraints when training is insufficient or when encountering unknown states. For power distribution network control scenarios with extremely high safety requirements, this "soft constraint" approach is unacceptable.

[0088] To address the aforementioned problems, this invention proposes a technical solution of "online verification of safety constraints and correction of constraint projection." The core design idea of ​​this solution is to add a safety verification step after the reinforcement learning action output and before the actual control command is issued, to forcibly correct actions that do not meet the physical constraints.

[0089] Specifically, the safety constraint verification module first calculates the system electrical quantities (node ​​voltage, branch current, equipment output, etc.) after executing the current joint control action vector based on the physical model of the distribution network, and then judges whether they meet the preset physical operation constraints one by one.

[0090] If all constraints are met, the action vector is directly output for control.

[0091] If any constraint is not met, the constraint projection correction mechanism is activated. The core of this mechanism is to solve a constraint optimization problem: under the premise of ensuring all physical constraints, find a corrected action vector that is "closest" to the original action vector. Here, "closest" is the objective function—aiming to minimize the adjustment amount of the control action, because the smaller the adjustment amount, the smaller the change in the agent's decision intention; and all physical constraints constitute the constraints of the optimization problem.

[0092] By solving this constrained projection optimization problem, the original action vector is "projected" to the point on the feasible region boundary that is closest to it. Mathematically, it can be proven that this modified action vector is a feasible solution that satisfies all physical constraints and is closest to the original action.

[0093] The technical advantages of the technical solution described in this embodiment are mainly reflected in the following aspects: First, the strictness of security guarantees. Unlike the "soft constraints" of the reward function penalty mechanism, the constraint projection correction mathematically guarantees that the final executed action satisfies all physical constraints, thus possessing formal security capabilities.

[0094] Second, minimal intervention in agent decision-making. Constraint projection correction only makes minimal adjustments when constraints are not met, preserving the optimal policy learned by the agent to the greatest extent possible, and avoiding the excessive damage to policy performance caused by simply and crudely pruning or limiting actions.

[0095] Third, the flexibility of modular design. The safety constraint verification module, as an independent module, is decoupled from the agent's policy network, allowing for independent design, testing, and upgrading without affecting the core framework of the reinforcement learning algorithm. It also facilitates customized configuration based on the specific constraint requirements of different power distribution networks in practical engineering.

[0096] In summary, this implementation method organically combines the decision-making flexibility of reinforcement learning with the mathematical guarantees of physical constraints, achieving beneficial technical effects such as strict guarantee of decision security, minimal intervention of agent intentions, and flexible modular deployment.

[0097] S6. Control command execution: Convert the final control action vector into control commands for each controlled device and issue them for execution to achieve real-time balance control of source, grid, load and storage.

[0098] In step S6, the real-time balance control of the source, grid, load, and storage includes active power balance control and reactive power balance control: The active power balance control includes: achieving real-time balance of active power in the distribution network by adjusting the active power output setpoint of the distributed power inverter, the charging and discharging power setpoint of the energy storage device, and the switching status of the controllable load; The reactive power balance control includes: achieving real-time stability of the voltage at distribution network nodes by adjusting the reactive power output setpoint of the distributed power inverter and the switching capacity of the reactive power compensation equipment; The active power balance control and reactive power balance control are jointly decided by the intelligent agent model of the corresponding controlled equipment based on a unified local state observation space. The joint control action vector contains both active power control commands and reactive power control commands.

[0099] Power distribution network load balancing control encompasses two dimensions: active power balance and reactive power balance. Active power balance involves frequency stability and power supply-demand matching, while reactive power balance involves voltage stability and power quality. These two dimensions are physically strongly coupled: the active and reactive power outputs of distributed generation inverters are constrained by the total inverter capacity, resulting in a trade-off between them; node voltage is affected by both reactive power injection and active power flow distribution. Therefore, separating and controlling these two dimensions independently may not only fail to achieve global optimization but could also lead to conflicting control strategies.

[0100] This invention achieves joint balance control of active and reactive power through the following design: First, a unified agent model. Each controlled device's agent model simultaneously undertakes decision-making functions for both active and reactive power control, rather than configuring separate agents for active and reactive power control. The policy network output dimension of the agent model includes both active and reactive power control commands.

[0101] Second, a unified local state observation space. The observation space of the agent model simultaneously includes electrical quantities related to active power (such as active power injected into nodes and active power flow in branches) and electrical quantities related to reactive power (such as node voltage amplitude and reactive power injected into nodes), enabling the agent to obtain information on both active and reactive power when making decisions.

[0102] Third, a unified reward function design. The reward function of the agent includes both active power balance reward indicators (such as the amount of active power imbalance in the system) and reactive power balance reward indicators (such as the degree to which the voltage of each node deviates from the rated value), driving the agent to optimize simultaneously in both active and reactive power dimensions.

[0103] Fourth, unified safety constraint verification. The constraint projection correction process simultaneously handles active power-related constraints (such as equipment active power output limits) and reactive power-related constraints (such as node voltage upper and lower limits) to ensure that the joint control actions meet safety requirements in both dimensions.

[0104] Through the above design, the intelligent agent can comprehensively weigh the objectives of active power control and reactive power control when making decisions, make full use of the four-quadrant regulation capability of distributed power inverters, and achieve synergistic optimization of multiple flexible resources such as source-grid-load-storage in the two dimensions of power balance and voltage stability.

[0105] Compared with the traditional approach of separately optimizing active and reactive power control, the joint control scheme described in this embodiment can effectively avoid control strategy conflicts, fully tap the multi-dimensional adjustment potential of distributed power inverters, and achieve beneficial technical effects such as active / reactive power coordinated optimization, full utilization of global resources, and consistency of control strategies.

[0106] This invention also provides a power distribution network source-load balance safety control system based on multi-agent reinforcement learning, comprising: Multiple agent models are configured on the controlled equipment side of the power distribution network. Each agent model includes a state perception module, a policy decision module, and a communication interaction module. The state perception module is used to collect electrical state information of local nodes and physically connected neighboring nodes to construct a local state observation space. The policy decision module integrates a policy network obtained using a two-stage training architecture to generate candidate action decisions based on the local state observation space. The communication interaction module is used to interact with neighboring agents through a preset communication topology to obtain candidate action decision information and state information of neighboring agents. The multi-agent collaborative decision engine is connected to the communication and interaction modules of each agent model. It is used to receive the candidate action decisions and state information of each agent model, calculate the influence weights between neighboring agents based on the attention mechanism and perform collaborative corrections to generate the collaborative action decisions of each controlled device. A centralized evaluation network, deployed on a cloud server, communicates with the multi-agent collaborative decision engine. It is used to acquire global state information of the distribution network and the joint actions of each agent during the training phase, calculate the Q value of the joint actions, and feed them back to each agent model for policy network parameter updates. The safety constraint verification module is connected to the multi-agent collaborative decision engine and is used to perform physical operation constraint verification and constraint projection correction on the joint control action vector formed by the aggregation of collaborative action decisions. The control command execution module is connected to the safety constraint verification module and each controlled device, and is used to convert the final control action vector after verification or correction into control commands for each controlled device and issue them for execution.

[0107] The system adopts a centralized training and distributed execution architecture: During the training phase, the centralized evaluation network acquires the global state information of the distribution network and the joint actions of each agent model. Based on the global state information, it calculates the global Q value of the joint actions and distributes the global Q value as a training signal to each agent model through the multi-agent collaborative decision engine. Each agent model updates the policy network parameters synchronously based on the training signal using the policy gradient algorithm. During the execution phase, each agent model is decoupled from the centralized evaluation network. The state perception module, strategy decision-making module, and communication interaction module of each agent model run independently on the edge computing unit of each controlled device. Each agent model makes distributed decisions based only on its own local state observation space and the limited neighborhood information obtained through the communication interaction module. The safety constraint verification module and the control command execution module run on the local controller of the distribution network. All modules work together to complete the real-time balance control of source-grid-load-storage. When the operating status of the distribution network changes significantly, the centralized evaluation network and each agent model are triggered to undergo rapid retraining. The rapid retraining is completed by a small number of iterations and fine-tuning based on the network parameters obtained by offline pre-training. The retrained network parameters are then updated to the agent models of each controlled device through cloud delivery.

[0108] The system of this invention adopts a three-layer architecture of cloud-edge-device collaboration.

[0109] A centralized evaluation network is deployed in the cloud, responsible for global information aggregation, joint action evaluation, and policy parameter updates during the training phase. The cloud has powerful computing capabilities and global data access capabilities, making it suitable for executing computationally intensive centralized training tasks.

[0110] Each intelligent agent model (including a state perception module, a policy decision-making module, and a communication interaction module) is deployed at the edge and runs on the edge computing units on each controlled device. The edge computing units have real-time computing capabilities and can complete local state perception, policy forward inference, and neighborhood communication interaction with millisecond-level latency, meeting the time constraints of real-time control.

[0111] The safety constraint verification module and control command execution module are deployed on the end side and run on the local controller of the distribution network. They are responsible for verifying and correcting the actions output by the intelligent agent and sending the final control command to each controlled device for execution.

[0112] The state awareness module is responsible for collecting electrical state information from local and neighboring nodes and constructing a standardized local state observation space. This module interacts with local sensors and neighboring devices through industrial communication protocols (such as IEC61850, Modbus, etc.) to ensure the real-time performance and accuracy of the state information.

[0113] The strategy decision module integrates a policy network and is responsible for generating candidate action decisions based on the local state observation space. This module employs a lightweight neural network inference engine to achieve efficient model inference under the limited computing power of edge computing units.

[0114] The communication and interaction module is responsible for information exchange between intelligent agents, including sending its own candidate action decisions and state information, and receiving the same information from neighboring intelligent agents. The communication and interaction module supports multiple communication methods (including industrial Ethernet, 5G, etc.) and has the capability to handle degradation in case of communication failures.

[0115] The multi-agent collaborative decision engine receives candidate action decisions and state information from each agent, calculates neighborhood influence weights based on graph attention mechanism, and generates collaborative action decisions.

[0116] The centralized evaluation network works collaboratively with each agent model during the training phase and decouples from each agent model during the execution phase, achieving a flexible switch between "centralized during training and distributed during execution".

[0117] The safety constraint verification module and the control command execution module constitute the system's "safety valve," ensuring that all control commands issued to the controlled equipment meet physical safety constraints.

[0118] This invention has the following systemic advantages over the prior art: First, in terms of training efficiency, through a two-stage training architecture of "generative adversarial imitation learning pre-training + multi-agent reinforcement learning online fine-tuning", and a feature extraction and decision architecture based on Transformer and prototype matching, the agent has reasonable initial decision-making ability before deployment. Fine-tuning can be completed with only a small number of samples in the online stage, and the training convergence speed is several times faster than the existing de novo training method.

[0119] Second, at the level of decision-making collaboration. By using a graph attention mechanism to achieve dynamic collaboration among multiple agents based on real-time states, each agent can adaptively adjust its collaboration weights according to the topology and operating status of the distribution network. The precision of collaborative decision-making is superior to existing global splicing or fixed-weight schemes.

[0120] Third, in terms of security assurance. By using online verification of security constraints and constraint projection correction, the final executed control actions are guaranteed to meet all physical operational constraints in a mathematically rigorous manner. The rigor of its security assurance is superior to existing methods based on penalty mechanisms.

[0121] Fourth, in terms of scenario adaptability. Through the dynamic update mechanism and rapid retraining mechanism of the prototype library, the agent can continuously absorb new operating modes and quickly adapt to changes in the power distribution network scenario, and its long-term operating performance is better than that of the static model solution.

[0122] Fifth, in terms of engineering feasibility. Through the cloud-edge-device collaborative system architecture and the CTDE training / execution separation design, the computational advantages of centralized training and the real-time advantages of distributed execution are unified, demonstrating good scalability and engineering implementation capabilities.

[0123] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Those skilled in the art can make various modifications and variations to the present invention without departing from its spirit and scope, and these modifications and variations should also be covered within the protection scope of the present invention.

Claims

1. A power distribution network source-load balance security control method based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Construct a multi-agent system: Configure an independent agent model for each controlled device in the power distribution network, and obtain each agent model using a two-stage training architecture; The controlled equipment includes distributed power supply equipment, energy storage equipment, controllable load equipment, and reactive power compensation equipment. The two-stage training architecture includes: In the first stage, based on generative adversarial imitation learning, the policy network of the agent model is pre-trained offline using historical operating data of the power distribution network, so that the policy network learns the distribution characteristics of expert control actions. In the second stage, the pre-trained policy network is deployed to the power distribution network environment and fine-tuned and optimized online based on the multi-agent reinforcement learning algorithm. S2. Obtain the local state observation space: Each controlled device collects the electrical state information of its local node and the electrical state information of its physically connected neighbor nodes, splices and normalizes the collected information, and constructs the local state observation space at the current moment. The electrical status information includes node voltage amplitude, node injected active power, node injected reactive power, branch current amplitude, and power factor. S3. Agent forward decision-making: Each agent model inputs the local state observation space into its policy network for forward propagation calculation, and the policy network outputs the candidate action decision for the corresponding controlled device. S4. Multi-agent collaborative decision-making: Each agent model interacts with neighboring agents through a preset communication topology to obtain candidate action decision information and state information of the neighboring agents. Based on its own local state observation space and the obtained neighborhood information, each agent model uses an attention mechanism to calculate the influence weight of the neighboring agents on the current agent's decision. Based on the influence weight, it collaboratively corrects its own candidate action decision to generate a collaborative action decision. S5. Joint Action Optimization and Safety Verification: The action decisions of each intelligent agent model after collaboration are aggregated into a joint control action vector. The joint control action vector is globally evaluated through a centralized evaluation network. The evaluated joint control action vector is input into the safety constraint verification module. Verification and constraint projection correction are performed according to the physical operation constraints of the distribution network to generate the final control action vector that satisfies all physical operation constraints. S6. Control command execution: Convert the final control action vector into control commands for each controlled device and issue them for execution to achieve real-time balance control of source, grid, load and storage.

2. The power distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The policy network in step S1 integrates an encoder based on the Transformer architecture and a decision maker based on prototype matching. The encoder of the Transformer architecture uses a multi-head self-attention mechanism to jointly encode the temporal features and spatial topological features in the local state observation space to generate a state feature vector. The prototype matching-based decision maker has a prototype library pre-installed. The prototype library is constructed in the following way: collecting state-action pair data of the power distribution network under various typical operating scenarios, using a clustering algorithm to cluster the state-action pair data, storing each cluster center as a prototype in the prototype library, and each prototype contains a typical state feature vector and its corresponding typical action feature vector. The decision-maker performs similarity matching between the state feature vector and each prototype in the prototype library. Based on the similarity calculation result, it activates at least one active prototype that best matches the current state from the prototype library. The decision-maker then performs feature fusion between the state feature vector and the typical action feature vector of the active prototype to generate the candidate action decision.

3. The distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 2, characterized in that, The prototype library is dynamically updated during the operation of the power distribution network. This dynamic updating includes: During the operation of the power distribution network, the actual control effect data of each controlled device is collected in real time, and the similarity between the current state-action pair and each prototype in the prototype library is calculated. If the similarity between the current state-action pair and all prototypes is lower than a preset threshold, then the current state-action pair will be added to the prototype library as a new prototype. If the similarity between the current state-action pair and a certain prototype is higher than a preset threshold, but the deviation between the actual control effect and the typical control effect corresponding to the prototype exceeds a preset deviation threshold, then the corresponding prototype is updated with weight based on the current state-action pair. The weights for the weighted update are dynamically determined based on the current state and the confidence level of the action pair.

4. The power distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The offline pre-training based on generative adversarial imitation learning in step S1 includes: Collect historical operating data of the distribution network. The historical operating data includes the state sequence of each controlled device under various source and load scenarios and the corresponding expert control action sequence. The expert control action sequence is obtained based on the optimal power flow calculation of the distribution network or human expert experience. A generative adversarial imitation learning framework is constructed, comprising a generator and a discriminator; the generator adopts the same network structure as the policy network and is used to generate imitation actions based on the current state; the discriminator is used to determine the degree of similarity between the generated actions and the expert-controlled actions; The generator and discriminator are trained adversarially based on the historical operating data. During the training process, the generator aims to maximize the probability of misjudgment by the discriminator, while the discriminator aims to maximize the accuracy of distinguishing between generated actions and expert-controlled actions. Training stops when the discrimination accuracy of the discriminator converges to a preset threshold range. The network parameters of the generator after training convergence are used as the initialization parameters of the policy network to complete offline pre-training.

5. The distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The attention mechanism in step S4 is a graph attention mechanism. Each agent model calculates the influence weights of neighboring agents on the current agent's decision through the graph attention mechanism, specifically including: Each agent model uses its own local state observation space features and the state information features of each neighboring agent as query vector and key vector, respectively. The attention coefficients between the current agent and each neighboring agent are calculated through a graph attention layer. These attention coefficients are calculated based on the similarity between the query vector and the key vector. The attention coefficients are normalized to obtain the influence weights of each neighboring agent on the current agent's decision. Each agent model performs weighted aggregation of the candidate action decisions of each neighboring agent according to the influence weight, and performs residual connection and fusion with its own candidate action decisions to generate collaborative action decisions. The network parameters of the graph attention mechanism are jointly optimized with the policy network parameters during the online fine-tuning process of multi-agent reinforcement learning in the second stage.

6. The power distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The constraint projection correction of the safety constraint verification module in step S5 includes: The physical operation constraints include upper and lower limits of node voltage amplitude, thermal stability limit of branch current, upper and lower limits of active power output of distributed power sources, upper and lower limits of state of charge of energy storage devices, and node power balance constraints. The safety constraint verification module first determines whether the joint control action vector satisfies all physical operation constraints. If it does, it directly outputs the joint control action vector as the final control action vector. If the joint control action vector does not satisfy any physical operation constraint, the safety constraint verification module constructs a constraint projection optimization problem with the objective function of minimizing the control action adjustment and with all physical operation constraints as constraints. Solve the constraint projection optimization problem by projecting the joint control action vector to the nearest point on the feasible region boundary to obtain a modified control action vector that satisfies all physical operation constraints, which is then output as the final control action vector.

7. The distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 1, characterized in that, The second stage of multi-agent reinforcement learning online fine-tuning optimization in step S1 adopts a centralized training and distributed execution framework, specifically including: During the training phase, each agent model sends its local state observation space, candidate action decisions, and collaborative action decisions to a centralized evaluation network in the cloud. The centralized evaluation network acquires the global state information of the distribution network, calculates the Q value of the joint action based on the global state information and the joint action of each agent, and feeds back the Q value as a reward signal to each agent model. Each agent model updates its own policy network parameters based on the reward signal using a policy gradient algorithm. During the execution phase, each intelligent agent model runs independently on the side of each controlled device, making distributed decisions based only on its own local state observation space and limited neighborhood information obtained through communication interaction. It does not need to obtain global state information, nor does it need to rely on the centralized evaluation network. When a change in the distribution network topology or a change in the access status of distributed power sources is detected, rapid retraining of each agent model is triggered. The rapid retraining is completed by a small number of iterations and fine-tuning based on the network parameters obtained from offline pre-training.

8. The distribution network source-load balance security control method based on multi-agent reinforcement learning according to claim 1, characterized in that, In step S6, the real-time balance control of the source, grid, load, and storage includes active power balance control and reactive power balance control: The active power balance control includes: achieving real-time balance of active power in the distribution network by adjusting the active power output setpoint of the distributed power inverter, the charging and discharging power setpoint of the energy storage device, and the switching status of the controllable load; The reactive power balance control includes: achieving real-time stability of the voltage at distribution network nodes by adjusting the reactive power output setpoint of the distributed power inverter and the switching capacity of the reactive power compensation equipment; The active power balance control and reactive power balance control are jointly decided by the intelligent agent model of the corresponding controlled equipment based on a unified local state observation space. The joint control action vector contains both active power control commands and reactive power control commands.

9. A power distribution network source-load balance safety control system based on multi-agent reinforcement learning, characterized in that, include: Multiple agent models are configured on the controlled equipment side of the power distribution network. Each agent model includes a state perception module, a policy decision module, and a communication interaction module. The state perception module is used to collect electrical state information of local nodes and physically connected neighboring nodes to construct a local state observation space. The policy decision module integrates a policy network obtained using a two-stage training architecture to generate candidate action decisions based on the local state observation space. The communication interaction module is used to interact with neighboring agents through a preset communication topology to obtain candidate action decision information and state information of neighboring agents. The multi-agent collaborative decision engine is connected to the communication and interaction modules of each agent model. It is used to receive the candidate action decisions and state information of each agent model, calculate the influence weights between neighboring agents based on the attention mechanism and perform collaborative corrections to generate the collaborative action decisions of each controlled device. A centralized evaluation network, deployed on a cloud server, communicates with the multi-agent collaborative decision engine. It is used to acquire global state information of the distribution network and the joint actions of each agent during the training phase, calculate the Q value of the joint actions, and feed them back to each agent model for policy network parameter updates. The safety constraint verification module is connected to the multi-agent collaborative decision engine and is used to perform physical operation constraint verification and constraint projection correction on the joint control action vector formed by the aggregation of collaborative action decisions. The control command execution module is connected to the safety constraint verification module and each controlled device, and is used to convert the final control action vector after verification or correction into control commands for each controlled device and issue them for execution.

10. A power distribution network source-load balance safety control system based on multi-agent reinforcement learning according to claim 9, characterized in that, The system adopts a centralized training and distributed execution architecture: During the training phase, the centralized evaluation network acquires the global state information of the distribution network and the joint actions of each agent model. Based on the global state information, it calculates the global Q value of the joint actions and distributes the global Q value as a training signal to each agent model through the multi-agent collaborative decision engine. Each agent model updates the policy network parameters synchronously based on the training signal using the policy gradient algorithm. During the execution phase, each agent model is decoupled from the centralized evaluation network. The state perception module, strategy decision-making module, and communication interaction module of each agent model run independently on the edge computing unit of each controlled device. Each agent model makes distributed decisions based only on its own local state observation space and the limited neighborhood information obtained through the communication interaction module. The safety constraint verification module and the control command execution module run on the local controller of the distribution network. All modules work together to complete the real-time balance control of source-grid-load-storage. When the operating status of the distribution network changes significantly, the centralized evaluation network and each agent model are triggered to undergo rapid retraining. The rapid retraining is completed by a small number of iterations and fine-tuning based on the network parameters obtained by offline pre-training. The retrained network parameters are then updated to the agent models of each controlled device through cloud delivery.