Power distribution network voltage control method, system, device and medium based on multi-agent

Abstract

The application discloses a power distribution network voltage control method, system, device and medium based on multiple agents, which can automatically adjust the reactive power of the photovoltaic inverter and maintain the voltage within a safe range in most scenarios. Compared with the traditional scheme, the application can greatly reduce the computing burden, and the strategy based on the neural network can quickly reason and respond to various situations in the power grid scheduling in real time. At the same time, compared with the existing multi-agent reinforcement learning scheme for voltage control tasks, the mechanism of autonomous correction of dangerous actions is added, which can effectively reduce the voltage out-of-limit situation, and can make decisions only relying on local power grid information, which can well meet the needs of distributed deployment. Therefore, the application can be conveniently deployed in local voltage control equipment, has wide application scenarios, provides strong technical support for the development of intelligent power grid scheduling, and also promotes the popularization of distributed renewable energy in the power generation end of the power grid.

Description

Technical Field

[0001] This invention relates to the field of distribution network voltage automation control technology, and in particular to a distribution network voltage control method, system, device and medium based on multi-agent systems. Background Technology

[0002] With the increasing penetration of distributed renewable energy in distribution networks, significant challenges are posed to the operation and control of these networks. The randomness, volatility, and intermittency of distributed power output alter the power flow distribution of traditional distribution networks, causing voltage fluctuations and even exceeding limits. This introduces numerous uncertainties to the stable operation of the grid, impacting the overall quality and reliability of the power supply. Both overvoltage and undervoltage reduce energy efficiency, cause inrush currents in equipment, and even damage user equipment. Furthermore, this leads to decreased production efficiency and increased energy consumption for businesses, causing serious harm to human production and daily life. These unstable distributed renewable energy sources present unprecedented challenges to distribution network voltage control, making it a crucial topic in current intelligent dispatching technologies for distribution networks.

[0003] Traditional voltage control strategies can be broadly categorized into two types: reactive power dispatch based on optimal power flow methods and droop control based on local voltage and power measurements. Optimal power flow algorithms have high computational complexity, requiring significant computational resources and time, making them impractical for real-time grid control and suitable only for static analysis and planning. Droop control relies solely on local observations, and its performance is highly dependent on manually designed parameters. Furthermore, due to a lack of global information, its calculated results are often suboptimal. Therefore, traditional distribution network voltage control algorithms have many limitations and cannot meet the requirements of smart grids, such as real-time response, adaptability, and reliability.

[0004] Currently, in practical voltage dispatching, a dispatching method primarily relies on manual operation, supplemented by traditional simulation and computational tools, to ensure the safe and stable operation of the power grid. This requires significant investment in equipment and manpower. Furthermore, due to the vast operational space, extremely long decision-making steps, overly complex topologies, and constraints from various random unforeseen events in distribution network dispatching, it is difficult to guarantee whether the currently adopted strategies are conducive to low-loss and safe long-term power grid operation. In addition, the installation of numerous measuring devices in the power grid results in a massive amount of power grid monitoring data. How to effectively analyze and utilize this big data and apply it to practical decision-making is also a significant challenge in practical power grid dispatching.

[0005] In recent years, multi-agent reinforcement learning based on reactive power control of photovoltaic inverters has been introduced into the field of distribution network voltage control. Compared with traditional voltage control methods, it utilizes data-driven learning to determine the optimal strategy, requiring less computational overhead during decision-making. By continuously interacting with the power grid simulation environment and applying penalties and rewards based on the interaction results, the neural network gradually learns scheduling strategies to cope with various changes in the power grid state. During the inference phase, given global power grid information or local power grid information of the area where the control equipment is located, the policy network can output the actions that the control equipment should take under the corresponding local power grid information. These actions ensure low losses and safety during power grid operation.

[0006] In addition, researchers have proposed a safety reinforcement learning algorithm based on constrained Markov decision processes for voltage control.

[0007] In general, the above three types of technical solutions currently have the following technical problems:

[0008] 1. Currently, the voltage control algorithms actually used in power grids (i.e., the aforementioned traditional voltage control strategies) are still mainly based on manual experience. Power grid dispatchers need to manually design numerous parameters and then revise the strategies based on the results of other auxiliary calculation software. This voltage control method cannot fully utilize the power grid's big data and is easily influenced by the cognitive and experience levels of power grid dispatchers. It struggles to comprehensively consider the overall system characteristics of the power grid, which may lead to unstable algorithm performance and incur significant manpower and computational costs. Furthermore, traditional voltage control algorithms are difficult to automate, meaning they cannot respond promptly to changes in the power grid's state, such as sudden line faults, leading to grid instability and impacting daily production and life.

[0009] 2. Existing voltage control algorithms based on multi-agent reinforcement learning require extensive trial and error during the exploration process, and lack effective measures to ensure the safety of the learned strategy during deployment. In power grid dispatching tasks, multi-agent reinforcement learning algorithms need to try different actions under different conditions to discover the optimal control strategy, which places high demands on the modeling of the power simulation environment, requiring consideration of numerous possible states. Furthermore, the cost of agent trial and error operations can be high, making the learning process slow and time-consuming. Moreover, if the agent's strategy is not constrained during trial and error, the agent may learn strategies containing unsafe behaviors, making it difficult to guarantee the safety of the strategy. Current technical solutions lack effective research on the safe exploration and deployment of multi-agent reinforcement learning, requiring further exploration and improvement.

[0010] 3. Existing safety reinforcement learning algorithms based on constrained Markov decision processes are mostly used for voltage control in centralized single-agent settings, limiting their applicability and making them difficult to apply in practical distributed renewable energy scenarios. In actual scheduling scenarios, agents (controllable devices) in each region of the power grid typically cannot obtain global grid information, but only local grid information for the area that the measuring equipment can measure. In other words, existing single-agent safety reinforcement learning methods cannot make decisions under conditions of limited local information. Summary of the Invention

[0011] The purpose of this invention is to provide a distribution network voltage control method, system, device, and medium based on multi-agent technology. This method can efficiently adjust the reactive power of photovoltaic inverters according to the local grid information of the current area, and correct dangerous action strategies that may cause voltage overruns (i.e., the current reactive power value of the photovoltaic inverter may cause certain voltage overruns in the next time step). This ensures that the final action strategy maintains all buses in the grid within the safe voltage range as much as possible, thereby better adapting to the actual grid dispatch requirements.

[0012] The objective of this invention is achieved through the following technical solution:

[0013] A multi-agent-based distribution network voltage control method includes:

[0014] A distribution network voltage control model containing multiple agents is constructed. Each agent includes a strategy network module and an action correction sub-network module. The strategy network module is responsible for outputting corresponding actions based on local grid information, and the action correction sub-network module is responsible for calculating the first action correction amount by combining the local grid information and the actions output by the strategy network module.

[0015] The distribution network voltage control model containing multiple agents is trained centrally. A safety layer is introduced during the centralized training process. The safety layer is responsible for correcting the input actions based on global grid information and outputting safe actions. Based on global grid information, the actions output by the policy network module in all agents, and the first action correction amount output by the action correction sub-network module, a centralized action value function is trained for each agent. For each agent, the parameters of the policy network module are updated by maximizing the centralized action value function. Then, the parameters of the action correction sub-network module are updated based on the actions output by the policy network module and the safe actions output by the safety layer. The process is iterated repeatedly until the parameters of the policy network module and the action correction sub-network module of each agent converge, thus completing the centralized training process.

[0016] After training is completed, each agent generates and executes a corrected action based on the action output by the trained policy network module and the action correction sub-network module and the first action correction amount; where the action refers to the reactive power adjustment action.

[0017] A multi-agent-based distribution network voltage control system includes:

[0018] The model building unit is used to build a distribution network voltage control model containing multiple agents. Each agent includes a strategy network module and an action correction sub-network module. The strategy network module in the agent is responsible for outputting corresponding actions based on local grid information, and the action correction sub-network module is responsible for calculating the first action correction amount by combining the local grid information and the actions output by the strategy network module.

[0019] A centralized training unit is used to centrally train the distribution network voltage control model containing multiple agents. A safety layer is introduced during the centralized training process. The safety layer is responsible for correcting the input actions based on global grid information and outputting safe actions. Based on global grid information, the actions output by the policy network module in all agents, and the first action correction amount output by the action correction sub-network module, a centralized action value function is trained for each agent. For each agent, the parameters of the policy network module are updated by maximizing the centralized action value function. Then, the parameters of the action correction sub-network module are updated based on the actions output by the policy network module and the safe actions output by the safety layer. The process is iterated repeatedly until the parameters of the policy network module and the action correction sub-network module of each agent converge, thus completing the centralized training process.

[0020] The distributed execution unit, composed of trained agents, is used to generate and execute corrected actions by each agent based on the actions output by the trained policy network module and the action correction sub-network module and the first action correction amount after training is completed; where the action refers to the reactive power adjustment action.

[0021] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0022] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0023] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0024] As can be seen from the technical solution provided by this invention, it can automatically adjust the reactive power of photovoltaic inverters and maintain the voltage within a safe range in most scenarios. Compared with traditional voltage control methods, this invention can significantly reduce the computational burden, and the neural network-based strategy can perform fast inference and respond to various situations in grid dispatch in real time. Furthermore, compared with existing multi-agent reinforcement learning schemes for voltage control tasks, it incorporates a mechanism for autonomously correcting dangerous actions, effectively reducing voltage exceedance situations. It can also rely solely on local grid information for decision-making, well meeting the needs of distributed deployment. Therefore, this invention can be portablely deployed in local voltage control equipment, has a wide range of applications, provides strong technical support for the intelligent development of grid dispatch, and promotes the popularization of distributed renewable energy at the grid generation end. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A flowchart of a power distribution network voltage control method based on multi-agent provided in an embodiment of the present invention;

[0027] Figure 2 A schematic diagram of a centralized training and distributed execution architecture provided in an embodiment of the present invention;

[0028] Figure 3 A schematic diagram illustrating the strategy network module and action correction sub-network module and their interaction with the environment, provided in an embodiment of the present invention;

[0029] Figure 4 A schematic diagram of a multi-agent-based power distribution network voltage control system provided in an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0032] First, the following explanations are provided for the terms that may be used in this article:

[0033] The terms “including,” “comprising,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0034] The following provides a detailed description of a multi-agent-based distribution network voltage control method, system, device, and medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0035] Example 1

[0036] This invention provides a multi-agent-based distribution network voltage control method, such as... Figure 1 As shown, it mainly includes:

[0037] Step 1: Construct a distribution network voltage control model that includes multiple intelligent agents.

[0038] In this embodiment of the invention, each intelligent agent includes a policy network module and an action correction sub-network module. The policy network module in the intelligent agent is responsible for outputting corresponding actions based on local power grid information, and the action correction sub-network module is responsible for calculating the first action correction amount by combining the local power grid information and the actions output by the policy network module.

[0039] In this embodiment of the invention, the policy network module includes: a gated recurrent unit and a first linear layer; the input of the gated recurrent unit is local power grid information, and the output is the features of the local power grid information; the first linear layer outputs an initial action based on the features of the local power grid information, and then the initial action is pruned according to the allowed action range to obtain the final output action.

[0040] In this embodiment of the invention, the action correction sub-network module includes: a second linear layer and a third linear layer; the input of the second linear layer is the action output by the policy network module, and the output is action features; the input of the third linear layer is the sum of the action features and the features of the local power grid information, and the output is a first action correction amount.

[0041] In this embodiment of the invention, the action refers to the reactive power adjustment action.

[0042] In this embodiment of the invention, the local power grid information mainly includes: reactive power, active power, voltage amplitude, voltage phase, and other information of all nodes in the area where the agent is located (this area refers to the local topology of the distribution network, or a subgraph of the distribution network topology).

[0043] Step 2: Perform centralized training on the power distribution network voltage control model containing multiple intelligent agents.

[0044] In this embodiment of the invention, a safety layer is introduced during centralized training. This safety layer is responsible for correcting the input actions based on global power grid information and outputting safe actions. Specifically, the safety layer includes a distribution network voltage prediction module and a first-order dangerous action approximation module.

[0045] (1) Distribution network voltage prediction module.

[0046] The distribution network voltage prediction module is responsible for predicting the voltage values ​​of all bus nodes in the next time step based on global grid information and input actions; these predicted voltage values ​​are referred to as the predicted voltage values.

[0047] In this embodiment of the invention, the global power grid information mainly includes: reactive power, active power, voltage amplitude, voltage phase, and other information of all nodes in the distribution network.

[0048] In this embodiment of the invention, the distribution network voltage prediction module needs to be trained in advance. Through interaction with the simulation environment, each time step interaction generates a (o, a, v) triple, where v is the actual voltage value of all bus nodes in the next time step after taking action a under the global grid information o. All triples generated by the interaction are collected to form a training set of the voltage prediction network. The global grid information o and the action a taken in the training set are input to the distribution network voltage prediction module. The difference between the voltage value predicted by the distribution network voltage prediction module and the actual voltage value in the training set is used to construct a loss function. The distribution network voltage prediction module is trained using the loss function.

[0049] (2) First-order dangerous action approximation module.

[0050] The first-order dangerous action approximation module is used to determine whether the input action is dangerous based on whether the predicted voltage value is within the safe range; if the voltage value is not within the safe range, the input action is identified as dangerous, and a second action correction amount is calculated in combination with the predicted voltage value; if the voltage value is within the safe range, the second action correction amount is 0; and a safe action is obtained by combining the second action correction amount with the input action.

[0051] Specifically: the calculation of the second action correction amount based on the predicted voltage value includes: obtaining the Jacobian matrix of the predicted voltage value with respect to the action, establishing a first-order approximate expression of the action with respect to the predicted voltage value, constructing a convex optimization problem using the first-order approximate expression, solving the convex optimization problem based on the Jacobian matrix of the predicted voltage value with respect to the action, and obtaining the second action correction amount.

[0052] In this embodiment of the invention, the centralized training method uses common multi-agent reinforcement learning algorithms in continuous action space (such as MADDPG, MAPPO, MATD3, etc.). The specific process is as follows: (1) Based on global power grid information, the actions output by the policy network module in all agents and the first action correction amount output by the action correction sub-network module, a centralized action value function, i.e., Q value, is trained for each agent; (2) For each agent, the parameters of its respective policy network module are updated by maximizing the centralized action value function; (3) For each agent, the parameters of the action correction sub-network module are updated based on the actions output by the policy network module and the safe actions output by the security layer. Specifically, the parameters of the action correction sub-network module are updated by minimizing the distance between the actions output by the policy network module and the corrected actions in the Q value space (i.e., the set of action value function values) and minimizing the Euclidean distance between the corrected actions and the safe actions; where the corrected actions refer to the actions obtained by adding the first action correction amount to the actions output by the policy network module.

[0053] In this embodiment of the invention, the above three steps are iterated repeatedly for each agent until the parameters of the policy network module and the action correction sub-network module converge; that is, each iteration includes the above three steps (1) to (3), of which (1) to (2) are the basic flow of common multi-agent reinforcement learning algorithms in continuous action space.

[0054] Step 3: After training is complete, each agent adds the action output by the trained policy network module to the first action correction amount output by the action correction sub-network module to generate the corrected action and execute it.

[0055] The solution provided in this invention can automatically adjust the reactive power of photovoltaic inverters and maintain voltage within a safe range in most scenarios. Compared to traditional voltage control methods, this invention significantly reduces computational burden, and the neural network-based strategy enables rapid inference and real-time response to various situations in grid dispatch. Furthermore, compared to existing multi-agent reinforcement learning schemes for voltage control tasks, it incorporates a mechanism for autonomously correcting dangerous actions, effectively reducing voltage exceedances. It can also make decisions based solely on local grid information, well meeting the needs of distributed deployments. Therefore, this invention can be portablely deployed in local voltage control devices, has a wide range of applications, provides strong technical support for the intelligent development of grid dispatch, and promotes the widespread adoption of distributed renewable energy at the grid generation end.

[0056] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0057] I. The purpose of the invention and the various parts involved in achieving the purpose of the invention.

[0058] 1. Purpose of the invention.

[0059] To address the shortcomings of existing technologies, this invention proposes a multi-agent safety reinforcement learning voltage control framework based on a safety layer. This framework effectively overcomes the deficiencies of existing solutions, allowing agents to perform distributed control based solely on local information while also correcting potentially dangerous actions. Using this framework, the voltage exceedance rate during distribution network voltage dispatch can be effectively reduced. Furthermore, this framework is implemented based on deep neural networks and is entirely data-driven, requiring no prior knowledge of the power grid, such as its topology or detailed equipment parameters. Therefore, compared to existing technologies, this invention has broader applicability and higher robustness, effectively improving the practical application of distributed multi-agent safety reinforcement learning schemes in voltage control tasks under conditions of high renewable energy penetration. Specifically, it can efficiently adjust the reactive power of photovoltaic inverters based on local grid information in the current region and correct dangerous strategies that may lead to voltage exceedances (i.e., the current reactive power value of the photovoltaic inverter may cause certain voltage exceedances in the next time step). This ensures that the final strategy maintains all buses in the power grid within a safe voltage range as much as possible, thus better adapting to actual power grid dispatch requirements.

[0060] 2. Security layer.

[0061] In this embodiment of the invention, the safety layer is mainly used during the centralized training phase and can be directly removed after training is completed. The safety layer mainly includes: a distribution network voltage prediction module and a first-order dangerous action approximation module.

[0062] (1) Distribution network voltage prediction module.

[0063] For the distribution network voltage prediction module, this invention employs a residual network module to predict the voltage forecast values ​​of all bus nodes in the power grid at the next time step based on the input global power grid information o and a given action a. This residual network module consists of a linear input layer, two residual blocks, and a linear output layer.

[0064] In this embodiment of the invention, a random uniform sampling strategy is used to interact with the simulation environment. Each interaction at each time step generates a (o, a, v) triplet, where v is the actual voltage value of all bus nodes in the next time step after taking action a under grid information o. All triplets generated by the interaction are collected to form the training set of the voltage prediction network. The loss function can be the mean square error between the output value of the voltage prediction network and the actual voltage value of the grid in the next time step.

[0065] (2) First-order dangerous action approximation module.

[0066] The first-order dangerous action approximation module aims to utilize the voltage prediction module to obtain a first-order approximate expression for the voltage prediction value at the next time step, thereby transforming the dangerous action correction problem into a quadratic programming (QP) problem. This invention obtains the predicted voltage vector for each bus at the (t+1)th time step (i.e., the next time step) by using the backpropagation algorithm on the distribution network voltage prediction module. Regarding the action vectors a of all agents at time step t... t Jacobi matrix If the voltage prediction value from the voltage prediction module exceeds the safe voltage range, then the currently taken action a is considered to be... t If it is dangerous, then the following quadratic programming problem is solved using a QP solver to obtain the small action correction amount Δa that corrects the dangerous action to a safe action. t .

[0067]

[0068]

[0069] Here, τ1 and τ2 are two set thresholds, and τ1 > τ2. For example, τ1 = 1.05 and τ2 = 0.95 can be set.

[0070] In this embodiment of the invention, the minute movement correction amount Δa t It is a vector, where each term corresponds to a second action correction amount for an agent.

[0071] 3. Intelligent agents.

[0072] In this embodiment of the invention, all intelligent agents have the same structure, which includes a policy network module and an action correction sub-network module.

[0073] (1) Policy Network Module.

[0074] Each agent i possesses a policy network module π i This invention uses the strategy network module π i The gated recurrent unit (GRU) extracts local power grid information. i The features are extracted, and then the extracted features are input into a linear layer to obtain the initial action. Finally, the initial action is cropped according to the allowed action range to obtain the final action. During training, the movements As Gaussian distribution The average value in the data, the final action output by the policy network module Sampling is performed from this Gaussian distribution; during deployment and execution, actions are directly obtained by pruning the policy network module of each agent. Actions taken in the current state

[0075] (2) Action correction subnetwork module.

[0076] The action correction sub-network module is connected after the policy network module. It performs minor corrections to the actions given by the policy network module to reduce the possibility of voltage over-limit caused by the policy network module's output actions. This module shares local power grid information with the policy network module. i The features are extracted simultaneously using a linear layer to extract the action output of the strategy network module. Features, action characteristics and local power grid information i The feature dimensions are the same. The action... Features and local power grid information i The features are summed and then fed into a linear layer to extract their common representations. Finally, the summation is passed through another linear layer to obtain the action representation. Correction amount Δa i , will Δa i and Adding them together gives the corrected action a i .

[0077] II. A framework of centralized training and decentralized execution.

[0078] In this embodiment of the invention, a centralized training and distributed execution framework is used to train each agent. The centralized training process mainly includes the following steps:

[0079] (1) Based on global power grid information o and the joint action a of all agents (the joint action output by the policy network modules of all agents) Adding the motion correction amount Δa = (Δa1, Δa2, ..., Δa) output from the motion correction sub-network module n Train a centralized action value function for each agent, where the centralized action value function for the i-th agent is denoted as...

[0080] (2) For the i-th agent, maximize the action value function of the i-th agent. This update updates the network parameters in the policy network module of the i-th agent. At this time, the network parameters of the action correction sub-network module are not updated. The loss function of the policy network module in this part is expressed as: Specifically: All actions used in this section are corrected actions, and the power grid information is global power grid information. The right-hand side of the loss function... express Used only for updating the policy network π of the i-th agent. i The parameters, i.e., gradient information, are obtained through a. i To spread and thus update π i The parameters are frozen, while other actions are frozen, meaning gradient information is not transmitted through these actions.

[0081] (3) Input global grid information o and the joint action a of all agents into the safety layer (composed of distribution network voltage prediction module and first-order dangerous action approximation module) to obtain the action correction amount of the i-th agent. By minimizing the original action in the Q-value space (i.e., the action output by the policy network module in the i-th agent) and the corrected action The distance and the action after minimizing the correction a i and the safety actions given by the safety layer The Euclidean distance is used to update the network parameters in the action correction sub-network module of the i-th agent. At this time, the network parameters of the policy network module are not updated. The loss function of the action correction sub-network module in this part is expressed as: Where d is the distance function, Indicates calculation action With the corrected action a i Q-value spatial distance, Indicates action The Q-value (the value of the action value function), This represents the Q-value (value of the action value function) of the corrected action 'a'; α is the set proportionality coefficient; ||.||2 is the L2 norm. Indicates the calculated corrected action a i and safety actions The Euclidean distance is calculated. This process is iterated repeatedly until the parameters of the policy network module and the action correction sub-network module converge.

[0082] During execution, the trained strategy network module and action correction sub-network module only need to be deployed on the photovoltaic inverter. It can respond in real time by relying solely on the local power grid information of the area and take safe reactive power regulation actions. Specifically, the trained strategy network module first outputs the action, and then the trained action correction sub-network module outputs the first action correction amount, which is added to the action output by the strategy network module. The corrected action obtained is the safe reactive power regulation action.

[0083] Figure 2 This paper demonstrates the principle of a centralized training and distributed execution architecture. The security layer and its processing described in the first part are only applied to the centralized training process. In the multi-agent reinforcement learning algorithm, 'r' refers to the reward, which is a reward signal that the environment provides after the agent's action is applied at each time step. The algorithm adjusts the agent's strategy based on the reward signal. In this embodiment of the invention, the reward signal 'r' at each time step consists of two parts: the overall power loss of the power grid and the voltage over-limit penalty of all nodes. The multi-agent reinforcement learning algorithm (i.e., the first two steps of the three-step iterative process described above) adjusts the agent's strategy based on the reward signal.

[0084] Figure 3 The diagram illustrates the policy network module and the action correction sub-network module, as well as their interaction with the environment. The left side shows the policy network module and the action correction sub-network module and their interaction with the environment (distribution network). n represents the number of agents, O is the vector formed by the local power grid information observed by all agents, Δa is the vector composed of the first action correction quantities output by the action correction sub-network module in all agents, and a is the action vector composed of the corrected actions of all agents. The right side shows the principle of the policy network module and the action correction sub-network module. The local power grid information encoder in the right-hand policy network module is the gated loop unit mentioned earlier. h is the feature of the local power grid information, t represents the time step t, and the action encoder in the right-hand action correction sub-network module is the second linear layer mentioned earlier.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0086] Example 2

[0087] This invention also provides a multi-agent-based distribution network voltage control system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 4 As shown, the system mainly includes:

[0088] The model building unit is used to build a distribution network voltage control model containing multiple agents. Each agent includes a strategy network module and an action correction sub-network module. The strategy network module in the agent is responsible for outputting corresponding actions based on local grid information, and the action correction sub-network module is responsible for calculating the first action correction amount by combining the local grid information and the actions output by the strategy network module.

[0089] A centralized training unit is used to centrally train the distribution network voltage control model containing multiple agents. A safety layer is introduced during the centralized training process. The safety layer is responsible for correcting the input actions based on global grid information and outputting safe actions. Based on global grid information, the actions output by the policy network modules of all agents, and the first action correction amount output by the action correction sub-network modules, a centralized action value function is trained for each agent. For each agent, the parameters of the policy network module are updated by maximizing the centralized action value function. Then, the parameters of the action correction sub-network module are updated based on the actions output by the policy network module and the safe actions output by the safety layer. The process is iterated repeatedly until the parameters of the policy network module and the action correction sub-network module of each agent converge, thus completing the centralized training process.

[0090] The distributed execution unit, composed of trained agents, is used to generate and execute corrected actions by each agent based on the actions output by the trained policy network module and the action correction sub-network module and the first action correction amount after training is completed; where the action refers to the reactive power adjustment action.

[0091] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0092] Example 3

[0093] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0094] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0095] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0096] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0097] The output device can be a display terminal;

[0098] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0099] Example 4

[0100] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0101] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0102] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A distribution network voltage control method based on multi-agent systems, characterized in that, include: A distribution network voltage control model containing multiple agents is constructed. Each agent includes a strategy network module and an action correction sub-network module. The strategy network module is responsible for outputting corresponding actions based on local grid information, and the action correction sub-network module is responsible for calculating the first action correction amount by combining the local grid information and the actions output by the strategy network module. The distribution network voltage control model containing multiple agents is trained centrally. A safety layer is introduced during the centralized training process. The safety layer is responsible for correcting the input actions based on global grid information and outputting safe actions. Based on global grid information, the actions output by the policy network module in all agents, and the first action correction amount output by the action correction sub-network module, a centralized action value function is trained for each agent. For each agent, the parameters of the policy network module are updated by maximizing the centralized action value function. Then, the parameters of the action correction sub-network module are updated based on the actions output by the policy network module and the safe actions output by the safety layer. The process is iterated repeatedly until the parameters of the policy network module and the action correction sub-network module of each agent converge, thus completing the centralized training process. After training is completed, each agent generates and executes a corrected action based on the action output by the trained policy network module and the action correction sub-network module and the first action correction amount; where the action refers to the reactive power adjustment action.

2. The distribution network voltage control method based on multi-agent system according to claim 1, characterized in that, The safety layer includes: a distribution network voltage prediction module and a first-order dangerous action approximation module; wherein: The distribution network voltage prediction module is responsible for predicting the voltage values ​​of all bus nodes in the next time step based on global grid information and input actions. These predicted voltage values ​​are called the predicted voltage values. The first-order dangerous action approximation module is used to determine whether the input action is dangerous based on whether the predicted voltage value is within the safe range. If the voltage value is not within the safe range, the input action is identified as dangerous, and a second action correction amount is calculated based on the predicted voltage value. If the voltage value is within the safe range, the second action correction amount is 0. A safe action is obtained by combining the second action correction amount with the input action.

3. The distribution network voltage control method based on multi-agent system according to claim 2, characterized in that, The power distribution network voltage prediction module needs to be trained in advance. By interacting with the simulation environment, each time step of the interaction generates a (o,a,v) triplet, where v is the actual voltage value of all bus nodes in the next time step after taking action a under the global power grid information o; all triplets generated by the interaction are collected to form the training set of the voltage prediction network. The global power grid information o and the action a taken in the training set are input into the distribution network voltage prediction module. The difference between the voltage value predicted by the distribution network voltage prediction module and the actual voltage value in the training set is used to construct a loss function, and the distribution network voltage prediction module is trained using the loss function.

4. The distribution network voltage control method based on multi-agent system according to claim 2, characterized in that, The calculation of the second action correction amount based on the predicted voltage value includes: Obtain the Jacobian matrix of the predicted voltage value with respect to the action, establish a first-order approximate expression of the action with respect to the predicted voltage value, construct a convex optimization problem using this first-order approximate expression, solve the convex optimization problem using the Jacobian matrix of the predicted voltage value with respect to the action, and obtain the second action correction amount.

5. The distribution network voltage control method based on multi-agent system according to claim 1, characterized in that, The policy network module includes: a gated recurrent unit and a first linear layer; The input of the gated loop unit is local power grid information, and the output is the characteristics of the local power grid information; The first linear layer outputs an initial action based on the characteristics of the local power grid information. Then, the initial action is trimmed according to the allowed action range to obtain the final output action.

6. The distribution network voltage control method based on multi-agent system according to claim 5, characterized in that, The motion correction subnetwork module includes: a second linear layer and a third linear layer; The input to the second linear layer is the action output by the policy network module, and the output is action features. The input to the third linear layer is the sum of the action features and the features of the local power grid information, and the output is the first action correction amount.

7. The distribution network voltage control method based on multi-agent system according to claim 1, characterized in that, The parameters of the action-based policy network module output and the security action update and correction sub-network module output by the security layer include: The parameters of the action correction sub-network module are updated by minimizing the distance between the action output by the policy network module and the corrected action in the Q-value space, and by minimizing the Euclidean distance between the corrected action and the safety action. The Q-value space is a set of action value function values, and the corrected action refers to the action obtained by adding the first action correction amount to the action output by the policy network module.

8. A distribution network voltage control system based on multi-agent systems, characterized in that, The system for implementing the method according to any one of claims 1 to 7 comprises: The model building unit is used to build a distribution network voltage control model containing multiple agents. Each agent includes a strategy network module and an action correction sub-network module. The strategy network module in the agent is responsible for outputting corresponding actions based on local grid information, and the action correction sub-network module is responsible for calculating the first action correction amount by combining the local grid information and the actions output by the strategy network module. A centralized training unit is used to centrally train the distribution network voltage control model containing multiple agents. A safety layer is introduced during the centralized training process. The safety layer is responsible for correcting the input actions based on global grid information and outputting safe actions. Based on global grid information, the actions output by the policy network module in all agents, and the first action correction amount output by the action correction sub-network module, a centralized action value function is trained for each agent. For each agent, the parameters of the policy network module are updated by maximizing the centralized action value function. Then, the parameters of the action correction sub-network module are updated based on the actions output by the policy network module and the safe actions output by the safety layer. The process is iterated repeatedly until the parameters of the policy network module and the action correction sub-network module of each agent converge, thus completing the centralized training process. The distributed execution unit, composed of trained agents, is used to generate and execute corrected actions by each agent based on the actions output by the trained policy network module and the action correction sub-network module and the first action correction amount after training is completed; where the action refers to the reactive power adjustment action.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images