A multi-region active power distribution network distributed agent construction method and device, terminal equipment and distributed cooperative voltage regulation method

Abstract

The application discloses a kind of multi-region active power distribution network distributed intelligent agent construction method, device, terminal equipment and distributed collaborative voltage regulation method, belong to active power distribution network field, the method is: input corresponding local observation state to each intelligent agent, obtain the original action of each intelligent agent;The safety projection verification is carried out to each original action, and an effective projection action set is generated;Effective projection action set is applied to active power distribution network environment, to make active power distribution network environment return current global reward value and the global state of next interactive operation;Update interaction step number, and current global state, the original action of each intelligent agent, effective projection action set and the current global reward value and the global state of next interactive operation returned are stored in current experience playback pool;Repeat execution interactive operation until cumulative expected reward converges, obtain several training completed intelligent agents. By implementing the present application, the problem of poor voltage regulation effect of active power distribution network in the prior art can be solved.

Description

Technical Field

[0001] This invention relates to the field of active distribution networks, and in particular to a method, apparatus, terminal equipment, and distributed collaborative voltage regulation method for constructing distributed intelligent agents in multi-regional active distribution networks. Background Technology

[0002] Distribution network voltage control is a crucial step in reducing network losses and enhancing the absorption capacity of new energy sources. With the integration of distributed power sources such as photovoltaics, which have a high proportion and strong randomness, the voltage stability problem of active distribution networks with multi-regional interconnection is becoming increasingly prominent, and traditional centralized optimization and localized control methods are facing severe challenges.

[0003] In recent years, Multi-Agent Reinforcement Learning (MARL) has emerged as a data-driven approach. Through autonomous interaction between multiple agents and the environment, it learns collaborative strategies, theoretically balancing response speed and global optimization. However, existing MARL methods, when applied to multi-region active distribution network voltage regulation, fail to fully consider the heterogeneity of different regional grid structures and the mutual influence of agent policy updates. This leads to unstable learning and low coordination efficiency in non-stationary environments. Furthermore, the actions directly output by MARL agents (such as reactive power commands for photovoltaic inverters) often only pursue long-term reward maximization, failing to strictly meet the physical safety constraints of real-time distribution network operation, and potentially outputting unsafe actions that endanger grid security. Summary of the Invention

[0004] This invention provides a method, apparatus, terminal equipment, and distributed collaborative voltage regulation method for constructing distributed intelligent agents in a multi-region active distribution network. The method can solve the problem of poor voltage regulation effect in the existing active distribution network.

[0005] To address the aforementioned technical problems, one embodiment of the present invention provides a method for constructing a distributed intelligent agent in a multi-region active distribution network, comprising: Initialize the experience replay pool, the Critic network, and the Actor networks for each agent; Repeat the training rounds until the cumulative expected reward converges, resulting in several trained agents; Each training round includes: Initialize the global reward value, reset the global state of the active distribution network environment and the interaction steps of the experience replay pool; Repeatedly execute the interactive operations until the number of interactive steps in the experience replay pool reaches a preset number, completing the current training round. Based on the global reward value of each interactive operation within the current training round, determine the cumulative expected reward for the current training round; wherein, the interactive operations include: Input the corresponding local observation state to each agent to obtain the original action output by the Actor network for each agent; where each agent is used to represent the photovoltaic inverters and gas turbine units in different regions of the multi-region active distribution network; Perform secure projection verification on each original action to generate a set of valid projected actions; The effective projection action set is applied to the active distribution network environment so that the active distribution network environment returns the current global reward value and the global state of the next interaction operation; Update the interaction steps, and store the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool.

[0006] Furthermore, the step of performing secure projection verification on each original action to generate a set of valid projected actions includes: The original actions of each agent are substituted into a preset low-carbon optimal power flow model of the distribution network to verify whether the low-carbon optimal power flow model of the distribution network has a solution; wherein, the objective of the low-carbon optimal power flow model of the distribution network is to minimize the total operating cost of the distribution network. If a solution exists, the original actions of each agent are taken as the corresponding valid projected actions, and the set of valid projected actions is generated. If no solution is found, the low-carbon optimal power flow model of the distribution network is solved by a preset optimization solver to obtain a feasible solution that satisfies the preset distribution network security constraints on the reactive power output of the photovoltaic inverter and the active power output of each gas turbine unit. The feasible solution is used as the effective projection action of each intelligent agent to generate the set of effective projection actions.

[0007] Furthermore, the objective function of the low-carbon optimal power flow model for the distribution network is specifically: ; ; ; ; In the formula, This represents the voltage difference across all nodes. For reference voltage, Represents a set of nodes. Represents intelligent agents Photovoltaic reactive power deployed at nodes in an active distribution network; Represents the active power loss of all nodes; parameters This indicates the weights allocated to balance voltage deviation and active power loss; This represents the cost that the distribution network purchases or sells to the superior power grid. for The price of electricity for buying or selling during a specific time period. for Electricity purchased during a specific time period For the scheduling period, for Electricity prices during specific time periods. for Electricity sold during a specific time period; This indicates the cost of power generation by thermal power units within the distribution network. This indicates the total number of thermal power units. for Electricity generation at any given moment; , and These represent the thermal power units in the system. The power generation cost coefficient; This indicates the cost of excess carbon emissions. This represents the cost coefficient for excess carbon emissions. This indicates the carbon emission intensity corresponding to purchasing electricity from the upstream power grid. Indicates the first The carbon emission intensity corresponding to each unit Indicates the initial state of the entire system Emission quotas.

[0008] Furthermore, after storing the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool, it also includes: When the amount of data in the experience replay pool reaches a preset threshold, a number of sampled data are extracted from the experience replay pool, and the strategy parameters of each Actor network and the parameters of the Critic network are updated based on the sampled data. Before each update, the update order of each agent is randomly generated, and the policy parameters of each Actor network are updated sequentially according to the update order of each agent.

[0009] Furthermore, based on the sampled data, the parameters of the Critic network are updated, including: Based on the sampled data, a feature vector is encoded; The feature vectors are mapped to the Query space, Key space, and Value space of each attention head using a learnable linear projection function. Based on the Query space, Key space, and Value space of each attention point, the state-action representation vector is calculated; The parameters of the Critic network are updated based on the state-action representation vector.

[0010] Furthermore, the cumulative expected reward for each training round is calculated using the following formula: ; In the formula, Indicates the discount factor; Indicates the first The global reward value obtained from each interactive operation.

[0011] An embodiment of the present invention also provides a distributed intelligent agent construction device for a multi-region active distribution network, comprising: an initialization module and a training module; the training module includes an initialization unit and an interactive operation unit; The initialization module is used to initialize the experience replay pool, the Critic network, and the Actor network of each agent. The training module is used to repeatedly execute training rounds until the cumulative expected reward converges, resulting in several trained agents. The initialization unit is used to initialize the global reward value and reset the global state of the active distribution network environment and the interaction steps of the experience replay pool. The interactive operation unit is used to repeatedly execute interactive operations until the number of interactive steps in the experience replay pool reaches a preset number, completing the current training round. Based on the global reward value of each interactive operation within the current training round, the cumulative expected reward for the current training round is determined. The interactive operations include: Input the corresponding local observation state to each agent to obtain the original action output by the Actor network for each agent; where each agent is used to represent the photovoltaic inverters and gas turbine units in different regions of the multi-region active distribution network; Perform secure projection verification on each original action to generate a set of valid projected actions; The effective projection action set is applied to the active distribution network environment so that the active distribution network environment returns the current global reward value and the global state of the next interaction operation; Update the interaction steps, and store the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool.

[0012] Furthermore, the interactive operation unit is also used for: When the amount of data in the experience replay pool reaches a preset threshold, a number of sampled data are extracted from the experience replay pool, and the strategy parameters of each Actor network and the parameters of the Critic network are updated based on the sampled data. Before each update, the update order of each agent is randomly generated, and the policy parameters of each Actor network are updated sequentially according to the update order of each agent.

[0013] This application also provides a terminal device, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed intelligent agent construction method for multi-region active distribution networks as described in the above embodiments of the invention.

[0014] Another embodiment of the present invention provides a distributed coordinated voltage regulation method for a multi-region active distribution network, comprising: Select a target active distribution network and divide the target active distribution network into several regions according to its topology; For each region, a distributed intelligent agent is constructed corresponding to all gas turbine units and photovoltaic inverters within the region; wherein, the distributed intelligent agent is constructed based on the distributed intelligent agent construction method of the multi-region active distribution network described above; Distributed agents corresponding to all gas turbine units and photovoltaic inverters in each region are deployed to the corresponding regions, so that the Actor network corresponding to each agent can output corresponding adjustment commands in real time based on the local observation status of the region; wherein, the adjustment commands include the active power adjustment command of the corresponding gas turbine unit and the reactive power adjustment command of the corresponding photovoltaic inverter. According to the adjustment command, the corresponding gas turbine unit and photovoltaic inverter are controlled to perform power output.

[0015] The following benefits can be obtained by implementing the present invention: This invention provides a method, apparatus, terminal equipment, and distributed collaborative voltage regulation method for constructing distributed agents in a multi-region active distribution network. The method inputs local observation states to each agent to obtain its original actions, allowing each agent to independently learn the operating characteristics of its region. Each agent makes decisions based on its local observation states, avoiding the reliance on system-wide data in centralized methods. Then, each original action is verified by secure projection, generating a set of effective projected actions. This secure action projection mechanism ensures that the actions output by the agents strictly comply with the operating requirements of distribution network voltage safety and power balance, mitigating system operational risks. The set of effective projected actions is then applied to the environment to obtain a global reward value and the next global state. Finally, based on the updated interaction steps and the stored interaction data in an experience replay pool, the cumulative expected reward converges. This achieves diversified sample accumulation and iterative optimization during the offline training phase. Using the cumulative expected reward as a convergence evaluation index, it ensures that the trained distributed agents possess stable and efficient voltage regulation performance, enabling independent deployment in multi-region active distribution networks and achieving fast and accurate distributed collaborative voltage regulation. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a method for constructing a distributed intelligent agent in a multi-region active distribution network according to a certain embodiment of this application; Figure 2 This is a schematic diagram of the training process of a distributed intelligent agent provided in a certain embodiment of this application; Figure 3 This is a schematic diagram of the structure of a distributed intelligent agent construction device for a multi-region active distribution network provided in a certain embodiment of this application; Figure 4 This is a schematic diagram of the structure of a terminal device provided in a certain embodiment of this application; Figure 5 This is a flowchart illustrating a distributed coordinated voltage regulation method for a multi-region active distribution network provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0020] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0022] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0023] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0024] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0025] See Figure 1 To address the problem of poor voltage regulation performance in existing active distribution networks, an embodiment of the present invention provides a method for constructing a distributed intelligent agent for a multi-regional active distribution network, comprising: S101. Initialize the experience replay pool, the Critic network, and the Actor network for each agent. Specifically, this involves a large active distribution network divided into multiple zones, each equipped with multiple photovoltaic (PV) inverters and gas turbine (MT) units, managed by the responsible distribution network owner. Each PV inverter has a reactive power generation unit to control the voltage at a level called... The photovoltaic inverters and gas turbines operate near a fixed value (with a control interval of 3 minutes), while the gas turbines can generate active power. All photovoltaic inverters and gas turbines within a region share the observations of that region, so each agent can only observe a portion of the information of the entire power grid. However, maintaining the security and economy of the power grid is a common goal of the agents, so the challenge of active voltage control in the distribution network can be effectively modeled as a Markov game.

[0026] Specifically, the Critic network dynamically measures the importance of different agents' decisions by projecting global state and action information into the Query space, Key space, and Value space, captures the complex dependencies between agents, and provides a reliable state-action value evaluation for policy optimization.

[0027] Specifically, the Actor networks of each agent adopt a heterogeneous design with non-shared parameters. Each agent independently corresponds to an Actor network, representing photovoltaic inverters or gas turbine units in different areas of the active distribution network. Corresponding nodes deployed in the active distribution network The initial weights are randomly initialized, and each Actor network independently learns the operating characteristics of its region, adapting to the heterogeneous operating conditions of multi-regional distribution networks.

[0028] S102. Repeat the training rounds until the cumulative expected reward converges, and obtain several trained agents. See Figure 2 To illustrate, in order for the agent to learn to collaboratively optimize voltage, reduce network losses and carbon emissions while meeting preset distribution network safety constraints, multiple training rounds are needed to continuously optimize its strategy until convergence. Specifically, a training round refers to a complete sequence of repeated interactive operations starting from the initial state of the environment until the termination condition is met; each training round includes: S1021. Initialize the global reward value and reset the global state of the active distribution network environment and the interaction steps of the experience replay pool; Indicatively, the entire active distribution network is modeled as an environment, including In each region, agents make decisions based on the node information they observe in the distribution network of their respective regions. After the agents make decisions, the voltage of each node is estimated and the reward obtained by the agents is determined.

[0029] Specifically, the state space of the entire environment is described as The global reward value is initialized to 0, and the interaction step counter of the experience replay pool is set to zero.

[0030] S1022. Repeatedly execute the interactive operation until the number of interactive steps in the experience replay pool reaches the preset number of steps, and complete the current training round. Based on the global reward value of each interactive operation in the current training round, determine the cumulative expected reward of the current training round. In a schematic way, the voltage control problem of the distribution network is modeled as a Markov game. Each agent continuously interacts with the distribution network environment based on the local observation state of its region. The safe projection mechanism ensures that the actions meet the safety constraints of the distribution network. At the same time, the data such as the state, actions and rewards generated by the interaction are stored in the experience replay pool. After accumulating to a preset number of interaction steps, a single training round is completed. Then, the cumulative expected reward is calculated based on the global reward value of each interaction operation in the training round, thereby evaluating the policy learning effect.

[0031] Specifically, after initializing the global state, global reward value, and interaction steps, the following interaction operations are executed repeatedly until the interaction steps reach a preset threshold, completing the current training iteration and calculating the cumulative expected reward. The interaction operations include: S10211. Input the corresponding local observation state to each agent to obtain the original action output by the Actor network corresponding to each agent; wherein, each agent is used to represent the photovoltaic inverters and gas turbine units in different regions of the multi-region active distribution network. Specifically, for each intelligent agent Input the corresponding local observation state into it. For a given intelligent agent The observed local state variables Depend on Composition, in which, and These are active and reactive loads, respectively. This represents the active power output of the photovoltaic system during the current period. This represents the reactive power output of the photovoltaic inverter in the previous time period; each agent calculates based on the aforementioned local observation state variables. The original action set is output through the Actor network corresponding to each agent. Action set This includes all possible actions of the agent, for a single agent. If it is a photovoltaic inverter, its operation is indicated as follows: ,action This indicates the maximum proportion of reactive power generated by the photovoltaic inverter. For a gas turbine unit, its operation is indicated as... The action indicates the maximum percentage of active power output from the gas turbine unit relative to its capacity.

[0032] also, The specific calculation formula is as follows: (1) in, Represents intelligent agents The physical capacity (i.e., apparent power) of photovoltaic inverters deployed at nodes in an active distribution network, when At that time, the photovoltaic inverter injects reactive power into the node. It absorbs reactive power.

[0033] S10212. Perform safety projection verification on each original action to generate a set of valid projection actions; In illustrative terms, in order to ensure that the actions output by the intelligent agents meet the safety constraints of the distribution network and that the photovoltaic output can maintain the system voltage within a controllable range at all times, while taking into account low network losses and carbon emissions, this application introduces a safety action projection mechanism for the original actions of each intelligent agent. Through constraint verification and optimization, an effective set of projected actions that meets the requirements of the distribution network model is generated.

[0034] In a preferred embodiment, the step of performing secure projection verification on each original action to generate a set of valid projected actions includes: The original actions of each agent are substituted into a preset low-carbon optimal power flow model of the distribution network to verify whether the low-carbon optimal power flow model of the distribution network has a solution; wherein, the objective of the low-carbon optimal power flow model of the distribution network is to minimize the total operating cost of the distribution network. If a solution exists, the original actions of each agent are taken as the corresponding valid projected actions, and the set of valid projected actions is generated. If no solution is found, the low-carbon optimal power flow model of the distribution network is solved by a preset optimization solver to obtain a feasible solution that satisfies the preset distribution network security constraints on the reactive power output of the photovoltaic inverter and the active power output of each gas turbine unit. The feasible solution is used as the effective projection action of each intelligent agent to generate the set of effective projection actions.

[0035] Specifically, the initial objective function of the low-carbon optimal power flow model for the distribution network is: (2) However, in order to address the challenges of low-carbon voltage control in distribution networks, especially for nodes... In scenarios where active power, reactive power, and photovoltaic power generation cause voltage fluctuations exceeding the safety constraints of the distribution network, the overall network voltage deviation is mitigated by adjusting the reactive power output of the photovoltaic inverter, while minimizing reactive power generation to reduce power loss. Therefore, in this embodiment, equation (2) is replaced with the objective function of equations (3)-(6): In a preferred embodiment, the objective function of the low-carbon optimal power flow model for the distribution network is specifically: (3) (4) (5) (6) In the formula, This represents the voltage difference across all nodes. For reference voltage, Represents a set of nodes. Indicating in intelligent agents Photovoltaic reactive power deployed at nodes in an active distribution network; Represents the active power loss of all nodes; parameters This indicates the weights allocated to balance voltage deviation and active power loss; This represents the cost that the distribution network purchases or sells to the superior power grid. for The price of electricity for buying or selling during a specific time period. for Electricity purchased during a specific time period For the scheduling period, for Electricity prices during specific time periods. for Electricity sold during a specific time period; This indicates the cost of power generation by thermal power units within the distribution network. This indicates the total number of thermal power units. for Electricity generation at any given moment; , and These represent the thermal power units in the system. The power generation cost coefficient; This represents the cost of excess carbon emissions (yuan / kg). This represents the cost coefficient for excess carbon emissions. This indicates the carbon emission intensity corresponding to purchasing electricity from the upstream power grid. Indicates the first Carbon emission intensity (kg / kWh) for each unit Indicates the initial state of the entire system Emission quota (kg / h).

[0036] Specifically, the safety constraints for the power distribution network are as follows: (7) (8) (9) (10) (11) In the formula, This represents the set of branches, where node 0 is the node connecting to the main network (used to balance active and reactive power in the distribution network); nodes The voltage amplitude and phase angle are expressed as follows: and , Represents nodes A set of indices for connected nodes; Indicates a branch Electrical conductivity; Indicates a branch The susceptance on; Represents a node and The phase difference; Represents a node Photovoltaic active power; Represents a node Photovoltaic active and reactive power (node) (0 when there is no load).

[0037] For the safe operation of the distribution network system, voltage is typically allowed. Deviation is ( per unit value ( ), , ). The active power of the reference node. Represents a node The capacity of the photovoltaic inverter is located here. The maximum reactive power of the photovoltaic inverter is Since the high penetration of photovoltaic inverters may not be feasible, slack variables can be added to the voltage constraints in this case.

[0038] Specifically, after the agent outputs the original action based on its own local observation state, it substitutes the original action into equations (3)-(11) for feasibility verification. If the verification result shows that there is a feasible solution after substituting the original action, it indicates that the set of original actions will not cause distribution network safety problems. At this time, the set of original actions is directly determined as the set of effective projected actions. No additional adjustments are required.

[0039] If the verification results show that there is no solution to the constraints after substituting the original actions, the preset Gurobi optimization solver is started. Based on the distribution network low-carbon optimal power flow model and distribution network security constraints constructed by equations (3)-(11), a feasible solution is obtained, and the feasible solution is used as the effective set of projected actions. This feasible solution satisfies the requirements for safe operation of the distribution network while also taking into account the goals of low-carbon emission reduction and network loss optimization. Therefore, it is used as the effective projection action of each intelligent agent, and after integration, it generates a set of effective projection actions to ensure that subsequent actions acting on the distribution network environment are safe and reasonable.

[0040] S10213. Apply the effective projection action set to the active distribution network environment so that the active distribution network environment returns the current global reward value and the global state of the next interaction operation; As an illustration, since the objective function represented by equation (3) cannot handle the complexity of voltage constraints well, a potential function is used to represent voltage constraints and network loss constraints, and carbon emissions are considered in the reward function. The specific reward function is shown below: (12) (13) (14) In the formula, It is the probability density function of a normal distribution, and its mean is The standard deviation is 0.1. It is a voltage potential function. For reactive power generation losses, It is a weighting parameter that balances voltage difference and active power loss; These represent four hyperparameters used to adjust the shape and smoothness of the function, and are set to 2, 0.095, 0.01, and 0.04 respectively. Represents a set of intelligent agents.

[0041] Specifically, based on equations (12)-(14), the first... The global reward value for each interaction.

[0042] In a preferred embodiment, the cumulative expected reward for the current training round is calculated using the following formula: (15) In the formula, Indicates the discount factor; Indicates the first The global reward value obtained from each interactive operation; Specifically, after all interactive operations are completed in the current training round, all global reward values ​​stored in the round are extracted from the experience replay pool, and the cumulative expected reward is calculated according to formula (15) to obtain the cumulative expected reward of the training round. This cumulative expected reward is compared with the cumulative expected reward of the previous round. If the fluctuation range of the cumulative expected reward for W consecutive rounds is less than the preset threshold, then convergence is determined, training is stopped and a deployable distributed agent model is output. If convergence is not achieved, step S1021 is returned to start the next training round.

[0043] S10214. Update the interaction steps and store the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool. Specifically, first, the interaction step counter in the experience replay pool is incremented by 1; then, a complete data sample of a single interaction is stored in the experience replay pool: the data sample includes the current global state. The original action set of each intelligent agent Effective projection action set Current global reward value The global state for the next interaction. The experience replay pool provides high-quality training data to support subsequent parameter updates for the Actor and Critic networks.

[0044] In a preferred embodiment, after storing the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool, the method further includes: When the amount of data in the experience replay pool reaches a preset threshold, a number of sampled data are extracted from the experience replay pool, and the strategy parameters of each Actor network and the parameters of the Critic network are updated based on the sampled data. Before each update, the update order of each agent is randomly generated, and the policy parameters of each Actor network are updated sequentially according to the update order of each agent.

[0045] Indicatively, the policy sequential update mechanism progressively optimizes the agents within each update cycle, treating already updated policies as known information to refresh the experience data of the remaining agents. This allows each agent to adapt to the latest policies of other agents, thereby improving cooperation.

[0046] This approach maintains a more stable learning environment and helps guide the system towards higher global rewards. It is well-suited for voltage regulation in multi-region active distribution network systems with varying state space dimensions because it does not assume homogeneity among agents.

[0047] Specifically, before each Actor network is updated, an update order is randomly generated, and then each Actor network is updated cyclically according to this order. The policy update method is as follows: set up , and These represent Actor networks that have been updated, are being updated, or are yet to be updated, respectively. The joint policy during the Actor network update is denoted as... The state-action value function is then defined as: (16) In the formula, , express The time has been updated The action output by the Actor network, This refers to the action being performed by the Actor network that is currently being updated. This represents the Actor network output action to be updated (i.e., the old corresponding Actor network output action). express Of the intelligent agents, excluding those already updated... and the intelligent agent that is being updated Other intelligent agents besides This indicates that the joint policy of the agents has not been updated.

[0048] Based on the update order The next iteration updates the entire Actor network. To avoid local optima, similar to the SAC method, an entropy regularization term is added to the objective function: (17) In the formula, Represents policy entropy, Using the temperature coefficient, the state-action value function of the agent being updated is now represented as: (18) In the formula, For the agent that is being updated The output action.

[0049] In a preferred embodiment, updating the parameters of the Critic network based on the sampled data includes: Based on the sampled data, feature vectors are encoded; through a learnable linear projection function, the feature vectors are mapped to the Query space, Key space, and Value space of each attention head; based on the Query space, Key space, and Value space of each attention head, state-action representation vectors are calculated. The parameters of the Critic network are updated based on the state-action representation vector.

[0050] Indicatively, each agent makes independent decisions using its own Actor network based solely on its local observations, thereby achieving distributed control.

[0051] This algorithm designs an attention-enhanced critic network that uses a multi-head self-attention mechanism to dynamically measure the importance of different agents, thereby accurately assessing the interdependencies of complex systems and guiding policy optimization loops. In these loops, the Actor network is updated sequentially. In each cycle, the agents update their policies sequentially, with each update conditioned on the latest policies of the previous agents, thus significantly improving coordination and training stability.

[0052] Specifically, by projecting the sampled data into the Query, Key, and Value spaces, the critic network can dynamically measure the importance of different inputs, focusing on rewarding key information, thereby better guiding the policy learning of the actor network. (19) In the formula, , and It is a learnable projection function that projects the feature vectors respectively. Mapping to Query space, Key space, and Value space; scaling factor It can prevent large dot product distortion. Output; This represents the state-action representation vector.

[0053] A shared central Critic network is used to accurately evaluate the contribution of each agent to policy updates, and its loss function is defined as: (20) ; (twenty one) In the formula, It is the central Critic network. It is the target network of the central Critic network; It is an agent's strategy. It is a discount factor. It is a temperature parameter.

[0054] To alleviate The problem of overestimating the value is addressed by using the smaller of the two central Critics. Value; finally, the strategy The optimization objective is set as follows: ; (twenty two) In the formula, To update the intelligent agent The joint strategy at the time For intelligent agents The observed local observation state, This is a strategy that is currently being updated; this makes it possible to perform efficient estimations through sampling. The value becomes feasible.

[0055] To guide the output of the Actor network to approximate the effective projected action that satisfies the constraints, the Euclidean distance between the original action and the effective projected action is used. The policy is incorporated into the Actor network's update loss function, rather than into the reward function using the traditional DRL method, thus incorporating the policy... The optimization objectives are improved as follows: ;(twenty three) At this point, in order to better execute the original movements With projection action To bring it closer, update the parameters of the critic network as follows: ; (twenty four) (25) At this point, the projection action is used to update the Critic network (in fact, the original action). With projection action Most of them are the same, only a small part Replaced if constraints are not met Through the above-mentioned action safety projection mechanism and the guidance of consistent operation between the original action and the effective projected action, the Critic network... Value estimation can accurately reflect the quality of the Actor network's output actions and guide the Actor network to generate actions that meet the distribution network's security constraints and optimize strategy performance.

[0056] See Figure 3 This invention provides a distributed intelligent agent construction device for a multi-region active distribution network, comprising: an initialization module and a training module; the training module includes an initialization unit and an interactive operation unit. The initialization module is used to initialize the experience replay pool, the Critic network, and the Actor network of each agent. The training module is used to repeatedly execute training rounds until the cumulative expected reward converges, resulting in several trained agents. The initialization unit is used to initialize the global reward value and reset the global state of the active distribution network environment and the interaction steps of the experience replay pool. The interactive operation unit is used to repeatedly execute interactive operations until the number of interactive steps in the experience replay pool reaches a preset number, completing the current training round. Based on the global reward value of each interactive operation within the current training round, the cumulative expected reward for the current training round is determined. The interactive operations include: Input the corresponding local observation state to each agent to obtain the original action output by the Actor network for each agent; where each agent is used to represent the photovoltaic inverters and gas turbine units in different regions of the multi-region active distribution network; Perform secure projection verification on each original action to generate a set of valid projected actions; The effective projection action set is applied to the active distribution network environment so that the active distribution network environment returns the current global reward value and the global state of the next interaction operation; Update the interaction steps, and store the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool.

[0057] In a preferred embodiment, the interactive operation unit is specifically used for: The original actions of each agent are substituted into a preset low-carbon optimal power flow model of the distribution network to verify whether the low-carbon optimal power flow model of the distribution network has a solution; wherein, the objective of the low-carbon optimal power flow model of the distribution network is to minimize the total operating cost of the distribution network. If a solution exists, the original actions of each agent are taken as the corresponding valid projected actions, and the set of valid projected actions is generated. If no solution is found, the low-carbon optimal power flow model of the distribution network is solved by a preset optimization solver to obtain a feasible solution that satisfies the preset distribution network security constraints on the reactive power output of the photovoltaic inverter and the active power output of each gas turbine unit. The feasible solution is used as the effective projection action of each intelligent agent to generate the set of effective projection actions.

[0058] In a preferred embodiment, the objective function of the low-carbon optimal power flow model for the distribution network is specifically: ; ; ; ; In the formula, This represents the voltage difference across all nodes. For reference voltage, Represents a set of nodes. Represents intelligent agents Photovoltaic reactive power deployed at nodes in an active distribution network; Represents the active power loss of all nodes; parameters This indicates the weights allocated to balance voltage deviation and active power loss; This represents the cost that the distribution network purchases or sells to the superior power grid. for The price of electricity for buying or selling during a specific time period. for Electricity purchased during a specific time period For the scheduling period, for Electricity prices during specific time periods. for Electricity sold during a specific time period; This indicates the cost of power generation by thermal power units within the distribution network. This indicates the total number of thermal power units. for Electricity generation at any given moment; , and These represent the thermal power units in the system. The power generation cost coefficient; This indicates the cost of excess carbon emissions. This represents the cost coefficient for excess carbon emissions. This indicates the carbon emission intensity corresponding to purchasing electricity from the upstream power grid. Indicates the first The carbon emission intensity corresponding to each unit Indicates the initial state of the entire system Emission quotas.

[0059] In a preferred embodiment, the interactive operation unit is further configured to: When the amount of data in the experience replay pool reaches a preset threshold, a number of sampled data are extracted from the experience replay pool, and the strategy parameters of each Actor network and the parameters of the Critic network are updated based on the sampled data. Before each update, the update order of each agent is randomly generated, and the policy parameters of each Actor network are updated sequentially according to the update order of each agent.

[0060] In a preferred embodiment, updating the parameters of the Critic network based on the sampled data includes: Based on the sampled data, a feature vector is encoded; The feature vectors are mapped to the Query space, Key space, and Value space of each attention head using a learnable linear projection function. Based on the Query space, Key space, and Value space of each attention point, the state-action representation vector is calculated; The parameters of the Critic network are updated based on the state-action representation vector.

[0061] In a preferred embodiment, the cumulative expected reward for the current training round is calculated using the following formula: ; In the formula, Indicates the discount factor; Indicates the first The global reward value obtained from each interactive operation.

[0062] See Figure 4 One embodiment of this application also provides a terminal device, including: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed intelligent agent construction method for multi-region active distribution networks as described above.

[0063] The processor is used to control the overall operation of the terminal device to complete all or part of the steps of the above-described method for constructing a distributed intelligent agent for a multi-regional active distribution network.

[0064] The memory is used to store various types of data to support the operation of the terminal device. This data may include, for example, instructions for any application or method to operate on the terminal device, as well as application-related data.

[0065] The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0066] In an exemplary embodiment, the terminal device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the distributed intelligent agent construction method for multi-region active distribution networks as described in any of the above embodiments, and achieve the same technical effects as the methods described above.

[0067] See Figure 5 A distributed coordinated voltage regulation method for a multi-region active distribution network, provided in one embodiment of the present invention, includes: S201. Select the target active distribution network and divide the target active distribution network into several regions according to the topology of the target active distribution network; Specifically, based on the physical topology nodes of the target active distribution network, consecutive nodes on the same feeder are divided into the same area, while ensuring that the difference in total load and photovoltaic installed capacity in each area does not exceed a preset threshold.

[0068] Ultimately, several independently operating and information-autonomous regions are obtained. Each region is electrically connected to other regions through interconnection switches, and each region is equipped with an independent local data acquisition unit to acquire operational data such as load, photovoltaic output, and voltage of nodes within the region.

[0069] S202. For each region, construct a distributed intelligent body corresponding to all gas turbine units and photovoltaic inverters within the region; wherein, the distributed intelligent body is constructed based on the distributed intelligent body construction method of the multi-region active distribution network described above; Specifically, an independent intelligent agent is constructed for each gas turbine unit and each photovoltaic inverter within the region: Each agent corresponds to an Actor network with non-shared parameters, and all agents share the central Critic network within the region. Through the distributed agent construction method of the multi-region active distribution network, the parameters of the Actor network and the Critic network are optimized, and finally, agents adapted to the operating characteristics of the region are obtained. Among them, the agents corresponding to the gas turbine units take active power output as the action dimension, and the agents corresponding to the photovoltaic inverters take reactive power output as the action dimension.

[0070] S203. Deploy the distributed agents corresponding to all gas turbine units and photovoltaic inverters in each region to the corresponding region, so that the Actor network corresponding to each agent can output the corresponding adjustment command in real time based on the local observation state of the region; wherein, the adjustment command includes the active power adjustment command of the corresponding gas turbine unit and the reactive power adjustment command of the corresponding photovoltaic inverter. Specifically, the trained distributed agent model is deployed to the local edge computing device in the region. The agent collects the local observation status of the region in real time (including the active load, reactive load, photovoltaic active power in the current period, and inverter reactive power in the previous period of the node).

[0071] The system outputs the original action through its own Actor network; then, through the locally deployed safety action projection mechanism, it verifies whether the original action meets the distribution network safety constraints and generates a valid projected action as an adjustment command: the agent corresponding to the gas turbine outputs the active power adjustment command, and the agent corresponding to the photovoltaic inverter outputs the reactive power adjustment command.

[0072] S204. According to the adjustment command, control the corresponding gas turbine unit and photovoltaic inverter to perform power output; Specifically, the adjustment commands output by the intelligent agent are converted into control signals that the device can recognize and sent to the corresponding gas turbine or photovoltaic inverter: after receiving the active power adjustment command, the gas turbine adjusts its output power; after receiving the reactive power adjustment command, the photovoltaic inverter adjusts its reactive power output.

[0073] Meanwhile, the local data acquisition unit monitors the node voltage, power and other operating data in real time after the equipment executes the command, and feeds it back to the agent as the local observation status for the next round of decision-making.

[0074] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for constructing a distributed intelligent agent in a multi-region active distribution network, characterized in that, include: Initialize the experience replay pool, the Critic network, and the Actor networks for each agent; Repeat the training rounds until the cumulative expected reward converges, resulting in several trained agents; Each training round includes: Initialize the global reward value, reset the global state of the active distribution network environment and the interaction steps of the experience replay pool; Repeatedly execute the interactive operations until the number of interactive steps in the experience replay pool reaches a preset number, completing the current training round. Based on the global reward value of each interactive operation within the current training round, determine the cumulative expected reward for the current training round; wherein, the interactive operations include: Input the corresponding local observation state to each agent to obtain the original action output by the Actor network for each agent; where each agent is used to represent the photovoltaic inverters and gas turbine units in different regions of the multi-region active distribution network; Perform secure projection verification on each original action to generate a set of valid projected actions; The effective projection action set is applied to the active distribution network environment so that the active distribution network environment returns the current global reward value and the global state of the next interaction operation; Update the interaction steps, and store the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool.

2. The method for constructing a distributed intelligent agent in a multi-region active distribution network as described in claim 1, characterized in that, The step of performing secure projection verification on each original action to generate a set of valid projected actions includes: The original actions of each agent are substituted into a preset low-carbon optimal power flow model of the distribution network to verify whether the low-carbon optimal power flow model of the distribution network has a solution; wherein, the objective of the low-carbon optimal power flow model of the distribution network is to minimize the total operating cost of the distribution network. If a solution exists, the original actions of each agent are taken as the corresponding valid projected actions, and the set of valid projected actions is generated. If no solution is found, the low-carbon optimal power flow model of the distribution network is solved by a preset optimization solver to obtain a feasible solution that satisfies the preset distribution network security constraints on the reactive power output of the photovoltaic inverter and the active power output of each gas turbine unit. The feasible solution is used as the effective projection action of each intelligent agent to generate the set of effective projection actions.

3. The method for constructing a distributed intelligent agent in a multi-region active distribution network as described in claim 2, characterized in that, The objective function of the low-carbon optimal power flow model for the distribution network is as follows: ； ； ； ； In the formula, This represents the voltage difference across all nodes. For reference voltage, Represents a set of nodes. Represents intelligent agents Photovoltaic reactive power deployed at nodes in an active distribution network; Represents the active power loss of all nodes; parameters This indicates the weights allocated to balance voltage deviation and active power loss; This represents the cost that the distribution network purchases or sells to the superior power grid. for The price of electricity for buying or selling during a specific time period. for Electricity purchased during a specific time period For the scheduling period, for Electricity prices during specific time periods. for Electricity sold during a specific time period; This indicates the cost of power generation by thermal power units within the distribution network. This indicates the total number of thermal power units. for Electricity generation at any given moment; , and These represent the thermal power units in the system. The power generation cost coefficient; This indicates the cost of excess carbon emissions. This represents the cost coefficient for excess carbon emissions. This indicates the carbon emission intensity corresponding to purchasing electricity from the upstream power grid. Indicates the first The carbon emission intensity corresponding to each unit Indicates the initial state of the entire system Emission quotas.

4. The method for constructing a distributed intelligent agent in a multi-region active distribution network as described in claim 1, characterized in that, After storing the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool, it also includes: When the amount of data in the experience replay pool reaches a preset threshold, a number of sampled data are extracted from the experience replay pool, and the strategy parameters of each Actor network and the parameters of the Critic network are updated based on the sampled data. Before each update, the update order of each agent is randomly generated, and the policy parameters of each Actor network are updated sequentially according to the update order of each agent.

5. The method for constructing a distributed intelligent agent in a multi-region active distribution network as described in claim 4, characterized in that, Based on the sampled data, the parameters of the Critic network are updated, including: Based on the sampled data, a feature vector is encoded; The feature vectors are mapped to the Query space, Key space, and Value space of each attention head using a learnable linear projection function. Based on the Query space, Key space, and Value space of each attention point, the state-action representation vector is calculated; The parameters of the Critic network are updated based on the state-action representation vector.

6. The method for constructing a distributed intelligent agent in a multi-region active distribution network as described in claim 1, characterized in that, The cumulative expected reward for each training round is calculated using the following formula: ； In the formula, Indicates the discount factor; Indicates the first The global reward value obtained from each interactive operation.

7. A distributed intelligent agent construction device for a multi-region active distribution network, characterized in that, include: An initialization module and a training module; the training module includes an initialization unit and an interactive operation unit. The initialization module is used to initialize the experience replay pool, the Critic network, and the Actor network of each agent. The training module is used to repeatedly execute training rounds until the cumulative expected reward converges, resulting in several trained agents. The initialization unit is used to initialize the global reward value and reset the global state of the active distribution network environment and the interaction steps of the experience replay pool. The interactive operation unit is used to repeatedly execute interactive operations until the number of interactive steps in the experience replay pool reaches a preset number, completing the current training round. Based on the global reward value of each interactive operation within the current training round, the cumulative expected reward for the current training round is determined. The interactive operations include: Input the corresponding local observation state to each agent to obtain the original action output by the Actor network for each agent; where each agent is used to represent the photovoltaic inverters and gas turbine units in different regions of the multi-region active distribution network; Perform secure projection verification on each original action to generate a set of valid projected actions; The effective projection action set is applied to the active distribution network environment so that the active distribution network environment returns the current global reward value and the global state of the next interaction operation; Update the interaction steps, and store the current global state, the original actions of each agent, the set of effective projected actions, the returned current global reward value, and the global state of the next interaction operation into the current experience replay pool.

8. The distributed intelligent agent construction device for a multi-region active distribution network as described in claim 7, characterized in that, The interactive operation unit is also used for: When the amount of data in the experience replay pool reaches a preset threshold, a number of sampled data are extracted from the experience replay pool, and the strategy parameters of each Actor network and the parameters of the Critic network are updated based on the sampled data. Before each update, the update order of each agent is randomly generated, and the policy parameters of each Actor network are updated sequentially according to the update order of each agent.

9. A terminal device, characterized in that, include: One or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed agent construction method for a multi-region active distribution network as described in any one of claims 1-6.

10. A distributed coordinated voltage regulation method for a multi-regional active distribution network, characterized in that, include: Select a target active distribution network and divide the target active distribution network into several regions according to its topology; For each region, a distributed intelligent agent is constructed corresponding to all gas turbine units and photovoltaic inverters within the region; wherein, the distributed intelligent agent is constructed based on the distributed intelligent agent construction method of the multi-region active distribution network as described in any one of claims 1-6; Distributed agents corresponding to all gas turbine units and photovoltaic inverters in each region are deployed to the corresponding regions, so that the Actor network corresponding to each agent can output corresponding adjustment commands in real time based on the local observation status of the region; wherein, the adjustment commands include the active power adjustment command of the corresponding gas turbine unit and the reactive power adjustment command of the corresponding photovoltaic inverter. According to the adjustment command, the corresponding gas turbine unit and photovoltaic inverter are controlled to perform power output.

Images