Power distribution network and microgrid cluster collaborative voltage control method

Abstract

The application provides a power distribution network and micro-grid cluster collaborative voltage control method, and belongs to the technical field of power system operation control, and comprises the following steps: building a mathematical model of power distribution network and micro-grid cluster collaborative voltage control with the minimum voltage fluctuation and the optimal economy of the power distribution network and the micro-grid as the target; equivalently converting the built mathematical model of power distribution network and micro-grid cluster collaborative voltage control into a Markov decision process of reinforcement learning; combining a multi-agent reinforcement learning algorithm with an evolutionary algorithm, introducing a scenario memory mechanism, solving the converted Markov decision process based on the improved multi-agent reinforcement learning algorithm, and realizing the collaborative voltage control and strategy online learning between the power distribution network and the micro-grid cluster. The application effectively improves the utilization efficiency of samples and the global stability of the algorithm.

Description

Technical Field

[0001] This invention belongs to the field of power system operation and control technology, specifically relating to a method for coordinated voltage control of distribution networks and microgrid clusters. Background Technology

[0002] The increasing scale of distributed energy resources integrated into the power grid poses significant challenges to grid operation. Microgrids, as an effective means of absorbing new energy sources, demonstrate superior performance in localized absorption, multi-energy complementarity and coordinated optimization, as well as flexibility and adaptability. However, due to the numerous, widespread, and dispersed locations of microgrids within the distribution network, the strong randomness of distributed energy sources leads to prominent issues such as voltage exceeding limits and power flow overload in the distribution network. Furthermore, the diverse stakeholders within each microgrid contribute to the growing conflict between the economic efficiency of microgrid operation and the voltage control of the distribution network. Therefore, comprehensively considering challenges such as incomplete information exchange between distribution and microgrids, source-load uncertainty, and conflicting control objectives to achieve efficient coordinated voltage control and improve the penetration rate of renewable energy integration has become a critical issue that urgently needs to be addressed.

[0003] Currently, many scholars have explored different operation and control methods to address this problem, including centralized optimization, distributed optimization, game theory, and artificial intelligence algorithms, and have achieved certain results. Chinese invention patent application CN119905991A provides a centralized two-layer scheduling optimization method for distribution microgrids. It uses second-order cone technology to solve the lower-level microgrid grid-connected optimal scheduling model, and calls a digital optimization solver combined with the microgrid access point type to solve the upper-level distribution network optimal scheduling model to obtain the scheduling strategy. Although centralized optimization methods have good global convergence, they have certain limitations, such as difficulty in modeling and the requirement that the model has a specific form for equivalent transformation. For models that do not meet the specific form, extensive simplification and approximation are often required, leading to higher control errors. Chinese invention patent application CN120150197A provides a distributed two-layer collaborative optimization scheduling method for distribution microgrids based on active power aggregation domain. The microgrid locally calculates and submits the power feasible domain boundary, and the distribution network verifies the boundary validity based on the goal of minimizing network losses. It iterative feedback adjustments are made through a row and column generation algorithm until the boundary meets the global optimization requirements, which can improve scheduling efficiency even under conditions of limited information interaction. While such distributed algorithms significantly alleviate model dependency and privacy issues, their solution efficiency decreases considerably as the number of decision-makers increases, making it difficult to meet the requirements of online real-time operation and control. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a collaborative voltage control method for distribution networks and microgrid clusters. The distribution network layer aims to minimize voltage fluctuations and optimize economic efficiency, using network power flow constraints as conditions. The microgrid layer aims to optimize its own economic benefits, considering the operational constraints of internal distributed energy resources and energy storage devices. A mathematical model for collaborative voltage control between the distribution network and microgrid clusters is constructed. Furthermore, this invention proposes a novel improved multi-agent reinforcement learning algorithm that integrates an evolutionary algorithm and a contextual memory mechanism. On one hand, by combining the multi-agent reinforcement learning algorithm with the evolutionary algorithm, the robustness of the algorithm to reward signals is improved, enhancing its exploration capability in the vast nonlinear policy space of collaborative control between distribution networks and microgrid clusters. On the other hand, by introducing a contextual memory mechanism, the explored information and experience are fully utilized to promote policy training, effectively improving sample utilization efficiency and the global stability of the algorithm.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for coordinated voltage control of distribution networks and microgrid clusters includes the following steps:

[0007] Step 1: To achieve the goals of minimizing voltage fluctuations and optimizing economic efficiency in the distribution network and the microgrid, establish a mathematical model for coordinated voltage control of the distribution network and microgrid clusters.

[0008] Step 2: Treat the distribution network and each microgrid as independent intelligent agents. Based on the mathematical model of the coordinated voltage control of the distribution network and microgrid cluster constructed in Step 1, construct the state space, action space and reward function for reinforcement learning training of the control strategies of each intelligent agent, and form the training environment and reinforcement learning model of each intelligent agent.

[0009] Step 3: Based on the training environment built in Step 2, the multi-agent reinforcement learning algorithm and evolutionary algorithm are integrated and a context memory mechanism is introduced to train and solve the reinforcement learning models of each agent in Step 2 offline and call them online in real time to generate a collaborative voltage control strategy, so as to minimize voltage fluctuations and optimize economic efficiency in the distribution network and microgrid.

[0010] Furthermore, in step 1, the mathematical model includes: power flow constraints of the distribution network, upper and lower limit constraints of node voltage, and state of charge and power balance constraints of microgrid energy storage.

[0011] Furthermore, in step 2, the state space includes: the electricity price of the distribution network and the main grid, the output of distributed energy resources in the microgrid, and the state of charge of energy storage.

[0012] Furthermore, in step 2, the action space includes: the electricity sales price signal from the distribution network to the microgrid, and the gas turbine power and energy storage charging and discharging commands within the microgrid.

[0013] Furthermore, in step 2, the reward function is composed of the distribution network revenue and the node voltage fluctuation penalty, and the microgrid reward is composed of the difference between the electricity sales revenue and the generation cost.

[0014] Furthermore, in step 3, the multi-agent reinforcement learning algorithm saves high-reward experiences through an episodic memory mechanism and modifies the Q function with memory target regularization.

[0015] Furthermore, in step 3, the evolutionary algorithm periodically shares strategies with the multi-agent reinforcement learning algorithm to provide diverse samples for the experience pool.

[0016] Furthermore, step 3 includes: obtaining the current state, selecting an action, executing the action, updating the experience pool and the scenario replay area, until the training ends.

[0017] Furthermore, the cooperative voltage control strategy can be directly used for real-time regulation after training, without the need for model linearization or quadratic solution.

[0018] Furthermore, the mathematical model, training environment, and the integration of multi-agent reinforcement learning algorithm and evolutionary algorithm are implemented within the same software framework.

[0019] Beneficial effects:

[0020] This invention adds a memory table to the existing multi-agent reinforcement learning architecture to store high-reward action values, thereby memorizing the target regularization correction Q-function. This allows the agent to quickly utilize successful experiences, accelerating the agent's training. Secondly, in the multi-agent policy exploration phase, this invention utilizes evolutionary algorithms to interact with the environment, providing diverse samples to the shared experience pool. The multi-agent reinforcement learning algorithm's policies are periodically shared with the evolutionary algorithm, guiding its evolutionary direction. This provides more and better policy samples and drives the multi-agent reinforcement learning algorithm to escape local optima, enhancing its exploration capabilities in complex environments. Attached Figure Description

[0021] Figure 1 This is a flowchart of the power distribution network and microgrid cluster coordinated voltage control method of the present invention;

[0022] Figure 2 A schematic diagram of the upgraded IEEE 33-node power distribution system with multiple microgrids connected.

[0023] Figure 3 A diagram illustrating the convergence of global reward values ​​for different algorithms;

[0024] Figure 4a A schematic diagram comparing the convergence of the distribution network reward function under different algorithms;

[0025] Figure 4b A schematic diagram comparing the convergence of the reward function of the first microgrid under different algorithms;

[0026] Figure 4c A schematic diagram comparing the convergence of the reward function of the third microgrid under different algorithms;

[0027] Figure 5a This is a schematic diagram comparing the voltage of the 33 nodes after optimization by different algorithms at 5:00.

[0028] Figure 5b This is a diagram showing the voltage comparison of the 33 nodes after optimization by different algorithms at 10:00.

[0029] Figure 5c This is a schematic diagram comparing the voltage of 33 nodes after different algorithm optimizations at 23:00. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of the invention described below can be combined with each other as long as they do not conflict with each other.

[0031] like Figure 1 As shown, the present invention provides a method for coordinated voltage control of a power distribution network and a microgrid cluster, comprising the following steps:

[0032] Step 1: To achieve the goals of minimizing voltage fluctuations and optimizing economic efficiency in the distribution network and the microgrid, establish a mathematical model for coordinated voltage control of the distribution network and microgrid clusters.

[0033] Step 2: Treat the distribution network and each microgrid as independent intelligent agents. Based on the mathematical model of the coordinated voltage control of the distribution network and microgrid cluster constructed in Step 1, construct the state space, action space and reward function for reinforcement learning training of the control strategies of each intelligent agent, and form a training environment.

[0034] Step 3: Based on the reinforcement learning environment (i.e. training environment) constructed in Step 2, the multi-agent reinforcement learning algorithm and evolutionary algorithm are integrated and a context memory mechanism is introduced. The reinforcement learning models of each agent in Step 2 are trained and solved offline and called online in real time to generate a collaborative voltage control strategy, so as to minimize voltage fluctuations and optimize economic efficiency in the distribution network and microgrid.

[0035] Further, step 1 includes:

[0036] Step 1-1: Establish a microgrid operation optimization control model:

[0037] Microgrids aim for optimal economic efficiency while being constrained by factors such as the state of charge (SBC) of energy storage devices and power balance. The mathematical model for the operation and control of a microgrid can be described as follows:

[0038] (1)

[0039] in, Let T represent the objective function for microgrid operation control, and let T represent the complete training time. This indicates the electricity price traded between the microgrid and the distribution network. This indicates the power interaction between the microgrid and the distribution network. This represents the profit or cost generated from transactions between the microgrid and the distribution network, where N represents the total number of electrical loads within the microgrid. This represents the power consumed by the f-th load. This represents the revenue generated by a microgrid selling electricity to its internal loads. Indicates the number of gas turbines. Cost per unit power generated by a gas turbine This represents the gas turbine power at time t. This represents the cost required for gas turbine power generation, where r represents the index value of the gas turbine.

[0040] Micro gas turbines generate electricity by consuming natural gas. The power output of a micro gas turbine is expressed as:

[0041] (2)

[0042] in, This represents the power generation of the micro gas turbine at time t. This represents the power generation efficiency coefficient of a micro gas turbine. This indicates the fuel flow rate.

[0043] To improve the absorption capacity of renewable energy, energy storage devices are typically connected to microgrids to mitigate the volatility of renewable energy and to perform peak shaving and valley filling. The dynamic process can be represented as follows:

[0044] (3)

[0045] in, This represents the state of charge of the energy storage device at time t. This indicates the state of charge of the energy storage device at time t-1. [0,1] represent the charging efficiency coefficient and discharging efficiency coefficient of the energy storage device, respectively. This represents the charging power of the energy storage device at time t. Indicates the energy storage capacity of the energy storage device. This represents the discharge power of the energy storage device at time t.

[0046] The operation and control of microgrids also need to meet the following constraints:

[0047] State of charge constraints for energy storage devices:

[0048] (4)

[0049] in, This represents the state of charge of the energy storage device in the p-th microgrid at hour t. This represents the minimum state of charge of the energy storage device in the p-th microgrid. This represents the maximum state of charge (SOC) of the energy storage device in the p-th microgrid. This is determined by the initial SOC. With final state of charge The constraints ensure the backup capacity of the energy storage device.

[0050] The upper and lower limits of output for micro gas turbines and microgrids are as follows:

[0051] (5)

[0052] in, This represents the maximum power output of the gas turbine at time t. This represents the grid-connected power of the p-th microgrid. This represents the minimum allowable grid connection power for the p-th microgrid. This represents the maximum allowable grid-connected power for the p-th microgrid.

[0053] Microgrid power balance constraints:

[0054] (6)

[0055] in, Represents the energy provided by distributed energy sources. This represents the electrical energy traded between the microgrid and the distribution network. This represents the power consumption of the loads within the microgrid at time t. This represents the charging power of the energy storage device at time t.

[0056] Step 1-2: Build a voltage control model for the power distribution network:

[0057] The objective of distribution network optimization control is to minimize voltage fluctuations at each node and achieve optimal system economy. as follows:

[0058] (7)

[0059] in, Represents voltage fluctuations in the power distribution network. Represents the first in the distribution network Node voltage, The standard voltage for each node is represented by n, which represents the total number of nodes in the distribution network. Indicates the time step; Represents the total economic benefits of the power distribution network. This indicates the total number of microgrids connected to the distribution network. , This represents the price at which the distribution network sells electricity to the main grid and the price at which it purchases electricity at time t. , This represents the amount of electricity the distribution network needs to sell to and purchase from the main grid after power flow calculation and balancing. q is the node number to which the microgrid is connected. , These represent the electricity price when the distribution network sells electricity to the microgrid and the electricity price when the microgrid sells electricity to the distribution network, respectively. This represents the amount of electricity sold by the distribution network to the q-th microgrid. This represents the amount of electricity that the distribution network purchases from the q-th microgrid.

[0060] The power flow equation constraints for the distribution network are:

[0061] (8)

[0062] in, This refers to nodes that include microgrids or distributed generation. The active power of power generation, This indicates a node that includes microgrid loads. The active load power. Nodes including microgrids or distributed generation The reactive power of power generation, This refers to nodes that include microgrids or distributed generation. The reactive load power. Representative node With nodes The voltage phase angle difference at time t. , Represents distribution network nodes ,node The magnitude of the voltage at time t. Represents a node With nodes The electrical conductance between them Represents a node With nodes The susceptance between [variables]. The upper and lower limits of voltage and active and reactive power at each node in the distribution network are:

[0063] (9)

[0064] in, , Representing distribution network nodes The upper and lower limits of active power at time t; , Represents distribution network nodes The upper and lower limits of reactive power at time t; Represents distribution network nodes The upper and lower limits of the voltage amplitude.

[0065] Further, step 2 includes:

[0066] Step 2-1: Set up the state space:

[0067] Each distribution network and microgrid can be considered as an intelligent agent, and the observable state variables of the distribution network are... This includes the distribution network's own time-series information and the electricity price information for the transaction between the distribution network and the main grid, as shown in formula (10):

[0068] (10)

[0069] in, The current time-series information consists of the current day and the current hour. These represent the purchase price and sales price of electricity in the distribution network and the main grid, respectively.

[0070] Observable state variables of microgrids As shown in equation (11):

[0071] (11)

[0072] in, This represents the state variables of distributed energy storage within a microgrid. This represents the load size of the microgrid at that moment. This represents the amount of distributed energy output received by the microgrid at that moment.

[0073] Step 2-2: Set the motion space:

[0074] The distribution network intelligent agent primarily uses a price incentive mechanism to guide the power injection into each microgrid, thereby achieving voltage optimization target control. Therefore, the control action variables of the distribution network intelligent agent... As shown in formula (12):

[0075] (12)

[0076] in, This indicates the electricity price when the distribution network sells electricity to the microgrid.

[0077] The operational space of each microgrid is a three-dimensional discrete space, mainly including the output control of each unit within the microgrid and the control of electricity sales prices. Microgrid control actions. As shown in equation (13):

[0078] (13)

[0079] in, This is the control signal for the power generation of the gas turbine. This indicates the electricity price when the microgrid sells electricity to the distribution network. This represents the charging and discharging signal of distributed energy storage.

[0080] Steps 2-3: Define the reward function:

[0081] To maximize the benefits of the distribution network and minimize voltage fluctuations at each node, the reward function of the distribution network agent is... The design is as follows:

[0082] (14)

[0083] in, The revenue generated by the distribution network from selling electricity to microgrids and the main grid. The cost of electricity purchased by the distribution network from various microgrids and the main grid. This is a penalty term for voltage fluctuations at each node.

[0084] Distribution network revenue It consists of two parts: one part comes from selling electricity to the main grid, and the other part comes from selling electricity to the microgrid cluster. The calculation is shown in formula (15):

[0085] (15)

[0086] in, This represents the price at which the distribution network sells electricity to the main grid at time t. This indicates the amount of electricity sold from the distribution network to the main grid. This indicates the electricity price when the distribution network sells electricity to the microgrid. This represents the amount of electricity sold by the distribution network to the q-th microgrid.

[0087] Cost of obtaining electricity from the distribution network It consists of two parts, as shown in equation (16):

[0088] (16)

[0089] in, This represents the price at which the distribution network purchases electricity from the main grid at time t. This indicates the amount of electricity the distribution network needs to purchase from the main grid. This represents the electricity price when the q-th microgrid sells electricity to the distribution network. This represents the amount of electricity that the distribution network purchases from the i-th microgrid.

[0090] To maximize the revenue of each microgrid, a reward function as shown in equation (17) is designed. The expression is:

[0091] (17)

[0092] in, For the revenue from selling electricity to the q-th microgrid, The cost of purchasing electricity for the q-th microgrid.

[0093] Revenue from microgrid electricity sales It consists of two parts: one part is the revenue from selling electricity to the distribution network, and the other part is the revenue from selling electricity to users within the microgrid.

[0094] (18)

[0095] in, This represents the number of load users within the microgrid. This represents the load of the m-th user at time t.

[0096] Generation costs of each microgrid As shown in equation (19);

[0097] (19)

[0098] in, Cost per unit power generated by a gas turbine This represents the power generation of the micro gas turbine at time t.

[0099] Further, step 3 includes:

[0100] Step 3-1: Initialize the experience pool, scenario replay area, and network parameters for the multi-agent reinforcement learning and evolutionary algorithms:

[0101] The actions of each agent are randomly initialized, and the state transition process of each agent is recorded. The current state, the current action taken, the next state, and the reward function of each agent are stored in the experience pool until the experience pool is full. At the same time, the scenario buffer stores the state-action pairs of the current best policy and initializes the weights of the target Q network of each agent.

[0102] Step 3-2: Each agent obtains the current environmental state. :

[0103] Take the local state observations of each agent within the current time period. This includes information on electricity prices between distribution network smart agents and the main grid, the status of distributed energy storage in microgrid smart agents, load size, and output of distributed energy units. The shared policy observer extracts the global environmental status for each smart agent. .

[0104] Step 3-3: Using an improved multi-agent reinforcement learning algorithm that combines evolutionary algorithms with episodic memory mechanisms, select the current action. :

[0105] The population strategy of the evolutionary algorithm is evaluated, and the population fitness is converted into a reward. The reward of the multi-agent reinforcement learning algorithm is calculated, and the non-policy experience generated by the evolutionary algorithm and the multi-agent reinforcement learning algorithm is stored in the shared experience replay pool. The loss functions of the evolutionary algorithm and the multi-agent reinforcement learning algorithm are calculated according to the formulas (20) to (21), and the Q function is updated.

[0106] (20)

[0107] (twenty one)

[0108] in, This represents the loss function of the evolutionary algorithm. This represents the loss function for multi-agent reinforcement learning. Representing state In distribution Expectations This represents the non-policy experience collected by all multi-agent reinforcement learning algorithm teams and all evolutionary algorithm teams. and Let each represent a parameter of the Q-function of the i-th agent. Represents the j-th team in the evolutionary algorithm. These represent the strategies generated by the evolutionary algorithm team and the strategies generated by the multi-agent reinforcement learning algorithm, respectively.

[0109] Crossover mutation is performed on the evolutionary algorithm population, and the Q function in the multi-agent reinforcement learning algorithm is modified by memory target regularization in the context memory mechanism. The context memory target added to the temporal sorting loss calculation is shown in Equations (22)-(23). After comprehensively considering the magnitude of the temporal sorting loss in the current step and the deviation between the current policy and the historical best policy samples, the comprehensive loss function of the improved multi-agent reinforcement learning algorithm is shown in Equation (24).

[0110] (twenty two)

[0111] (twenty three)

[0112] (twenty four)

[0113] in, These are parameters of the Q function. This represents the state-action value estimation function. It is a weighting term for balancing the target reference effect of contextual memory. This indicates the magnitude of the time-series lookup loss at the current step. This represents the loss function obtained with the episodic memory target as a reference. Represents the state of the agent in the i-th round of training. Take action below The best historical reward value that can be obtained, where E represents the number of training rounds the agent has undergone. This represents the state of the agent during the y-th training round. Take action below The rewards that can be obtained.

[0114] Steps 3-4: Update the experience pool and scenario replay area:

[0115] Each agent executes an improved multi-agent reinforcement learning algorithm that integrates evolutionary algorithms and episodic memory mechanisms to obtain its current action. To obtain the local state s at the next time step. t+1 and reward value r t The state transition process is stored in the experience pool. If the experience pool is full, the oldest experience record is deleted. If the experience pool is not full, the next step is performed. At the same time, the state-action pair of the current best strategy is stored in the scenario experience replay area.

[0116] Steps 3-5: Integrating the loss functions of multi-agent reinforcement learning and evolutionary algorithms:

[0117] The optimized multi-agent reinforcement learning algorithm strategy is shared with the evolutionary algorithm population, and the shared policy encoder is updated according to Equation (25).

[0118] (25)

[0119] in, Indicates the update strategy, Let represent the local observations of the i-th agent at time t, and act(·) denote the activation function. The matrix... This represents the linear strategy for each team. Represents a nonlinear shared observation encoder. express A 1-dimensional real vector This represents the action dimension of the i-th agent. Representation matrix line d+1, Representation matrix Lines 1 to d.

[0120] Finally, repeat steps 3-2 to 3-5 until the maximum number of training iterations is reached.

[0121] Example:

[0122] 1. Example setup:

[0123] The modified 33-node distribution system containing multiple microgrids is selected as an example, such as... Figure 2 As shown (MG represents a microgrid), five microgrids are connected to five nodes (nodes 10, 15, 20, 22, and 28). The system voltage reference value is 12.66 kV, the total voltage control duration is set to one day, and the decision control step size is one hour. The hardware platform for simulation testing includes: CPU: Intel(R) Core(TM) i7-9750H CPU @260HZ, GPU: NVIDIA 1650TX; the software platform includes: Python 3.9; PyTorch 1.9.1. For the improved multi-agent reinforcement learning algorithm that integrates the proposed evolutionary algorithm and the contextual memory mechanism, the hyperparameters are set as follows: experience pool capacity is set to 5000, batch size is set to 128, discount factor is set to 0.99, learning rate is set to 0.0001, and episode is set to 2500 rounds.

[0124] 2. Analysis of Implementation Results:

[0125] To demonstrate the effectiveness of the proposed improved multi-agent reinforcement learning algorithm (EM-RACE) that integrates the evolutionary algorithm with the context memory mechanism, the traditional multi-agent dual-delay deep deterministic policy gradient algorithm (MATD3) and the multi-agent reinforcement learning algorithm integrated with the evolutionary algorithm (RACE) without considering the context memory mechanism are selected as comparison algorithms. The reward function curves of the distribution network and some microgrids during the training process of the three algorithms are shown in the figure. Figure 3 , Figure 4a , Figure 4b , Figure 4c As shown. Among them, Figure 3 A comparison chart showing the convergence of global reward values ​​for different algorithms. Figure 4a , Figure 4b , Figure 4c This image shows a comparison of the convergence of the reward function in a distribution network and some microgrids under different algorithms. Figure 3It is evident that as the number of training rounds increases, the global reward function curves of each algorithm gradually stabilize. During training, the agents continuously learn control strategies that comply with network security constraints, and the global reward value steadily increases until convergence. However, the multi-agent dual-delay deep deterministic policy gradient algorithm suffers from insufficient exploration rate in the microgrid collaborative voltage control environment, and conflicts of interest exist among the agents, causing the global reward value to oscillate slightly near local optima. The proposed improved multi-agent reinforcement learning algorithm, which integrates evolutionary algorithms and contextual memory mechanisms, utilizes diverse experiences and historical best strategies for guidance. Compared to the multi-agent reinforcement learning algorithm integrated with evolutionary algorithms without considering contextual memory mechanisms, and the multi-agent dual-delay deep deterministic policy gradient algorithm, it can more smoothly explore the optimal voltage collaborative control strategy.

[0126] Furthermore, from Figure 4a , Figure 4b , Figure 4c As can be seen, although all agents in the three multi-agent reinforcement learning algorithms eventually converged their reward values, the multi-agent dual-delay deep deterministic policy gradient algorithm, which uses a traditional multi-agent algorithm to explore the policy space, suffers from conflicting policies between the agents in the microgrid and distribution network, leading to drastic fluctuations in reward values. The first microgrid even explored a poorer policy at one point, getting stuck in a local optimum. In contrast, the multi-agent reinforcement learning algorithm that integrates with an evolutionary algorithm, without considering contextual memory, introduces an evolutionary algorithm. During the multi-agent policy exploration phase, it utilizes the interaction between the evolutionary algorithm and the environment to provide diverse samples for the shared experience pool, guiding the agents out of the local optimum search space. Compared to the multi-agent dual-delay deep deterministic policy gradient algorithm, it learns a better global voltage control policy and obtains a higher reward value. However, the multi-agent reinforcement learning algorithm that integrates with an evolutionary algorithm without considering contextual memory relies solely on traditional time-series differential errors to guide policy selection, failing to efficiently utilize the explored historical experience samples. This results in the first and third microgrids requiring longer training times to learn superior control policies. The improved multi-agent reinforcement learning algorithm, which integrates the proposed evolutionary algorithm with an episodic memory mechanism, incorporates this mechanism, using the remembered target as a regularization term to modify the loss function. This allows the agents to learn high-reward behaviors more quickly. Enhanced exploration capabilities and improved sample efficiency lead to a higher global reward, resulting in a superior voltage control strategy. Ultimately, the improved multi-agent reinforcement learning algorithm with the integration of the evolutionary algorithm and episodic memory mechanism achieves an 11.3% improvement in global reward function value compared to the algorithm without episodic memory mechanism, and a 19.7% improvement compared to the multi-agent dual-delay deep deterministic policy gradient algorithm.

[0127] Figure 5a , Figure 5b , Figure 5c The voltage distribution of 33 nodes in the power distribution system at three time periods—5:00, 10:00, and 23:00—is presented using different algorithms. Figure 5a , Figure 5b , Figure 5c As can be seen, under the control of the traditional multi-agent dual-delay deep deterministic policy gradient algorithm, the voltage deviation at the end nodes 17-19 and 30-33 of the distribution network still shows significant deviations below the lower limit. The algorithm combining multi-agent reinforcement learning and evolutionary algorithms without considering context memory mechanisms improves voltage support to some extent by controlling the output of internal equipment in the microgrid, but still cannot meet the voltage control requirements of all nodes in the distribution network. The improved multi-agent reinforcement learning algorithm, by using existing historical experience samples to guide the microgrid agents in learning, allows each microgrid agent to optimize the output of its internal equipment units based on historical optimal strategies, resulting in stronger voltage support capabilities. This not only improves the voltage at the end nodes but also enhances the overall voltage level of the 33 nodes. Overall, compared to the multi-agent dual-delay deep deterministic policy gradient algorithm and the algorithm combining multi-agent reinforcement learning and evolutionary algorithms without considering context memory mechanisms, the improved multi-agent reinforcement learning algorithm combining the proposed evolutionary algorithm and context memory mechanism reduces the controlled voltage deviation by 18.38% and 8.21%, respectively.

Claims

1. A method for coordinated voltage control of distribution networks and microgrid clusters, characterized in that, Includes the following steps: Step 1: To achieve the goals of minimizing voltage fluctuations and optimizing economic efficiency in the distribution network and the microgrid, establish a mathematical model for coordinated voltage control of the distribution network and microgrid clusters. The dynamic process of connecting energy storage devices to each microgrid to alleviate the volatility of renewable energy and to play a role in peak shaving and valley filling is represented as follows: (3) in, This represents the state of charge of the energy storage device at time t. This indicates the state of charge of the energy storage device at time t-1. [0,1] represent the charging efficiency coefficient and discharging efficiency coefficient of the energy storage device, respectively. This represents the charging power of the energy storage device at time t. Indicates the energy storage capacity of the energy storage device. This represents the discharge power of the energy storage device at time t; The operation and control of microgrids also need to meet the following constraints: State of charge constraints for energy storage devices: (4) in, This represents the state of charge of the energy storage device in the p-th microgrid at hour t. This represents the minimum state of charge of the energy storage device in the p-th microgrid. This represents the maximum state of charge (SOC) of the energy storage device in the p-th microgrid; by analyzing the initial SOC... With final state of charge The constraints ensure the backup capacity of the energy storage device; The upper and lower limits of output for micro gas turbines and microgrids are as follows: (5) in, This represents the maximum power generation of the gas turbine at time t; This represents the grid-connected power of the p-th microgrid. This represents the minimum allowable grid connection power for the p-th microgrid. This represents the maximum allowable grid-connected power for the p-th microgrid; Microgrid power balance constraints: (6) in, Represents the energy provided by distributed energy sources. Represents the electrical energy traded between the microgrid and the distribution network; This represents the power consumption of the loads within the microgrid at time t. This represents the charging power of the energy storage device at time t; Step 2: Treat the distribution network and each microgrid as independent intelligent agents. Based on the mathematical model of the coordinated voltage control of the distribution network and microgrid cluster constructed in Step 1, construct the state space, action space and reward function for reinforcement learning training of the control strategies of each intelligent agent, and form the training environment and reinforcement learning model of each intelligent agent. Control action variables of distribution network intelligent agents As shown in formula (12): (12) in, This indicates the electricity price when the distribution network sells electricity to the microgrid; The operational space of each microgrid is a three-dimensional discrete space, including the output control of each unit within the microgrid and the control of electricity sales prices; the control actions of the microgrid As shown in equation (13): (13) in, This is the control signal for the power generation of the gas turbine. This indicates the electricity price when the microgrid sells electricity to the distribution network. This represents the charging and discharging signal of distributed energy storage; reward function The expression is: (17) in, For the revenue from selling electricity to the q-th microgrid, The cost of purchasing electricity for the q-th microgrid; Step 3: Based on the training environment built in Step 2, the multi-agent reinforcement learning algorithm and evolutionary algorithm are integrated and a context memory mechanism is introduced to train and solve the reinforcement learning models of each agent in Step 2 offline and call them online in real time to generate a collaborative voltage control strategy to minimize voltage fluctuations and optimize economic efficiency in the distribution network and microgrid. Calculate the loss functions of evolutionary algorithms and multi-agent reinforcement learning algorithms, and update the Q-function: (20) (21) in, This represents the loss function of the evolutionary algorithm. This represents the loss function for multi-agent reinforcement learning. Representing state In distribution Expectations This represents the non-policy experience collected by all multi-agent reinforcement learning algorithm teams and all evolutionary algorithm teams. and Let these represent the Q-function parameters of the i-th agent, respectively. Represents the j-th team in the evolutionary algorithm. These represent the strategies generated by the evolutionary algorithm team and the strategies generated by the multi-agent reinforcement learning algorithm, respectively. Crossover mutation is performed on the population of the evolutionary algorithm. At the same time, the Q function in the multi-agent reinforcement learning algorithm is modified by memory target regularization in the context memory mechanism. The context memory target added to the time-series search loss calculation is as shown in Equations (22)-(23): (22) (23) Taking into account both the magnitude of the time-series search loss at the current step and the deviation between the current policy and the historical best policy samples, the comprehensive loss function of the improved multi-agent reinforcement learning algorithm is shown in equation (24): (24) in, These are Q function parameters. This represents the state-action value estimation function. It is a weighting term that balances the target reference effect of episodic memory; This indicates the magnitude of the time-series lookup loss at the current step. This represents the loss function obtained with the episodic memory target as a reference. Represents the state of the agent in the i-th round of training. Take action a t The best historical reward value that can be obtained, where E represents the number of training rounds the agent has undergone. This represents the state of the agent during the y-th training round. Take action a t The rewards that can be obtained; The optimized multi-agent reinforcement learning algorithm policy is shared with the evolutionary algorithm population, and the shared policy encoder is updated according to equation (25): (25) in, Indicates the update strategy, Let represent the local observations of the i-th agent at time t, and act(·) denote the activation function. The matrix... This represents the linear strategy for each team. Represents a nonlinear shared observation encoder. express A 1-dimensional real vector This represents the action dimension of the i-th agent. Representation matrix line d+1, Representation matrix The lower 1st row to the dth row.

2. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, In step 1, the mathematical model includes: power flow constraints of the distribution network, upper and lower limits of node voltage, and state of charge and power balance constraints of microgrid energy storage.

3. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, In step 2, the state space includes: the electricity price of the distribution network and the main grid, the output of distributed energy resources in the microgrid, and the state of charge of energy storage.

4. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, In step 2, the action space includes: the electricity sales price signal from the distribution network to the microgrid, and the power of the gas turbine and the charging and discharging commands of the energy storage within the microgrid.

5. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, In step 2, the reward function is composed of the distribution network revenue and the node voltage fluctuation penalty, and the microgrid reward is composed of the difference between the electricity sales revenue and the generation cost.

6. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, In step 3, the multi-agent reinforcement learning algorithm saves high-reward experiences through the contextual memory mechanism and modifies the Q function with memory target regularization.

7. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, In step 3, the evolutionary algorithm periodically shares strategies with the multi-agent reinforcement learning algorithm to provide diverse samples for the experience pool.

8. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, Step 3 includes: obtaining the current state, selecting an action, executing the action, updating the experience pool and the scenario replay area, until the training ends.

9. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, The cooperative voltage control strategy can be directly used for real-time regulation after training, without the need for model linearization or secondary solution.

10. The method for coordinated voltage control of distribution networks and microgrid clusters according to claim 1, characterized in that, The mathematical model, training environment, and the integration of multi-agent reinforcement learning and evolutionary algorithms within the same software framework are all implemented.

Images