Coordinated voltage control method and system for grid-connected microgrid cluster

Abstract

The application provides a kind of coordinated voltage control method and system for grid-connected microgrid cluster, method includes collecting the common connection point voltage of each sub-microgrid, photovoltaic active output, energy storage state of charge and load data, calculates net load and forms current state vector after inputting the strategy network of training completion, obtains continuous action vector;Proportional mapping and boundary clipping are performed on continuous action vector, and control instructions are obtained;The control instructions are respectively issued to the energy storage converter and photovoltaic inverter of each sub-microgrid, and active and reactive power coordinated regulation is executed;Control instructions are executed and updated grid-connected microgrid cluster state is collected, and the next control cycle is entered;The updated grid-connected microgrid cluster state is input into the strategy network as the current state of the next control cycle.The application realizes active-reactive multi-resource coordinated voltage regulation by simultaneously outputting energy storage active power and photovoltaic inverter reactive power, and improves controllable resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence control technology, specifically to a collaborative voltage control method and system for grid-connected microgrid clusters. Background Technology

[0002] As the penetration rate of distributed photovoltaic (PV) power in distribution networks and microgrids continues to increase, PV output is affected by cloud cover, temperature changes, and rapid fluctuations in irradiance, potentially causing significant changes within a short period. For low-voltage distribution systems and grid-connected microgrid clusters, due to the high line resistance and reactance, rapid changes in PV active power can cause voltage shifts, short-term fluctuations, and even exceeding limits at the point of common coupling. After multiple sub-microgrids are clustered together via a common bus or coupling lines, changes in the net injected power of one sub-microgrid can also affect other sub-microgrids through electrical coupling. Therefore, control methods relying solely on local information from a single node are insufficient to fully utilize the global operating status at the cluster level.

[0003] Existing voltage regulation methods include on-load tap-changing transformers, switched capacitors, static var compensators (SVCs), reactive power control of photovoltaic (PV) inverters, active power regulation of energy storage, and droop control. PV inverters can quickly provide reactive power support using surplus capacity, and energy storage systems can change active power injection at nodes through charging and discharging; both have relatively fast dynamic response capabilities. Droop control has a simple structure and does not require complex communication, but it typically uses a fixed droop coefficient and mainly generates control quantities based on local voltage deviations, making it difficult to simultaneously consider PV output, energy storage state of charge, net load, and the coordination relationships between multiple sub-microgrids.

[0004] Voltage control methods based on optimization models can achieve good control performance when network parameters are accurate. However, their online solution relies on accurate power flow models, resulting in high computational complexity and model maintenance costs when dealing with ZIP loads, photovoltaic intermittency, continuous control of multiple devices, and constantly changing operating scenarios. Reinforcement learning can learn control strategies from states to actions through interaction with the simulation environment. After training, it can quickly infer online, making it suitable for continuous action control problems. However, existing deep reinforcement learning voltage control techniques are mostly aimed at single distribution networks or single types of voltage regulating equipment. Some methods focus on minimizing network losses or local voltage as the main objective. There is still a lack of a continuous control scheme for grid-connected microgrid clusters that simultaneously coordinates the active power of energy storage and the reactive power of photovoltaic inverters, and explicitly considers photovoltaic intermittency and voltage over-limit penalties.

[0005] Therefore, it is necessary to propose a collaborative voltage control method that can integrate the global state of multi-sub-microgrids, adapt to different weather conditions, jointly utilize the regulation capabilities of energy storage and photovoltaic inverters, and can be quickly deployed after training.

[0006] Patent document CN115588998A discloses a distribution network voltage reactive power optimization method based on graph reinforcement learning. This patent document constructs the distribution network voltage reactive power optimization problem as an MDP, adopts GCN-PPO, and uses SVC and photovoltaic inverter reactive power as the main control variables, focusing on using graph convolution to adapt the distribution network topology. In contrast, this invention is aimed at grid-connected microgrid clusters composed of multiple sub-microgrids. The state explicitly includes the energy storage SOC and net load of each sub-microgrid, and the action includes both the active power of energy storage and the reactive power of photovoltaic inverters. The control objective directly addresses cluster voltage deviation and over-limit.

[0007] Patent document CN120222494A discloses a voltage regulation method for photovoltaic-storage coordinated operation in distribution networks based on deep reinforcement learning. This patent document uses safety-constrained multi-agent deep reinforcement learning to control photovoltaic inverters and employs a deep deterministic strategy for gradient control of energy storage, belonging to the category of photovoltaic-storage coordinated voltage regulation in distribution networks. In comparison, this invention uses a unified PPOActor-Critic strategy to simultaneously output the active power of energy storage and the reactive power of photovoltaics at the cluster level, constructing a 12-dimensional global state and a 6-dimensional continuous action, and combines sunny, cloudy, and rainy weather intermittent photovoltaic scenarios for training and verification.

[0008] Patent document CN112186799A discloses an autonomous control method and system for distributed energy systems based on deep reinforcement learning. This patent document addresses the autonomous control of multi-energy systems such as electricity, gas, cooling, and heating, emphasizing integrated energy dispatching and simulation-based decision-making. In contrast, this invention focuses on the voltage quality problem of grid-connected microgrid clusters, establishes explicit voltage deviation and over-limit reward functions, and provides constraints for photovoltaic, energy storage, ZIP loads, and continuous operation.

[0009] Patent document CN118739383A discloses a microgrid control method, control device, and microgrid. This patent document generates dispatch parameters based on energy storage data, load data, and predicted load trends from multiple sub-microgrids, primarily focusing on power allocation and overall efficiency. In contrast, this invention employs PPO learning to map real-time state to continuous voltage regulation actions, directly coordinating energy storage charging and discharging with reactive power output from the photovoltaic inverter, and using voltage reference values ​​and allowable ranges as control constraints. Summary of the Invention

[0010] In view of the deficiencies in the prior art, the purpose of this invention is to provide a collaborative voltage control method and system for grid-connected microgrid clusters.

[0011] A cooperative voltage control method for grid-connected microgrid clusters provided by the present invention includes: Step S1: Collect the common coupling point voltage, photovoltaic active power output, energy storage state of charge and load data of each sub-microgrid, calculate the net load and form the current state vector; Step S2: Input the current state vector into the trained policy network to obtain a continuous action vector composed of the energy storage charging and discharging power of each sub-microgrid and the reactive power of the photovoltaic inverter; perform proportional mapping and boundary clipping on the continuous action vector to obtain control commands; the control commands include energy storage power commands and photovoltaic inverter reactive power commands. Step S3: Send the control commands to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation; Step S4: Execute the control command and collect the updated state of the grid-connected microgrid cluster, and enter the next control cycle; use the updated state of the grid-connected microgrid cluster as the current state input to the strategy network for the next control cycle.

[0012] Preferably, the load data includes active power and reactive power of the load; by establishing photovoltaic, energy storage, ZIP load, power balance and voltage coupling models between sub-microgrids, a microgrid cluster simulation environment for reinforcement learning agents to interact is constructed.

[0013] Preferably, the voltage control problem of a microgrid cluster is constructed as a Markov decision process; for a cluster containing N sub-microgrids, the state vector s includes the voltage, photovoltaic output, energy storage state of charge, and net load of each sub-microgrid:

[0014] in, This represents the common point voltage of the i-th sub-microgrid. This represents the photovoltaic active power output of the i-th sub-microgrid. This represents the state of charge of the i-th sub-microgrid energy storage system. This represents the net load of the i-th sub-microgrid, where i represents the sub-microgrid index; The policy network outputs a continuous action vector a based on the state vector:

[0015] in, For energy storage charging and discharging power, The reactive power of the photovoltaic inverter. This represents the active power of energy storage charging and discharging in the i-th sub-microgrid. This represents the reactive power of the photovoltaic inverter in the i-th sub-microgrid; the controller performs physical constraint processing on the action before issuing the action, including the rated power constraint of energy storage, the SOC boundary constraint and the apparent power constraint of the photovoltaic inverter.

[0016] Preferably, the reactive power of the photovoltaic inverter satisfies the following formula: ; Among them, Sinv This represents the rated apparent power capacity of the photovoltaic inverter in the i-th sub-microgrid. The reactive power of the photovoltaic inverter. This indicates the active power output of the photovoltaic system.

[0017] Preferably, a reward function is constructed based on the voltage reference value, the allowable upper and lower limits, and the weight of the penalty for exceeding the limit; the reward function Including voltage deviation penalty and voltage over-limit penalty, the expression is:

[0018] In the formula, For reference voltage, and To allow for upper and lower voltage limits, The voltage over-limit penalty weight is used, and N represents the number of sub-microgrid clusters. This represents the common point voltage of the i-th sub-microgrid during the t-th control cycle.

[0019] Preferably, a proximal policy optimization agent, including a policy network and a value network, is used to interact with the simulation environment in a training scenario consisting of different irradiance and load sequences, obtain state, action, reward and next state samples, and update network parameters based on the pruning agent target.

[0020] Preferably, the proximal policy optimization agent adopts an Actor-Critic structure; the Actor network outputs the mean and standard deviation of the continuous action distribution based on the state, and the Critic network estimates the state value.

[0021] Preferably, data is collected during the training process. The sequence is used to calculate the advantage function, and the magnitude of change between the old and new strategies is limited by pruning the agent objective. :

[0022] in, This represents the probability ratio between the new strategy and the old strategy. For the sake of advantage estimation, For the clipping factor, This represents the expected value at time t. This represents the network parameters of the current policy, and clip(·) represents the probability ratio constraint function. Indicates the current state. Indicates continuous actions, Represents the reward function, This indicates the new voltage and power state.

[0023] A cooperative voltage control system for grid-connected microgrid clusters provided by the present invention includes: Module M1: Collects the common coupling point voltage, photovoltaic active power output, energy storage state of charge and load data of each sub-microgrid, calculates the net load and forms the current state vector; Module M2: Inputs the current state vector into the trained policy network to obtain a continuous action vector composed of the energy storage charging and discharging power of each sub-microgrid and the reactive power of the photovoltaic inverter; performs proportional mapping and boundary clipping on the continuous action vector to obtain control commands; the control commands include energy storage power commands and photovoltaic inverter reactive power commands. Module M3: Sends the control commands to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation; Module M4: Executes the control command and collects the updated state of the grid-connected microgrid cluster, then enters the next control cycle; the updated state of the grid-connected microgrid cluster is used as the current state input to the strategy network for the next control cycle.

[0024] Preferably, the load data includes active power and reactive power of the load; by establishing photovoltaic, energy storage, ZIP load, power balance and voltage coupling models between sub-microgrids, a microgrid cluster simulation environment for reinforcement learning agents to interact is constructed.

[0025] Preferably, the voltage control problem of a microgrid cluster is constructed as a Markov decision process; for a cluster containing N sub-microgrids, the state vector s includes the voltage, photovoltaic output, energy storage state of charge, and net load of each sub-microgrid:

[0026] in, This represents the common point voltage of the i-th sub-microgrid. This represents the photovoltaic active power output of the i-th sub-microgrid. This represents the state of charge of the i-th sub-microgrid energy storage system. This represents the net load of the i-th sub-microgrid, where i represents the sub-microgrid index; The policy network outputs a continuous action vector a based on the state vector:

[0027] in, For energy storage charging and discharging power, The reactive power of the photovoltaic inverter. This represents the active power of energy storage charging and discharging in the i-th sub-microgrid. This represents the reactive power of the photovoltaic inverter in the i-th sub-microgrid; the controller performs physical constraint processing on the action before issuing the action, including the rated power constraint of energy storage, the SOC boundary constraint and the apparent power constraint of the photovoltaic inverter.

[0028] Preferably, the reactive power of the photovoltaic inverter satisfies the following formula: ; Among them, S inv This represents the rated apparent power capacity of the photovoltaic inverter in the i-th sub-microgrid. The reactive power of the photovoltaic inverter. This indicates the active power output of the photovoltaic system.

[0029] Preferably, a reward function is constructed based on the voltage reference value, the allowable upper and lower limits, and the weight of the penalty for exceeding the limit; the reward function Including voltage deviation penalty and voltage over-limit penalty, the expression is:

[0030] In the formula, For reference voltage, and To allow for upper and lower voltage limits, The voltage over-limit penalty weight is used, and N represents the number of sub-microgrid clusters. This represents the common point voltage of the i-th sub-microgrid during the t-th control cycle.

[0031] Preferably, a proximal policy optimization agent, including a policy network and a value network, is used to interact with the simulation environment in a training scenario consisting of different irradiance and load sequences, obtain state, action, reward and next state samples, and update network parameters based on the pruning agent target.

[0032] Preferably, the proximal policy optimization agent adopts an Actor-Critic structure; the Actor network outputs the mean and standard deviation of the continuous action distribution based on the state, and the Critic network estimates the state value.

[0033] Preferably, data is collected during the training process. The sequence is used to calculate the advantage function, and the magnitude of change between the old and new strategies is limited by pruning the agent objective. :

[0034] in, This represents the probability ratio between the new strategy and the old strategy. For the sake of advantage estimation, For the clipping factor, This represents the expected value at time t. This represents the network parameters of the current policy, and clip(·) represents the probability ratio constraint function. Indicates the current state. Indicates continuous actions, Represents the reward function, This indicates the new voltage and power state.

[0035] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention incorporates the voltage, photovoltaic output, energy storage SOC and net load of each sub-microgrid into a unified state vector, enabling coordinated control using global cluster information and overcoming the limitation that fixed droop parameters rely solely on local voltage.

[0036] 2. This invention improves the efficiency of controllable resource utilization by simultaneously outputting the active power of energy storage and the reactive power of photovoltaic inverters, thereby achieving coordinated voltage regulation of active and reactive power resources.

[0037] 3. This invention allows for the adjustment of control target weights according to engineering requirements by simultaneously setting voltage deviation and voltage limit exceedance terms in the reward function, and directly embeds the allowable voltage range into the training process.

[0038] 4. This invention limits the policy update range through the PPO pruning update mechanism, thereby improving the stability of continuous action training; after training is completed, only forward inference of the policy network is required, making it suitable for online execution under shorter control cycles.

[0039] 5. By constructing photovoltaic power output scenarios under various weather conditions, the control strategy can learn the impact of photovoltaic intermittency on voltage and dynamically adjust the control actions according to changes in photovoltaic power output, energy storage status, and net load.

[0040] 6. The method provided by this invention can be extended to different numbers of sub-microgrids and can be adapted to different grid-connected microgrid clusters by adjusting the state dimension, action dimension, equipment constraints and voltage coupling model. Attached Figure Description

[0041] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a grid-connected microgrid cluster structure.

[0042] Figure 2 Typical irradiance curves are shown for sunny, cloudy, and rainy days.

[0043] Figure 3 This is a diagram of a microgrid cluster collaborative voltage control framework based on near-end strategy optimization.

[0044] Figure 4 This is a training reward curve under different weather conditions.

[0045] Figure 5 This is a comparison chart of the voltage trajectories of PPO control and droop control under sunny conditions.

[0046] Figure 6 This is a comparison chart of the voltage trajectories of PPO control and droop control under cloudy conditions.

[0047] Figure 7 This is a comparison chart of the voltage trajectories of PPO control and droop control under rainy conditions.

[0048] Figure 8 This is a flowchart of the method of the present invention.

[0049] In the picture: Main Grid refers to the main power grid.

[0050] Common bus refers to the common bus line.

[0051] MG1 represents microgrid 1.

[0052] MG2 represents microgrid 2.

[0053] MG3 stands for Microgrid 3.

[0054] PV stands for photovoltaic power generation unit.

[0055] ESS stands for Energy Storage System.

[0056] ZIP load refers to a ZIP load, which is a constant impedance, constant current, and constant power load.

[0057] Simplified voltage coupling among sub-microgrids represents the simplified voltage coupling relationship between sub-microgrids.

[0058] Irradiance represents irradiance.

[0059] Time indicates the duration of time.

[0060] Clear means sunny.

[0061] Cloudy means overcast.

[0062] Rainy means rainy day.

[0063] State represents the state.

[0064] Current state represents the current state.

[0065] Observation represents the state of observation.

[0066] N: number of microgrids indicates that N represents the number of microgrids.

[0067] PPO Agent represents a proximal policy optimization agent.

[0068] Actor-Critic represents a policy network-value network structure.

[0069] Actor represents the policy network.

[0070] Critic stands for value network.

[0071] Policy refers to a strategy.

[0072] Value represents worth.

[0073] Action represents a movement or action.

[0074] Control Action refers to the action taken to control something.

[0075] Control command refers to a control instruction.

[0076] Environment refers to the surrounding environment.

[0077] Microgrid Cluster refers to a cluster of microgrids.

[0078] Next state indicates the next state.

[0079] "become the current state at the next step" means that the current state will be reached at the next time step.

[0080] Training feedback for policy update refers to the training feedback used for policy updates.

[0081] Reward Function refers to the reward function.

[0082] Voltage deviation / violation penalty refers to penalties for voltage deviation and penalties for exceeding voltage limits.

[0083] System response / voltage information refers to system response and voltage information.

[0084] Interaction flow refers to the process of interaction.

[0085] The control loop represents a closed-loop control system.

[0086] Training feedback flow refers to the training feedback process.

[0087] "Policy update" means policy is updated.

[0088] "At each time step" means at each time step.

[0089] "Reward" means bonus or incentive.

[0090] "Next state" refers to the next state.

[0091] "At the next time step" indicates the next time step.

[0092] "become the new current state" means to become the new current state.

[0093] Average reward refers to the average reward received.

[0094] Episode refers to a training round.

[0095] Instantaneous voltage under PPO refers to the instantaneous voltage under near-end policy optimization control.

[0096] Instantaneous voltage under droop control refers to the instantaneous voltage under droop control.

[0097] Time step refers to a time step.

[0098] MG1_PPO represents the near-end strategy optimization control curve of microgrid 1.

[0099] MG2_PPO represents the near-end strategy optimization control curve of microgrid 2.

[0100] MG3_PPO represents the near-end strategy optimization control curve of microgrid 3.

[0101] MG1_Droop represents the droop control curve of microgrid 1.

[0102] MG2_Droop represents the droop control curve of microgrid 2.

[0103] MG3_Droop represents the droop control curve of microgrid 3. Detailed Implementation

[0104] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0105] To address the problems of existing local droop control methods, such as difficulty in utilizing global cluster information, high computational complexity of model optimization methods, and voltage fluctuations caused by photovoltaic intermittency, this invention provides a grid-connected microgrid cluster collaborative voltage control method based on near-end strategy optimization, taking into account photovoltaic intermittency, to achieve the following objectives: A grid-connected microgrid cluster model is constructed that includes photovoltaic, energy storage, and ZIP loads, and the voltage of the common coupling point of each sub-microgrid, photovoltaic output, energy storage state of charge, and net load are uniformly incorporated into the control state.

[0106] The design ensures continuous coordinated operation of the energy storage charging and discharging power with the reactive power of the photovoltaic inverter, enabling active and reactive power regulation to cooperate at the cluster level.

[0107] A reward function is constructed that includes both voltage deviation penalty and voltage over-limit penalty, so that the controller can reduce the average voltage deviation while maintaining the allowable range of 0.95 to 1.05 pu.

[0108] The pruning and updating mechanism of the proximal policy optimization algorithm is used to improve training stability, and control instructions are quickly output through the policy network after training to reduce the online computation burden.

[0109] By training and validating the control strategy under different photovoltaic power output scenarios such as sunny, cloudy, and rainy days, the method's adaptability to photovoltaic intermittency and load changes can be improved.

[0110] like Figure 8 As shown, a collaborative voltage control method for grid-connected microgrid clusters includes: Step S1: Collect the common coupling point voltage, photovoltaic active power output, energy storage state of charge and load data of each sub-microgrid, calculate the net load and form the current state vector; Step S2: Input the current state vector into the trained policy network to obtain a continuous action vector composed of the energy storage charging and discharging power of each sub-microgrid and the reactive power of the photovoltaic inverter; perform proportional mapping and boundary clipping on the continuous action vector to obtain control commands; the control commands include energy storage power commands and photovoltaic inverter reactive power commands. Step S3: Send the control commands to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation; Step S4: Execute the control command and collect the updated state of the grid-connected microgrid cluster, and enter the next control cycle; use the updated state of the grid-connected microgrid cluster as the current state input to the strategy network for the next control cycle.

[0111] The method can be executed by a microgrid cluster energy management controller, edge controller, or dispatch master station. The microgrid cluster includes at least two sub-microgrids; in one embodiment, it includes three sub-microgrids, each of which includes a photovoltaic power generation unit, an energy storage system, and a ZIP load, and is connected to the main grid via a common bus.

[0112] In one embodiment, the method includes the following steps: Step S1: Obtain the common coupling point voltage, photovoltaic active power output, energy storage state of charge, load active power and reactive power of each sub-microgrid, and calculate the net load based on photovoltaic output and load.

[0113] Step S2: Establish photovoltaic, energy storage, ZIP load, power balance and voltage coupling models between sub-microgrids, and construct a microgrid cluster simulation environment that can be used for reinforcement learning agents to interact.

[0114] Step S3: Combine the voltage, photovoltaic output, energy storage state of charge and net load of each sub-microgrid into a state vector, and combine the energy storage charging and discharging power and photovoltaic inverter reactive power of each sub-microgrid into a continuous action vector.

[0115] Step S4: Construct a reward function based on the voltage reference value, the allowable upper and lower limits, and the penalty weight for exceeding the limit; wherein, the reward function jointly penalizes voltage deviation and voltage exceeding the limit.

[0116] Step S5: The agent is optimized using a proximal policy network including a policy network and a value network. It interacts with the simulation environment in a training scenario consisting of different irradiance and load sequences to obtain state, action, reward and next state samples, and updates the network parameters based on the pruning agent target.

[0117] Step S6: Deploy the trained policy network to the microgrid cluster controller; in each control cycle, generate energy storage power and photovoltaic inverter reactive power commands online according to the current state, and limit the power of the equipment, capacity, state of charge and the apparent power of the inverter within the constraints.

[0118] Step S7: Execute the control command, collect the updated system status, and enter the next control cycle; during the testing or actual operation phase, the policy network only performs forward inference, and the reward signal is used for performance evaluation or subsequent offline updates.

[0119] Example 1 I. Grid-connected microgrid cluster structure and data acquisition: like Figure 1 As shown, the grid-connected microgrid cluster is connected to the main grid via a common bus. The main grid provides a voltage reference and absorbs or supplements the cluster's net power difference. Each sub-microgrid includes at least a photovoltaic unit, an energy storage system, an inverter, and a load. The controller acquires the voltage at the point of common coupling (PCC), photovoltaic output, energy storage state of charge, and load information of each sub-microgrid through measurement devices, and inputs these measurements into the strategy network after per-unit, normalization, or filtering.

[0120] II. Photovoltaic, Energy Storage and Load Models: like Figure 2 As shown, in one specific embodiment, the photovoltaic output power P is calculated based on photovoltaic conversion efficiency, light-receiving area, solar irradiance, and ambient temperature:

[0121] In the formula, η is the photovoltaic conversion efficiency, S is the effective light-receiving area, G is the solar irradiance, and T is the ambient temperature. For reference temperature. In one embodiment, η = 0.18, S = 200 m². =25 ℃, standard peak irradiance =1000 W / m². By superimposing random attenuation and perturbation on the irradiance sequence, sunny, cloudy, and rainy day scenarios were constructed respectively.

[0122] The energy storage system uses the state of charge (SOC) as the primary state variable. Assuming the stored power Pbat is positive for charging and negative for discharging, the following formula can be used for discrete updates:

[0123] In the formula, η is the equivalent efficiency of energy storage charging and discharging, and Δt is the control period. This refers to the rated capacity of the energy storage. To avoid overcharging and over-discharging, the State of Charge (SOC) should meet preset upper and lower limits. This represents the energy storage state of charge at the start of the (t+1)th control cycle. This represents the state of charge of the energy storage at the start of the t-th control cycle, where t represents the control cycle index. This represents the active power of the energy storage system during the t-th control cycle. In practical engineering, the charging efficiency and discharging efficiency can also be modeled separately.

[0124] The load is modeled using a ZIP static load model, including constant impedance, constant current, and constant power components:

[0125]

[0126] in, , Cluster power balance can be expressed as the sum of grid power and photovoltaic output equaling the sum of load power and energy storage charging power, i.e. ;when A negative value corresponds to energy storage discharge. Net load is defined as... .

[0127] Indicates the active power of the load. This indicates the rated active load under rated voltage. This indicates the proportion of the constant impedance component in the active load. This indicates the proportion of the constant current component in the active load. This indicates the proportion of constant power component in the active load. This represents the actual voltage at the load connection point. Indicates the rated voltage. Indicates the reactive power of the load. This indicates the rated reactive load under rated voltage. This indicates the proportion of the constant impedance component in the reactive load. This indicates the proportion of the constant current component in the reactive load. This indicates the proportion of constant power component in reactive load. This represents the active power exchanged between the main power grid and the microgrid cluster. Indicates the active power output of photovoltaic power. This indicates the active power of energy storage charging and discharging.

[0128] III. Construction of Markov Decision Processes: The voltage control problem of a microgrid cluster is constructed as a Markov decision process. For a cluster containing N sub-microgrids, the state vector s includes the voltage, photovoltaic output, energy storage state of charge, and net load of each sub-microgrid:

[0129] in, This represents the common point voltage of the i-th sub-microgrid. This represents the photovoltaic active power output of the i-th sub-microgrid. This represents the state of charge of the i-th sub-microgrid energy storage system. Let i represent the net load of the i-th sub-microgrid, where i represents the sub-microgrid index. When N=3, the dimension of the state vector is 12.

[0130] The policy network outputs a continuous action vector a based on the state vector:

[0131] When N=3, the dimension of the action vector is 6. For energy storage charging and discharging power, The reactive power of the photovoltaic inverter. This represents the active power of energy storage charging and discharging in the i-th sub-microgrid. This represents the reactive power of the photovoltaic inverter in the i-th sub-microgrid. Before issuing an action, the controller performs physical constraint processing on the action, including energy storage rated power constraints, SOC boundary constraints, and photovoltaic inverter apparent power constraints. Preferably, the photovoltaic inverter reactive power satisfies: .

[0132] Among them, S inv This represents the rated apparent power capacity of the i-th sub-microgrid photovoltaic inverter.

[0133] IV. Reward Function: The reward function Including voltage deviation penalty and voltage over-limit penalty, the expression is:

[0134] In the formula, For reference voltage, and These are the upper and lower limits of the permissible voltage, respectively. This is a weighted penalty for voltage exceeding limits. In one embodiment, =1.0 pu, =1.05 pu, =0.95 pu, =10. By adjusting... It can alter the trade-off between average voltage deviation and strict over-limit suppression. This represents the common point voltage of the i-th sub-microgrid during the t-th control cycle.

[0135] V. Proximal Policy Optimization Agent and Training Method: like Figure 3 As shown, the proximal policy optimization agent adopts an Actor-Critic structure. The Actor network outputs the mean and standard deviation of the continuous action distribution based on the state, and the Critic network estimates the state value. Data is collected during training (…). The sequence is used to calculate the advantage function, and the magnitude of change between the old and new strategies is limited by pruning the agent objective. :

[0136] in, This represents the probability ratio between the new strategy and the old strategy. For the sake of advantage estimation, For the clipping factor, This represents the expected value at time t. This represents the network parameters of the current policy, and clip(·) represents the probability ratio constraint function.

[0137] The training feedback is only used for updating network parameters; during the policy deployment phase, the current state is directly input into the trained Actor network, which then outputs control actions without the need for online backpropagation.

[0138] VI. Online Collaborative Control Process: 1. At the start of the control cycle, collect voltage, photovoltaic output, energy storage SOC, and load data for each sub-microgrid, calculate the net load, and form the current state. .

[0139] 2. Input the current state into the trained Actor network to obtain continuous actions. The actions are then subjected to proportional mapping and boundary clipping to obtain the energy storage power command and the photovoltaic inverter reactive power command.

[0140] 3. Control commands are sent to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation.

[0141] 4. The environment or the actual microgrid cluster forms new voltage and power states according to control commands. In the next control cycle, As a new current state input policy network.

[0142] 5. During offline retraining or incremental training phases, calculate rewards based on system response. and will ( It is used to update the policy network and value network.

[0143] Example 2 This embodiment mainly focuses on the training and deployment of a three-sub-microgrid grid-connected cluster.

[0144] A grid-connected microgrid cluster comprising MG1, MG2, and MG3 was selected as the implementation target. Each sub-microgrid is equipped with photovoltaic units, energy storage systems, and ZIP loads, with parameters shown in Table 1. The cluster is connected to a common bus, and the main grid provides a voltage reference. During the simulation control cycle, the voltage, photovoltaic output, SOC, and net load of the three sub-microgrids are collected to form a 12-dimensional state vector; the Actor network outputs the power of three energy storage units and the reactive power of three photovoltaic inverters, forming a 6-dimensional continuous action vector.

[0145] Table 1. Main parameters of the microgrid cluster

[0146] Photovoltaic output is calculated using the following formula:

[0147] In the formula, η is the photovoltaic conversion efficiency, S is the effective light-receiving area, G is the solar irradiance, and T is the ambient temperature. This is a reference temperature.

[0148] like Figure 4 As shown, the sunny scene uses a smooth sunshine curve; the cloudy scene has random attenuation and rapid disturbances caused by cloud cover superimposed on the sunshine curve; the rainy scene has low overall irradiance with small random fluctuations. Training is performed separately for each of the three weather conditions or jointly in a mixed scenario.

[0149] During training, the agent reads the current state at each time step, samples actions according to the policy distribution, applies them to the simulation environment after device boundary pruning, and the environment performs power flow or simplified voltage coupling calculations, returning the next state and reward. The advantage function is calculated using generalized advantage estimation, and the Actor and Critic parameters are updated according to the pruning objective:

[0150] in, This represents the probability ratio between the new strategy and the old strategy. For the sake of advantage estimation, For the clipping factor, This represents the expected value at time t. This represents the network parameters of the current policy, and clip(·) represents the probability ratio constraint function.

[0151] When the preset number of rounds or reward convergence conditions are reached, the policy network is saved.

[0152] After training, the Actor network is deployed to the cluster controller. In each control cycle, the controller directly calculates the control action based on the current state and sends it to the devices. The test duration is 500 time steps, with voltage pass rate, average voltage deviation, and maximum voltage deviation used as evaluation metrics. Average voltage deviation... and maximum voltage deviation These represent the average absolute deviation and maximum absolute deviation of all sub-microgrids and all time steps relative to the reference voltage, respectively:

[0153]

[0154] Table 2 Examples of PPO training parameters

[0155] Table 3 Performance Comparison of PPO Control and Droop Control

[0156] Combining Tables 2 and 3, and Figure 5 , Figure 6 and Figure 7 The results show that the PPO controller can reduce the average voltage deviation under all three weather conditions, indicating that it can utilize global state coordination between energy storage and photovoltaic inverters to make the overall voltage closer to the reference value. The voltage compliance rate and maximum voltage deviation in the current embodiment can still be further improved by increasing the weight of the over-limit penalty, adding a safety filter, employing constraint reinforcement learning, or adding action projection at the output. All of the above improvements can be considered as optional embodiments of the present invention.

[0157] The method provided by this invention addresses the intermittent and fluctuating characteristics of distributed photovoltaic power output due to the influence of irradiance, temperature, and weather conditions. It constructs the microgrid cluster voltage control problem as a Markov decision process and uses a near-end strategy optimization algorithm to coordinate and control the charging and discharging power of each sub-microgrid energy storage system and the reactive power of the photovoltaic inverter, so as to reduce cluster voltage deviation and suppress voltage over-limit.

[0158] The present invention also provides a coordinated voltage control system for grid-connected microgrid clusters. The coordinated voltage control system for grid-connected microgrid clusters can be implemented by executing the process steps of the coordinated voltage control method for grid-connected microgrid clusters. That is, those skilled in the art can understand the coordinated voltage control method for grid-connected microgrid clusters as a preferred embodiment of the coordinated voltage control system for grid-connected microgrid clusters.

[0159] Specifically, a collaborative voltage control system for grid-connected microgrid clusters includes: Module M1: Collects the common coupling point voltage, photovoltaic active power output, energy storage state of charge and load data of each sub-microgrid, calculates the net load and forms the current state vector; Module M2: Inputs the current state vector into the trained policy network to obtain a continuous action vector composed of the energy storage charging and discharging power of each sub-microgrid and the reactive power of the photovoltaic inverter; performs proportional mapping and boundary clipping on the continuous action vector to obtain control commands; the control commands include energy storage power commands and photovoltaic inverter reactive power commands. Module M3: Sends the control commands to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation; Module M4: Executes the control command and collects the updated state of the grid-connected microgrid cluster, then enters the next control cycle; the updated state of the grid-connected microgrid cluster is used as the current state input to the strategy network for the next control cycle.

[0160] The load data includes active and reactive power; by establishing photovoltaic, energy storage, ZIP load, power balance, and voltage coupling models between sub-microgrids, a microgrid cluster simulation environment for reinforcement learning agents to interact is constructed.

[0161] The voltage control problem of a microgrid cluster is constructed as a Markov decision process; for a cluster containing N sub-microgrids, the state vector includes the voltage, photovoltaic output, energy storage state of charge, and net load of each sub-microgrid:

[0162] The policy network outputs a continuous action vector based on the state vector:

[0163] in, For energy storage charging and discharging power, The reactive power of the photovoltaic inverter; before issuing the action, the controller performs physical constraint processing on the action, including the rated power constraint of the energy storage, the SOC boundary constraint and the apparent power constraint of the photovoltaic inverter.

[0164] The reactive power of a photovoltaic inverter satisfies the following formula: .

[0165] A reward function is constructed based on the voltage reference value, allowable upper and lower limits, and over-limit penalty weights; the reward function includes voltage deviation penalty and voltage over-limit penalty, and its expression is:

[0166] In the formula, For reference voltage, and For the upper and lower limits of allowable voltage, The voltage over-limit penalty weight is denoted by N, which represents the number of sub-microgrid clusters.

[0167] An agent is optimized using a proximal policy network that includes a policy network and a value network. It interacts with the simulation environment in a training scenario consisting of different irradiance and load sequences to obtain samples of state, action, reward, and next state, and updates the network parameters based on the pruning agent objective.

[0168] The proximal policy optimization agent adopts an Actor-Critic structure; the Actor network outputs the mean and standard deviation of the continuous action distribution based on the state, and the Critic network estimates the state value.

[0169] Data collected during training ( The sequence is used to calculate the advantage function, and the magnitude of change between the old and new strategies is limited by pruning the agent objective:

[0170] in, This represents the probability ratio between the new strategy and the old strategy. For the sake of advantage estimation, This is the clipping factor.

[0171] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0172] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A collaborative voltage control method for grid-connected microgrid clusters, characterized in that, include: Step S1: Collect the common coupling point voltage, photovoltaic active power output, energy storage state of charge and load data of each sub-microgrid, calculate the net load and form the current state vector; Step S2: Input the current state vector into the trained policy network to obtain a continuous action vector composed of the energy storage charging and discharging power of each sub-microgrid and the reactive power of the photovoltaic inverter; perform proportional mapping and boundary clipping on the continuous action vector to obtain control commands; the control commands include energy storage power commands and photovoltaic inverter reactive power commands. Step S3: Send the control commands to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation; Step S4: Execute the control command and collect the updated state of the grid-connected microgrid cluster, and enter the next control cycle; use the updated state of the grid-connected microgrid cluster as the current state input to the strategy network for the next control cycle.

2. The cooperative voltage control method for grid-connected microgrid clusters according to claim 1, characterized in that, The load data includes active and reactive power; by establishing photovoltaic, energy storage, ZIP load, power balance, and voltage coupling models between sub-microgrids, a microgrid cluster simulation environment for reinforcement learning agents to interact is constructed.

3. The cooperative voltage control method for grid-connected microgrid clusters according to claim 1, characterized in that, The voltage control problem of a microgrid cluster is constructed as a Markov decision process; for a cluster containing N sub-microgrids, the state vector s includes the voltage, photovoltaic output, energy storage state of charge, and net load of each sub-microgrid: in, This represents the common point voltage of the i-th sub-microgrid. This represents the photovoltaic active power output of the i-th sub-microgrid. This represents the state of charge of the i-th sub-microgrid energy storage system. This represents the net load of the i-th sub-microgrid, where i represents the sub-microgrid index; The policy network outputs a continuous action vector a based on the state vector: in, For energy storage charging and discharging power, The reactive power of the photovoltaic inverter. This represents the active power of energy storage charging and discharging in the i-th sub-microgrid. This represents the reactive power of the photovoltaic inverter in the i-th sub-microgrid; the controller performs physical constraint processing on the action before issuing the action, including the rated power constraint of energy storage, the SOC boundary constraint and the apparent power constraint of the photovoltaic inverter.

4. The cooperative voltage control method for grid-connected microgrid clusters according to claim 1, characterized in that, The reactive power of a photovoltaic inverter satisfies the following formula: ； Among them, S inv This represents the rated apparent power capacity of the photovoltaic inverter in the i-th sub-microgrid. The reactive power of the photovoltaic inverter. This indicates the active power output of the photovoltaic system.

5. The cooperative voltage control method for grid-connected microgrid clusters according to claim 1, characterized in that, A reward function is constructed based on the voltage reference value, allowable upper and lower limits, and the weight of the penalty for exceeding the limit; the reward function Including voltage deviation penalty and voltage over-limit penalty, the expression is: In the formula, For reference voltage, and To allow for upper and lower voltage limits, The voltage over-limit penalty weight is used, and N represents the number of sub-microgrid clusters. This represents the common point voltage of the i-th sub-microgrid during the t-th control cycle.

6. The cooperative voltage control method for grid-connected microgrid clusters according to claim 2, characterized in that, An agent is optimized using a proximal policy network that includes a policy network and a value network. It interacts with the simulation environment in a training scenario consisting of different irradiance and load sequences to obtain samples of state, action, reward, and next state, and updates the network parameters based on the pruning agent objective.

7. The cooperative voltage control method for grid-connected microgrid clusters according to claim 6, characterized in that, The proximal policy optimization agent adopts an Actor-Critic structure; the Actor network outputs the mean and standard deviation of the continuous action distribution based on the state, and the Critic network estimates the state value.

8. The cooperative voltage control method for grid-connected microgrid clusters according to claim 7, characterized in that, Data collected during training ( The sequence is used to calculate the advantage function, and the magnitude of change between the old and new strategies is limited by pruning the agent objective. : in, This represents the probability ratio between the new strategy and the old strategy. For the sake of advantage estimation, For the clipping factor, This represents the expected value at time t. This represents the network parameters of the current policy, and clip(·) represents the probability ratio constraint function. Indicates the current state. Indicates continuous actions, Represents the reward function, This indicates the new voltage and power state.

9. A collaborative voltage control system for grid-connected microgrid clusters, characterized in that, include: Module M1: Collects the common coupling point voltage, photovoltaic active power output, energy storage state of charge and load data of each sub-microgrid, calculates the net load and forms the current state vector; Module M2: Inputs the current state vector into the trained policy network to obtain a continuous action vector composed of the energy storage charging and discharging power of each sub-microgrid and the reactive power of the photovoltaic inverter; performs proportional mapping and boundary clipping on the continuous action vector to obtain control commands; the control commands include energy storage power commands and photovoltaic inverter reactive power commands. Module M3: Sends the control commands to the energy storage converters and photovoltaic inverters of each sub-microgrid to perform active and reactive power coordinated regulation; Module M4: Executes the control command and collects the updated state of the grid-connected microgrid cluster, then enters the next control cycle; the updated state of the grid-connected microgrid cluster is used as the current state input to the strategy network for the next control cycle.

10. The collaborative voltage control system for grid-connected microgrid clusters according to claim 9, characterized in that, The load data includes active and reactive power; by establishing photovoltaic, energy storage, ZIP load, power balance, and voltage coupling models between sub-microgrids, a microgrid cluster simulation environment for reinforcement learning agents to interact is constructed.

Images