Micro-grid multi-time scale voltage optimization method based on deep reinforcement learning

Abstract

The micro-grid multi-time scale voltage optimization method based on deep reinforcement learning comprises the steps of providing a multi-time scale collaborative optimization framework based on deep reinforcement learning for a voltage out-of-limit problem caused by distributed photovoltaic access in a micro-grid; an improved hierarchical DQN (HRL-Dueling DQN) algorithm is adopted to carry out long-time scale optimization control on discrete capacitor equipment, so that the voltage out-of-limit risk is reduced; and performing short-time scale fine adjustment on reactive output of the continuous photovoltaic inverter by using a depth deterministic strategy gradient (DDPG) algorithm to suppress voltage deviation. According to the micro-grid multi-time scale voltage optimization method based on deep reinforcement learning, the real-time performance and robustness of voltage optimization are improved, the voltage out-of-limit risk is effectively restrained, voltage fluctuation is finely adjusted, and reliable guarantee is provided for safe and stable operation of a micro-grid with new energy access.

Description

Technical Field

[0001] This invention relates to the field of microgrid voltage control technology, specifically to a multi-timescale voltage optimization method for microgrids based on deep reinforcement learning. Background Technology

[0002] In the power system sector, the stable operation of microgrids is crucial for ensuring power supply quality, and voltage optimization is a key aspect of microgrid operation and management. With the large-scale integration of distributed energy resources, especially distributed photovoltaics, into microgrids, voltage control faces many new challenges.

[0003] In existing research, scholars have proposed various optimization strategies to address the voltage and line loss problems caused by the integration of new energy sources. The paper "Reactive Power and Voltage Optimization Technology for Large-Scale New Energy Grid Connection" (Niu Haoming, Lu Yilan, Zhang Liqing, Wang Weizhou. Reactive Power and Voltage Optimization Technology for Large-Scale New Energy Grid Connection [J]. Electronic Design Engineering, 2025, 33(03): 115-119.) proposes a reactive power and voltage optimization method combining graph convolutional neural networks and particle swarm optimization algorithm. This method improves optimization efficiency by identifying key control nodes to compress the solution space, but it does not fully consider the response differences of different regulating devices on different time scales. The paper "Optimization Strategy Analysis of Line Loss and Voltage Coordinated Management for New Energy Integration" (Han Jing. Optimization Strategy Analysis of Line Loss and Voltage Coordinated Management for New Energy Integration [J]. Integrated Circuit Applications, 2024, 41(12): 116-117.) constructs a multi-objective model of line loss and voltage deviation and uses simulated annealing particle swarm optimization algorithm to achieve coordinated optimization. However, the coordinated control mechanism for continuously regulating devices such as photovoltaic inverters and discrete devices such as capacitor banks still needs further development. The paper “Optimization Operation Strategy of Microgrid Based on Photovoltaic Energy Storage” (Zhang Ji. Optimization Operation Strategy of Microgrid Based on Photovoltaic Energy Storage [J]. Energy Storage Science and Technology, 2024, 13(11): 4062-4064.) discusses the application of photovoltaic energy storage system in voltage regulation and proposes to suppress voltage fluctuations by regulating active / reactive power, but it has shortcomings in the fine regulation strategy under multiple time scales.

[0004] The following technical problems still exist in the existing technology. (1) In the multi-device coordinated voltage regulation of microgrids, the voltage over-limit problem caused by distributed photovoltaic access stems from the differences in regulation characteristics between devices: capacitors are discretely regulated with long regulation cycles; photovoltaic inverters are continuously regulated and can dynamically adjust reactive power output. The differences in time scale and regulation form between the two lead to the inability to respond to short-term fluctuations in a timely manner, ultimately causing the risk of voltage over-limit.

[0005] (2) When dealing with voltage optimization problems on a long time scale, traditional methods rely on numerical optimization of mathematical models, but face the problems of large computational load and easy getting trapped in local optima. The traditional DQN algorithm has two major shortcomings in this process: first, the high-dimensional discrete action space leads to the curse of dimensionality, resulting in slow training convergence; second, it does not separate long-term decision-making and real-time adjustment requirements, resulting in poor policy generalization.

[0006] (3) The traditional DQN algorithm relies on a discrete action space sampling mechanism to generate reactive power regulation commands for photovoltaic inverters, which leads to insufficient regulation accuracy. At the same time, the single-valued function estimation structure used by the DQN algorithm is difficult to capture the complex mapping relationship between grid state and continuous actions, resulting in low strategy convergence efficiency. Summary of the Invention

[0007] To address the aforementioned technical issues, this invention proposes a multi-timescale voltage optimization method for microgrids based on deep reinforcement learning. This method helps improve the real-time performance and robustness of voltage optimization, effectively suppresses the risk of voltage exceeding limits, and finely regulates voltage fluctuations, providing a reliable guarantee for the safe and stable operation of microgrids with new energy access.

[0008] The technical solution adopted in this invention is as follows: A multi-time-scale voltage optimization method for microgrids based on deep reinforcement learning includes: S1: To address the voltage limit violation problem caused by distributed photovoltaic access in microgrids, a multi-timescale collaborative optimization framework based on deep reinforcement learning is proposed. S2: An improved hierarchical DQN (HRL-Dueling DQN) algorithm is used to perform long-term time-scale optimization control on discrete capacitor devices to reduce the risk of voltage exceeding limits; S3: The Deep Deterministic Policy Gradient (DDPG) algorithm is used to perform short-time scale fine-tuning of the reactive power output of the continuous photovoltaic inverter to suppress voltage deviation.

[0009] In S1, the multi-timescale collaborative optimization framework based on deep reinforcement learning includes: S1.1: Constructing a hierarchical dual-agent cooperative control architecture: The first intelligent agent employs an improved hierarchical DQN (HRL-Dueling DQN) algorithm specifically designed to generate discrete control commands to regulate the binary switching operations of discrete capacitor devices. The second intelligent agent is based on the Deep Deterministic Policy Gradient (DDPG) algorithm and is dedicated to generating continuous power regulation commands to control the real-time reactive power output of the photovoltaic inverter. S1.2: Define the optimization objective for differentiated time scales: To address the long-term voltage limit violation problem, the optimization objective of the first agent is set as minimizing the voltage limit violation penalty value of all network nodes, with a decision cycle of 6 hours. To address the problem of short-time voltage dynamic fluctuations, the optimization objective of the second agent is set as minimizing the node normalized voltage deviation rate, with a decision cycle of 15 minutes. S1.3: Establish a shared microgrid environment model: The first and second intelligent agents, along with the system, undergo collaborative training through power flow calculations, sharing the power flow state of the microgrid; among them... The first intelligent agent observes the state space, which includes node power, load power, and capacitor power state. The second intelligent agent's observation state further incorporates the maximum output of the photovoltaic inverter and the voltage information of key nodes. S1.4: Design a distributed asynchronous decision-making mechanism: After training, the first and second agents operate independently. The first agent generates capacitor bank switching commands based on 6 hours of historical load data, while the second agent generates reactive power regulation commands based on 15 minutes of real-time photovoltaic output data. S1.5: Achieve hierarchical collaborative voltage optimization: Over long timescales, capacitor banks provide basic reactive power support through discrete switching, suppressing the risk of voltage exceeding limits; In the short timescale, photovoltaic inverters dynamically compensate for voltage fluctuations through continuous reactive power output, reducing node voltage deviation. Therefore, by coordinating the control of the first and second intelligent agents at different timescales, hierarchical and progressive voltage optimization is achieved, thus constructing a multi-timescale collaborative optimization framework.

[0010] In step S2, a hierarchical reinforcement learning framework is introduced to transform voltage optimization into a Markov model, which is then solved using an improved hierarchical DQN algorithm. Specifically, this includes the following: Hierarchical Reinforcement Learning (HRL) is introduced. In long-term voltage optimization, the improved DQN (Dueling DNQ) algorithm is used to effectively coordinate the switching operations of capacitor banks to reduce the risk of voltage exceedance. A hierarchical decision framework is adopted, decomposing the optimization task into two levels: Level 1 includes the following steps: A1: Define high-level strategies and optimization goals: ① Optimize target setting: based on the microgrid's , The state of nodes and power flow distribution of switched capacitors is considered. The goal is to minimize the voltage over-limit penalty value of all nodes in the network. The switching operation of capacitors is adjusted reasonably to ensure that the voltage of the power grid is within the safe range and to avoid excessive voltage over-limit.

[0011] ② Decision Cycle: The decision cycle of the high-level strategy is relatively long, set at 6 hours. Within each decision cycle, the high-level strategy formulates a global capacitor switching plan based on the power flow distribution of microgrid nodes and switched capacitors.

[0012] ③ Strategy Generation: High-level strategy generation control commands indicate the switching status of capacitors: turn capacitors on or off to optimize the voltage of all network nodes and reduce the risk of voltage exceeding limits; A2: Markov Decision Process (MDP): Voltage optimization is essentially a process of making action decisions based on the current state. The voltage optimization problem in high-level policies is transformed into a Markov decision process, which includes states, actions, transition functions, and rewards. The specific implementation is as follows: (1) Define the state space of a long-time microgrid: ; In the formula, For long-term microgrid state space; and They are nodes Active power and reactive power, and For nodes Active power and reactive power, and The active and reactive power of the switched capacitor are... This is a time-based indicator.

[0013] (2) Define the action space of a long-term agent: ; In the formula, For long-term intelligent agents, this represents their action space. and These are adjustments to the active and reactive power outputs of the capacitor, respectively.

[0014] (3) Define the long-term agent transfer function: ; In the formula, For long-term intelligent agents, the transfer function is used. Indicates the current state of the power grid, including , Power of nodes and switched capacitors; This refers to the switching action of the switched capacitor; , and To supply power to the next load and photovoltaic system; The power flow equations are solved using the Newton-Raphson method, and the node voltage and power distribution are updated.

[0015] (4) Define the reward function for a long-term agent: (4) In the formula, For the reward function of the first agent on a long timescale; This represents the total number of nodes in the microgrid. It is a node The actual voltage, of which and These are the minimum and maximum values ​​of the node voltage, respectively. This is the activation function for the lower voltage safety limit. This is the activation function for the voltage safety upper limit.

[0016] A3: Constructing a Dueling DQN network architecture: Dueling DQN is a reinforcement learning algorithm that improves upon the traditional DQN structure by changing the value function. (Value Function) and Advantage Function The advantage function is separated to improve the learning efficiency of the policy. Combining a Markov decision process (MDP), Dueling DQN optimizes the Q-value more stably and efficiently. In Dueling DQN, the network is divided into two parts: the value function and the advantage function, as detailed below: 1) Value function: Outputs the state space of the microgrid over a long time scale. value function , It is a parameter of the value function, which reflects an estimate of the long-term cumulative reward that the agent can obtain in the current state; 2) Advantage function: Output the switching action of each capacitor. Advantages relative to average action value , This is a parameter of the advantage function, representing the additional benefit of taking a capacitor switching action compared to the average capacitor switching action. This separation allows for a clearer distinction between good and bad states and the relative advantages of actions, thus enabling more effective learning of action value.

[0017] 3) Calculate the Q-value of the action: Based on the outputs of the value function and the advantage function, the Q-value of the action is calculated using the following formula: ; In the formula, For state The following measures The value of the action; This indicates a traversal of all capacitor switching actions; These are the parameters of the value function; These are the parameters of the dominance function; The value function for the state space of a long-term microgrid; For state Lower capacitor switch action Advantage function relative to average action value; The capacitor switch operates; It indicates the size of the action space.

[0018] This formula combines state value and action advantage to obtain an estimate of the Q-value for each action in the current state.

[0019] Compared to the traditional DQN method that directly estimates the Q-value, this approach can more accurately evaluate the value of actions, especially in high-dimensional state spaces, where it can better capture the complex relationship between states and actions.

[0020] A4: Training and Optimization Process During the training of Dueling DQN, an experience replay mechanism is employed. The agent interacts with the microgrid environment, recording the state obtained from each interaction. Capacitor switching action adopted Rewards received And the next state Stored in the experience replay pool, as follows: 1) Calculate the target value : ; In the formula, It is a discount factor. It is the maximum Q value that the first agent can obtain among all the capacitor switching actions it can take in the next moment.

[0021] 2) Calculate the loss function, using the mean squared error loss function: ; In the formula, The loss function is obtained by optimization. and This enables the Q-value calculation model to accurately predict the Q-value corresponding to each capacitor switching action; This is the expected value; 3) Select the largest Value actions: High-level strategies involve selecting a control action that maximizes future returns; that is, selecting the action that corresponds to the maximum. Capacitor switching operation of value.

[0022] 4) Generate control commands: After selecting the optimal action, the high-level strategy will generate control instructions to adjust the switching of the capacitors in the current decision cycle (6 hours) to coordinate the timing of the discrete capacitors and reduce the risk of voltage exceeding limits.

[0023] In S3, when dealing with voltage optimization on short timescales, a Deep Deterministic Policy Gradient (DDPG) algorithm is introduced. By constructing a continuous action space optimization framework based on the Actor-Critic architecture, the voltage deviation suppression problem is transformed into a deterministic policy search process. Similarly, the mathematical model of voltage optimization is transformed into a Markov model to avoid easily converging to local optima during the solution process. A hierarchical decision framework is used for solving the problem. The details of level two are as follows: B1: Define low-level strategies and optimization objectives: ① Optimization Target Setting: The goal of the low-level strategy is to adjust the state of capacitors and continuously regulating photovoltaic inverters in real time, guided by the macro-level goals set by the high-level strategy. The optimization objective of the low-level strategy is to minimize node voltage deviation and maintain real-time voltage stability.

[0024] ② Decision Cycle: The decision cycle for the low-level strategy is relatively short, set to once every 15 minutes. The low-level strategy adjusts the reactive power output of the photovoltaic inverter in the continuous adjustment equipment based on real-time data on photovoltaic output and load fluctuations.

[0025] ③ Strategy Generation: The low-level strategy continuously adjusts the reactive power output of the photovoltaic inverter based on the actual voltage conditions. The action mode is the reactive power regulation action of the photovoltaic inverter. Further optimize voltage stability to suppress voltage deviation.

[0026] B2: Markov Decision Process (MDP): Short-timescale voltage optimization and long-timescale optimization share the same microgrid environment, but the states, actions, and rewards observed by the second agent differ. This is because the short-timescale state design supports rapid and continuous adjustment of the photovoltaic inverter, requiring real-time monitoring of photovoltaic output and other states to constrain the action space, as detailed below: (1): Define the state space of a short-timescale agent: ; In the formula, For short-timescale intelligent agents, the state space is defined. and These are the active power output and reactive power output of the photovoltaic inverter, respectively. This represents the maximum active power output of the photovoltaic inverter.

[0027] (2): Define the action space of a short-timescale agent: ; In the formula, For short-timescale intelligent agent action space; and These are adjustments to the active and reactive power outputs of the photovoltaic inverter, respectively.

[0028] (3): Define the transfer function of the agent in a short time scale: ; In the formula, For short-timescale agent transfer functions; For reactive power regulation of photovoltaic inverters; , and To provide power for the next load and photovoltaic output.

[0029] (4): Define the reward function for a short-timescale agent: ; In the formula, For short-timescale agent reward functions; This represents the total number of nodes in the microgrid. It is a node The actual voltage; where and The minimum and maximum values ​​of the node voltage are represented by negative values ​​to indicate the "penalty" for the deviation, guiding the second agent to actively avoid high deviation states.

[0030] B3: Building the DDPG network architecture: Considering that the DQN algorithm cannot meet the requirements of rapid and continuous regulation of photovoltaic systems, while the DDPG algorithm directly outputs continuous action Q values ​​through an Actor-Critic architecture, where the Actor network (policy network) is responsible for generating reactive power regulation commands. The Critic network (value network) is responsible for evaluating the Q-value of the actions that drive the photovoltaic inverter. Through the synergy of both, strategy optimization is achieved, enabling high-precision, real-time reactive power regulation. A higher Q-value indicates that after the photovoltaic inverter increases reactive power output, the voltage deviation at future times will be smaller, the secondary agent will receive less negative reward (penalty), and the voltage stability will be better. Specifically: 1) Construct an Actor network (policy network): used to directly generate continuous reactive power regulation commands. It drives the photovoltaic inverter to operate and regulate reactive power.

[0031] Input variable: Current microgrid state space ; Output variable: The continuous action output by a single neuron, namely the reactive power regulation action of the photovoltaic inverter. ; 2) Build a Critic network (value network): used to evaluate the actions output by the policy network. The quality, calculate the Q value To guide Actor strategy optimization; Input variable: The current state space of the microgrid Reactive power regulation operation of photovoltaic inverters ; Output variable: Evaluate the value of these state actions, outputting the Q-value. ; 3) Define the main network and the target network: Main networks: Actor main network and Critic main network, which directly participate in real-time decision-making and value assessment, generate current actions and calculate Q-values.

[0032] Target networks: Actor target network and Critic target network provide stable target Q values, avoiding training oscillations caused by the main network directly optimizing real-time Q values.

[0033] 4) Batch training and parameter updates: a. Data collection and experience review: Actions generated by Actor network By performing actions Observation Rewards Next state , sample Store in experience replay pool D; b. Sampling batch data: Randomly draw batches of samples from the experience replay pool D. M represents the batch size, ensuring that the Actor-Critic network can effectively learn the reactive power regulation strategy of the photovoltaic inverter.

[0034] c. Update the Critic network: Calculate the target Q value: ; In the formula, For the Critic target network output; The reactive power regulation action generated for the next photovoltaic inverter in the Actor target network; and These are the target network parameters for Critic and Actor, respectively, and the main network is tracked through a soft update mechanism.

[0035] Calculate the loss function: ; In the formula, and Reward functions and discount factors for agents on short time scales; It is the maximum Q value obtained among all the reactive power regulation actions of the photovoltaic inverter that the second intelligent agent can take in the next moment.

[0036] d. Update the Actor network: Calculate the policy gradient: ; In the formula, The gradient of the objective function of the Actor network is used to update the parameters of the policy network so that the actions generated by the policy can maximize the long-term cumulative reward. and These are the policy network parameters and the value network parameters, namely the weights and biases of the policy network and the value network. These parameters determine how to determine the state. Generate Actions N represents the total number of microgrid nodes, indicating the number of nodes participating in voltage optimization. Q-value pairs of actions The gradient represents the gradient at the current... Adjust actions in the state Sensitivity to Q-values ​​guides the policy network in adjusting its actions to improve returns; It is the equivalent policy function of the policy network; Indicates policy network parameters In state Influences the magnitude of the action. The value of an action is assessed using a Critic (value network). The Actor network (policy network) updates the policy along the Q-value gradient direction to maximize the action reward.

[0037] 5) Introduce a target network soft update mechanism: periodically update the main network parameters Slowly synchronize to target network parameters To avoid sudden changes in Q value: ; ; In the formula, The target network parameters for the policy network; The target network parameters for the value network; These are the main network parameters for the policy network; These are the main network parameters of the value network; The learning rate of the policy network; The learning rate of the value network; For policy networks in The gradient of the objective function at time step; For value networks in The loss function at time step; Update the symbols for the parameters.

[0038] 6) Training Termination: The trained Actor network generates reactive power adjustment instructions every 15 minutes based on the real-time status. Training continues until the Q value converges to a stable range or the voltage deviation is less than 0.01 pu and no longer fluctuates significantly.

[0039] Through the above architecture design and training process, the DDPG algorithm achieves efficient modeling of continuous reactive power regulation of photovoltaic inverters, and significantly improves the real-time performance and accuracy of microgrid voltage deviation suppression in the short time scale.

[0040] This invention presents a multi-time-scale voltage optimization method for microgrids based on deep reinforcement learning, with the following technical advantages: 1) This invention constructs a multi-timescale collaborative optimization framework, combining hierarchical Dueling DQN and deep deterministic policy gradient algorithm to handle the differentiated control requirements of discrete and continuous equipment, significantly improving the real-time performance and robustness of voltage optimization, and providing technical support for the safe and stable operation of new energy microgrids.

[0041] 2) This invention quantifies the nonlinear relationship between microgrid voltage and equipment regulation through deep reinforcement learning algorithms, which significantly improves the real-time performance and robustness of voltage optimization, and provides a guarantee for the safe operation of microgrids with a high proportion of new energy access. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and examples; Figure 1 A flowchart for a multi-timescale collaborative optimization framework.

[0043] Figure 2 This is a diagram of a dual-agent control system architecture.

[0044] Figure 3 This is a diagram showing the voltage optimization at the end nodes of a microgrid. Detailed Implementation

[0045] This invention proposes a multi-timescale voltage optimization method for microgrids based on deep reinforcement learning. First, to address the voltage exceedance problem caused by distributed photovoltaic (PV) integration in microgrids, a multi-timescale collaborative optimization framework based on deep reinforcement learning is proposed. Second, to address the issue of long adjustment cycles and difficulty in matching the discrete operation characteristics of reactive power compensation equipment with rapid changes in PV output, an improved hierarchical Reinforcement Learning Dueling Deep Q-network (HRL-Dueling DQN) algorithm is employed. This algorithm coordinates the timing of discrete capacitor equipment operation through long-term optimization control strategies to reduce the risk of voltage exceedance. Finally, to address the difficulty of traditional methods adapting to the continuous operation space and rapid response requirements of PV inverters, this invention uses a deep deterministic policy gradient (DDPG) algorithm to perform fine-grained adjustment of reactive power output on short-term timescales, thereby suppressing voltage deviation.

[0046] Figure 1 A multi-timescale collaborative optimization framework is proposed. First, to address the voltage exceedance problem caused by distributed photovoltaic (PV) integration in microgrids, a multi-timescale collaborative optimization framework based on reinforcement learning is proposed. Second, an improved hierarchical DQN (HRL-Dueling DQN) algorithm is employed to coordinate the timing of capacitor operation by optimizing the control strategy over a long timescale, thereby reducing the risk of voltage exceedance. Finally, the DDPG algorithm is used to finely adjust the reactive power output of the PV inverter over a short timescale to suppress voltage deviation.

[0047] Figure 2 This is a diagram of a dual-agent control architecture. (See diagram for example.) Figure 2 As shown, firstly, a dual-agent cooperative control strategy is adopted to solve the problem of coordinated optimization of reactive power equipment across multiple time scales. Secondly, during training, an agent based on Dueling DQN generates discrete control commands every 6 hours to optimize the capacitor bank switching strategy. Finally, an agent based on DDPG generates continuous power commands every 15 minutes to finely adjust the photovoltaic inverter output; the two interact in a shared environment. During the execution phase, the agents adjust equipment power based on real-time data, collaboratively resolving voltage over-limit and deviation issues, and improving the voltage control performance of the microgrid.

[0048] Figure 3 This is a diagram showing the voltage optimization at the end nodes of a microgrid. Figure 3The line graph visually illustrates the voltage optimization effect of the microgrid's end nodes over 24 hours. The horizontal axis represents time (0-24 hours, 15-minute intervals), and the vertical axis represents the per-unit voltage (pu). The dashed line in the graph represents the voltage curve before optimization: a significant drop occurs during peak hours (10:00-23:00), with the lowest value dropping to 0.945 pu at 17:00 (circled), below the safe lower limit of 0.95 pu (dotted line). The solid line represents the voltage curve after optimization, showing that after multi-timescale collaborative control, the voltage remains stable within the range of 0.96-1.02 pu, with the voltage increasing to 0.993 pu during critical periods (triangled), completely eliminating the over-limit phenomenon. The rated voltage (1.0 pu) is marked with a black dashed line, highlighting the slight fluctuation of the optimized voltage around the rated value. The graph, with annotations of key points and threshold auxiliary lines, clearly verifies the effectiveness of the deep reinforcement learning algorithm in suppressing photovoltaic fluctuations and improving voltage stability, providing a visual basis for the real-time optimization of high-proportion renewable energy microgrids.

Claims

1. A multi-time-scale voltage optimization method for microgrids based on deep reinforcement learning, characterized in that... include: S1: A multi-temporal-scale collaborative optimization framework based on deep reinforcement learning is proposed; S2: An improved hierarchical DQN algorithm is used to perform long-term time-scale optimization control on discrete capacitor devices to reduce the risk of voltage exceeding limits; S3: Use a deep deterministic strategy gradient algorithm to perform short-time scale fine-tuning of the reactive power output of the photovoltaic inverter in order to suppress voltage deviation.

2. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 1, characterized in that: In S1, the multi-timescale collaborative optimization framework based on deep reinforcement learning includes: S1.1: Constructing a hierarchical dual-agent cooperative control architecture: The first intelligent agent employs an improved hierarchical DQN algorithm to generate discrete control commands to regulate the binary switching operations of discrete capacitor devices. The second intelligent agent is based on a deep deterministic policy gradient algorithm to generate continuous power regulation commands to control the real-time reactive power output of the photovoltaic inverter. S1.2: Define the optimization objective for differentiated time scales: To address the long-term voltage limit violation problem, the optimization objective of the first agent is set as minimizing the voltage limit violation penalty value of all network nodes; To address the problem of short-time voltage dynamic fluctuations, the optimization objective of the second intelligent agent is set as minimizing the node normalized voltage deviation rate. S1.3: Establish a shared microgrid environment model: The first and second intelligent agents, along with the system, undergo collaborative training through power flow calculations, sharing the power flow state of the microgrid; among them... The first intelligent agent observes the state space, which includes node power, load power, and capacitor power state. The second intelligent agent's observation state further incorporates the maximum output of the photovoltaic inverter and the voltage information of key nodes. S1.4: Design a distributed asynchronous decision-making mechanism: After training, the first and second agents operate independently; the first agent generates capacitor bank switching instructions based on historical 6-hour load data, and the second agent generates reactive power adjustment instructions based on real-time 15-minute photovoltaic output data. S1.5: Achieve hierarchical collaborative voltage optimization: Over long timescales, capacitor banks provide basic reactive power support through discrete switching, suppressing the risk of voltage exceeding limits; In the short timescale, photovoltaic inverters reduce node voltage deviation by dynamically compensating for voltage fluctuations through continuous reactive power output; By coordinating control of the first and second agents at different time scales, hierarchical voltage optimization is achieved, and a multi-time-scale collaborative optimization framework is constructed.

3. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 2, characterized in that: In step S2, a hierarchical reinforcement learning framework is introduced to transform voltage optimization into a Markov model, which is then solved using an improved hierarchical DQN algorithm. Specifically, this includes the following steps: A1: Define high-level strategies and optimization goals; A2: Markov Decision Process: Voltage optimization is essentially a process of making action decisions based on the current state. It transforms the voltage optimization problem in high-level policies into a Markov decision process, which includes state, action, transition function and reward. A3: Constructing a Dueling DQN network architecture: Dueling DQN combines Markov decision processes, dividing the network into two parts: a value function and an advantage function. A4: Training and optimization process: During the training of Dueling DQN, an experience replay mechanism is employed; the agent interacts with the microgrid environment, recording the state obtained from each interaction. Capacitor switching action adopted Rewards received And the next state Stored in the experience replay pool.

4. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 3, characterized in that: In A1, high-level strategies and optimization objectives are defined, including: ① Optimize target setting: based on the microgrid's , The power flow distribution of nodes and switched capacitors is considered, and the goal is to minimize the penalty value for voltage overruns at all nodes in the network. The switching operation of capacitors is then adjusted accordingly. ② Decision cycle: The decision cycle of the high-level strategy is relatively long, set at 6 hours; within each decision cycle, the high-level strategy formulates a global capacitor switching plan based on the power flow distribution of microgrid nodes and switched capacitors. ③ Strategy Generation: High-level strategy generation control commands indicate the switching status of capacitors: turn capacitors on or off to optimize the voltage of all network nodes and reduce the risk of voltage exceeding limits.

5. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 3, characterized in that: The specific implementation of A2 is as follows: (1) Define the state space of a long-time microgrid: ； In the formula, For long-term microgrid state space; and They are nodes Active power and reactive power, and For nodes Active power and reactive power, and The active and reactive power of the switched capacitor are... For time indicators; (2) Define the action space of a long-term intelligent agent ； In the formula, For long-term intelligent agents, this represents their action space. and These are adjustments to the active and reactive power outputs of the capacitor, respectively. (3) Define the long-term agent transfer function: ； In the formula, For long-term intelligent agents, the transfer function is used. Indicates the current state of the power grid, including , Power of nodes and switched capacitors; This refers to the switching action of the switched capacitor; , and To supply power to the next load and photovoltaic system; The power flow equations are solved using the Newton-Raphson method, and the node voltages and power distributions are updated. (4) Define the reward function for a long-term agent: ；（4） In the formula, For the reward function of the first agent on a long timescale; This represents the total number of nodes in the microgrid. It is a node The actual voltage, of which and These are the minimum and maximum values ​​of the node voltage, respectively. This is the activation function for the lower voltage safety limit. This is the activation function for the voltage safety upper limit.

6. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 3, characterized in that: In A3, within Dueling DQN, the network is divided into two parts: a value function and a dominance function, as detailed below: 1) Value function: Outputs the state space of the microgrid over a long time scale. value function , It is a parameter of the value function, which reflects an estimate of the long-term cumulative reward that the agent can obtain in the current state; 2) Advantage function: Output the switching action of each capacitor. Advantages relative to average action value , It is a parameter of the advantage function, representing the additional benefit of taking capacitor switching action compared to the average capacitor switching action; 3) Calculate the Q-value of the action: Based on the outputs of the value function and the advantage function, the Q-value of the action is calculated using the following formula: ； In the formula, For state The following measures The value of the action; This indicates a traversal of all capacitor switching actions; These are the parameters of the value function; These are the parameters of the dominance function; The value function for the state space of a long-term microgrid; For state Lower capacitor switch action Advantage function relative to average action value; The capacitor switch operates; Indicates the size of the action space; This formula combines state value and action advantage to obtain an estimate of the Q-value for each action in the current state.

7. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 3, characterized in that: In A4, the training optimization process is as follows: 1) Calculate the target value : ； In the formula, It is a discount factor. It is the maximum Q value obtained among all the capacitor switching actions that the first agent can take in the next moment; 2) Calculate the loss function, using the mean squared error loss function: ； In the formula, The loss function is obtained by optimization. and This enables the Q-value calculation model to accurately predict the Q-value corresponding to each capacitor switching action; This is the expected value; 3) Select the largest Value actions: High-level strategies involve selecting a control action that maximizes future returns; that is, selecting the action that corresponds to the maximum. Capacitor switching operation of value; 4) Generate control commands: After selecting the optimal action, the higher-level strategy will generate control instructions to adjust the switching of the capacitor in the current decision cycle, coordinate the timing of the discrete capacitor's actions, and reduce the risk of voltage exceeding limits.

8. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 3, characterized in that: In S3, when dealing with voltage optimization on a short time scale, a deep deterministic policy gradient algorithm is introduced. By constructing a continuous action space optimization framework based on the Actor-Critic architecture, the voltage deviation suppression problem is transformed into a deterministic policy search process. Similarly, the mathematical model of voltage optimization is transformed into a Markov model.

9. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 8, characterized in that: S3 includes: B1: Define low-level strategies and optimization objectives: ① Optimization target setting: The goal of the low-level strategy is to adjust the state of the capacitors and the photovoltaic inverter of the continuous regulation equipment in real time under the guidance of the macro-level goals set by the high-level strategy; the optimization goal of the low-level strategy is to minimize the node voltage deviation and maintain the real-time stability of the voltage. ② Decision cycle: The decision cycle of the low-level strategy is relatively short, set to once every 15 minutes; the low-level strategy adjusts the reactive power output of the photovoltaic inverter of the continuous adjustment equipment based on real-time data photovoltaic output and load fluctuations. ③ Strategy Generation: The low-level strategy continuously adjusts the reactive power output of the photovoltaic inverter based on the actual voltage conditions. The action mode is the reactive power regulation action of the photovoltaic inverter. Further optimize voltage stability to suppress voltage deviation; B2: Markov Decision Process: Short-timescale voltage optimization and long-timescale optimization share the same microgrid environment, but the states, actions, and rewards observed by the second agent differ. This is because the short-timescale state design supports rapid and continuous adjustment of the photovoltaic inverter, requiring real-time monitoring of photovoltaic output and other states to constrain the action space, as detailed below: (1): Define the state space of a short-timescale agent: ； In the formula, For short-timescale intelligent agents, the state space is defined. and These are the active power output and reactive power output of the photovoltaic inverter, respectively. This represents the maximum active power output of the photovoltaic inverter. (2): Define the action space of a short-timescale agent: ； In the formula, For short-timescale intelligent agent action space; and These are adjustments to the active and reactive power outputs of the photovoltaic inverter, respectively. (3): Define the transfer function of the agent in a short time scale: ； In the formula, For short-timescale agent transfer functions; For reactive power regulation of photovoltaic inverters; , and To supply power to the next load and photovoltaic system; (4): Define the reward function for a short-timescale agent: ； In the formula, For short-timescale agent reward functions; This represents the total number of nodes in the microgrid. It is a node The actual voltage; where and The minimum and maximum values ​​of the node voltage are used, with negative values ​​representing the "penalty" for the deviation, guiding the second agent to actively avoid high deviation states; B3: Building the DDPG network architecture: The DDPG algorithm directly outputs the Q-values ​​of continuous actions through an Actor-Critic architecture, where the Actor network is responsible for generating reactive power regulation commands. The Critic network drives the photovoltaic inverter to take action, and evaluates the Q value of these actions. Through the synergy of the two, strategy optimization can be achieved, enabling high-precision and real-time reactive power regulation. The higher the Q value, the smaller the voltage deviation at future times after the photovoltaic inverter increases reactive output, the less penalty the second agent receives, and the better the voltage stability.

10. The microgrid multi-timescale voltage optimization method based on deep reinforcement learning according to claim 9, characterized in that: In B3, the DDPG network architecture is constructed as follows: 1) Construct an Actor network: used to directly generate continuous reactive power regulation commands. Drive the photovoltaic inverter to operate and regulate reactive power; Input variable: Current microgrid state space ; Output variable: The continuous action output by a single neuron, namely the reactive power regulation action of the photovoltaic inverter. ; 2) Build a Critic network: used to evaluate the actions output by the policy network. The quality, calculate the Q value To guide Actor strategy optimization; Input variable: The current state space of the microgrid Reactive power regulation operation of photovoltaic inverters ; Output variable: Evaluate the value of these state actions, outputting the Q-value. ; 3) Define the main network and the target network: Main networks: Actor main network and Critic main network, which directly participate in real-time decision-making and value assessment, generate current actions and calculate Q-values; Target networks: Actor target network and Critic target network, providing stable target Q values ​​to avoid training oscillations caused by the main network directly optimizing real-time Q values; 4) Batch training and parameter updates: a. Data collection and experience review: Actions generated by Actor network By performing actions Observation Rewards Next state , sample Store in experience replay pool D; b. Sampling batch data: Randomly draw batches of samples from the experience replay pool D. M represents the batch size, ensuring that the Actor-Critic network can effectively learn the reactive power regulation strategy of the photovoltaic inverter; c. Update the Critic network: Calculate the target Q value: ； In the formula, For the Critic target network output; The reactive power regulation action generated for the next photovoltaic inverter in the Actor target network; and These are the target network parameters for Critic and Actor, respectively, and the main network is tracked through a soft update mechanism. Calculate the loss function: ； In the formula, and Reward functions and discount factors for agents on short time scales; It is the maximum Q value obtained from all the reactive power regulation actions of the photovoltaic inverters taken by the second intelligent agent in the next moment; d. Update the Actor network: Calculate the policy gradient: ； In the formula, The gradient of the objective function of the Actor network is used to update the parameters of the policy network so that the actions generated by the policy can maximize the long-term cumulative reward. and These are the policy network parameters and the value network parameters, namely the weights and biases of the policy network and the value network. These parameters determine how to determine the state. Generate Actions N represents the total number of microgrid nodes, indicating the number of nodes participating in voltage optimization. Q-value pairs of actions The gradient represents the gradient at the current... Adjust actions in the state Sensitivity to Q-values ​​guides the policy network in adjusting its actions to improve returns; It is the equivalent policy function of the policy network; Indicates policy network parameters In state Influences the magnitude of the action; evaluates the value of the action through a Critic (value network). The Actor network updates the policy along the Q-value gradient direction to maximize the action reward; 5) Introduce a target network soft update mechanism: periodically update the main network parameters Slowly synchronize to target network parameters To avoid sudden changes in Q value: ； ； In the formula, The target network parameters for the policy network; The target network parameters for the value network; These are the main network parameters for the policy network; These are the main network parameters of the value network; The learning rate of the policy network; The learning rate of the value network; For policy networks in The gradient of the objective function at time step; For value networks in The loss function at time step; Update the symbols for the parameters; 6) Training Termination: The trained Actor network generates reactive power adjustment instructions every 15 minutes based on the real-time status. Training continues until the Q value converges to a stable range or the voltage deviation is less than 0.01 pu and no longer fluctuates significantly.

Images