Voltage regulation and peak demand control method based on deep learning

Abstract

The invention discloses a voltage regulation and peak demand control method based on deep learning, and relates to the technical field of power distribution networks, and the method comprises the steps: S1, building a power distribution system model, a photovoltaic inverter model and an energy storage model, and expressing the coordination control of a photovoltaic inverter and energy storage as an optimal power flow problem; s2, establishing a deep reinforcement learning framework; and S3, solving a multi-objective optimization problem in voltage regulation and peak load management by using a soft behavior-criticality algorithm, and designing two different reward functions for daytime and nighttime. According to the voltage regulation and peak demand control method based on deep learning, by coordinating control strategies of a photovoltaic inverter and a battery energy storage system, the problems of voltage overrun and peak load caused by photovoltaic power generation fluctuation and load change in a power distribution network are effectively solved, and the algorithm is improved through a two-stage deep reinforcement learning control framework. The dynamic coordination control of the photovoltaic inverter and the battery energy storage system is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of power distribution networks, and in particular to a voltage regulation and peak demand control method based on deep learning. Background Art

[0002] As the penetration rate of distributed photovoltaic (PV) power generation systems in distribution networks continues to increase, the voltage quality and load management of the power grid face severe challenges. In the distribution network, the volatility and intermittency of PV power generation may lead to voltage over-limit problems, especially during periods of strong sunshine and peak PV power generation, which may cause voltage rise or even overvoltage. On the other hand, during the evening peak period, as the load increases, the voltage may drop, affecting the stable operation of the system.

[0003] To solve these problems, existing technologies have proposed a variety of methods to coordinate the control of photovoltaic and battery energy storage systems (BESS) to achieve voltage regulation and peak shaving. However, existing methods often rely on complex rule control and precise system models, and cannot efficiently deal with the complex interactions between photovoltaic power generation and battery energy storage systems. Therefore, how to optimize voltage regulation and load management through intelligent control without relying on precise models has become the focus of current research.

[0004] Deep reinforcement learning (DRL), as a control method that does not require precise models, has been widely used in the optimization control of power systems. In particular, the soft actor-critic (SAC) algorithm based on deep reinforcement learning can continuously optimize the control strategy through trial and error learning, and has good convergence and performance. However, the performance of the single-agent DRL method is limited in multi-objective and complex scenarios, especially when there are different control objectives during the day and at night, the effect may not be ideal. Therefore, the control method based on two-stage deep reinforcement learning has become a new research direction.

[0005] The present invention studies and improves the existing technology and its shortcomings, and provides a voltage regulation and peak demand control method based on deep learning. Summary of the invention

[0006] The purpose of the present invention is to provide a voltage regulation and peak demand control method based on deep learning to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solution: a voltage regulation and peak demand control method based on deep learning, comprising the following steps:

[0008] S1. Establish distribution system model, photovoltaic inverter model, and energy storage model. The coordinated control of photovoltaic inverter and energy storage can be expressed as an optimal power flow problem. The goal is to achieve voltage regulation and peak load management while meeting voltage and power constraints by adjusting the power set points of photovoltaic and energy storage:

[0009] The power demand at the head end of the feeder is divided into two periods. The power demand in the peak load period T∈(18,24) is The power demand during the normal load period T∈(1,17) is T is time, 17 means 17:00;

[0010] The objective function is:

[0011]

[0012] Where: f i t (P i t ) is the utility function of the active power set point of the PV inverter, N vvn is the number of nodes with voltage exceeding the limit, is the power demand during peak load;

[0013] S2, formulate the coordinated control of active and reactive power set points of PV inverters and the actions of energy storage to regulate voltage and reduce peak load demand as a Markov decision process (MDP) and establish a deep reinforcement learning framework;

[0014] S3. The soft actor-critic algorithm is used to solve the multi-objective optimization problems in voltage regulation and peak load management. A multi-agent architecture is adopted, and a two-stage agent is used to control the photovoltaic and battery energy storage system operations during the day and night respectively. Two different reward functions are designed for daytime and nighttime to better adapt to the control needs of different time periods.

[0015] Furthermore, in step S1, the process of establishing the power distribution system model is as follows:

[0016] The distribution system model describes the relationship between node voltage, current, power flow and load. Assuming that the system consists of n nodes, the node set is N, and the branch set is M, in this system, the constraints of voltage and power flow are expressed as follows:

[0017]

[0018] Among them, V i is the voltage at node i, I i is the current at node i, P i and Q iis the active and reactive power of node i, PL i and QL i are the active and reactive loads of the node, H is the set of nodes with PV inverters;

[0019] At the same time, in order to ensure that the node voltage is within a given range to prevent voltage over-limit problems, the following constraints are imposed:

[0020]

[0021] or

[0022] Among them, N vvn It is the set of voltage-exceeding-limit nodes.

[0023] Furthermore, in step S1, the process of establishing the photovoltaic inverter model is as follows:

[0024] For each PV inverter, the power set point The constraints are as follows:

[0025]

[0026] in, The rated apparent power S of the inverter i and solar irradiance λ t The reactive power range of the inverter is defined as 30% of the rated apparent power:

[0027]

[0028] in is the maximum active power of the inverter.

[0029] Furthermore, in step S1, the process of establishing the energy storage model is as follows:

[0030] The dynamic model of energy storage is modeled as the following discrete time equation:

[0031]

[0032] Where, σ is the state of charge (SOC) of the energy storage, η i,c , η i,d is the charging and discharging efficiency of energy storage, is the charging power of the energy storage, is the discharge power of energy storage;

[0033] Since the charging and discharging actions of the energy storage cannot occur on the same energy storage at each time step, the charging and discharging action constraints are defined as:

[0034]

[0035] The SOC of energy storage is subject to upper and lower limits:

[0036]

[0037] For each energy storage, the upper and lower power constraints for charging and discharging are:

[0038]

[0039] in, and are the lower and upper limits of energy storage SOC respectively, and They are the rated charging power and rated discharging power of the energy storage respectively.

[0040] Furthermore, in step S2, deep reinforcement learning continuously improves the control strategy by optimizing the MDP, and the main components and design details of the MDP are as follows:

[0041] 1) Environment: The environment refers to the entire power distribution system, including the power distribution network, photovoltaic inverters, battery energy storage system and loads. The environment provides the state data of the system at each time step, which is used to describe the real-time operating status of the system. At each time step, the environment receives actions from the agent, updates the system status, and feeds back the new status information and corresponding reward value to the agent.

[0042] 2) Agent: The agent is the decision maker in the MDP and is responsible for making decisions (actions) at each time step based on the currently observed system state to control the PV inverter and battery energy storage system;

[0043] 3) Observation is the information about the state of the environment obtained by the agent, which is used to help the agent make decisions at each time step. The observation information covers the current load situation of the system, photovoltaic power generation capacity, battery status, etc.

[0044] The specific contents are as follows: Current time T, maximum output power of photovoltaic power generation Maximum reactive power Load demand at each node The state of charge (SOC) of the energy storage is used to construct an observation vector based on the above information and is used to input into the agent’s decision model;

[0045] 4) Actions are decisions made by the agent based on the current observed state, which are used to control the PV inverter and battery energy storage system; specifically, they include the active and reactive power settings of each PV inverter, and the charge and discharge power of the battery energy storage;

[0046] Among them, the action of the photovoltaic inverter is defined as: (αPV,P (i,t),α PV,Q (i,t)), where α PV,P (i,t)∈(0,1), used to control the active power output ratio of the photovoltaic inverter; α PV,Q (i,t)∈(-1,1), used to control the reactive power output ratio of the photovoltaic inverter;

[0047] Set the PV power to:

[0048]

[0049] The action of the energy storage system is defined as:

[0050] when When , it indicates charging, the calculation formula is:

[0051] when When , it indicates discharge, and the calculation formula is:

[0052] 5) The design of the reward function is the core of deep reinforcement learning, which is used to guide the agent to learn the optimal control strategy. The present invention designs two reward functions for different control objectives during the day and at night.

[0053] Furthermore, the environmental input includes time-varying load curves, photovoltaic power generation curves, charging and discharging power of battery energy storage, etc., and the output is the voltage value of the node and the power demand obtained by the distribution network from the main feeder; the open source power system simulation tool OpenDSS is used to simulate the distribution network to obtain changes in voltage and load demand.

[0054] Furthermore, the agent outputs the active and reactive power set points of each PV inverter and the charging and discharging strategy of each battery energy storage system. The agent learns the control strategy through interaction with the environment to maximize the cumulative reward, i.e., the voltage regulation and load reduction effects. According to the current system state, the agent outputs an action set to adjust the output power of the PV inverter and the charging and discharging operation of the battery energy storage system.

[0055] Furthermore, in step S3, it is proposed to use a soft actor-critic algorithm to solve the multi-objective friendly problem in voltage regulation and peak load management, as follows:

[0056] The soft actor-critic algorithm optimizes the control policy by maximizing the weighted sum of reward and policy entropy, and its objective function is:

[0057]

[0058] Among them, J(π) is the optimization target of the policy network; R(st ,a t ) is the current state s t and action a t The reward obtained; H(π(·s t )) is the policy entropy, which is used to encourage exploration behavior; α is the temperature parameter, which is used to balance the weight between reward and entropy; ω is the discount factor, which controls the importance of future rewards.

[0059] Furthermore, in step S3, the soft actor-critic algorithm relies on two important modules: an actor network and a critic network, which are used to generate actions and evaluate actions respectively;

[0060] The actor network, namely the policy network, outputs a probability distribution indicating the possibility of taking each action in a given state; it includes the following two parts:

[0061] 1) Action selection: a t ~π φ (a t s t );

[0062] 2) Gradient update for policy optimization:

[0063]

[0064] The critic network, i.e., the value function network, estimates the value (Q value) of the current state-action pair, which is used to evaluate the quality of the actor's actions. It mainly includes the following three parts:

[0065] 1) Q value update:

[0066]

[0067] 2) Target value update formula:

[0068] y t =r t +ω·(Q θ' (s t+1 ,a t+1 )-alogπ φ (a t+1 s t+1 ));

[0069] 3) Q function loss:

[0070]

[0071] Finally, using the stored interaction experience (s t ,a t ,r t ,s t+1) Optimize the actor and critic networks to improve the efficiency of sample utilization.

[0072] Furthermore, in step S3, a two-stage agent is used to control the photovoltaic and battery energy storage system operations during the day and night respectively, and its reward function is as follows:

[0073] Daytime reward function: Focus on voltage regulation and reduce active power cutoff, encourage battery charging to provide support during peak load periods;

[0074]

[0075] Among them, v i,violation is the penalty for the voltage exceeding the limit node, P ct is the photovoltaic power cutoff rate, ρ is the incentive coefficient to encourage energy storage batteries to charge during the day;

[0076] Nighttime reward function: focuses on shaving peak loads and filling valleys, encouraging batteries to discharge at night to reduce the pressure of peak loads;

[0077]

[0078] Among them, μ is the reward coefficient for battery discharge at night, which is used to balance the discharge amount during peak and off-peak periods.

[0079] The present invention provides a voltage regulation and peak demand control method based on deep learning, which has the following beneficial effects:

[0080] The present invention effectively solves the voltage over-limit and peak load problems caused by photovoltaic power generation fluctuations and load changes in the distribution network by coordinating the control strategies of the photovoltaic inverter and the battery energy storage system. The proposed algorithm realizes dynamic coordinated control of the photovoltaic inverter and the battery energy storage system through a two-stage deep reinforcement learning control framework. The algorithm optimizes voltage regulation and photovoltaic utilization during the day and reduces peak load at night, combined with an adaptive reward mechanism, so that the algorithm performs well in voltage stability, load management and energy utilization efficiency, and comprehensively improves the operating performance and reliability of the distribution system. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 A schematic diagram of a coordinated optimization model of a soft actor-critic algorithm combined with a multi-agent architecture for a voltage regulation and peak demand control method based on deep learning according to the present invention;

[0082] Figure 2 This is a table of specific implementation results of a voltage regulation and peak demand control method based on deep learning in the present invention. DETAILED DESCRIPTION

[0083] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0084] like Figure 1-Figure 2 As shown, a voltage regulation and peak demand control method based on deep learning includes the following steps:

[0085] S1. The distribution system model established describes the relationship between node voltage, current, power flow and load. Assume that the system consists of n nodes, the node set is N, and the branch set is M. In this system, the constraints of voltage and power flow are expressed as follows:

[0086]

[0087] Among them, V i is the voltage at node i, I i is the current at node i, P i and Q i is the active and reactive power of node i, PL i and QL i are the active and reactive loads of the node, and H is the set of nodes with PV inverters.

[0088] Since the node voltage must be kept within a given range to prevent voltage limit problems, the following constraints must be met:

[0089]

[0090] or

[0091] Among them, N vvn It is the set of voltage-exceeding-limit nodes.

[0092] Then the PV inverter model is established. For each PV inverter, the power set point The constraints are as follows:

[0093]

[0094] in, The rated apparent power S of the inverter i and solar irradiance λ t The reactive power range of the inverter is defined as 30% of the rated apparent power:

[0095]

[0096] in is the maximum active power of the inverter;

[0097] The dynamic model of energy storage can be modeled as the following discrete time equation:

[0098]

[0099] Where, σ is the state of charge (SOC) of the energy storage, η i,c , η i,d is the charging and discharging efficiency of energy storage, is the charging power of the energy storage, is the discharge power of energy storage;

[0100] Since the charging and discharging actions of the energy storage cannot occur on the same energy storage at each time step, the charging and discharging action constraints are defined as:

[0101]

[0102] The SOC of energy storage is subject to upper and lower limits:

[0103]

[0104] For each energy storage, the upper and lower power constraints for charging and discharging are:

[0105]

[0106] in, and are the lower and upper limits of energy storage SOC respectively, and They are the rated charging power and rated discharging power of the energy storage respectively.

[0107] Based on the above distribution system, photovoltaic inverter model and energy storage model, the coordinated control of photovoltaic inverter and energy storage is modeled as an optimal power flow problem. The goal is to achieve voltage regulation and peak load management while meeting voltage and power constraints by adjusting the power set points of photovoltaic and energy storage:

[0108] The power demand at the head end of the feeder is divided into two periods. The power demand in the peak load period T∈(18,24) is The power demand during the normal load period T∈(1,17) is T stands for time, and 17 means 17:00.

[0109] The objective function is:

[0110]

[0111] Where: f i t (P i t) is the utility function of the active power set point of the PV inverter, N vvn is the number of nodes with voltage exceeding the limit, is the power demand during peak load periods.

[0112] S2, the coordinated control of the active and reactive power set points of the PV inverter and the actions of the energy storage to regulate the voltage and reduce the peak load demand are expressed as a Markov decision process (MDP), and a deep reinforcement learning framework is established. Deep reinforcement learning continuously improves the control strategy through the optimization of the MDP. The following are the main components and design details of the MDP:

[0113] 1) Environment: The environment refers to the entire power distribution system, including the power distribution network, photovoltaic inverters, battery energy storage systems, and loads. The environment provides the state data of the system at each time step, which is used to describe the real-time operating status of the system. At each time step, the environment receives actions from the agent, updates the system state, and feeds back the new state information and corresponding reward value to the agent.

[0114] The inputs include time-varying load curves, photovoltaic power generation curves, battery energy storage charging and discharging power, etc. The outputs are the voltage values ​​of the nodes and the power demand obtained by the distribution network from the main feeder. The open source power system simulation tool OpenDSS is used to simulate the distribution network to obtain the changes in voltage and load demand.

[0115] 2) Agent: The agent is the decision maker in the MDP and is responsible for making decisions (actions) at each time step based on the currently observed system state to control the PV inverter and the battery energy storage system. Specifically, the agent outputs the active and reactive power set points for each PV inverter, as well as the charge and discharge strategy for each battery energy storage system. The agent learns the control strategy through interaction with the environment to maximize the cumulative reward, i.e., the voltage regulation and load reduction effects, and outputs an action set based on the current system state to adjust the output power of the PV inverter and the charge and discharge operations of the battery energy storage system.

[0116] 3) Observation is the information about the state of the environment obtained by the agent, which is used to help the agent make decisions at each time step. The observation information covers the current load of the system, photovoltaic power generation capacity, battery status, etc.

[0117] The specific contents are as follows: Current time T, maximum output power of photovoltaic power generation Maximum reactive power Load demand at each node The state of charge (SOC) of the energy storage is constructed based on the above information to construct an observation vector for input into the agent's decision model.

[0118] 4) Actions are decisions made by the agent based on the current observed state, which are used to control the PV inverter and battery energy storage system. Specifically, they include the active and reactive power settings of each PV inverter, and the charge and discharge power of the battery energy storage.

[0119] Among them, the action of the photovoltaic inverter is defined as: (α PV,P (i,t),α PV,Q (i,t)), where α PV,P (i,t)∈(0,1), used to control the active power output ratio of the photovoltaic inverter; α PV,Q (i,t)∈(-1,1), used to control the reactive power output ratio of the photovoltaic inverter;

[0120] Set the PV power to:

[0121]

[0122] The action of the energy storage system is defined as:

[0123] when When , it indicates charging, the calculation formula is:

[0124] when When , it indicates discharge, and the calculation formula is:

[0125] 5) The design of reward function is the core of deep reinforcement learning, which is used to guide the agent to learn the optimal control strategy. The present invention designs two reward functions for different control objectives during the day and at night.

[0126] S3. It is proposed to use the soft actor-critic algorithm to solve the multi-objective friendly problem in voltage regulation and peak load management, as follows:

[0127] The soft actor-critic algorithm optimizes the control policy by maximizing the weighted sum of reward and policy entropy, and its objective function is:

[0128]

[0129] Among them, J(π) is the optimization target of the policy network; R(s t ,a t ) is the current state s t and action a t The reward obtained; H(π(·s t )) is the policy entropy, which is used to encourage exploration behavior; α is the temperature parameter, which is used to balance the weight between reward and entropy; ω is the discount factor, which controls the importance of future rewards.

[0130] The soft actor-critic algorithm relies on two important modules: the actor network and the critic network, which are used to generate actions and evaluate actions respectively.

[0131] The actor network, also known as the policy network, outputs a probability distribution that indicates the likelihood of taking each action in a given state. It includes the following two parts:

[0132] 1) Action selection: a t ~π φ (a t s t );

[0133] 2) Gradient update for policy optimization:

[0134]

[0135] The critic network, also known as the value function network, estimates the value (Q value) of the current state-action pair and is used to evaluate the quality of the actor's actions. It mainly includes the following three parts:

[0136] 1) Q value update:

[0137]

[0138] 2) Target value update formula:

[0139] y t =r t +ω·(Q θ' (s t+1 ,a t+1 )-alogπ φ (a t+1 s t+1 ));

[0140] 3) Q function loss:

[0141]

[0142] Finally, using the stored interaction experience (s t ,a t ,r t ,s t+1 ) Optimize the actor and critic networks to improve the efficiency of sample utilization.

[0143] In this section, a two-stage agent is used to control the photovoltaic and battery energy storage system operations during the day and night respectively, and its reward function is as follows:

[0144] Daytime reward function: Focus on voltage regulation and reduce active power cutoff, encourage battery charging to provide support during peak load periods;

[0145]

[0146] Among them, v i,violation is the penalty for the voltage exceeding the limit node, P ct is the photovoltaic power cutoff rate, and ρ is the incentive coefficient to encourage the energy storage battery to charge during the day.

[0147] Nighttime reward function: focuses on shaving peak loads and filling valleys, encouraging batteries to discharge at night to reduce the pressure of peak loads;

[0148]

[0149] Among them, μ is the reward coefficient for battery discharge at night, which is used to balance the discharge amount during peak and off-peak periods.

[0150] In this embodiment, the designed soft actor-critic algorithm is combined with the coordinated optimization model of the multi-agent architecture as shown in Figure 1 As shown. During training, the agent continuously adjusts the PV active power set point in the neural network; however, it does not make sense to set the PV active power at night due to the absence of solar energy. The updates of weights and biases at night do not reflect any reward changes, which may mislead the updates of the soft actor-critic algorithm agent. Therefore, the agent seeks a balance between active power reduction, battery replacement, and voltage violations during the day. When night comes, the reward requires the agent to focus on solving the peak load demand reduction problem. The proposed solution achieves the generalization capability of 24-hour control, using a two-level agent to control different operating scenarios with different action dimensions during the day and night. R is used in the daytime stage. day Training is performed, while the night phase uses R night Conduct training.

[0151] The effects of the above embodiments are as follows: Figure 2 As shown in the table, the comparison results of different control schemes in terms of the number of voltage over-limit nodes, peak load, system voltage range and active power cut-off rate are shown. The proposed algorithm performs well in all indicators: it completely eliminates the voltage over-limit problem, reduces the peak load by about 30.2%, and keeps the maximum and minimum voltages within the ideal range. At the same time, the active power cut-off rate is only 0.6%, which is significantly better than other methods.

[0152] In summary, the effect of the present invention is as follows: the proposed algorithm realizes dynamic coordinated control of photovoltaic inverters and battery energy storage systems through a two-stage deep reinforcement learning control framework. The algorithm optimizes voltage regulation and photovoltaic utilization during the day and reduces peak load at night. Combined with an adaptive reward mechanism, the algorithm performs well in voltage stability, load management, and energy efficiency, and comprehensively improves the operating performance and reliability of the distribution system.

[0153] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

Claims

1. A voltage regulation and peak demand control method based on deep learning, characterized in that: The following steps are involved: S1. Establish distribution system model, photovoltaic inverter model and energy storage model, and express the coordinated control of photovoltaic inverter and energy storage as an optimal power flow problem; The power demand at the head end of the feeder is divided into two periods. The power demand in the peak load period T∈(18,24) is The power demand during the normal load period T∈(1,17) is T is time, 17 means 17:00; The objective function is: Where: f i t (P i t ) is the utility function of the active power set point of the PV inverter, N vvn is the number of nodes with voltage exceeding the limit, is the power demand during peak load; S2, formulate the coordinated control of active and reactive power set points of PV inverters and the actions of energy storage to regulate voltage and reduce peak load demand as a Markov decision process and establish a deep reinforcement learning framework; S3. The soft actor-critic algorithm is used to solve the multi-objective optimization problems in voltage regulation and peak load management, and a multi-agent architecture is adopted. A two-stage agent is used to control the photovoltaic and battery energy storage system operations during the day and night respectively, and two different reward functions are designed for daytime and nighttime.

2. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S1, the process of establishing the power distribution system model is as follows: Assume that the system consists of n nodes, the node set is N, and the branch set is M. In this system, the constraints of voltage and power flow are expressed as follows: Among them, V i is the voltage at node i, I i is the current at node i, P i and Q i is the active and reactive power of node i, PL i and QL i are the active and reactive loads of the node, H is the set of nodes with PV inverters; And make the following constraints: or Among them, N vvn It is the set of voltage-exceeding-limit nodes.

3. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S1, the process of establishing the photovoltaic inverter model is as follows: For each PV inverter, the power set point The constraints are as follows: in, The rated apparent power S of the inverter i and solar irradiance λ t The reactive power range of the inverter is defined as 30% of the rated apparent power: in is the maximum active power of the inverter.

4. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S1, the process of establishing the energy storage model is as follows: The dynamic model of energy storage is modeled as the following discrete time equation: Where, σ is the state of charge (SOC) of the energy storage, η i,c , η i,d is the charging and discharging efficiency of energy storage, is the charging power of the energy storage, is the discharge power of energy storage; And the charge and discharge action constraints are defined as: The SOC of energy storage is subject to upper and lower limits: For each energy storage, the upper and lower power constraints for charging and discharging are: in, and are the lower and upper limits of energy storage SOC respectively, and They are the rated charging power and rated discharging power of the energy storage respectively.

5. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S2, deep reinforcement learning continuously improves the control strategy by optimizing the Markov decision process, and the components and design details of the Markov decision process are as follows: 1) Environment: It provides the state data of the system at each time step, which is used to describe the real-time operation status of the system. At each time step, the environment receives the action from the agent, updates the system state, and feeds back the new state information and corresponding reward value to the agent. 2) Agent: responsible for making decisions (actions) at each time step based on the currently observed system state to control the PV inverter and battery energy storage system; 3) Observation is the information about the state of the environment obtained by the agent, which is used to help the agent make decisions at each time step. The observation information covers the current load situation of the system, photovoltaic power generation capacity, battery status, etc. The specific contents are as follows: Current time T, maximum output power of photovoltaic power generation Maximum reactive power Load demand at each node The state of charge (SOC) of the energy storage is used to construct an observation vector based on the above information and is used to input into the agent’s decision model; 4) Actions are decisions made by the agent based on the current observed state, which are used to control the PV inverter and battery energy storage system; specifically, they include the active and reactive power settings of each PV inverter, and the charge and discharge power of the battery energy storage; Among them, the action of the photovoltaic inverter is defined as: (α PV,P (i,t),α PV,Q (i,t)), where α PV,P (i,t)∈(0,1), used to control the active power output ratio of the photovoltaic inverter; α PV,Q (i,t)∈(-1,1), used to control the reactive power output ratio of the photovoltaic inverter; Set the PV power to: The action of the energy storage system is defined as: when When , it indicates charging, the calculation formula is: when When , it indicates discharge, and the calculation formula is: 5) The reward function is used to guide the agent to learn the optimal control strategy. Two reward functions are designed for different control objectives during the day and at night.

6. A voltage regulation and peak demand control method based on deep learning according to claim 5, characterized in that: The environmental inputs include time-varying load curves, photovoltaic power generation curves, battery energy storage charging and discharging power, etc. The outputs are the voltage values ​​of the nodes and the power demand obtained by the distribution network from the main feeder. The open source power system simulation tool OpenDSS is used to simulate the distribution network to obtain changes in voltage and load demand.

7. The voltage regulation and peak demand control method based on deep learning according to claim 6, characterized in that: The agent outputs the active and reactive power set points of each photovoltaic inverter, and the charging and discharging strategy of each battery energy storage system. The agent learns the control strategy through interaction with the environment to maximize the cumulative reward, namely the voltage regulation and load reduction effect. According to the current system state, the agent outputs an action set to adjust the output power of the photovoltaic inverter and the charging and discharging operation of the battery energy storage system.

8. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S3, it is proposed to use the soft actor-critic algorithm to solve the multi-objective friendly problem in voltage regulation and peak load management, as follows: The soft actor-critic algorithm optimizes the control policy by maximizing the weighted sum of reward and policy entropy, and its objective function is: Among them, J(π) is the optimization target of the policy network; R(s t ,a t ) is the current state s t and action a t The reward obtained; H(π(·s t )) is the policy entropy, which is used to encourage exploration behavior; α is the temperature parameter, which is used to balance the weight between reward and entropy; ω is the discount factor, which controls the importance of future rewards.

9. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S3, the soft actor-critic algorithm relies on two important modules: an actor network and a critic network, which are used to generate actions and evaluate actions respectively; The actor network: outputs a probability distribution representing the likelihood of taking each action in a given state; It includes the following two parts: 1) Action selection: a t ~π φ (a t s t ); 2) Gradient update for policy optimization: The critic network estimates the value (Q value) of the current state-action pair, which is used to evaluate the quality of the actor's actions; It includes the following three parts: 1) Q value update: 2) Target value update formula: y t =r t +ω·(Q θ' (s t+1 ,a t+1 )-alogπ φ (a t+1 s t+1 )); 3) Q function loss: Finally, using the stored interaction experience (s t ,a t ,r t ,s t+1 )Optimize the actor and critic network.

10. The voltage regulation and peak demand control method based on deep learning according to claim 1, characterized in that: In step S3, a two-stage agent is used to control the photovoltaic and battery energy storage system operations during the day and night respectively, and its reward function is as follows: Daytime reward function: Focus on voltage regulation and reduce active power cutoff, encourage battery charging to provide support during peak load periods; Among them, v i,violation is the penalty for the voltage exceeding the limit node, P ct is the photovoltaic power cutoff rate, ρ is the incentive coefficient to encourage energy storage batteries to charge during the day; Nighttime reward function: focuses on shaving peak loads and filling valleys, encouraging batteries to discharge at night to reduce the pressure of peak loads; Among them, μ is the reward coefficient for battery discharge at night, which is used to balance the discharge amount during peak and off-peak periods.

Images