A power distribution network double-time-scale voltage control method based on multi-agent deep reinforcement learning

Abstract

The application relates to a power distribution network double-time-scale voltage control method based on multi-agent deep reinforcement learning, which comprises the following steps: establishing a power distribution network slow-time-scale voltage control model based on a DDQN algorithm; establishing a Markov game process of slow-time-scale voltage control; establishing a power distribution network fast-time-scale voltage control model based on an EA-MASAC algorithm; establishing a Markov game process of fast-time-scale voltage control; respectively solving the power distribution network slow-time-scale voltage control model and the power distribution network fast-time-scale voltage control model, transmitting the scheduling instructions of the solved upper-layer agent to a lower-layer agent, and realizing double-time-scale voltage collaborative control. Compared with the prior art, the application has the advantages that not only the network loss of the power distribution network can be significantly reduced, but also voltage fluctuation caused by distributed power supply access to the power distribution network is relieved.

Description

Technical Field

[0001] The present invention relates to the field of power grid distribution, and in particular to a dual-time-scale voltage control method for a distribution network based on multi-agent deep reinforcement learning. Background Art

[0002] In recent years, with the influx of distributed generation (DGs) connected to distribution networks, voltage overshooting has become a frequent problem. Simply regulating the active power resources of the distribution network cannot guarantee safe and economical operation, necessitating coordinated optimization of both active and reactive power. Traditional distribution network reactive power optimization primarily relies on controlling reactive devices such as on-load tap-changing transformers (OLTCs), SVCs, and switched capacitor banks (CBs) to stabilize system voltage. With the large-scale integration of photovoltaic (PV) and water-powered vehicles (WTs) into the grid, the rise of electric vehicles, and the increasing maturity of vehicle-to-vehicle (V2G) technology, distributed generators (DGs) and electric vehicles are now participating in distribution network voltage regulation. However, the computational time of traditional distribution network optimization methods based on physical models increases exponentially with the size of the distribution network and the number of DGs.

[0003] At present, there are two main methods for optimizing the active and reactive power of distribution networks: optimization methods based on physical models and data-driven methods. The calculation time of the distribution network optimization method based on physical models will increase exponentially with the size of the distribution network and the number of DGs. In the data-driven method, the single-agent deep reinforcement learning algorithm has poor robustness and low learning efficiency; the multi-agent deep reinforcement learning (MADRL) algorithm has low sample sampling efficiency and unstable training results. Data enhancement technology can generate reliable samples to expand the original data set and improve the training efficiency of the MADRL algorithm. This technology has been widely used in single-agent training, but it is less common in multi-agent algorithms. Therefore, the present invention provides a dual-time-scale voltage control method for distribution networks based on multi-agent deep reinforcement learning. Summary of the Invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] In one aspect, the present invention provides a dual-time-scale voltage control method for a distribution network based on multi-agent deep reinforcement learning, the method comprising the following steps:

[0007] Step 1: Establish a slow time scale voltage control model for distribution network based on DDQN algorithm;

[0008] Step 2: For the distribution network slow time scale voltage control model, a Markov game process for slow time scale voltage control is established, including taking the on-load tap changer (OLTC), switched capacitor banks (CBs), and energy storage (ES) as upper-level agents, and setting the state space, action space, and reward function of the upper-level agents;

[0009] Step 3: Establish a fast time scale voltage control model for the distribution network based on the EA-MASAC algorithm;

[0010] Step 4: For the fast-time-scale voltage control model of the distribution network, a fast-time-scale voltage control Markov game process is established, including using distributed photovoltaic (PV), wind turbine (WT), electric vehicle (EV) charging station, and static VAR compensation device (SVC) as lower-level intelligent agents, and setting the state space, action space, and reward function of the lower-level intelligent agents;

[0011] Step 5: Solve the distribution network slow time scale voltage control model and the distribution network fast time scale voltage control model respectively, and transmit the solved dispatching instructions of the upper-level intelligent agent to the lower-level intelligent agent to realize dual-time scale voltage coordinated control.

[0012] Furthermore, the voltage control optimization objective function of the distribution network slow time scale voltage control model includes minimizing the network loss cost, voltage deviation cost, discrete device operation cost, and ES charging and discharging cost of the distribution network within one day, which is expressed by the following formula:

[0013]

[0014] Where: T is 24 hours; t is the number of decisions (t = 1, 2, 3...24); c loss is the network loss coefficient; P Loss,t is the distribution network loss at the tth decision; ξ is the voltage deviation cost coefficient; V i,t 、V i,N is the actual voltage and rated voltage of node i at the tth decision; V max With V min is the upper and lower voltage limits for safe operation of the distribution network; n is the number of distribution network nodes; C OLTC,t 、C CBs,t with C ES,t is the operation cost of OLTC and CBs and the charging and discharging cost of ES at the tth decision.

[0015] Furthermore, the voltage control optimization objective function of the distribution network slow time scale voltage control model satisfies the constraints, including: power flow constraint, voltage constraint, OLTC and CBs tap adjustment constraint, and ES constraint.

[0016] Further, the distribution network slow time scale voltage control model is solved by a DDQN algorithm.

[0017] Further, the distribution network slow time scale voltage control model is scheduled every hour.

[0018] Further, the voltage control optimization objective function of the distribution network fast time scale voltage control model is that the network loss cost, the voltage deviation cost, the PV and WT active power reduction cost, and the EV charging station operation cost in one cycle of the distribution network are minimum, and is expressed by the following formula:

[0019]

[0020] Wherein, c loss is a network loss coefficient; P Loss,Δt is the network loss of the distribution network at the Δtth decision; ξ is a voltage deviation cost coefficient; V i,t and V i,N are the actual voltage and the rated voltage of the node i at the tth decision; V max and V min are the upper and lower limits of the voltage for safe operation of the distribution network; n is the number of nodes of the distribution network; c PV and c WT are the PV and WT active power reduction cost coefficients; c TOU is a time-of-use electricity price; is the active power reduction amount of the PV and WT of the node i at the Δtth moment, is the charging power of the xth electric vehicle at the tth moment.

[0021] Further, the voltage control optimization objective function of the distribution network fast time scale voltage control model satisfies constraint conditions, including the PV and WT active and reactive power constraints, the SVC reactive power constraints, and the EV charging station active power constraints.

[0022] Further, the distribution network fast time scale voltage control model is solved by an EA-MASAC algorithm.

[0023] Further, the distribution network fast time scale voltage control model is scheduled every minute.

[0024] In a second aspect, the application provides a storage medium having a program stored thereon, the program being executed to implement the method described above.

[0025] Compared with the prior art, the application has the following beneficial effects:

[0026] 1. In the fast-scale voltage control model, the present invention integrates the attention mechanism and data augmentation technology into the EA-MASAC algorithm. The EA-MASAC algorithm solves the voltage optimization and power electronic equipment action optimization problems. Compared with the traditional voltage / reactive power control method, this method can not only regulate the voltage, but also reduce the power loss of the entire system, improve the algorithm robustness and sample sampling efficiency during training.

[0027] 2. This paper designs a dual-timescale active-reactive coordinated voltage control method for distribution networks. The DDQN algorithm is used to optimize discrete-action devices on the slow timescale, while the EA-MASAC algorithm is used to optimize continuously acting devices on the fast timescale. Comparing the proposed algorithm with traditional optimization algorithms in an improved IEEE-33 node system, the proposed algorithm reduces network losses and node voltage deviations, effectively smoothing voltage fluctuations.

[0028] 3. The present invention takes into account multiple intelligent entities such as OLTC, CBs, ES, PV, WT and EV when modeling dual-time-scale voltage control, and adopts deep reinforcement learning algorithm to intelligently coordinate the active and reactive optimization of the power grid at the same time, ensuring the safety and economy of distribution network operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the structural framework of the present invention;

[0030] Figure 2 Schematic diagram of the steps of the present invention;

[0031] Figure 3 Schematic diagram of the dual-time-scale MADRL algorithm training process of the present invention;

[0032] Figure 4 This is a schematic diagram of the power distribution system structure of the improved IEEE-33 node of the present invention;

[0033] Figure 5 Schematic diagram of charging power curves of three EV charging stations of the present invention;

[0034] Figure 6 Schematic diagram of the PV prediction curve of the test set of the present invention;

[0035] Figure 7 Schematic diagram of the WT prediction curve of the test set of the present invention;

[0036] Figure 8 Schematic diagram of the load change rate curve of the present invention;

[0037] Figure 9Schematic diagram of the dual-time-scale MADRL algorithm training results of the present invention, where (a) is a schematic diagram of the DDQN algorithm training results, and (b) is a schematic diagram of the EA-MASAC algorithm training results;

[0038] Figure 10 Schematic diagram of the day-ahead scheduling results of each discrete device of the present invention, where (a) is a schematic diagram of OLTC gear change, (b) is a schematic diagram of the CB reactive output of node 17, and (c) is a schematic diagram of the CB reactive output of node 33;

[0039] Figure 11 Schematic diagram of the ES day-ahead scheduling results of the present invention, wherein (a) is a schematic diagram of the ES charge and discharge power change, and (b) is a schematic diagram of the ES SOC change curve;

[0040] Figure 12 Schematic diagram of the intraday scheduling results of the continuous action equipment of the present invention, where (a) is the SVC output plan, (b) is the electric vehicle charging station decision plan curve, (c) is the PV power reduction change, and (d) is the WT power reduction change;

[0041] Figure 13 Schematic diagram of network loss changes at different times of the day in the distribution network under different solutions of the present invention;

[0042] Figure 14 This is a schematic diagram of the voltage distribution comparison analysis under different schemes at 12:00, 6:00, and 18:00 of the present invention, where (a) is the voltage distribution of each node at 12:00, (b) is the voltage distribution of each node at 6:00, and (c) is the voltage distribution of each node at 18:00. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0044] Example 1

[0045] This embodiment provides a dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning. Figure 1 and Figure 2 As shown, the specific steps include:

[0046] Step 1: Establish a slow-time-scale voltage control model for the distribution network based on the DDQN algorithm. The voltage control optimization objective of the slow-time-scale voltage control model for the distribution network is to minimize the network loss cost, voltage deviation cost, discrete device operation cost, and ES charging and discharging cost within one day of the distribution network, which is expressed as follows:

[0047]

[0048] Where: T is 24 hours; t is the number of decision-making times (t = 1, 2, 3...24); c loss is the network loss coefficient, set to 0.5 yuan / kWh; P Loss,t is the distribution network loss at the t-th decision; ξ is the voltage deviation cost coefficient, which is set to 500 yuan / pu; V i,t 、V i,N is the actual voltage and rated voltage of node i at the tth decision; V max With V min are the upper and lower voltage limits for safe operation of the distribution network, set to 1.05pu and 0.95pu; n is the number of distribution network nodes; C OLTC,t 、C CBs,t with C ES,t is the operation cost of OLTC and CBs and the charging and discharging cost of ES at the tth decision.

[0049] Specifically, P Loss,t 、C OLTC,t 、C CBs,t 、C ES,t As shown in the following formula:

[0050] P Loss,t =r ij ·(I ij，t ) 2

[0051]

[0052] Where: r ij is the resistance between nodes i and j, I ij,t is the current between nodes i and j at time t; OLTC , δ CBs The unit adjustment cost of OLTC and CBs is set to RMB 2 / time and RMB 1 / time respectively; M OLTC 、W CBs is the number of OLTCs and CBs; T m,t is the gear position of the mth OLTC at the tth decision; C w,t is the gear position of the w-th CB at the t-th decision; c ES is the ES charging and discharging cost coefficient, set to 0.1 yuan / kWh; Y ES is the total number of ES; is the charge and discharge power of the y-th ES during the t-th decision period.

[0053] The voltage control optimization objective of the power distribution network slow time scale voltage control model also needs to meet constraint conditions, including power flow constraints, voltage constraints, OLTC and CB tap adjustment constraints, ES constraints, as shown in the following formula.

[0054]

[0055] V i，min <V i,Δt <V i,max

[0056]

[0057]

[0058] SOC min ≤SOC i,Δt ≤SOC max

[0059] In the formula, Vi, V j , Gi j , Bi j , δi j are the first and last voltages, conductance, susceptance and phase angle difference of line ij; V i,Δt is the voltage of node i at time Δt; N OLTC,m,max , N CBs,w,max are the upper limits of the number of actions of the mth OLTC and the wth CB in a day; E i,Δt is the amount of ES at node i at time Δt; η char , ηdisc are the charging and discharging efficiencies of ES; SOC i,Δt is the charging and discharging power and state of charge of ES at node i at time Δt.

[0060] Step 2, according to the power distribution network slow time scale voltage control model, a slow time scale voltage control Markov game process is designed;

[0061] Under the slow time scale, the OLTC, CBs and ES are regarded as agents:

[0062] The agent state space is set as:

[0063] s m,t =[P m,t ,Q m,t ,V m,t ,T m,t ,C m,t ,N OLTC,m,t ,N CBs,m,t ,E m,t ]

[0064] In the formula, Pm,t , Q m,t 、V m,t N is the active and reactive power matrix and voltage matrix of each node when agent m makes the tth decision; OLTC,m,t 、N CBs,m,t is the cumulative number of actions of OLTC and CBs; E m,t is the ES power at the tth decision.

[0065] Set the agent action space to:

[0066]

[0067] When ES is charging, When discharging,

[0068] Set the agent reward function to

[0069]

[0070] Where φ is the penalty coefficient, which is set to 1000; σ(·) is the judgment function. If the voltage and the number of discrete device operations do not exceed the limit, the value of the judgment function is 0, otherwise it is 1.

[0071] Step 3: Establish a fast-time-scale voltage control model for the distribution network based on the EA-MASAC algorithm. The voltage control optimization objective of the fast-time-scale voltage control model for the distribution network is to minimize the network loss cost, voltage deviation cost, PV and WT active power reduction cost, and EV charging station operating cost within one cycle of the distribution network, which is expressed as follows:

[0072]

[0073] Where: c loss is the network loss coefficient; P Loss,Δt is the distribution network loss at the Δt-th decision; ξ is the voltage deviation cost coefficient; V i,t 、V i,N is the actual voltage and rated voltage of node i at the tth decision; V max With V min is the upper and lower voltage limits for safe operation of the distribution network; n is the number of distribution network nodes; c PV 、c WT is the cost coefficient of PV and WT active power reduction, set to 700 yuan / MWh and 800 yuan / MWh; c TOU Time-of-use electricity price; is the active power reduction of PV and WT at node i at time Δt, is the charging power of the x-th electric vehicle at time t.

[0074] The voltage control optimization objective of the distribution network fast-time-scale voltage control model must also meet the constraints, which include PV, WT active-reactive constraints, SVC reactive constraints, and EV charging station active constraints, as shown in the following formula:

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] Where: S PV is the PV rated capacity, which is generally 1.0 to 1.1 times the installed capacity; is the apparent power of WT at node i at time Δt.

[0084] Step 4: Design a Markov game process for fast-time-scale voltage control based on the fast-time-scale voltage control model of the distribution network;

[0085] In the fast time scale, PV, WT, EV charging station, and SVC are used as intelligent agents:

[0086] Set the agent state space to:

[0087] s m,Δt =[P m,Δt ,Q m,Δt ,V m,Δt ]

[0088] Set the agent action space to:

[0089]

[0090] Set the agent reward function to:

[0091]

[0092] Step 5: Solve the distribution network slow time scale voltage control model and the distribution network fast time scale voltage control model respectively, and coordinate the slow time scale voltage control and the fast time scale voltage control.

[0093] Specifically, in the slow time scale, OLTC, CBs and ES are scheduled by the predicted values of PV, WT and load per hour and solved by DDQN algorithm. The network loss cost, voltage deviation cost, discrete device action cost and ES charging and discharging cost of the distribution network in a day are minimized.

[0094] In the fast time scale, the EA-MASAC algorithm is used to train the agents in the distribution network offline to obtain the optimal voltage control method and pass it to each agent. The agents adjust the reactive power of PV, WT and SVC and the active power of EV charging station according to the local observation value, and are scheduled every minute. The network loss cost, voltage deviation cost, PV and WT active power reduction cost and EV charging station operation cost of the distribution network in an hour are minimized.

[0095] Because the time scales of the type device control in the two models are different, their collaborative control is usually modeled as a multi-time scale optimization problem.

[0096] The on-load tap-changing transformer (OLTC) and the shunt capacitor (CB) belong to mechanical devices, and their response speed is slow. Due to the life limit, they cannot act frequently, so the control time scale of such devices is usually longer. Distributed photovoltaic (PV), wind turbine inverter (WT) and static var compensator (SVC) belong to power electronic devices, which have the characteristics of fast response speed, frequent action and continuous adjustment, so such devices can act multiple times in a short time.

[0097] This embodiment comprehensively considers mechanical devices such as on-load tap-changing transformers and shunt capacitors, and power electronic devices such as photovoltaic and wind turbine inverters, and in addition, combines electric vehicles to carry out multi-time scale voltage collaborative control of the distribution network. The agents are divided into upper agents and lower agents. The upper agents handle the slow time scale problem, and the lower agents handle the fast time scale problem.

[0098] Specifically, the principle process of the double-time scale collaborative control is as follows:

[0099] First, the parameters of the upper agent action network and the lower multi-agent action network are loaded. At each time, the upper agent calculates the mechanical device scheduling instruction according to the global observation information. Then the state information of the mechanical device is transmitted to the lower multi-agent. In each time interval at this time, the lower agent calculates the inverter, SVC and EV scheduling instruction according to the state of the mechanical device and the local observation information and executes the instruction. Since the decision of the lower agent is made based on the local observation information and the state of the mechanical device, and the state signal of the mechanical device is transmitted by the upper agent in advance, the lower agent can quickly provide the scheduling instruction. Since the state of the optimized mechanical device is transmitted to the lower multi-agent, the lower agent considers the influence of the action of the upper agent when making decisions; since the reward value of the upper agent is calculated based on the reward value of the lower multi-agent, the upper agent will consider the influence of its action on the lower agent during the training process.

[0100] In addition, the double-time-scale MADRL algorithm (DDQN algorithm and EA-MASAC algorithm belong to MADRL algorithm) is also trained, and the training process is as shown in Figure 3 The algorithm training is performed from the initialization, action decision and training update three stages.

[0101] Initialization stage: first, initialize the power flow of the power distribution network and initialize the neural network parameters of the DDQN and EA-MASAC algorithms, and load the load, PV and WT data;

[0102] Action decision stage: in the action decision stage, the action of each agent is randomly selected, and the agent obtains the state information of the corresponding device and makes an action. The agent calculates the scheduling instruction, and after the actions of all agents are executed, the reward value is calculated, and then the system is transferred to the next state. Then, each agent stores the experience data obtained in the interaction in its own memory.

[0103] Training update stage: the slow time scale agent extracts a small batch of samples from the experience replay pool, updates the evaluation network parameters using the gradient descent method, and copies the updated evaluation network parameters to the target network at intervals to update, while minimizing the loss function. When the agent reaches the set number of training times, the training is ended.

[0104] The fast time scale agent first extracts a small batch of samples from the experience replay pool and artificially expands the sample data using experience enhancement technology. Then, the newly obtained sample set is used to update the target network parameters, policy network parameters and q1, q2 network parameters, while minimizing the loss function. When the agent reaches the set number of training times, the training is ended.

[0105] In order to verify the effect of the dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning provided in this embodiment, this embodiment applies the method to the improved IEEE-33 node distribution system for testing. Figure 4 As shown in Figure 1, the base voltage of the system is 12.66 kV. The EV charging stations include home area, work area and public area charging stations. The charging efficiency of each EV battery is 0.9, and the maximum charging power is 7 kW. In this embodiment, Monte Carlo sampling is performed on 1,000 EVs to generate charging power curves for three types of EV charging stations, as shown in Figure 1. Figure 5 shown.

[0106] The specific parameters of the equipment in this system and the time-of-use electricity price are shown in Tables 1 and 2.

[0107] Table 1 Parameters of each device in the IEEE-33 node system

[0108]

[0109] Table 2 Time-of-use electricity prices

[0110]

[0111] We selected the measured PV and WT data for a certain month in a certain area of ​​Shanghai, selected 20 days as the reinforcement learning training set, and the remaining 10 days as the reinforcement learning test set. The training set was used to train the multi-agent algorithm in the centralized learning phase. After the training was completed, the control performance of the proposed algorithm was evaluated in the distributed execution phase using the test set data. The PV and WT prediction curves of the test set are shown in Figure 2. Figure 6 and 7 shown.

[0112] The load of each node is based on the spatial load distribution of the original IEEE33 node system, and random Gaussian noise with a standard deviation of 5% is added to simulate the uncertainty of load demand. The load change rate curve is as follows: Figure 8 shown.

[0113] The power flow calculation was performed in Matlab, and the MADRL algorithm was implemented in Python's Pytorch neural network framework. The specific parameters are shown in Table 3.

[0114] Table 3 MADRL parameter settings

[0115]

[0116] The DDQN algorithm has a total exploration step count of 120,000, and updates the target network parameters every 24 steps. One round is one day, and one round contains 24 steps, corresponding to 24 hours, with a maximum round count of 5,000. The EA-MASAC algorithm has a total exploration step count of 7,200,000, and one round contains 1,440 steps per minute, corresponding to 5,000.

[0117] Figure 9 is the dual-time-scale MADRL algorithm training result, Figure 9 (a) is a schematic diagram of the DDQN algorithm training results. Figure 9 (b) is a diagram of the EA-MASAC algorithm training results. It can be seen that the cumulative rewards of the two algorithm agents did not increase significantly before the training rounds reached 200. At this time, the agents were in the exploration stage and the parameters of the deep neural network were not updated. The actions and states of the agents were stored in the experience pool. After 200 rounds, the cumulative rewards increased significantly, the agents began to train, and the parameters of the deep neural network began to update to adapt to environmental changes. At this time, the experience accumulated by the agents has filled the experience pool, and the agents can make better and better decisions, making network losses and various costs smaller and smaller. Figure 9 As shown in (a) and (b), the cumulative returns of the DDQN algorithm and the EA-MASAC algorithm gradually converge after 2000 and 1500 rounds, respectively, at which point the training is completed.

[0118] like Figure 10 (a) shows the day-ahead scheduling result of OLTC. The number of OLTC actions in one day is 4, which does not exceed the maximum number of actions.

[0119] like Figure 10 (b) and (c) show the day-ahead scheduling results for the two CBs. The number of actions taken by the two CBs is 5 and 3, respectively, which also falls within the maximum number of adjustments. This demonstrates that the agent, through trial and error, learned that excessive actions are penalized and thus learned to control discrete devices.

[0120] like Figure 11 Figure (a) shows the change in energy storage charging and discharging power over a 24-hour period. Positive values ​​indicate energy storage charging, while negative values ​​indicate energy storage discharging. This indicates that the energy storage is continuously charging from 10:00 AM to 7:00 PM, while discharging occurs during the remaining hours. This is because the distributed photovoltaic and wind turbine generation is high during this period, and the excess power is stored in the energy storage to reduce power backflow. During the remaining hours, the distributed photovoltaic and wind turbine generation is low, and the load demand is high.

[0121] like Figure 11 (b) shows the energy storage SOC change curve. The SOC fluctuates between 0.1 and 0.8, satisfying the constraints.

[0122] The daily scheduling results of continuous action equipment are as follows: Figure 12 shown.

[0123] like Figure 12 (a) shows the SVC output plan. Positive values ​​indicate the SVC outputs inductive reactive power, while negative values ​​indicate the SVC absorbs inductive reactive power. Around the 720-step decision time, both SVCs absorb reactive power, smoothing voltage fluctuations caused by PV and wind turbine generation and preventing voltage overshoot.

[0124] like Figure 12 (b) shows the charging power curves for the three types of electric vehicle charging stations. These curves are roughly similar to the predicted power curves for the charging stations. The charging power at the home charging station peaks during the evening rush hour around 6:00 PM, while the charging power at the work station peaks during the morning rush hour around 9:00 AM. The charging power at the public charging station is primarily concentrated in the mid-afternoon hours.

[0125] like Figure 12 Figures (c) and (d) show the changes in active power curtailment of distributed photovoltaic and wind turbine inverters. Distributed photovoltaics only generate power between approximately 6:00 AM and 7:00 PM. Around 12:00 PM, when photovoltaic power generation is high, the active power curtailment of the three PV inverters reaches its peak, thereby reducing voltage excursions. Wind power curtailment also exhibits a similar trend to the wind power output curve, with greater power curtailment occurring when DG output is high, thereby stabilizing system voltage levels.

[0126] In order to illustrate the effectiveness of the proposed dual-time-scale algorithm combination in voltage optimization, the following three schemes are compared with the method provided in this embodiment.

[0127] Solution 1: Use dual-time-scale PSO algorithm.

[0128] Solution 2: Use dual-time-scale genetic algorithm (GA) algorithm.

[0129] Solution 3: Keep DDQN unchanged for slow time scale and use MADDPG for fast time scale.

[0130] Network loss analysis

[0131] The changes of distribution network loss at different times of the day under different schemes are as follows Figure 13 The average daily network losses of Scheme 1, Scheme 2, Scheme 3 and the method provided in this embodiment are 50.13kW, 47.17kW, 46.04kW and 44.38kW, respectively. This shows the superiority of the method provided in this embodiment in reducing network losses.

[0132] Voltage deviation analysis

[0133] like Figure 14(a) shows the voltage distribution of 33 nodes at 12:00 under four different algorithms. As Figure 14 (b) shows the voltage distribution of 33 nodes at 6:00 under four different algorithms. As Figure 14 (c) shows the voltage distribution of 33 nodes at 18:00 under four different algorithms. This embodiment selects 12:00 with large PV and WT output, and 6:00 and 18:00 with small PV and WT output to compare and analyze the voltage distribution of 33 nodes under different schemes.

[0134] The average voltage deviation of 33 nodes under different schemes at 12:00, 6:00 and 18:00 is shown in Table 4.

[0135] Table 4 Average voltage deviation of each node under each scheme at 12:00, 6:00 and 18:00

[0136]

[0137] It can be seen that the average voltage deviation of each node is the smallest when the voltage control method provided in this embodiment is used, which provides a guarantee for the safe and stable operation of the power distribution network.

[0138] Embodiment 2

[0139] This embodiment provides a storage medium having a program stored thereon, and the program is executed to realize the power distribution network double-time-scale voltage control method based on multi-agent deep reinforcement learning mentioned in embodiment 1.

[0140] The preferred embodiments of the present application are described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the prior art according to the concept of the present application shall be within the protection scope defined by the claims.

Claims

1. A dual-time-scale voltage control method for distribution networks based on multi-agent deep reinforcement learning, characterized in that: The method comprises the following steps: Step 1: Establish a slow-time-scale voltage control model for the distribution network based on the DDQN algorithm. The voltage control optimization objective function of the slow-time-scale voltage control model for the distribution network is to minimize the network loss cost, voltage deviation cost, discrete device operation cost, and ES charging and discharging cost within one day of the distribution network, which is expressed as follows: Where: T is 24 hours; t is the number of decisions, t=1,2,3...24; c loss is the network loss coefficient; P Loss,t is the distribution network loss at the tth decision; ξ is the voltage deviation cost coefficient; V i,t 、 V i,N is the actual voltage and rated voltage of node i at the tth decision; V max and V min are the upper and lower voltage limits for safe operation of the distribution network; n is the number of nodes in the distribution network; C OLTC,t 、 C CBs,t and C ES,t is the action cost of OLTC and CBs and the charging and discharging cost of ES at the tth decision; Step 2: For the distribution network slow time scale voltage control model, a Markov game process for slow time scale voltage control is established, including taking OLTC, CBs, and ES as upper-layer agents, and setting the state space, action space, and reward function of the upper-layer agents; Step 3: Establish a fast-time-scale voltage control model for the distribution network based on the EA-MASAC algorithm. The voltage control optimization objective function of the fast-time-scale voltage control model for the distribution network is to minimize the network loss cost, voltage deviation cost, PV and WT active power reduction cost, and EV charging station operating cost within one cycle of the distribution network, which is expressed as follows: in: c loss is the network loss coefficient; is the distribution network loss at the Δt-th decision; ξ is the voltage deviation cost coefficient; V i,t 、 V i,N is the actual voltage and rated voltage of node i at the tth decision; V max and V min are the upper and lower voltage limits for safe operation of the distribution network; n is the number of nodes in the distribution network; c PV 、 c WT Active power reduction cost coefficient for PV and WT; c TOU Time-of-use electricity price; 、 is the active power reduction of PV and WT at node i at time Δt, is the charging power of the x-th electric vehicle at time t; Step 4: For the fast-time-scale voltage control model of the distribution network, a fast-time-scale voltage control Markov game process is established, including taking PV, WT, EV charging station, and SVC as lower-level intelligent agents, and setting the state space, action space, and reward function of the lower-level intelligent agents; Step 5: Solve the distribution network slow time scale voltage control model and the distribution network fast time scale voltage control model respectively, and transmit the solved dispatching instructions of the upper-level intelligent agent to the lower-level intelligent agent to realize dual-time scale voltage coordinated control.

2. The dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The voltage control optimization objective function of the distribution network slow time scale voltage control model satisfies the constraints, including: power flow constraint, voltage constraint, OLTC and CBs tap adjustment constraint, and ES constraint.

3. The dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning according to claim 2 is characterized in that: The slow time scale voltage control model of distribution network is solved by DDQN algorithm.

4. The dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: The slow time scale voltage control model of the distribution network is dispatched on an hourly basis.

5. The dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The voltage control optimization objective function of the distribution network fast time scale voltage control model satisfies the constraints, including: PV, WT active-reactive constraints, SVC reactive constraints, and EV charging station active constraints.

6. A dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning according to claim 5, characterized in that: The EA-MASAC algorithm is used to solve the fast time scale voltage control model of the distribution network.

7. The dual-time-scale voltage control method for distribution network based on multi-agent deep reinforcement learning according to claim 6 is characterized in that: The fast time scale voltage control model of the distribution network is dispatched every minute.

8. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 7 is implemented.

Images