Power distribution network multi-time scale voltage optimization method based on deep reinforcement learning

Abstract

The invention relates to the technical field of power distribution network voltage optimization, and discloses a power distribution network multi-time scale voltage optimization method based on deep reinforcement learning, and the method comprises the steps: dividing reactive power regulation equipment into long-time scale equipment and short-time scale equipment, training the action decision of the long-time scale equipment through employing a DQN intelligent agent, and carrying out the training of the action decision of the short-time scale equipment; a DDPG intelligent agent is adopted to train an action decision of short-time-scale equipment, then a decision strategy of the intelligent agent is adjusted according to voltage optimization feedback of the power distribution network, and finally multi-time-scale equipment participates in voltage optimization of the power distribution network cooperatively. According to the method, the cooperation output and voltage optimization of the equipment actions at different time scales is realized, the voltage can be effectively improved and the voltage is prevented from being out of limit at the long time scale, the voltage fluctuation range is also controlled at the short time scale, the voltage deviation is effectively optimized, and the electric energy quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distribution network voltage optimization, and in particular to a distribution network multi-time scale voltage optimization method based on deep reinforcement learning. Background Art

[0002] With the access of high-penetration distributed photovoltaics, the distribution network has changed from a single power flow system to a multi-terminal power supply system with bidirectional power. The resulting bidirectional power flow and voltage over-limit problems have become increasingly serious. Reasonable scheduling of diversified power system resources can help to suppress the impact of the randomness and intermittency of distributed photovoltaics on the distribution network, assist in regulating voltage levels by adjusting reactive power, optimize power flow, and effectively reduce power losses during transmission and distribution.

[0003] Common methods for reactive power and voltage optimization in distribution networks include centralized control, local control, and distributed control. However, a high proportion of photovoltaic access to distribution networks will significantly increase the optimization difficulty of traditional methods. A large number of power system resources with different adjustment time scales have multiplied the optimization constraints. Traditional optimization methods, due to their over-reliance on prediction and model accuracy, lack real-time optimization and are difficult to adapt to dynamic changes in grid structure and operation mode. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-time-scale voltage optimization method for distribution networks based on deep reinforcement learning, which is used to solve the problem of voltage optimization difficulty caused by traditional voltage optimization methods when dealing with a high proportion of distributed photovoltaic access to distribution networks.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] A multi-time-scale voltage optimization method for a distribution network based on deep reinforcement learning comprises the following steps:

[0007] S1. According to the equipment regulation characteristics, the reactive regulation equipment involved in the voltage optimization of the distribution network is divided into long-time scale equipment and short-time scale equipment. At the same time, DQN agents and DDPG agents are set to control the reactive output of long-time scale equipment and short-time scale equipment respectively.

[0008] S2. Establish the constraints of each electrical node in the distribution network, and use the DQN agent to obtain the distribution network status of long-time scale equipment, and use the DDPG agent to obtain the distribution network status of short-time scale equipment;

[0009] S3, setting the switching decision time of the long time scale equipment and the short time scale equipment, and according to the distribution network status of the long time scale equipment and the distribution network status of the short time scale equipment, obtaining the action decision of the DQN agent on the long time scale equipment and the action decision of the DDPG agent on the short time scale equipment at their respective corresponding switching decision times;

[0010] S4. According to the voltage over-limit and voltage offset conditions of the distribution network, feedback is given to the action decisions of the long-time scale equipment and the action decisions of the short-time scale equipment, and the distribution network feedback of the DQN agent to the long-time scale equipment and the distribution network feedback of the DDPG agent to the short-time scale equipment are obtained respectively;

[0011] S5. At the switching decision time of the long-time scale equipment, the network parameters of the DQN network are updated according to the distribution network state, action decision and distribution network feedback of the long-time scale equipment to obtain a trained DQN network, i.e., the optimal DQN agent, which is used to output the optimal action decision of the long-time scale equipment;

[0012] S6. At the switching decision time of the short-time scale equipment, the network parameters of the DDPG network are updated according to the distribution network state, action decision and distribution network feedback of the short-time scale equipment to obtain a trained DDPG network, i.e., the optimal DDPG agent, which is used to output the optimal action decision of the short-time scale equipment;

[0013] S7. Put the optimal DQN agent and the optimal DDPG agent into the distribution network to obtain the optimal action decision of the reactive power regulation equipment.

[0014] Furthermore, the constraints of each electrical node in the distribution network in step S2 are:

[0015]

[0016] Among them, P i , Q i They represent the active power and reactive power of electrical node i, n represents the total number of electrical nodes connected to electrical node i, v i 、v j Respectively represent the voltage of electrical node i and electrical node j, G ij represents the conductance of the line from electrical node i to electrical node j, B ij represents the susceptance of the line from electrical node i to electrical node j, cos represents the cosine function, sin represents the sine function, δ ij Represents the voltage phase difference between the two ends of the line from electrical node i to electrical node j, v i,min 、v i,maxThey represent the lower and upper limits of the voltage of the electrical node i, respectively. v(i,t) represents the voltage of the electrical node i at the tth moment. represents the reactive power output of the switched capacitor SCB at the electrical node i, Indicates the reactive power output corresponding to a single gear of the switched capacitor SCB. Indicates the capacitor level that electrical node i decides to switch on at time t, They represent the active output, reactive output and capacity of the photovoltaic PV at the electrical node i at the tth moment, It represents the upper limit of the active output of the photovoltaic PV at the electrical node i at the tth moment, min represents the minimum value, and max represents the maximum value.

[0017] Furthermore, in step S2, the distribution network state of the long-time scale equipment is obtained by using the DQN agent:

[0018]

[0019] in, represents the distribution network state of the long-time scale equipment obtained by the DQN agent at time t, P slack,t , Q slack,t They represent the active power and reactive power of the slack node at the tth moment, P load,t , Q load,t They represent the active power and reactive power of the loads of the electrical nodes except the relaxed nodes at the tth moment, P′ SCB,t , Q′ SCB,t They represent the active output and reactive output of the switched capacitor SCB at the tth moment respectively;

[0020] Among them, the slack node refers to the electrical node connected to the upper-level distribution network.

[0021] Furthermore, in step S2, the distribution network state of the short-time scale equipment is obtained by using the DDPG agent:

[0022]

[0023] in, represents the distribution network state of the short-time scale equipment obtained by the DDPG agent at time t, P′ PV,t , Q′ PV,t They represent the active and reactive output of photovoltaic PV at the tth moment, Indicates the maximum active output of photovoltaic PV.

[0024] Furthermore, in step S3, the action decision of the DQN agent for the long-time scale device is:

[0025]

[0026] in, Represents the action decision of the DQN agent on the long-time scale device at time t.

[0027] Furthermore, in step S3, the action decision of the DDPG agent for the short-time scale device is:

[0028]

[0029] in, Represents the action decision of the DDPG agent on the short-time-scale device at time t.

[0030] Furthermore, in step S4, the DQN agent’s feedback to the distribution network of the long-time scale equipment is:

[0031]

[0032] in, It represents the distribution network feedback of the DQN agent to the long-time scale equipment at the tth moment, and N represents the total number of electrical nodes in the distribution network.

[0033] Furthermore, in step S4, the DDPG agent provides the following feedback to the distribution network of the short-time scale equipment:

[0034]

[0035] in, represents the distribution network feedback of the DDPG agent to the short-time scale equipment at time t, v i,base Represents the nominal voltage at electrical node i.

[0036] Furthermore, step S5 specifically includes:

[0037] S51, at the switching decision time of the long-time scale device, input the distribution network state, action decision and distribution network feedback of the long-time scale device into the distribution network environment, and transfer to the next state at the same time;

[0038] S52, when updating the network parameters of the DQN network, the distribution network state, action decision and distribution network feedback of the long-time scale device are input into the DQN network, and the total loss function of the DQN network is updated by the gradient descent method. When the number of training times reaches the maximum value, a trained DQN network is obtained, that is, the optimal DQN agent, which is used to output the optimal action decision of the long-time scale device;

[0039] Among them, the total loss function of the DQN network is:

[0040]

[0041] Where L represents the total loss function of the DQN network, θ represents the network parameters, and θ t represents the network parameters of the DQN network at time t, θ t+1 represents the network parameters of the DQN network at time t+1, E represents the mean calculation, r DQN represents the distribution network feedback of long-time scale devices in the DQN network, γ DQN represents the reduction factor of the distribution network feedback of the long-time scale equipment in the DQN network, Q DQN represents the value evaluation value of the DQN network, a represents the action decision of the long-time scale device, s represents the distribution network state of the long-time scale device, s′ represents the distribution network state after adopting the action decision a under the distribution network state s of the long-time scale device, a′ represents the action decision of the distribution network state s′, α represents the learning rate, Represents the gradient.

[0042] Further, step S6 specifically includes:

[0043] S61, at the switching decision time of the short-time scale device, the distribution network state, action decision and distribution network feedback of the short-time scale device are input into the distribution network environment, and transferred to the next state at the same time;

[0044] S62, inputting the action decision of the current state into the value evaluation neural network of the DDPG network, and updating the total loss function of the value evaluation neural network using the gradient descent method to obtain the updated network parameters of the value evaluation neural network;

[0045] S63, according to the updated network parameters of the value evaluation neural network, the action neural network of the DDPG network is updated, by inputting the distribution network state, action decision and distribution network feedback of the short-time scale device into the action neural network, and using the gradient descent method to update the total loss function of the action neural network. When the number of training times reaches the maximum value, the trained action neural network and value evaluation neural network, i.e., the optimal DDPG agent, are obtained to output the optimal action decision of the short-time scale device;

[0046] Among them, the total loss function of the value assessment neural network is:

[0047]

[0048] The total loss function of the action neural network is:

[0049]

[0050] Where L′ represents the total loss function of the value assessment neural network, θ Q1 represents the network parameters of the value assessment neural network Q1, represents the network parameters of the value assessment neural network Q1 at time t-1, represents the network parameters of the value assessment neural network Q1 at the tth moment, r DDPG represents the distribution network feedback of short-time-scale devices in the DDPG network, γ DDPG represents the reduction factor of the distribution network feedback of short-time scale devices in the DDPG network, Q Q1 represents the value evaluation value of the value evaluation neural network Q1, s1 represents the distribution network state of the short-time scale device, a1 represents the action decision of the short-time scale device, s1′ represents the distribution network state after the action decision a1 is adopted under the distribution network state s1 of the short-time scale device, a1′ represents the action decision of the distribution network state s1′, α Q1 represents the learning rate of the value evaluation neural network Q1, J represents the total loss function of the action neural network, θ π represents the network parameters of the action neural network π, represents the network parameters of the action neural network π at time t-1, represents the network parameters of the action neural network π at the tth moment, α π represents the learning rate of the action neural network π.

[0051] The present invention has the following beneficial effects:

[0052] The present invention proposes a multi-time-scale voltage optimization method for distribution networks based on deep reinforcement learning. By considering the regulation characteristics of different reactive devices, a double-layer deep learning method of DQN and DDPG is adopted. Through the mutual collaborative decision-making of two intelligent agents, the output and voltage optimization of equipment actions at different time scales are achieved. At the same time, on a long time scale, the voltage can be effectively improved to prevent the voltage from exceeding the limit. On a short time scale, the voltage fluctuation range is controlled, which effectively optimizes the voltage deviation and improves the power quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic diagram of a flow chart of a multi-time-scale voltage optimization method for a distribution network based on deep reinforcement learning proposed in the present invention;

[0054] Figure 2 It is a structural schematic diagram of the distribution network system improved based on the IEEE33 node calculation example in the embodiment;

[0055] Figure 3 This is a schematic diagram of the convergence of intelligent agent training in the embodiment;

[0056] Figure 4 A schematic diagram of an action decision of a discrete capacitor made by a DQN agent in an embodiment;

[0057] Figure 5 A schematic diagram of a photovoltaic action decision made by a DDPG agent in an embodiment;

[0058] Figure 6 It is a box line diagram of all node voltages before and after optimization on the test day in the embodiment;

[0059] Figure 7 Schematic diagram of the voltage raising effect of the terminal node in the embodiment;

[0060] Figure 8 This is a schematic diagram of the voltage optimization effect at 14:00 on the test day in the embodiment. DETAILED DESCRIPTION

[0061] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0062] like Figure 1 As shown, a multi-time scale voltage optimization method for distribution network based on deep reinforcement learning includes the following steps S1-S7:

[0063] S1. According to the equipment regulation characteristics, the reactive regulation equipment participating in the voltage optimization of the distribution network is divided into long-time scale equipment and short-time scale equipment. At the same time, DQN agents and DDPG agents are set to control the reactive output of long-time scale equipment and short-time scale equipment respectively.

[0064] In this embodiment, the reactive regulation equipment involved in the voltage optimization of the distribution network is classified according to the equipment regulation characteristics, that is, discrete switching capacitors are defined as long-time scale equipment, and photovoltaics are defined as short-time scale equipment; for different types of equipment, the gear position of the discrete switching capacitor is controlled by the DQN intelligent agent, and the reactive output of the photovoltaic is controlled by the DDPG intelligent agent.

[0065] In addition, the distribution network in this embodiment is an improved distribution network system based on the IEEE33 node calculation example, as shown in the following figure: Figure 2 As shown, it includes distributed photovoltaics connected to electrical nodes 16, 18, 22, 24, 25, and 31, each of which has a capacity of 0.6MW. It also includes discrete switching capacitors connected to electrical nodes 6, 8, 18, and 25, with a total of four adjustable gears, each with a capacity of 0.2MW, and the gear changes do not exceed 6 times a day.

[0066] S2. Establish constraints for each electrical node in the distribution network, and use the DQN agent to obtain the distribution network status of long-time scale equipment, and use the DDPG agent to obtain the distribution network status of short-time scale equipment.

[0067] In this embodiment, the agent is switched to training mode, and the action decisions of the two agents are trained by simulating the daily operation status of the distribution network, and the status information of each electrical node in the distribution network is collected. In order to simulate the operation status of the distribution network, the distribution network needs to be modeled, and the constraints of each electrical node in the distribution network are established so that the DQN agent can obtain the distribution network status of long-time scale equipment and the DDPG agent can obtain the distribution network status of short-time scale equipment.

[0068] Specifically, the constraints of each electrical node in the distribution network in step S2 are:

[0069]

[0070] Among them, P i , Q i They represent the active power and reactive power of electrical node i, n represents the total number of electrical nodes connected to electrical node i, v i 、v j Respectively represent the voltage of electrical node i and electrical node j, G ij represents the conductance of the line from electrical node i to electrical node j, B ij represents the susceptance of the line from electrical node i to electrical node j, cos represents the cosine function, sin represents the sine function, δ ij Represents the voltage phase difference between the two ends of the line from electrical node i to electrical node j, v i,min 、v i,max They represent the lower and upper limits of the voltage of the electrical node i, respectively. v(i,t) represents the voltage of the electrical node i at the tth moment. represents the reactive power output of the switched capacitor SCB at the electrical node i, Indicates the reactive power output corresponding to a single gear of the switched capacitor SCB. Indicates the capacitor level that electrical node i decides to switch on at time t, They represent the active output, reactive output and capacity of the photovoltaic PV at the electrical node i at the tth moment, It represents the upper limit of the active output of the photovoltaic PV at the electrical node i at the tth moment, min represents the minimum value, and max represents the maximum value.

[0071] Specifically, in step S2, the distribution network state of the long-time scale equipment is obtained by using the DQN agent:

[0072]

[0073] in, represents the distribution network state of the long-time scale equipment obtained by the DQN agent at time t, P slack,t , Q slack,t They represent the active power and reactive power of the slack node at the tth moment, P load,t , Q load,t They represent the active power and reactive power of the loads of the electrical nodes except the relaxed nodes at the tth moment, P′ SCB,t , Q′ SCB,t They respectively represent the active output and reactive output of the switched capacitor SCB at the t moment.

[0074] Among them, the slack node refers to the electrical node connected to the upper-level distribution network.

[0075] Specifically, in step S2, the distribution network status of short-time scale equipment is obtained by using DDPG agent:

[0076]

[0077] in, represents the distribution network state of the short-time scale equipment obtained by the DDPG agent at time t, P′ PV,t , Q′ PV,t They represent the active and reactive output of photovoltaic PV at the tth moment, Indicates the maximum active output of photovoltaic PV.

[0078] S3. Set the switching decision time of long-time scale devices and short-time scale devices, and according to the distribution network status of long-time scale devices and the distribution network status of short-time scale devices, obtain the action decision of the DQN agent on the long-time scale devices and the action decision of the DDPG agent on the short-time scale devices at their corresponding switching decision times.

[0079] In this embodiment, the reactive power regulation equipment that needs to be decided is judged according to time, and the action decision of the reactive power regulation equipment is made by inputting the distribution network status information into the neural network of the corresponding intelligent agent; wherein, the DQN intelligent agent is set to make a switching decision every 6 hours to optimize the voltage over-limit, and the DDPG intelligent agent makes a photovoltaic reactive power processing decision every 15 minutes to reduce voltage deviation and suppress voltage fluctuation.

[0080] In addition, the switching decision time is set according to the action time of the equipment, that is, the long-time scale equipment is a discrete switching equipment, which is generally set to make decisions every 6 hours; the short-time scale equipment is photovoltaic, which is generally set to make decisions every 15 minutes.

[0081] Specifically, in step S3, the action decision of the DQN agent for the long-time scale device is:

[0082]

[0083] in, Represents the action decision of the DQN agent on the long-time scale device at time t.

[0084] Specifically, the action decision of the DDPG agent on the short-time scale device in step S3 is:

[0085]

[0086] in, Represents the action decision of the DDPG agent on the short-time-scale device at time t.

[0087] S4. According to the voltage over-limit and voltage offset conditions of the distribution network, feedback is provided to the action decisions of the long-time scale devices and the short-time scale devices, and the distribution network feedback of the DQN agent to the long-time scale devices and the distribution network feedback of the DDPG agent to the short-time scale devices are obtained respectively.

[0088] In this embodiment, according to the voltage over-limit and voltage offset of the distribution network, the action reward of the intelligent agent is fed back to the intelligent agent for updating the neural network, that is, updating the DQN network and the DDPG network, where the DQN network refers to the deep Q neural network, and the DDPG network refers to the action neural network and the value evaluation neural network.

[0089] Specifically, in step S4, the DQN agent's feedback to the distribution network of the long-time scale equipment is:

[0090]

[0091] in, It represents the distribution network feedback of the DQN agent to the long-time scale equipment at the tth moment, and N represents the total number of electrical nodes in the distribution network.

[0092] In this embodiment, the distribution network feedback of the long-time scale device is the action reward received by the DQN agent.

[0093] Specifically, in step S4, the DDPG agent's feedback to the distribution network of the short-time scale equipment is:

[0094]

[0095] in, represents the distribution network feedback of the DDPG agent to the short-time scale equipment at time t, v i,base Represents the nominal voltage at electrical node i.

[0096] In this embodiment, the distribution network feedback of the short-time scale device is the action reward received by the DDPG agent.

[0097] S5. At the switching decision time of the long-time scale equipment, the network parameters of the DQN network are updated according to the distribution network status, action decision and distribution network feedback of the long-time scale equipment to obtain a trained DQN network, i.e., the optimal DQN agent, which is used to output the optimal action decision of the long-time scale equipment.

[0098] In this embodiment, when performing long-time-scale voltage optimization Markov decision, the DQN agent inputs the action into the distribution network environment. The environment records the distribution network state, action decision and distribution network feedback of the long-time-scale equipment under this action, and transmits them to the experience replay pool for storage, and transfers to the next state at the same time; and each time the neural network parameters are updated, random sampling is performed from the experience replay pool, and stochastic gradient descent is used to update the agent network according to the total loss function, so that it can make better action decisions.

[0099] Specifically, step S5 includes S51-S52:

[0100] S51. At the switching decision time of the long-time scale device, the distribution network state, action decision and distribution network feedback of the long-time scale device are input into the distribution network environment, and transferred to the next state at the same time.

[0101] S52. When updating the network parameters of the DQN network, the distribution network state, action decision and distribution network feedback of the long-time scale device are input into the DQN network, and the total loss function of the DQN network is updated by the gradient descent method. When the number of training times reaches the maximum value, a trained DQN network is obtained, that is, the optimal DQN agent, which is used to output the optimal action decision of the long-time scale device.

[0102] Among them, the total loss function of the DQN network is:

[0103]

[0104] Where L represents the total loss function of the DQN network, θ represents the network parameters, and θ t represents the network parameters of the DQN network at time t, θ t+1 represents the network parameters of the DQN network at time t+1, E represents the mean calculation, r DQN represents the distribution network feedback of long-time scale devices in the DQN network, γ DQN represents the reduction factor of the distribution network feedback of the long-time scale equipment in the DQN network, Q DQNrepresents the value evaluation value of the DQN network, a represents the action decision of the long-time scale device, s represents the distribution network state of the long-time scale device, s′ represents the distribution network state after adopting the action decision a under the distribution network state s of the long-time scale device, a′ represents the action decision of the distribution network state s′, α represents the learning rate, Represents the gradient.

[0105] S6. At the switching decision time of the short-time scale equipment, the network parameters of the DDPG network are updated according to the distribution network status, action decision and distribution network feedback of the short-time scale equipment to obtain a trained DDPG network, that is, the optimal DDPG agent, which is used to output the optimal action decision of the short-time scale equipment.

[0106] In this embodiment, when performing short-time-scale voltage optimization Markov decision-making, the DDPG agent maintains two neural networks, namely the action neural network and the value evaluation neural network, which respectively perform action output and action evaluation; when the action neural network outputs the agent action to the environment, the environment records the distribution network state, action decision and distribution network feedback under this action, and transmits it to the experience replay pool for storage, and transfers it to the next state at the same time; at the same time, when the DDPG agent is updated, the value evaluation neural network is updated first according to the data in the experience replay pool, and guides the action neural network to update, so that the DDPG agent outputs better action decisions.

[0107] Specifically, step S6 specifically includes S61-S63:

[0108] S61. At the switching decision time of the short-time scale device, the distribution network state, action decision and distribution network feedback of the short-time scale device are input into the distribution network environment, and transferred to the next state at the same time.

[0109] S62. Input the action decision of the current state into the value evaluation neural network of the DDPG network, and use the gradient descent method to update the total loss function of the value evaluation neural network to obtain the updated network parameters of the value evaluation neural network.

[0110] S63. Update the action neural network of the DDPG network according to the updated network parameters of the value evaluation neural network, input the distribution network status, action decision and distribution network feedback of the short-time scale equipment into the action neural network, and use the gradient descent method to update the total loss function of the action neural network. When the number of training times reaches the maximum value, the trained action neural network and value evaluation neural network, i.e., the optimal DDPG agent, are obtained, which are used to output the optimal action decision of the short-time scale equipment.

[0111] Among them, the total loss function of the value assessment neural network is:

[0112]

[0113] The total loss function of the action neural network is:

[0114]

[0115] Where L′ represents the total loss function of the value assessment neural network, θ Q1 represents the network parameters of the value assessment neural network Q1, represents the network parameters of the value assessment neural network Q1 at time t-1, represents the network parameters of the value assessment neural network Q1 at the tth moment, r DDPG represents the distribution network feedback of short-time-scale devices in the DDPG network, γ DDPG represents the reduction factor of the distribution network feedback of short-time scale devices in the DDPG network, Q Q1 represents the value evaluation value of the value evaluation neural network Q1, s1 represents the distribution network state of the short-time scale device, a1 represents the action decision of the short-time scale device, s1′ represents the distribution network state after the action decision a1 is adopted under the distribution network state s1 of the short-time scale device, a1′ represents the action decision of the distribution network state s1′, α Q1 represents the learning rate of the value evaluation neural network Q1, J represents the total loss function of the action neural network, θ π represents the network parameters of the action neural network π, represents the network parameters of the action neural network π at time t-1, represents the network parameters of the action neural network π at the tth moment, α π represents the learning rate of the action neural network π.

[0116] like Figure 3 As shown, Figure 3 The reward convergence effect of agent training is shown. It can be seen that during the training process, the DQN agent and the DDPG agent cooperate with each other to make action decisions for reactive power regulation equipment in the distribution network, and the control objectives both include voltage optimization, so the rewards of the two agents have the same convergence trend; at the same time, between rounds 0-60, since the agents are in the exploration stage, there is no obvious convergence trend. However, after 70 rounds of training, the rewards of the agents tend to converge.

[0117] S7. Put the optimal DQN agent and the optimal DDPG agent into the distribution network to obtain the optimal action decision of the reactive power regulation equipment.

[0118] In this embodiment, it is determined whether the simulation operation time of the distribution network has reached the set training days requirement. If it has reached the requirement, the agent is switched to the test mode and put into the distribution network to receive the distribution network status and directly make reactive power regulation equipment action decisions. Specifically, one day is randomly selected to test the two trained agents (i.e., the optimal DQN agent and the optimal DDPG agent). The agent makes reactive power regulation equipment action decisions by receiving the load and voltage conditions of the distribution network; Figure 4-Figure 5 As shown, Figure 4 and Figure 5 The action decisions of discrete capacitors and photovoltaics made by the DQN agent and the DDPG agent. As can be seen from the figure, during the peak load stage, the voltage of each electrical node drops significantly. Therefore, the DQN agent increases the gear of the discrete capacitor to enhance the reactive support to the power grid. The DDPG agent effectively raises the voltage to prevent over-limit by increasing the reactive output of photovoltaics, and effectively reduces the voltage deviation.

[0119] In addition, if Figure 6 As shown, Figure 6 The box plots of all electrical nodes before and after voltage optimization on the test day; before optimization, there were voltage over-limit situations at electrical nodes from 6:00 to 20:00, and at 17:00, electrical node 18 had serious voltage over-limit, with the lowest voltage being 0.89; however, after voltage optimization was performed using a distribution network multi-time scale voltage optimization method based on deep reinforcement learning proposed in the present invention, the distribution network did not experience voltage over-limit situations throughout the day, with the highest voltage being 1.02 and the lowest being 0.96. At the same time, the terminal node 33 was selected to track its voltage changes throughout the day, as shown in Figure 7 As shown by Figure 7 It can be seen that this method raises the minimum voltage of the terminal node from 0.95 to 0.98, improving the power quality. Secondly, the voltage exceeding the limit seriously at 14:00 was selected, and the voltage of 33 nodes before and after optimization was measured, as shown in Figure 2. Figure 8 As shown in the figure, the intelligent agent increases the reactive output of the voltage regulating equipment near the over-limit nodes, such as nodes 16, 24, and 25, thereby effectively raising the voltage of the over-limit nodes, reducing the overall voltage deviation of the power grid, and improving the power quality.

[0120] In summary, the multi-time-scale voltage optimization method for distribution network based on deep reinforcement learning proposed in the present invention considers the regulation characteristics of different reactive devices, based on deep learning, and realizes the coordinated output of equipment actions at different time scales through the mutual collaborative decision-making of two intelligent agents, effectively avoids voltage over-limit, and improves the power quality of terminal nodes. Specifically: (1) The double-layer deep learning method of DQN and DDPG is adopted to effectively learn the reactive regulation strategy of distribution network, and the corresponding voltage optimization goal can be achieved without the accurate physical model of distribution network; (2) On the long time scale, the voltage can be effectively improved to prevent the voltage from over-limit; (3) On the short time scale, the voltage fluctuation range is also controlled, which effectively optimizes the voltage deviation and improves the power quality.

[0121] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0122] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A multi-time scale voltage optimization method for distribution network based on deep reinforcement learning, characterized in that: The following steps are involved: S1. According to the equipment regulation characteristics, the reactive regulation equipment involved in the voltage optimization of the distribution network is divided into long-time scale equipment and short-time scale equipment. At the same time, DQN agents and DDPG agents are set to control the reactive output of long-time scale equipment and short-time scale equipment respectively. S2. Establish the constraints of each electrical node in the distribution network, and use the DQN agent to obtain the distribution network status of long-time scale equipment, and use the DDPG agent to obtain the distribution network status of short-time scale equipment; S3, setting the switching decision time of the long time scale equipment and the short time scale equipment, and according to the distribution network status of the long time scale equipment and the distribution network status of the short time scale equipment, obtaining the action decision of the DQN agent on the long time scale equipment and the action decision of the DDPG agent on the short time scale equipment at their respective corresponding switching decision times; S4. According to the voltage over-limit and voltage offset conditions of the distribution network, feedback is given to the action decisions of the long-time scale equipment and the action decisions of the short-time scale equipment, and the distribution network feedback of the DQN agent to the long-time scale equipment and the distribution network feedback of the DDPG agent to the short-time scale equipment are obtained respectively; S5. At the switching decision time of the long-time scale equipment, the network parameters of the DQN network are updated according to the distribution network state, action decision and distribution network feedback of the long-time scale equipment to obtain a trained DQN network, i.e., the optimal DQN agent, which is used to output the optimal action decision of the long-time scale equipment; S6. At the switching decision time of the short-time scale equipment, the network parameters of the DDPG network are updated according to the distribution network state, action decision and distribution network feedback of the short-time scale equipment to obtain a trained DDPG network, i.e., the optimal DDPG agent, which is used to output the optimal action decision of the short-time scale equipment; S7. Put the optimal DQN agent and the optimal DDPG agent into the distribution network to obtain the optimal action decision of the reactive power regulation equipment.

2. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 1 is characterized in that: The constraints of each electrical node in the distribution network in step S2 are: Among them, P i , Q i They represent the active power and reactive power of electrical node i, n represents the total number of electrical nodes connected to electrical node i, v i 、v j Respectively represent the voltage of electrical node i and electrical node j, G ij represents the conductance of the line from electrical node i to electrical node j, B ij represents the susceptance of the line from electrical node i to electrical node j, cos represents the cosine function, sin represents the sine function, δ ij Represents the voltage phase difference between the two ends of the line from electrical node i to electrical node j, v i,min 、v i,max They represent the lower and upper limits of the voltage of the electrical node i, respectively. v(i,t) represents the voltage of the electrical node i at the tth moment. represents the reactive power output of the switched capacitor SCB at the electrical node i, Indicates the reactive power output corresponding to a single gear of the switched capacitor SCB. Indicates the capacitor level that electrical node i decides to switch on at time t, They represent the active output, reactive output and capacity of the photovoltaic PV at the electrical node i at the tth moment, It represents the upper limit of the active output of the photovoltaic PV at the electrical node i at the tth moment, min represents the minimum value, and max represents the maximum value.

3. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 2 is characterized in that: In step S2, the distribution network state of the long-term equipment is obtained by using the DQN agent: in, represents the distribution network state of the long-time scale equipment obtained by the DQN agent at time t, P slack,t , Q slack,t They represent the active power and reactive power of the slack node at the tth moment, P load,t , Q load,t They represent the active power and reactive power of the loads of the electrical nodes except the relaxed nodes at the tth moment, P′ SCB,t , Q′ SCB,t They represent the active output and reactive output of the switched capacitor SCB at the tth moment respectively; Among them, the slack node refers to the electrical node connected to the upper-level distribution network.

4. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 3 is characterized in that: In step S2, the distribution network status of short-time scale equipment is obtained by using DDPG agent: in, represents the distribution network state of the short-time scale equipment obtained by the DDPG agent at time t, P′ PV,t , Q′ PV,t They represent the active and reactive output of photovoltaic PV at the tth moment, Indicates the maximum active output of photovoltaic PV.

5. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 4 is characterized in that: In step S3, the action decision of the DQN agent for the long-time scale device is: in, Represents the action decision of the DQN agent on the long-time scale device at time t.

6. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 5 is characterized in that: In step S3, the DDPG agent's action decision for the short-time scale device is: in, Represents the action decision of the DDPG agent on the short-time-scale device at time t.

7. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 6 is characterized in that: In step S4, the DQN agent's feedback to the distribution network of the long-time scale equipment is: in, It represents the distribution network feedback of the DQN agent to the long-time scale equipment at the tth moment, and N represents the total number of electrical nodes in the distribution network.

8. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 7 is characterized in that: In step S4, the DDPG agent's feedback to the distribution network of the short-time scale equipment is: in, represents the distribution network feedback of the DDPG agent to the short-time scale equipment at time t, v i,base Represents the nominal voltage at electrical node i.

9. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 8, characterized in that: Step S5 specifically includes: S51, at the switching decision time of the long-time scale device, input the distribution network state, action decision and distribution network feedback of the long-time scale device into the distribution network environment, and transfer to the next state at the same time; S52, when updating the network parameters of the DQN network, the distribution network state, action decision and distribution network feedback of the long-time scale device are input into the DQN network, and the total loss function of the DQN network is updated by the gradient descent method. When the number of training times reaches the maximum value, a trained DQN network is obtained, that is, the optimal DQN agent, which is used to output the optimal action decision of the long-time scale device; Among them, the total loss function of the DQN network is: Where L represents the total loss function of the DQN network, θ represents the network parameters, and θ t represents the network parameters of the DQN network at time t, θ t+1 represents the network parameters of the DQN network at time t+1, E represents the mean calculation, r DQN represents the distribution network feedback of long-time scale devices in the DQN network, γ DQN represents the reduction factor of the distribution network feedback of the long-time scale equipment in the DQN network, Q DQN represents the value evaluation value of the DQN network, a represents the action decision of the long-time scale device, s represents the distribution network state of the long-time scale device, s′ represents the distribution network state after adopting the action decision a under the distribution network state s of the long-time scale device, a′ represents the action decision of the distribution network state s′, α represents the learning rate, Represents the gradient.

10. The method for multi-time-scale voltage optimization of distribution network based on deep reinforcement learning according to claim 9, characterized in that: Step S6 specifically includes: S61, at the switching decision time of the short-time scale device, the distribution network state, action decision and distribution network feedback of the short-time scale device are input into the distribution network environment, and transferred to the next state at the same time; S62, inputting the action decision of the current state into the value evaluation neural network of the DDPG network, and updating the total loss function of the value evaluation neural network using the gradient descent method to obtain the updated network parameters of the value evaluation neural network; S63, according to the updated network parameters of the value evaluation neural network, the action neural network of the DDPG network is updated, by inputting the distribution network state, action decision and distribution network feedback of the short-time scale device into the action neural network, and using the gradient descent method to update the total loss function of the action neural network. When the number of training times reaches the maximum value, the trained action neural network and value evaluation neural network, i.e., the optimal DDPG agent, are obtained to output the optimal action decision of the short-time scale device; Among them, the total loss function of the value evaluation neural network is: The total loss function of the action neural network is: Where L′ represents the total loss function of the value assessment neural network, θ Q1 represents the network parameters of the value assessment neural network Q1, represents the network parameters of the value assessment neural network Q1 at time t-1, represents the network parameters of the value assessment neural network Q1 at the tth moment, r DDPG represents the distribution network feedback of short-time-scale devices in the DDPG network, γ DDPG represents the reduction factor of the distribution network feedback of short-time scale devices in the DDPG network, Q Q1 represents the value evaluation value of the value evaluation neural network Q1, s1 represents the distribution network state of the short-time scale device, a1 represents the action decision of the short-time scale device, s1′ represents the distribution network state after the action decision a1 is adopted under the distribution network state s1 of the short-time scale device, a1′ represents the action decision of the distribution network state s1′, α Q1 represents the learning rate of the value evaluation neural network Q1, J represents the total loss function of the action neural network, θ π represents the network parameters of the action neural network π, represents the network parameters of the action neural network π at time t-1, represents the network parameters of the action neural network π at the tth moment, α π represents the learning rate of the action neural network π.

Images