Two-stage reactive power and voltage optimization method for distribution network based on double time scale

Abstract

The application provides a two-stage reactive power and voltage optimization method for a power distribution network based on a double time scale, adopts historical power generation data of a photovoltaic power station, fully considers the uncertainty of day-ahead prediction of distributed photovoltaics in the power distribution network, and establishes a charging station electric vehicle cluster participation power grid reactive power optimization scheduling model based on a Monte Carlo algorithm, and establishes a two-stage optimization model of electric vehicles participating in reactive power optimization of the power distribution network based on various power flow constraint conditions of the power distribution network operation and constraint conditions of the capacitor bank and electric vehicle power, formulates a switching plan of the reactive power compensation capacitor bank in the power distribution network in a long time scale, and formulates the output of the electric vehicle connected to the power grid in a short time scale; a deep network is added to the DQN and DDPG agent, samples in a dangerous state are trained, and the agent is limited to explore in the direction of the dangerous state, so as to solve the voltage out-of-limit problem caused by the grid connection of a large number of distributed power sources and electric vehicles.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power dispatching, and in particular to a double-time-scale reactive power and voltage optimization method for distribution networks based on security reinforcement learning. BACKGROUND

[0002] With a large number of distributed power sources and electric vehicles connected to the distribution network, the uncertainty, dynamics and randomness in the operation of the power grid are becoming more and more obvious, bringing unprecedented challenges to the formulation and optimization of real-time control decisions of the power system.

[0003] At the same time, electric vehicles are also a flexible power resource that can be charged and discharged, and if they are guided to cooperate with distributed power sources and grid loads and timely do reactive power consumption or reactive power compensation, the load peak-valley difference can be eliminated and the voltage quality in the distribution network can be improved.

[0004] Most of the current reactive power optimization methods rely on dispatching models, so the accurate parameters of the power grid need to be known, and the original deep reinforcement learning may make decisions that exceed the voltage limit due to random exploration. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a double-time-scale two-stage reactive power and voltage optimization method for distribution networks, which considers the uncertainty of the day-ahead prediction of distributed photovoltaic power in the distribution network, formulates the switching plan of the reactive power compensation capacitor bank in the distribution network on a long time scale, and formulates the output of the electric vehicles connected to the power grid on a short time scale. A deep network is added to the original deep reinforcement learning network to train samples in a dangerous state and limit the exploration of the agent in the direction of the dangerous state, so as to solve the voltage limit problem caused by the connection of a large number of distributed power sources and electric vehicles.

[0006] The technical scheme of the present application is: a double-time-scale two-stage reactive power and voltage optimization method for distribution networks, comprising the following steps:

[0007] S1), using the historical power generation data of photovoltaic power stations as a training set;

[0008] S2), training a GANs generation model based on Wasserstein distance using the training set collected in step S1);

[0009] S3), generating scenarios using the trained GANS generation model to obtain a photovoltaic output scenario set;

[0010] S4), establishing a charging station electric vehicle cluster participation power grid reactive power optimization dispatching model based on the Monte Carlo algorithm;

[0011] S5), based on various power flow constraints of power distribution network operation and constraints of capacitor banks and electric vehicle power, a two-stage optimization model of electric vehicle participation in power distribution network reactive power optimization is established;

[0012] S6), a training environment for the two-stage optimization model is constructed, and the DQN agent and the DDPG agent are interactively trained with the two-stage optimization model, and historical dangerous state samples are input to the agents to limit the exploration of the agents to dangerous states and avoid voltage out-of-limit conditions;

[0013] S7), a double-time-scale decision time is set, and in each long time scale, the DQN agent gives corresponding actions based on the environment perception of the two-stage optimization model to obtain the capacitor bank switching plan;

[0014] S8), in a long time scale, based on the previous capacitor bank switching plan, the DDPG agent gives corresponding actions according to the two-stage optimization model environment for each short time scale to regulate the charging and discharging power of electric vehicles;

[0015] S9), after decision-making for all time scales in a day, the optimized reactive compensation capacitor bank switching plan and electric vehicle charging and discharging regulation scheme are obtained.

[0016] Preferably, in step S2), the GANs generation model based on Wasserstein distance has a generator network and a discriminator network, wherein the weights of the generator network and the discriminator network are θ (G) and θ (D) , respectively.

[0017] Preferably, in step S2), during the training process of the GANs generation model based on Wasserstein distance, the generator network is trained to accept a batch of inputs and output scenes through a series of upsampling operations by different functional neurons.

[0018] Preferably, in step S2), during the training process of the GANs generation model based on Wasserstein distance, the discriminator network obtains input samples from real historical power generation data or the generator network, and outputs a continuous value p real to measure the degree to which the input sample belongs to the real distribution P X , and the expression of the discriminator network is:

[0019] p real =D(x;θ (D) ).

[0020] As preferred, in step S2), during the training process, a batch of samples drawn from the distribution P Z is input into the generator network, while a batch of real historical samples is input into the discriminator network, a smaller L G indicates that the generated samples are more real in the perspective of the discriminator network, i.e., the generated scene looks like a historical scene. Similarly, a smaller L D indicates that the discriminator network is good at distinguishing the generated scene and the historical scene, which reflects that there is a large difference between the probability distribution P G of the data samples generated by the generator network and the probability distribution P X of the real observation data; wherein the loss functions of the generator network and the discriminator network are respectively:

[0021] L G =-E Z [D(G(Z))];

[0022] L D =-E X [D(X)]+E Z [D(G(Z))];

[0023] wherein L G and L D are the loss functions of the generator network and the discriminator network respectively, E Z denotes taking the expectation over all possible noise vectors Z, G(Z) denotes the data generated by the generator network according to the input noise vector Z, D() denotes that the discriminator network outputs a probability between 0 and 1 according to the input data, indicating whether the data is real data or data generated by the generator network; X denotes real data, and E X denotes taking the expectation value over all real data X.

[0024] As preferred, in step S2), in order to enable the generator network G and the discriminator network D to be trained simultaneously, a value function V(G, D) is constructed, and the training target of the GANs generation model is a two-person minimax game, i.e.:

[0025]

[0026] V(G,D)=-L D ;

[0027] Consider two random variables X and Y, whose marginal distributions are f X and f Y , let Γ represent the set of all possible joint distributions with marginal distributions f X and f Y ; the Wasserstein distance between the two random variables X and Y is defined as:

[0028]

[0029] where W(X, Y) represents the Wasserstein distance between two random variables X and Y, Γ represents all possible joint probability distributions, inf represents finding the minimum of the integral in Γ XY ; x, y represents the specific value of random variables X and Y, f XY represents the joint probability distribution of X and Y.

[0030] The training is continuously iterated, when the Wasserstein distance is close to 0, the empirical distribution of the generated scene is very close to the empirical distribution of the training scene, the generated scene has the same statistical characteristics as the historical data, and the diversity of pattern changes can also be well captured in the generated scene.

[0031] As preferred, in step S4), the expression of the charging station electric vehicle cluster participating in the reactive power optimization scheduling model of the power grid is:

[0032]

[0033] where E EV,t , E EV,t-1 represent the total adjustable charging capacity of the charging station electric vehicle cluster at t, t-1 time, and respectively represent the charging power and discharging power of the charging station electric vehicle cluster at t time, η c and η d respectively represent the charging efficiency and discharging efficiency of the electric vehicle, Δt represents the time step, i ∈ EV represents the i-th vehicle in the electric vehicle cluster, c i,t and d i,t are Boolean variables representing the charging state and discharging state of the i-th vehicle at t time, and respectively represent the charging power and discharging power of each electric vehicle in the electric vehicle cluster, SOC EV represents the state of charge of the electric vehicle, is the minimum state of charge, is the maximum state of charge.

[0034] As preferred, in step S5), the two-stage optimization model includes optimization in a long time scale and optimization in a short time scale.

[0035] As preferred, in step S5), the optimization in the long time scale takes the minimum total network loss of the distribution network and the minimum voltage deviation of the pivotal node in the distribution network as the optimization target:

[0036]

[0037] wherein F loss is the objective function of network loss, n is the number of branches in the distribution network, G i,j is the conductance between nodes i, j, U i , U j are the voltage amplitudes of nodes i, j, respectively, θ i,j is the voltage phase angle difference between nodes i, j, F ΔU is the objective function of voltage deviation, U i,N is the rated voltage of node i, U i,max , U i,min are the upper and lower voltage limits that node i can withstand, respectively, and N is the number of hub nodes of the distribution network.

[0038] The above objective function is minimized by adjusting the switching plan of the reactive compensation capacitor bank to achieve the optimal operating state.

[0039] As a preferred, in step S5), the optimization objective in the short time scale has three, respectively: the minimum voltage deviation of the charging station node at the current time, the maximum charging power when the electric vehicles participating in the regulation in the cluster still have charging demand, and the minimum average deviation of all node voltages in the distribution network at the current time; the objective function is:

[0040]

[0041] wherein t represents the sampling time of the short time scale, m represents the number of charging station nodes that can participate in voltage regulation; μ 1,i , μ 2,i , μ3 are weight coefficients; U p,i (t) is the voltage of the i-th participating regulation charging station node accessed at time t, N i (t) is the total number of electric vehicles accessed by the i-th charging station at time t; P EV,n,i is the charging active power of the n-th electric vehicle of the i-th charging station; P EV,st,i is the rated charging active power of the i-th charging station; U j (t) is the voltage of the j-th node at time t; S is the number of nodes of the distribution network.

[0042] As preferred, in step S6), a two-stage optimization model is constructed using the distribution network node voltage, node injected power, photovoltaic output scene generated by GANs, charging data of the electric vehicle cluster of the charging station, and a long-time scale training environment; and a deep network is added to the DQN agent; the deep network is used to train the dangerous samples in the dangerous state; a safe DQN agent is obtained, which outputs the current long-time scale capacitor bank switching plan according to the perceived distribution network environment, and transmits the switching plan to each short-time scale under the current long-time scale.

[0043] As preferred, in step S6), a short-time scale training environment is constructed using the distribution network node voltage, node injected power, photovoltaic output scene set generated by GANs, charging data of the electric vehicle cluster of the charging station, and long-time scale capacitor bank switching plan, and a deep network is added to the DDPG agent, the deep network is used to train the dangerous samples in the dangerous state, and a safe DDPG agent is obtained, which can effectively avoid the occurrence of voltage out-of-limit, and outputs the charging power of each electric vehicle in the electric vehicle cluster of the charging station in each short-time scale under the current long-time scale according to the perceived distribution network environment and the capacitor bank switching plan, to obtain an electric vehicle charging regulation scheme.

[0044] As preferred, in step S6), the electric vehicle charging regulation scheme of the last short-time scale under the current long-time scale is transmitted to the next long-time scale, and the DQN agent gives the capacitor bank switching plan of the next long-time scale according to the distribution network state and the electric vehicle charging regulation scheme.

[0045] The beneficial effects of the present application are:

[0046] 1、The present application proposes a distribution network double-time scale reactive voltage optimization method based on safe reinforcement learning under the premise of considering the requirements of safe operation of the distribution network, the uncertainty of day-ahead photovoltaic power generation prediction and the charging demand of electric vehicle users, which can make a reasonable reactive compensation capacitor bank switching plan and regulate the charging power of the electric vehicle cluster of the charging station, avoid the occurrence of voltage out-of-limit, realize the consumption of photovoltaic power, and solve the voltage out-of-limit problem caused by the large number of distributed power and electric vehicles connected to the grid, which is of great significance to improve the safety and stability of the operation of the distribution network.

[0047] 2. The application considers the uncertainty of distributed photovoltaic day-ahead prediction in the power distribution network, formulates the switching plan of the reactive compensation capacitor bank in the distribution network on a long time scale, formulates the output of the electric vehicle connected to the power grid on a short time scale, adds a deep network to the original deep reinforcement learning network, trains the samples in the dangerous state, and limits the exploration of the agent to the dangerous state direction, so as to solve the voltage out-of-limit problem caused by the connection of a large number of distributed power sources and electric vehicles. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 A flowchart of the steps of the method of the application;

[0049] Figure 2 A schematic diagram of the framework of the method of the application;

[0050] Figure 3 A schematic diagram of the reinforcement learning environment of the method of the application. DETAILED DESCRIPTION

[0051] The specific embodiments of the application will be further described below in conjunction with the accompanying drawings:

[0052] Example 1

[0053] As shown in Figure 1 and 2 , the embodiment provides a two-stage reactive voltage optimization method for power distribution network based on double time scale, which comprises the following steps:

[0054] S1), using the historical power generation data of the photovoltaic power station as the training set;

[0055] S2), training the GANs generation model based on Wasserstein distance using the training set collected in step S1); specifically:

[0056] The GANs generation model based on Wasserstein distance has a generator network and a discriminator network, wherein the weights of the generator network and the discriminator network are θ (G) and θ (D) ; and setting G represents the generation function with θ (G as the weight, and setting D represents the generation function with θ (D) as the weight.

[0057] In the training process of the GANs generation model based on Wasserstein distance of the embodiment, the generator network G is trained to accept a batch of inputs and output scenes through a series of upsampling operations of different functional neurons. The discriminator network D obtains input samples from the real historical power generation data or the generator network, and outputs a continuous value p through a series of downsampling operations using another deep neural network.real , to measure the degree to which the input sample belongs to the real distribution P X , the expression of the discriminator network is:

[0058] p real = D(x; 0 (D) ).

[0059] In the training process of the embodiment, a batch of samples drawn from the distribution P Z is input into the generator network, while a batch of real historical samples is input into the discriminator network. A smaller L G indicates that the generated sample is more real in the perspective of the discriminator network, i.e., the generated scene looks like a historical scene. Similarly, a smaller L D indicates that the discriminator network is good at distinguishing between the generated scene and the historical scene, which reflects that there is a large difference between the probability distribution P G of the data sample generated by the generator network and the probability distribution P X of the real observed data; wherein the loss functions of the generator network and the discriminator network are respectively:

[0060] L G = -E Z [D(G(Z))]; and

[0061] L D = -E X [D(X)] + E Z [D(G(Z))]; and

[0062] In the formula, L G and L D are the loss functions of the generator network and the discriminator network respectively, E Z denotes the expectation over all possible noise vectors Z, G(Z) denotes the data generated by the generator network according to the input noise vector Z, D() denotes the output of the discriminator network according to the input data, which is a probability between 0 and 1, indicating whether the data is real data or data generated by the generator network; X denotes real data, and E X denotes the expectation over all real data X.

[0063] In the embodiment, in order to train the generator network G and the discriminator network D simultaneously, a value function V(G, D) is constructed, and the training target of the GANs generation model is a two-person minimax game, i.e.:

[0064]

[0065] V(G, D) = -L D ;

[0066] Consider two random variables X and Y, whose marginal distributions are f X and f Y , let Γ represent the set of all possible joint distributions with f X and f Y ; the Wasserstein distance between two random variables X and Y is defined as:

[0067]

[0068] Where W(X,Y) represents the Wasserstein distance between two random variables X and Y, Γ represents all possible joint probability distributions, inf represents finding the minimum f XY in Γ; x, y represent specific values of random variables X and Y, f XY represents the joint probability distribution of X and Y.

[0069] When the Wasserstein distance is close to 0, the empirical distribution of the generated scene is very close to the empirical distribution of the training scene, and the generated scene has the same statistical characteristics as the historical data, and the diversity of pattern changes can also be well captured in the generated scene.

[0070] S3), using the trained GANS generation model to generate scenes to obtain a set of photovoltaic output scenes; specifically:

[0071] This embodiment samples multiple noise vectors Z from a predefined Gaussian distribution Z~P Z ; then inputs the sampled noise vector Z into the trained generator network G, and the generator network D generates a set of photovoltaic output scenes; the empirical distribution of the generated scene is very close to the empirical distribution of the training scene, and the generated scene has the same statistical characteristics as the historical data, and the diversity of pattern changes can also be well captured in the generated scene.

[0072] S4), establish a charging station electric vehicle cluster participating in power grid reactive power optimization scheduling model based on Monte Carlo algorithm; specifically:

[0073] S41), assuming that the starting SoC and parking time length of the electric vehicle access satisfy the Weibull distribution, then the probability density function f(x, l, k) is as follows:

[0074]

[0075] Where x represents a specific value of random variable X, l represents a scale parameter, and k represents a shape parameter.

[0076] S42), assuming that the SOC distribution of the vehicle at the end of charging conforms to the generalized extreme value distribution, the probability density function is as follows:

[0077]

[0078] In the formula, x represents a specific value of the random variable X, μ represents a position parameter, σ represents a scale parameter, and ξ represents a shape parameter.

[0079] S43), the expression of the charging station electric vehicle cluster participating in the reactive power optimization scheduling model of the power grid is:

[0080]

[0081] In the formula, E EV,t , E EV,t-1 represent the total adjustable charging capacity of the charging station electric vehicle cluster at times t and t-1, and respectively represent the charging power and discharging power of the charging station electric vehicle cluster at time t, η c and η d respectively represent the charging efficiency and discharging efficiency of the electric vehicle, Δt represents a time step, i ∈ EV represents the ith vehicle in the electric vehicle cluster, c i,t and d i,t are Boolean variables representing the charging state and discharging state of the ith vehicle at time t, and respectively represent the charging power and discharging power of each electric vehicle in the electric vehicle cluster, SOC EV represents the state of charge of the electric vehicle, is the minimum state of charge, is the maximum state of charge.

[0082] S5), based on various power flow constraints of the distribution network operation and the constraints of the capacitor bank and the electric vehicle power, a two-stage optimization model of the electric vehicle participating in the reactive power optimization of the distribution network is established; specifically:

[0083] In this embodiment, the radial distribution network power flow constraints satisfying the various power flow constraints of the distribution network operation and the constraints of the capacitor bank and the electric vehicle are as follows:

[0084]

[0085] In the formula, P Gi,t and Q Gi,t are the active and reactive power of the generator at the ith node at time t; P PVi,t is the active power injected into the node i by the photovoltaic at the ith node at time t; P Bi,t and Q Bi,trespectively, are the normal active and reactive power of the i th node at time t; P EVi,t and Q EVi,t respectively, are the active power of all charging loads connected to the i th node at time t and the reactive power provided by the charger; U i,t , U j,t respectively, are the voltages of node i and node j at time t, G i,j , B i,j , θ i,j respectively, are the conductance, susceptance and voltage phase angle difference between node i and node i;

[0086] Wherein, the electric vehicle power constraint is as follows:

[0087]

[0088] In the formula, P EV,n,i is the charging active power of the n th electric vehicle of the i th charging station; Q EV,n,i is the charging reactive power of the n th electric vehicle of the i th charging station; S EV is the rated charging capacity of the electric vehicle;

[0089] The capacitor bank constraint is as follows:

[0090] 0≤N CBi ≤N CBi,max ;

[0091] In the formula, N CBi is the number of capacitor banks put into node i, N CBi,max is the upper limit of the parallel capacitor banks that can be put into node i;

[0092] The two-stage optimization model described in the embodiment includes optimization in a long time scale and optimization in a short time scale.

[0093] In the embodiment, the optimization in the long time scale takes the minimum total network loss of the distribution network and the minimum voltage deviation of the pivotal node in the distribution network as the optimization target, adjusts the switching plan of the reactive compensation capacitor bank to minimize the value of the above-mentioned objective function, and reaches the optimal operation state, wherein the expression of the objective function of the optimization in the long time scale is:

[0094]

[0095] Wherein, F loss is the objective function of the network loss, n is the branch number in the distribution network, G i,j is the conductance between node i and node j, U i , U j are the voltage amplitudes of node i and node j, respectively, θ i,jis the voltage phase angle difference between nodes i, j, F ΔU is the target function of voltage deviation, U i,N is the rated voltage of node i, U i,max , U i,min are the upper and lower voltage limits that node i can withstand, and N is the number of hub nodes of the power distribution network.

[0096] In the embodiment, the optimization objectives on the short time scale are three, and the three optimization objectives are respectively: the minimum voltage deviation of the charging station node at the current moment, the maximum charging power when the electric vehicles in the cluster still have charging demand participating in regulation and control, and the minimum average deviation of the voltage of all nodes of the power distribution network at the current moment. In the embodiment, the objective function of the optimization on the short time scale is:

[0097]

[0098] Wherein, t represents the sampling time of the short time scale, and m represents the number of charging station nodes that can participate in voltage regulation and control; μ 1,i , μ 2,i , μ3 are weight coefficients; U p,i (t) is the voltage of the i th charging station node participating in regulation and control accessed at t; N i (t) is the total number of electric vehicles accessed by the i th charging station at t; P EV,n,i is the charging active power of the n th electric vehicle of the i th charging station; P EV,st,i is the rated charging active power of the i th charging station; U j (t) is the voltage of the j th node at t; S is the number of nodes of the power distribution network.

[0099] S6), a training environment of the two-stage optimization model is constructed, the DQN agent and the DDPG agent are interactively trained with the two-stage optimization model, and historical dangerous state samples are input to the agent to limit the exploration of the agent to the dangerous state direction and avoid voltage out-of-limit conditions; Specifically:

[0100] As shown in Figure 3 , the embodiment combines the power distribution network node voltage, the node injected power, the photovoltaic output scene generated by the GANs, and the charging data of the charging station electric vehicle cluster to form an input matrix, and the historical voltage out-of-limit power distribution network node voltage and power information are used as dangerous samples.

[0101] Wherein, the charging data of the charging station electric vehicle cluster includes the arrival time of the charging station electric vehicle, the charging start SOC, the stay duration, and the expected end SOC.

[0102] The embodiment adds a deep network in the DQN agent; the added deep network is used for training the dangerous state sample decided by the DQN agent, and the training result is used as a penalty term, so as to fully utilize the dangerous state sample, limit the exploration of the agent to the dangerous state direction, and avoid the situation that the unreasonable capacitor bank switching causes the voltage out-of-limit. The embodiment uses the time difference of the action value function as the network loss function L(θ) of the DQN, and the expression is as follows:

[0103]

[0104] Wherein, is output by a target network with the same structure as the estimated value network structure for a fixed step length, r(t) is a reward value obtained according to the action a(t) of the state s(t-1) at t; γ is a discount factor, θ, θ target are respectively the parameters of the current network and the parameters of the target network, and a represents a possible action.

[0105] And the random gradient descent is used to update the parameters, that is:

[0106]

[0107] In the formula, θ t+1 , θ t are respectively the network parameters at t+1 and the network parameters at t, is the gradient of the function, and L(θ) is the network loss function of the DQN,

[0108] The added deep network is called a dangerous state detector, which is used to identify the dangerous state that the agent may encounter in the environment, and further improve the safety and robustness of the agent. The loss function L(θ1) of the added deep network is as follows:

[0109] L(θ1)=(y1-Q(s,u;θ1)) 2 ;

[0110]

[0111] Wherein, y1 is a target value of taking an action under a given state, Q is an action value function, s is a state, and u and u' are respectively a current action and a future action; θ1 and θ2 are respectively parameters of a first Q network and parameters of a second Q network; η is a regularization coefficient; φ i+1 , φ i ' +1 are respectively a state at i+1 and a next state at i+1; avg represents an average value; γ is a discount factor; and r1 is a reward obtained by the agent by executing an action in a current state.

[0112] The dangerous state detector receives state input and outputs a dangerous score to determine whether the current state is a dangerous state. The newly added deep network is trained by supervised learning using labeled dangerous state samples.

[0113] The output of the dangerous state detector is integrated into the loss function of the DQN, and the Q value estimate is adjusted by the output of the dangerous state detector. Therefore, the loss function of the DQN agent with an added deep network is as follows:

[0114]

[0115] wherein, is the expected value, which is the statistical average of all possible state-action pairs; r is the immediate reward obtained after performing action a in state s; γ is the discount factor; D(s') is the dangerous score for the next state s'; is the maximum estimated value of the Q value of all possible actions a' in the next state s', i.e. the target Q value; Q(s, a) is the Q value estimate of the current state s and action a; D(s) is the dangerous score of the current state s; λ is the regularization coefficient.

[0116] As preferred in the embodiment, the embodiment constructs a training environment in the short time scale with the distribution network node voltage, node injection power, photovoltaic output scene set generated by GANs, and capacitor bank switching plan in the long time scale, and adds a deep network in the DDPG agent. The newly added deep network is used to train dangerous samples in dangerous states to obtain a safe DDPG agent. The safe DDPG agent can effectively avoid the occurrence of voltage out-of-limit conditions. According to the perceived distribution network environment and capacitor bank switching plan, the charging power of each electric vehicle in each short time scale charging station electric vehicle cluster in the current long time scale is output, and an electric vehicle charging regulation scheme is obtained.

[0117] The newly added deep network in the embodiment is also called a dangerous state detector. The dangerous state detector receives state input and outputs a dangerous score to determine whether the current state is a dangerous state.

[0118] The Critic network in the DDPG agent is modified. When calculating the Q value, the dangerous score is introduced for adjustment. The loss function of the modified Critic network will include a dangerous score item. Therefore, the loss function of the DDPG agent with an added deep network is as follows:

[0119]

[0120] wherein, is the expected value, referring to the statistical average of all possible state-action pairs; r is the immediate reward obtained after performing action a in state s; g is the discount factor; D(s') is the danger score for the next state s'; Q'(s', a') is the Q-value estimate for action a' selected according to policy n'(s') in the next state s'; Q(s, a) is the Q-value estimate for the current state s and action a; D(s) is the danger score for the current state s; and l is the weight coefficient.

[0121] S7), set the long time scale, short time scale decision time, in each long time scale, based on the DQN agent for two-stage optimization model environment perception to give the corresponding action, get the capacitor group switching plan; Specifically:

[0122] Set the long time scale decision time, for example, 1h, set the short time scale decision time, for example, 15min, in each long time scale, the distribution network node voltage, node injection active power, node injection reactive power into a state vector, as the input of the trained DQN agent, the DQN agent makes a decision according to these variables, get the capacitor group switching plan under the current long time scale.

[0123] S8), in a long time scale, based on the previous capacitor group switching plan, use DDPG agent for each short time scale to give the corresponding action according to the two-stage optimization model environment to regulate the charging and discharging power of electric vehicles; Specifically as follows:

[0124] In a long time scale, based on the capacitor group switching plan decided by DQN, adjust the state of capacitor group in distribution network, for each short time scale in the long time scale, according to the electric vehicle arrival time, charging start SOC, expected vehicle charging end SOC and vehicle maximum charging power, monte carlo charging simulation is carried out to get the charging demand of electric vehicle cluster.

[0125] At the same time, the distribution network node voltage, node injection active power, node injection reactive power and vehicle charging demand are combined into a state vector input to the trained Actor network in DDPG, and the Actor network makes a decision according to the state variables of the distribution network to get the charging power regulation plan of the electric vehicle cluster under the current short time scale.

[0126] All short time scales in a long time scale are traversed to get the charging power regulation plan of the electric vehicle cluster in each short time scale, and the charging power regulation plan of the last short time scale is added to the state vector of the next long time scale as one of the decision bases for the capacitor group switching plan of the next long time scale.

[0127] S9), after the decision of all time scales in a day, the optimized reactive compensation capacitor bank switching plan and the electric vehicle charging and discharging regulation scheme are obtained.

[0128] The embodiment is based on the capacitor bank switching plan of all long time scales obtained in step S7) and the electric vehicle cluster charging power regulation plan of all short time scales obtained in step S8). The long time scale decision and the short time scale decision influence each other. After the decision of all time scales in a day, the optimized reactive compensation capacitor bank switching plan and the electric vehicle charging and discharging regulation scheme are obtained.

[0129] Embodiment 2

[0130] The embodiment provides a power distribution network double-time-scale reactive voltage optimization system based on security reinforcement learning, comprising:

[0131] The photovoltaic scene generation module generates a photovoltaic output scene using a trained GANS generation model based on Wasserstein distance, wherein the GANS generation model based on Wasserstein distance uses historical power generation data of a photovoltaic power station as a training set;

[0132] The electric vehicle random charging module realizes the regulation of electric vehicle charging by establishing an electric vehicle cluster charging regulation model of a charging station using a Monte Carlo algorithm;

[0133] The long time scale optimization module optimizes the decision of the long time scale by using a trained DQN agent to obtain an optimized capacitor bank switching plan, and simultaneously transmits the optimization result of the long time scale to the short time scale optimization module;

[0134] The short time scale optimization module optimizes the decision of the short time scale according to a state vector by using a trained DDPG agent to obtain an optimized electric vehicle charging regulation scheme;

[0135] The full time scale optimization module is used to return the short time scale optimization scheme to the long time scale optimization module for full time scale iteration to obtain the capacitor bank switching plan of all long time scales and the electric vehicle cluster charging regulation scheme of all short time scales.

[0136] The above embodiment and the description in the specification only illustrate the principles and the best mode of the present application. Without departing from the spirit and the scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application.

Claims

1. A two-stage reactive power and voltage optimization method for distribution network based on dual time scales, characterized in that: The steps include: Based on various power flow constraints of distribution network operation and power constraints of capacitor banks and electric vehicles, a two-stage optimization model for electric vehicles to participate in reactive power optimization of distribution network is established. Build a training environment for the two-stage optimization model, allowing DQN and DDPG agents to interactively train with the two-stage optimization model, while simultaneously feeding historical dangerous state samples into the DQN and DDPG agents. Set the decision time of dual time scales and use the DQN agent to obtain the capacitor bank switching plan in the long time scale; Based on the capacitor bank switching plan, the DDPG agent is used to control the charging and discharging power of electric vehicles on a short time scale; After making decisions at all time scales throughout the day, the optimized reactive power compensation capacitor bank switching plan and electric vehicle charging and discharging control scheme are obtained; Among them, there are three optimization goals on the short time scale, which are: minimizing the voltage deviation of the charging station node at the current moment, maximizing the charging power when the electric vehicles participating in the control in the cluster still have charging demand, and minimizing the average voltage deviation of all nodes in the distribution network at the current moment; the objective function of the optimization on the short time scale is: Where t represents the sampling time of the short time scale, m represents the number of charging station nodes connected to the network that can participate in voltage regulation, and μ 1,i 、μ 2,i , μ3 are weight coefficients; U p,i (t) is the node voltage of the i-th charging station participating in the regulation connected at time t, N i (t) is the total number of electric vehicles connected to the i-th charging station at time t; P EV,n,i is the charging active power of the nth electric vehicle at the i-th charging station; P EV,st,i is the rated charging active power of the charger at the i-th charging station; U j (t) is the voltage of the jth node at time t; S is the number of nodes in the distribution network.

2. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 1 is characterized in that: The two-stage optimization model includes optimization on a long time scale and optimization on a short time scale.

3. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 2 is characterized in that: The optimization on a long time scale aims to minimize the total network loss of the distribution network and the voltage deviation of the central node of the distribution network. The switching plan of the reactive compensation capacitor group is adjusted to minimize the value of the objective function and achieve the optimal operating state. The expression of the objective function of the optimization on a long time scale is: Among them, F loss is the objective function of network loss, n is the number of branches in the distribution network, G i,j is the conductance between nodes i and j, U i 、U j are the voltage amplitudes of nodes i and j, θ i,j is the voltage phase angle difference between nodes i and j, F ΔU is the target function of voltage deviation, U i,N is the rated voltage of node i, U i,max 、U i,min are the upper and lower voltage limits that node i can withstand, and N is the number of central nodes in the distribution network.

4. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 1 is characterized in that: The interactive training of the DQN and DDPG agents with the two-stage optimization model refers to adding a corresponding neural network to the DQN and DDPG agents, using the constructed training environment as the input of the neural network, and training dangerous samples in a dangerous state through the newly added neural network; thus, a trained DQN and DDPG agent is obtained.

5. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 4 is characterized in that: The loss functions of the DQN and DDPG agents with added deep networks are: in, is the expected value, which is the statistical average of all state-action pairs; r is the immediate reward obtained after performing action a in state s; γ is the discount factor; D(s′) is the risk score for the next state s′; is the maximum estimated Q value of all actions a' in the next state s', that is, the target Q value; Q'(s',π'(s')) is the estimated Q value of action a' selected according to strategy π'(s') in the next state s'; Q(s,a) is the estimated Q value of the current state s and action a; D(s) is the risk score of the current state s; λ is the regularization coefficient.

6. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 5 is characterized in that: The DQN agent is trained in a long-term environment using distribution network node voltage, node injection power, photovoltaic output scenarios generated by GANs, and charging data of electric vehicle clusters at charging stations; The DDPG agent is trained in a short-time-scale environment using distribution network node voltage, node injection power, a set of photovoltaic output scenarios generated by GANs, charging data of electric vehicle clusters at charging stations, and capacitor bank switching plans at long-time-scales.

7. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 6 is characterized in that: The photovoltaic output scene set generated by GANs is a scene set generated by a GANs generation model based on Wasserstein distance.

8. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 7 is characterized in that: The charging data of the electric vehicle cluster at the charging station is obtained through a reactive power optimization dispatch model of the electric vehicle cluster at the charging station participating in the grid based on the Monte Carlo algorithm. The expression of the reactive power optimization dispatch model of the electric vehicle cluster at the charging station participating in the grid is: Where, E EV,t 、E EV,t-1 represents the total chargeable capacity of the electric vehicle cluster at the charging station at time t and t-1, and They represent the charging power and discharging power of the electric vehicle cluster at the charging station at time t, respectively, η c and η d represent the charging efficiency and discharging efficiency of electric vehicles respectively, Δt represents the time step, i∈EV represents the i-th vehicle in the electric vehicle cluster, c i,t and d i,t is a Boolean variable representing the charging and discharging status of the i-th vehicle at time t. and Represents the charging power, discharging power, SOC of each electric vehicle in the electric vehicle cluster. EV Represents the state of charge of the electric vehicle, is the minimum state of charge, The maximum state of charge.

9. The two-stage reactive power and voltage optimization method for distribution network based on dual time scales according to claim 1 is characterized in that: By setting the decision time for the long time scale and the short time scale in the dual time scale, the distribution network node voltage, node injected active power, and node injected reactive power are combined into a state vector in each long time scale as the input of the trained DQN agent. The DQN agent makes decisions based on these variables to obtain the capacitor bank switching plan in the current long time scale. Over a long time scale, the capacitor bank switching plan determined by the DQN is used to adjust the state of the capacitor banks in the distribution network. For each short time scale within the long time scale, a Monte Carlo charging simulation is performed based on the electric vehicle arrival time, the starting state of charge (SOC), the desired end state of charge (SOC), and the maximum charging power of the vehicle to determine the charging demand of the electric vehicle cluster. At the same time, the distribution network node voltage, node injected active power, node injected reactive power, and vehicle charging demand are combined into a state vector and input into the Actor network trained in DDPG. The Actor network makes decisions based on the state variables of the distribution network to determine the charging power control plan for the electric vehicle cluster over the current short time scale. All short time scales under a long time scale are traversed to obtain the electric vehicle cluster charging power control plan for each short time scale, and the last short time scale charging power control plan is added to the state vector of the next long time scale as the decision for the capacitor bank switching plan for the next long time scale.

Images