Three-level converter system control method based on deep reinforcement learning

Abstract

The invention discloses a three-level converter system control method based on deep reinforcement learning, and belongs to the technical field of power electronics. The three-level converter system comprises a T-type three-level converter and a Buck-Boost converter; designing a three-level converter system controller by adopting a deep reinforcement learning algorithm: constructing an intelligent agent based on a TD3 algorithm according to the environment state of the converter system, designing a reward function, and training the intelligent agent; transplanting the trained intelligent agent model into a main control chip of the converter system, controlling the T-type three-level converter by adopting a CBPWM strategy according to the output of the intelligent agent, and controlling the Buck-Boost converter to realize neutral-point potential automatic balance at the same time. The method is high in adaptive capacity and robustness to environmental changes, can effectively cope with the influence of fluctuation and uncertainty of the voltage of the power grid on system control, realizes rapid and stable control of the three-level converter system, and can be widely applied to systems of new energy, energy storage, charging piles and the like.

Description

Technical Field

[0001] The present invention relates to a control method, in particular to a three-level converter system control method based on deep reinforcement learning, and belongs to the technical field of power electronics. Background Art

[0002] The T-type three-level converter has been applied to new energy systems such as distributed photovoltaic power generation, wind power, and energy storage systems due to its advantages such as small harmonics, low voltage stress of switching devices, low cost, and the ability to achieve bidirectional energy transmission.

[0003] However, the topology of the existing T-type three-level converter has the inherent defect of unbalanced midpoint potential, which increases the complexity of the existing control methods. At present, the commonly used control methods for T-type three-level converters are mainly PID control, model predictive control (MPC), and finite control set model predictive control (FCS-MPC). Due to the fixed parameters of PI or PID control, it is difficult to achieve the optimal control effect under different working conditions. MPC achieves the optimal performance of the converter by designing an appropriate cost function, but MPC selects the action of the bidirectional switch tube at the next moment by comparing the cost size of different switch states. Therefore, MPC needs to have a modulation function, which requires MPC to be implemented by hardware with a high computing speed. FCS-MPC has the advantages of fast dynamic response speed, simple and easy to implement control algorithm, no modulation link, and easy to achieve multi-objective and constraint synchronization processing. It is increasingly used in converter systems, but there are problems such as large algorithm calculation volume, cumbersome weight factor design, low control accuracy at low frequency, and non-fixed switching frequency.

[0004] In order to solve the inherent midpoint potential imbalance problem, existing methods include PI control method, zero-sequence component injection method, finite control set model predictive control method (FCS-MPC), finite time control method (FTC), etc. However, these methods have defects such as complex calculations and long time consumption, so their rapid response ability is poor, and they cannot effectively deal with interference to the stable operation of the system caused by light, wind speed fluctuations (and thus power fluctuations), grid voltage fluctuations, etc.

[0005] The deep reinforcement learning control algorithm effectively combines the perception ability of deep learning and the decision-making ability of reinforcement learning. It can control according to the input information and is an artificial intelligence control method that is closer to human thinking. The reward function of deep reinforcement learning (DRL) is similar to the cost function of MPC, so it has the same advantages as MPC. However, DRL control is equivalent to a special PID control whose parameters can change with the external environment and does not require modulation, so its computational cost is much lower than that of MPC. In addition, the control effect of MPC depends on accurate mathematical models and device parameters, while the optimal action of DRL control is obtained through exploration and optimization in a random and uncertain environment. Therefore, DRL control has strong robustness to both internal parameter misalignment and external environmental disturbances. As one of the most advanced training algorithms for DRL, double-delayed deep deterministic policy gradient (TD3) has been used for power electronics and motor control. The results show that TD3 control also has good control effects in uncertain environments. Therefore, it is particularly suitable for AC / DC converter systems with unknown time-varying disturbances (such as volatility and uncertainty in grid voltage), changes in their own structural parameters after long-term operation, and nonlinear and strongly coupled characteristics. However, there is currently little research on the application of deep reinforcement learning control in converter systems. Summary of the invention

[0006] The main purpose of the present invention is: to address the deficiencies in the above-mentioned prior art, the present invention provides a three-level converter system capable of automatically adjusting the midpoint potential balance and a control method thereof based on the deep reinforcement learning TD3 algorithm, so as to achieve optimal control of multiple objective tasks such as reducing the harmonic content of the AC side current, reducing the DC side voltage fluctuation, and suppressing the midpoint potential imbalance and oscillation, thereby ensuring the effective control and efficient operation of the three-level converter system.

[0007] In order to achieve the above objectives, the three-level converter system of the present invention comprises: a T-type three-level converter and a Buck-Boost DC converter; one end of the T-type three-level converter is connected to a power grid, and the other end is connected to the Buck-Boost DC converter; the other end of the Buck-Boost DC converter is connected to a DC bus.

[0008] The T-type three-level converter includes three LCL filters, three bridge arms, and two output capacitors; the bridge arm has four power switch tubes: S x1 ~S x4, x=a, b, c; one end of the LCL filter is connected to the power grid; the two output capacitors are connected in series, wherein the output capacitor connected to the positive pole of the DC side of the T-type three-level converter is called an upper capacitor, the output capacitor connected to the negative pole of the DC side of the T-type three-level converter is called a lower capacitor, and the connection point of the two output capacitors is called the midpoint O of the T-type three-level converter; the Buck-Boost DC converter includes a DC inductor L, two power switch tubes VT1, VT2 and an output capacitor C; one end of the DC inductor L is connected to the midpoint of the T-type three-level converter.

[0009] The present invention provides a three-level converter system control method based on deep reinforcement learning, comprising the following steps:

[0010] Step 1: Obtain the three-phase grid voltage e through the phase-locked loop PLL a 、e b 、e c The phase angle θ is calculated, and the phase information is provided to the Park transformation and iPark transformation of the system to ensure that the three-phase modulation wave of the T-type three-level converter is synchronized with the grid voltage;

[0011] Step 2: The points after the three-phase alternating current passes through the three LCL filters of the T-type three-level converter are respectively recorded as Ta, Tb, and Tc points, and the voltages at Ta, Tb, and Tc points are respectively recorded as u a 、u b 、u c ; The current flowing through points Ta, Tb, and Tc are respectively denoted as i a 、i b 、i c ; will u a 、u b 、u c After abc / dq transformation, we get u d and u q ;change i a 、i b 、i c After abc / dq transformation, we get i d and i q ;

[0012] Step 3: Determine the environmental state S of the three-level converter system t The environmental state includes the DC side voltage U of the T-type three-level converter. dc and its reference value U dc * ; The voltage U2 of the lower capacitor of the T-type three-level converter; The dq-axis component i of the AC side current of the T-type three-level converter d and its reference value i d* 、i q and its reference value i q * ; The dq-axis component u of the AC side voltage of the T-type three-level converter d and u q ; The DC inductor current i of the Buck-Boost DC converter L ; The environmental state is recorded as S t , then S t =[U dc U dc * U2 L i d i d * i q i q * u d u q ].

[0013] Step 4: The environmental state S of the three-level converter system determined in step 3 t Build an agent based on the deep reinforcement learning TD3 algorithm:

[0014] The agent includes an Actor network structure and a Critic network structure, which contain 6 deep neural networks, specifically:

[0015] The Actor network structure has two deep neural networks: the policy network Actor Main Network π(s|θ') and its target network Actor Target Network π ta (s|θ' ta ):

[0016] The Critic network structure has four deep neural networks: It consists of two evaluation networks for value function estimation, namely the Critic main network and its target network Critic Target network Where s and a are general representations of the environment state and the action value of the deep neural network, respectively, θ', θ' ta They are the weight parameters of the Actor network and its target network Actor Target network respectively; They are the weight parameters of the Critic main network and its target network Critic Target network respectively;

[0017] Its working process is:

[0018] The input of the Actor main network is the environment state S t, add action exploration noise N to the output of the Actor main network t Then get action A t (S t |θ'), where θ' is the weight parameter of the Actor main network, and the action exploration noise N t Obeys normal distribution; the input of the Actor Target network is the environment state S at the next moment t+1 , add the target smooth noise ε to the output of the ActorTarget network to obtain the action estimate A at the next moment t+1 (S t+1 |θ' ta ), where θ' ta is the weight parameter of the Actor Target network, and ε obeys the normal distribution.

[0019] The Critic main network is based on the input S of the current Actor main network. t and output action A t (S t |θ'), and output the value function respectively The Critic Target network is based on the input S of the Actor Target network. t+1 And output action estimate A t+1 (S t+1 |θ' ta ), output the target value function respectively

[0020] Step 5: Design reward function R t :

[0021] The reward function R t Designed for:

[0022] R t =R1+R2 (1)

[0023] Wherein, R1 is the AC side current distortion rate, DC side voltage and midpoint potential tracking reward function of the T-type three-level converter, and R2 is the AC side current and voltage fluctuation suppression, DC side voltage fluctuation suppression, and midpoint potential fluctuation and oscillation suppression reward function of the T-type three-level converter;

[0024] R1 is defined as:

[0025]

[0026] Wherein, w1 is the AC side current tracking weight coefficient of the T-type three-level converter, which is used to quickly reduce the current tracking error; w2 is the DC side voltage tracking weight coefficient of the T-type three-level converter, which is used to quickly reduce the DC side voltage tracking error; w3 is the midpoint potential tracking weight coefficient of the T-type three-level converter, which is used to quickly reduce the midpoint potential tracking error;

[0027] R2 is defined as:

[0028]

[0029] Wherein, w4 is the AC side current stability weight coefficient of the T-type three-level converter, which is used to reduce the AC side current fluctuation; w5 is the AC side voltage stability weight coefficient of the T-type three-level converter, which is used to reduce the AC side voltage fluctuation; w6 is the DC side voltage stability weight coefficient of the T-type three-level converter, which is used to reduce the DC side voltage fluctuation; w7 is the midpoint potential stability weight coefficient of the T-type three-level converter, which is used to reduce the midpoint potential fluctuation; e Udc is the DC side voltage tracking error of the T-type three-level converter, e Udc =U dc * -U dc ;e U2 is the midpoint potential tracking error of the T-type three-level converter, e U2 =U2 * -U2, let U2 * =U dc / 2;i d (tl),i q (tl),u d (tl),u q (tl), e Udc (tl), e U2 (t1) are respectively the i of the first moment before the T-type three-level converter d 、i q 、u d 、u q 、e Udc 、e U2 , K is the upper limit of the time for suppressing fluctuations, l and K are both positive integers;

[0030] Step 6: Determine action set A t (S t |θ'): The agent is based on the environment state S t and the reward function R t , directly output the optimal action A through a fast response strategy t (S t |θ'), At (S t |θ') There are four actions, namely: the dq-axis component u of the AC side voltage of the T-type three-level converter d 、u q The reference value u d * 、u q * , the dq-axis component i of the AC side current of the T-type three-level converter d The reference value i d * , and the modulation signal m of the two power switch tubes VT1 and VT2 of the Buck-Boost DC converter.

[0031] Step 7: Use the deep reinforcement learning TD3 algorithm to train the agent:

[0032] 71) Initialize the network weight parameters of the agent and set the relevant parameters for the agent training, including: initializing the weight parameters of the Actor main network and the Critic main network And the weight parameters of the corresponding target network Actor Target network and Critic Target network Set the learning rate μ of the Actor main network; set the discount factor γ of deep reinforcement learning; set the maximum number of training rounds N, the number of consecutive training rounds n, the maximum number of steps T in each training round, and the update frequency D of the target network Actor Target network and CriticTarget network; set and initialize the experience database.

[0033] 72) Based on the current environment state S t , the output of the Actor network adds action exploration noise N t , get action A t (S t |θ'):

[0034] A t (S t |θ′)=π(S t θ|′)+N t (4)

[0035] In the formula, π(S t |θ') is the output of the Actor network.

[0036] 73) After each step is executed, the next environment state S is obtained t+1 , and R is calculated according to formula (1) t , the experience gained (S t ,At ,R t ,S t+1 ) is stored in the experience database.

[0037] 74) Randomly and uniformly sample M small batches of samples (S ti ,A ti ,R ti ,S ti+1 ), that is, M sets of empirical data, where i = 1, 2, 3, …, M.

[0038] 75) Update the weight parameters of the Critic main network in a way that minimizes the error Right now

[0039]

[0040] Where M represents the size of the mini-batch sample; Y i represents the expected target value function, which is defined as:

[0041]

[0042] In the formula, γ is the discount factor; ε represents the target smooth noise, which is a truncated normal distribution noise, that is,

[0043]

[0044] In the formula, σ clip is the standard deviation of ε, and c is the cutoff factor.

[0045] 76) If the number of training steps is a multiple of the update frequency D of the Critic Target network, proceed to step 77), otherwise go to step 79).

[0046] 77) Update θ' by performing gradient ascent on the expected reward J, i.e.

[0047]

[0048] Where μ is the learning rate; the expression of J is:

[0049]

[0050] It means that J is derived with respect to θ':

[0051]

[0052] In the formula, represents the derivative of π with respect to θ'.

[0053] 78) Update the weight parameters θ' of the Actor Target network by exponential smoothing according to Equation (11) ta and the weight parameters θ' of the Critic Target network tak as follows:

[0054]

[0055] where τ represents the soft update factor, and τ ≤ 1.

[0056] 79) If the number of training steps has not reached the maximum number of steps T in each training episode, return to step 72); otherwise, the current training episode ends, and the agent model obtained in this training episode is saved, and the average reward function value R av and its change rate are calculated.

[0057] 710) If the absolute value of R av is less than or equal to R * av for n (n < N) consecutive training episodes, where R * av is a positive real number close to 0, and the change rate of R av is less than 5% for n (n < N) consecutive training episodes, then end the entire training process; otherwise, go to step 711);

[0058] 711) Determine whether the maximum number of training episodes N has been reached. If it has, end the entire training process. If not, return to step 72) to continue training in the next training episode.

[0059] Step 8: Transplant the trained agent model into the main control chip of the three-level converter system, establish an actual control system based on the main control chip of the three-level converter system, and achieve stable control of the three-level converter system, and output A t (S t |θ'), that is: the dq-axis components u d and u q of the AC-side voltage of the T-type three-level converter, the reference values u d * and u q * , the dq-axis components i d of the AC-side current of the T-type three-level converter, the reference values i d * , and the modulation signals m of the two power switches VT1 and VT2 of the Buck-Boost DC converter;

[0060] Step 9: The AC side voltage reference value u of the T-type three-level converter obtained in step 8 is d * and u q * , after dq / abc coordinate transformation, the AC side voltage u of the T-type three-level converter is obtained a 、u b 、u c The reference value u a * 、u b * 、u c * ; Then, u a * 、u b * 、u c * After modulation by the CBPWM module, the driving signal s is generated a 、s b 、s c , driving the power switch tube S of the T-type three-level converter x1 ~S x4 , x=a, b, c, controls the operation of the T-type three-level converter.

[0061] Step 10: Send the modulation signal m of the two power switch tubes VT1 and VT2 of the Buck-Boost DC converter obtained in step 8 to the PWM module to generate a driving signal of the Buck-Boost DC converter, control the Buck-Boost DC converter, extract current from the midpoint O of the T-type three-level converter and inject current into the midpoint O of the T-type three-level converter, so as to realize automatic balance control of the midpoint potential.

[0062] The present invention uses a deep reinforcement learning TD3 controller to control a T-type three-level converter to achieve AC side current tracking and DC side voltage stability, and controls a Buck-Boost DC converter to achieve midpoint potential balance. Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1. The entire system only needs one controller, which is simple to control and overcomes the limitations of traditional controllers in completing complex tasks. The present invention designs a reward function that matches the modulation method, realizes the optimal control of multi-objective tasks, and can effectively reduce the harmonic distortion rate of the current on the AC side of the converter and reduce the power loss of the converter itself.

[0064] 2. Overcome the influence of midpoint potential fluctuation on the control performance of the T-type three-level converter. The present invention adopts a Buck-Boost DC converter, which can automatically achieve the midpoint potential balance of the T-type three-level converter, simplify the control algorithm of the T-type three-level converter, greatly improve the performance of the converter system, and ensure its fast and stable control. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the topological structure of the photovoltaic power generation and wind power generation system containing energy storage equipment of the present invention.

[0066] Figure 2 It is a schematic diagram of the topological structure of the photovoltaic power generation system including the energy storage device of the present invention.

[0067] Figure 3 It is a schematic diagram of the topological structure of the wind power generation system including the energy storage device of the present invention.

[0068] Figure 4 Schematic diagram of the topological structure of the three-level converter system of the present invention.

[0069] Figure 5 This is a control structure block diagram of a three-level converter system based on the TD3 algorithm adopted in the present invention.

[0070] Figure 6 This is a structural diagram of the intelligent agent described in the present invention.

[0071] Figure 7 The figure is a flow chart of training the intelligent agent described in the present invention.

[0072] Among them, 1-T type three-level converter, 2-Buck-Boost DC converter; 3-grid; 4-DC bus; 5-photovoltaic module system; 6-wind turbine system; 7-energy storage equipment; 8-intelligent body. DETAILED DESCRIPTION

[0073] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0074] like Figure 1 As shown, one end of the main converter system of the present invention is connected to the power grid 3 through a circuit breaker, and the other end is connected to the DC bus 4; as an example, the photovoltaic module system 5, the wind turbine system 6, and the energy storage device 7 are connected to the DC bus 4. Of course, the system can also be only a photovoltaic power generation system (such as Figure 2 ), or only wind power generation systems (such as Figure 3 shown).

[0075] like Figure 4As shown, the converter system of the present invention includes a T-type three-level converter 1 and a Buck-Boost DC converter 2; one end of the T-type three-level converter 1 is connected to a power grid 3, and the other end is connected to the Buck-Boost DC converter 2; the other end of the Buck-Boost DC converter 2 is connected to a DC bus 4;

[0076] The T-type three-level converter 1 includes three LCL filters (L x -C x -L gx , x=a,b,c), three bridge arms, two output capacitors (C1, C2); each bridge arm has four power switches: S x1 ~S x4 , x=a, b, c; one end of the LCL filter is connected to the grid 3; two output capacitors C1 and C2 are connected in series, wherein the output capacitor C1 connected to the positive pole of the DC side of the T-type three-level converter 1 is called the upper capacitor, the output capacitor C2 connected to the negative pole of the DC side of the T-type three-level converter 1 is called the lower capacitor, and the connection point of C1 and C2 is called the midpoint O of the T-type three-level converter 1; the bidirectional switch units S of the three bridge arms x2 -S x3 The other end of (x=a,b,c) is short-circuited and connected to point O.

[0077] The DC converter 2 includes a DC inductor L and two power switches VT1 and VT2. One end of the DC inductor L is connected to the midpoint O of the T-type three-level converter. The function of the Buck-Boost DC converter 2 is to extract current from the midpoint O of the T-type three-level converter 1 or inject current into the midpoint O of the T-type three-level converter 1 to achieve automatic midpoint potential balance.

[0078] The present invention provides a three-level converter system control method based on deep reinforcement learning, such as Figure 4 , Figure 5 As shown, the following steps are included:

[0079] Step 1: Obtain the three-phase grid voltage e through the phase-locked loop PLL a 、e b 、e c The phase angle θ is determined, and the phase information is provided to the Park transformation and iPark transformation of the system to ensure that the three-phase modulation wave of the T-type three-level converter 1 is synchronized with the grid voltage.

[0080] Step 2, such as Figure 4 As shown, the voltages at points Ta, Tb, and Tc in the T-type three-level converter 1 are respectively denoted as u a 、u b 、u c , the current flowing through it is recorded as i a 、ib 、i c ; will u a 、u b 、u c After abc / dq transformation, we get u d and u q ;change i a 、i b 、i c After abc / dq transformation, we get i d and i q , where i d Related to active power, i q is related to reactive power, so i q The reference value i q * Can be set to 0, that is, let i q * =0 to achieve unity power factor operation.

[0081] Step 3: Determine the environmental state S of the three-level converter system t is: the DC side voltage U of the T-type three-level converter 1 dc and its reference value U dc * ; The voltage U2 of the lower capacitor C2 of the T-type three-level converter 1; The dq-axis component i of the AC side current of the T-type three-level converter 1 d andi d The reference value i d * 、i q andi q The reference value i q * ; The dq-axis component u of the AC side voltage of the T-type three-level converter 1 d and u q ; The current i of the DC inductor L of the Buck-Boost DC converter 2 L ; Let the environmental state be S t , then S t =[U dc U dc * U2 L i d i d * i q i q * u d u q ].

[0082] Step 4, such as Figure 6As shown, the environmental state S of the three-level converter system determined in step 3 t Building an agent based on the deep reinforcement learning TD3 algorithm 8:

[0083] Agent 8 includes an Actor network structure and a Critic network structure, and contains 6 deep neural networks, specifically:

[0084] The Actor network structure has two deep neural networks: the policy network Actor main network π(s|θ') and its target network Actor Target network π ta (s|θ' ta ) composition: The Critic network structure has 4 deep neural networks: It consists of two evaluation networks for value function estimation, namely the Critic main network and its target network Critic Target network k = 1, 2, where s, a are general representations of the state and action values ​​of the deep neural network, θ', θ' ta They are the weight parameters of the Actor network and its target network Actor Target network respectively; They are the weight parameters of the Critic main network and its target network Critic Target network respectively.

[0085] Its working process is:

[0086] The input of the Actor main network is the environment state S t , add action exploration noise N to the output of the Actor main network t Then get action A t (S t |θ'), where θ' is the weight parameter of the Actor main network, and the action exploration noise N t Obeys normal distribution; the input of the Actor Target network is the environment state S at the next moment t+1 , add the target smooth noise ε to the output of the Actor Target network to obtain the action estimate A at the next moment t+1 (S t+1 |θ' ta ), where θ' ta is the weight parameter of the Actor Target network, and ε follows a normal distribution. The Critic main network is based on the input S of the current Actor main network. t and output action A t (S t |θ'), and output the value function respectively The Critic Target network is based on the input S of the Actor Target network.t+1 And output action estimate A t+1 (S t+1 |θ' ta ), output the target value function respectively

[0087] Step 5: Design reward function R t :

[0088] The reward function R t Designed for:

[0089] R t =R1+R2 (1)

[0090] Wherein, R1 is the AC side current distortion rate, DC side voltage and midpoint potential tracking reward function of the T-type three-level converter 1, and R2 is the AC side current and voltage fluctuation, DC side voltage fluctuation, midpoint potential fluctuation and oscillation suppression reward function of the T-type three-level converter 1;

[0091] R1 is defined as:

[0092]

[0093] Wherein, w1 is the AC side current tracking weight coefficient of the T-type three-level converter 1, which is used to quickly reduce the current tracking error; w2 is the DC side voltage tracking weight coefficient of the T-type three-level converter 1, which is used to quickly reduce the DC side voltage tracking error; w3 is the midpoint potential tracking weight coefficient of the T-type three-level converter 1, which is used to quickly reduce the midpoint potential tracking error;

[0094] Reward function R t The variable weight structure is adopted to determine the weight value according to the power P transmitted by the three-level converter system: 1) When the power P is less than the critical power P c (determined according to actual conditions), the power loss of the converter system is mainly caused by the harmonics of the AC side current. The primary task of the control strategy is to reduce the harmonics. At this time, w1 is set to the maximum; 2) When the power P is greater than P c When the control strategy is used, the primary task is to stabilize the DC output voltage U dc , at this time, w2 is set to the maximum. 3) The midpoint potential fluctuation and oscillation suppression are required in the entire power range, so w3 is set to change slightly with the power P.

[0095] R2 is defined as:

[0096]

[0097] Wherein, w4 is the AC side current stability weight coefficient of the T-type three-level converter 1, which is used to reduce the AC side current fluctuation; w5 is the AC side voltage stability weight coefficient of the T-type three-level converter 1, which is used to reduce the AC side voltage fluctuation; w6 is the DC side voltage stability weight coefficient of the T-type three-level converter 1, which is used to reduce the DC side voltage fluctuation; w7 is the midpoint potential stability weight coefficient of the T-type three-level converter 1, which is used to reduce the midpoint potential fluctuation; e Udc is the DC side voltage tracking error of the T-type three-level converter 1, e Udc =U dc * -U dc ;e U2 is the midpoint potential tracking error of the T-type three-level converter 1, e U2 =U2 * -U2, let U2 * =U dc / 2;i d (tl),i q (tl),u d (tl),u q (tl), e Udc (tl), e U2 (t1) are the i of the T-type three-level converter 1 at the first moment. d 、i q 、u d 、u q 、e Udc 、e U2 , K is the upper limit of the time for suppressing fluctuations, l and K are both positive integers;

[0098] Step 6: Determine action set A t (S t |θ'): Agent 8 based on the environment state S t and the reward function R t , directly output the optimal action A through a fast response strategy t (S t |θ'), contains four parameters: the dq-axis component u of the AC side voltage of the T-type three-level converter 1 d 、u q The reference value u d * 、u q * , the dq-axis component i of the AC side current of the T-type three-level converter 1 d The reference value i d * , and the modulation signal m of the two power switch tubes VT1 and VT2 of the Buck-Boost DC converter 2.

[0099] Step 7, such as Figure 6 and Figure 7 As shown, the deep reinforcement learning TD3 algorithm is used to train agent 8:

[0100] To solve the continuous state and action problem, agent 8 uses the Actor main network π(s|θ') and its target network π ta (s|θ' ta ) to fit the deterministic policy π(s); use the Critic main network and its target network To fit the action value function Q(s,a). The training process of agent 8 is the process of updating the neural network parameters in one round, which is divided into action exploration and iterative optimization.

[0101] 71) Initialize the network weight parameters of agent 8 and set the relevant parameters for training agent 8, including:

[0102] Initialize the weight parameters of the Actor main network and the Critic main network And the weight parameters of the corresponding target network Actor Target network and Critic Target network Set the learning rate μ of the Actor main network; set the discount factor γ of deep reinforcement learning; set the maximum number of training rounds N, the number of consecutive training rounds n, the maximum number of steps in each training round T, and the update frequency D of the target network Actor Target network and Critic Target network; set and initialize the experience database.

[0103] 72) Based on the current environment state S t , the output of the Actor network adds action exploration noise N t , get action A t (S t |θ'):

[0104] A t (S t |θ′)=π(S t θ|′)+N t (4)

[0105] In the formula, π(S t |θ') is the output of the Actor network.

[0106] 73) After each step is executed, the next environment state S is obtained t+1 , and R is calculated according to formula (1) t , the experience gained (S t ,A t ,Rt ,S t+1 ) is stored in the experience database.

[0107] 74) Randomly and uniformly sample M small batches of samples (S ti ,A ti ,R ti ,S ti+1 ), that is, M sets of empirical data, where i = 1, 2, 3, …, M.

[0108] 75) Update the weight parameters of the Critic main network in a way that minimizes the error Right now

[0109]

[0110] Where M represents the size of the mini-batch sample; Y i represents the expected target value function, which is defined as:

[0111]

[0112] In the formula, ε represents the target smooth noise, which is a truncated normal distribution noise, that is,

[0113]

[0114] In the formula, σ clip is the standard deviation of ε, and c is the cutoff factor.

[0115] 76) If the number of training steps is a multiple of the update frequency D of the Critic Target network, proceed to step 77), otherwise go to step 79).

[0116] 77) Update θ' by performing gradient ascent on the expected reward J, i.e.

[0117]

[0118] In the formula, the expression of J is:

[0119]

[0120] It means that J is derived with respect to θ':

[0121]

[0122] In the formula, represents the derivative of π with respect to θ'.

[0123] 78) The weight parameter θ' of the Actor Target network is calculated by exponential smoothing according to formula (11). taand the weight parameters θ' of the CriticTarget network tak are updated as follows:

[0124]

[0125] where τ represents the soft update factor and τ ≤ 1.

[0126] 79) If the number of training steps has not reached the maximum number of steps T in each training episode, return to step 72); otherwise, the current training episode ends, and the agent model obtained in this training episode is saved, and the average reward function value R of this episode is calculated av and its rate of change.

[0127] 710) If the absolute value of R av is less than or equal to R * av for n (n < N) consecutive training episodes, where R * av is a positive real number close to 0, and the rate of change of R av is less than 5% for n (n < N) consecutive training episodes, then end the entire training process; otherwise, proceed to step 711);

[0128] 711) Determine whether the maximum number of training episodes N has been reached. If it has, end the entire training process. If not, return to step 72) to continue training in the next training episode.

[0129] Step 8: Transplant the trained agent model into the main control chip of the three-level converter system, establish an actual control system based on the main control chip of the three-level converter system, realize the stable control of the three-level converter system, and output A t (S t |θ'), that is: the dq-axis components u d and u q of the AC-side voltage of the T-type three-level converter 1, the reference values u d * and u q * , and the modulation signals m of the two power switches VT1 and VT2 of the Buck-Boost DC converter 2;

[0130] Step 9: Transform the AC-side voltage reference values u d * and u q * obtained in step 8 through dq / abc coordinate transformation to obtain the AC-side voltages u a and u b and u cThe reference value u a * 、u b * 、u c * ; Then, u a * 、u b * 、u c * After modulation by the CBPWM module, the driving signal s is generated a 、s b 、s c , driving the power switch tube S of the T-type three-level converter 1 x1 ~S x4 , x=a,b,c, controls the T-type three-level converter 1 to operate.

[0131] Step 10, sending the modulation signal m of the two power switch tubes VT1 and VT2 of the Buck-Boost DC converter 2 obtained in step 8 to the PWM module to generate a drive signal of the Buck-Boost DC converter 2, and controlling the Buck-Boost DC converter 2 to work: when the midpoint potential is higher than its reference value, current is drawn from the midpoint O to reduce the midpoint potential, and the converter works in the Boost state at this time; when the midpoint potential is lower than its reference value, current is injected into the midpoint O to increase the midpoint potential, and the converter works in the Buck state at this time, thereby achieving automatic balance of the midpoint potential.

[0132] In summary, the present invention uses only one controller based on deep reinforcement learning to achieve optimal control of multi-objective tasks. The control is simple, and can suppress the harmonic distortion of the AC side current of the main converter, reducing power consumption. At the same time, the midpoint potential of the T-type three-level converter can be quickly and automatically balanced, and has strong adaptability and robustness to environmental changes. It can effectively cope with the impact of volatility and uncertainty of grid voltage on system control, and realize rapid and stable control of the three-level converter system. It can be widely used in new energy power generation, energy storage, charging piles and other systems.

Claims

1. A three-level converter system control method based on deep reinforcement learning, the three-level converter system comprising a T-type three-level converter and a Buck-Boost DC converter; one end of the T-type three-level converter is connected to a power grid, and the other end is connected to the Buck-Boost DC converter; the other end of the Buck-Boost DC converter is connected to a DC bus; the T-type three-level converter comprises three LCL filters, three bridge arms, and two output capacitors; the bridge arm has four power switch tubes; the two output capacitors are connected in series, and the connection point is called the midpoint O of the T-type three-level converter, wherein, The output capacitor connected to the negative electrode of the DC side of the T-type three-level converter is called a lower capacitor; the Buck-Boost DC converter includes a DC inductor and two power switch tubes, and one end of the DC inductor is connected to the midpoint O of the T-type three-level converter; it is characterized in that the following steps are adopted: Step 1: Obtain the three-phase grid voltage e through the phase-locked loop PLL a 、e b 、e c The phase angle θ is obtained, and the phase information is provided to the Park transformation and iPark transformation of the system; Step 2: The points after the three-phase alternating current passes through the three LCL filters of the T-type three-level converter are respectively recorded as Ta, Tb, and Tc points, and the voltages at Ta, Tb, and Tc points are respectively recorded as u a 、u b 、u c ; The current flowing through points Ta, Tb, and Tc are respectively denoted as i a 、i b 、i c ; will u a 、u b 、u c After abc / dq transformation, we get u d and u q ;change i a 、i b 、i c After abc / dq transformation, we get i d and i q ; Step 3: Determine the environmental state S of the three-level converter system t ; Step 4: The environmental state S of the three-level converter system determined in step 3 t , build an agent based on deep reinforcement learning TD3 algorithm: The agent includes an Actor network structure and a Critic network structure, which contain 6 deep neural networks, specifically: The Actor network structure has two deep neural networks: the policy network Actor Main Network π(s|θ') and its target network Actor Target Network π ta (s|θ' ta ): The Critic network structure has four deep neural networks: It consists of two evaluation networks for value function estimation, namely the Critic main network and its target network Critic Target network k = 1, 2, where s and a are general representations of the environment state and the action value of the deep neural network, respectively, and θ', θ' ta They are the weight parameters of the Actor network and its target network Actor Target network respectively; They are the weight parameters of the Critic main network and its target network Critic Target network respectively; Its working process is: the input of the Actor main network is the environment state S t , add action exploration noise N to the output of the Actor main network t Then get action A t (S t |θ'), where θ' is the weight parameter of the Actor main network, and the action exploration noise N t Obeys normal distribution; the input of the Actor Target network is the environment state S at the next moment t+1 , add the target smooth noise ε to the output of the Actor Target network to obtain the action estimate A at the next moment t+1 (S t+1 |θ' ta ), where θ' ta is the weight parameter of the Actor Target network, ε follows a normal distribution; the Critic main network is based on the input S of the current Actor main network t and output action A t (S t |θ'), and output the value function respectively k=1,2;The Critic Target network is based on the input S of the Actor Target network t+1 And output action estimate A t+1 (S t+1 |θ' ta ), output the target value function respectively k = 1, 2; Step 5: Design reward function R t : Step 6: Determine action set A t (S t |θ'); Step 7, using deep reinforcement learning TD3 algorithm to train the intelligent agent; Step 8: transplant the trained agent model into the main control chip of the three-level converter system, establish an actual control system based on the main control chip of the three-level converter system, realize stable control of the three-level converter system, and output A t (S t |θ'), that is: the dq-axis component u of the AC side voltage of the T-type three-level converter d 、u q The reference value u d * 、u q * , the dq-axis component i of the AC side current of the T-type three-level converter d The reference value i d * , and the modulation signal m of the two power switch tubes of the Buck-Boost DC converter; Step 9: The AC side voltage reference value u of the T-type three-level converter obtained in step 8 is d * and u q * , after dq / abc coordinate transformation, the AC side voltage u of the T-type three-level converter is obtained a 、u b 、u c The reference value u a * 、u b * 、u c * ; Then, u a * 、u b * 、u c * A driving signal is generated after modulation by a carrier pulse width modulation (CBPWM) module to drive the power switch tube of the T-type three-level converter to control the operation of the T-type three-level converter; Step 10, sending the modulation signal m of the two power switch tubes of the Buck-Boost DC converter obtained in step 8 to the PWM module to generate a driving signal of the Buck-Boost DC converter, control the Buck-Boost DC converter, and realize automatic balance control of the midpoint potential.

2. A three-level converter system control method based on deep reinforcement learning according to claim 1, characterized in that: The environmental state S of the three-level converter system in step 3 t Determined as: including the DC side voltage U of the T-type three-level converter dc and its reference value U dc * ; The voltage U2 of the lower capacitor of the T-type three-level converter; The dq-axis component i of the AC side current of the T-type three-level converter d and its reference value i d * 、i q and its reference value i q * ; The dq-axis component u of the AC side voltage of the T-type three-level converter d 、u q ; The current i of the DC inductor of the Buck-Boost DC converter L ; The environmental state is recorded as S t , then S t =[U dc U dc * U2 L i d i d * i q i q * u d u q ].

3. A three-level converter system control method based on deep reinforcement learning according to claim 1, characterized in that: The reward function R in step 5 t Designed for: R t =R1+R2 (1) Wherein, R1 is the AC side current distortion rate, DC side voltage and midpoint potential tracking reward function of the T-type three-level converter, and R2 is the AC side current and voltage fluctuation suppression, DC side voltage fluctuation suppression, and midpoint potential fluctuation and oscillation suppression reward function of the T-type three-level converter; R1 is defined as: Wherein, w1 is the AC side current tracking weight coefficient of the T-type three-level converter, which is used to quickly reduce the current tracking error; w2 is the DC side voltage tracking weight coefficient of the T-type three-level converter, which is used to quickly reduce the DC side voltage tracking error; w3 is the midpoint potential tracking weight coefficient of the T-type three-level converter, which is used to quickly reduce the midpoint potential tracking error; R2 is defined as: Wherein, w4 is the AC side current stability weight coefficient of the T-type three-level converter, which is used to reduce the AC side current fluctuation; w5 is the AC side voltage stability weight coefficient of the T-type three-level converter, which is used to reduce the AC side voltage fluctuation; w6 is the DC side voltage stability weight coefficient of the T-type three-level converter, which is used to reduce the DC side voltage fluctuation; w7 is the midpoint potential stability weight coefficient of the T-type three-level converter, which is used to reduce the midpoint potential fluctuation; e Udc is the DC side voltage tracking error of the T-type three-level converter, e Udc =U dc * -U dc ;e U2 is the midpoint potential tracking error of the T-type three-level converter, e U2 =U2 * -U2, let U2 * =U dc / 2;i d (tl),i q (tl),u d (tl),u q (tl), e Udc (tl), e U2 (t1) are respectively the i of the first moment before the T-type three-level converter d 、i q 、u d 、u q 、e Udc 、e U2 , K is the upper limit of the time for suppressing fluctuations, and l and K are both positive integers.

4. A three-level converter system control method based on deep reinforcement learning according to claim 1, characterized in that: The action set A in step 6 t (S t |θ') is determined as follows: the agent is determined according to the environment state S t and the reward function R t , directly output the optimal action A through a fast response strategy t (S t |θ'), A t (S t |θ') There are four actions, namely: the dq-axis component u of the AC side voltage of the T-type three-level converter d 、u q The reference value u d * 、u q * , the dq-axis component i of the AC side current of the T-type three-level converter d The reference value i d * , and the modulation signal m of the two power switch tubes of the Buck-Boost DC converter.

5. A three-level converter system control method based on deep reinforcement learning according to claim 1, characterized in that: The specific method for training the agent in step 7 is: 71) Initialize the network weight parameters of the agent and set the relevant parameters for the agent training, including: initializing the weight parameters θ' of the Actor main network and the Critic main network, And the corresponding target network Actor Target network, Critic Target network weight parameter θ' ta , k=1,2; set the learning rate μ of the Actor main network; set the discount factor γ of deep reinforcement learning; set the maximum number of training rounds N, the number of consecutive training rounds n, the maximum number of steps T in each training round, and the update frequency D of the target network Actor Target network and Critic Target network; set and initialize the experience database; 72) Based on the current environment state S t , the output of the Actor network adds action exploration noise N t , get action A t (S t |θ'): A t (S t |θ′)=π(S t θ|′)+N t (4) In the formula, π(S t |θ') is the output of the Actor network; 73) After each step is executed, the next environment state S is obtained t+1 , and R is calculated according to formula (1) t , the experience gained (S t ,A t ,R t ,S t+1 ) is stored in the experience database; 74) Randomly and uniformly sample M small batches of samples (S ti ,A ti ,R ti ,S ti+1 ), i.e., M empirical data sets, where i = 1, 2, 3, ..., M; 75) Update the weight parameters of the Critic main network in a way that minimizes the error Right now Where M represents the size of the mini-batch sample; Y i represents the expected target value function, which is defined as: In the formula, ε represents the target smooth noise, which is a truncated normal distribution noise, that is, In the formula, σ clip is the standard deviation of ε, c is the cutoff factor; 76) If the number of training steps is a multiple of the update frequency D of the Critic Target network, proceed to step 77), otherwise go to step 79); 77) Update θ' by performing gradient ascent on the expected reward J, i.e. θ′=θ′+μ▽ θ′ J (8) In the formula, the expression of J is: ▽ θ′ J represents the derivative of J with respect to θ': In the formula, ▽ θ′ π(S ti |θ′) represents the derivative of π with respect to θ'; 78) The weight parameter θ' of the Actor Target network is calculated by exponential smoothing according to formula (11). ta and the Critic Target network weight parameter θ' tak To update, that is: Where, τ represents the soft update factor, τ≤1; 79) If the number of training steps does not reach the maximum number of steps T in each training round, return to step 72); otherwise, the current training round ends, and the agent model obtained in this training round is saved, and the average reward function value R of this round is calculated av and its rate of change; 710) If R av has an absolute value less than or equal to R * av for n (n < N) consecutive training rounds, where R * av is a positive real number close to 0, and the change rate of R av is less than 5% for n (n < N) consecutive rounds, then end the entire training process; otherwise, proceed to step 711). 711) Determine whether the maximum number of training rounds N has been reached. If reached, the entire training process is terminated. If not reached, return to step 72) to enter the next training round to continue training.

Images