DC-DC Converter Control Method Based on TD3 Reinforcement Learning Algorithm

Abstract

The present invention discloses a DC-DC converter control method based on the TD3 reinforcement learning algorithm, comprising: obtaining a state observation vector based on the DC-DC converter; constructing a reinforcement learning controller model based on the TD3 reinforcement learning algorithm, wherein the reinforcement learning control model includes a reinforcement learning agent and a reward module, the state observation vector is input into the reinforcement learning agent to obtain a behavior action and form a new state observation vector; the reward module obtains an instant reward based on the behavior action; the parameters of the reinforcement learning agent are updated based on the instant reward until convergence, thereby obtaining a trained reinforcement learning controller model; and the DC-DC converter is controlled based on the trained reinforcement learning controller model. The present invention can effectively improve the dynamic response performance of the converter.

Description

Technical Field

[0001] The present invention relates to the technical field of DC-DC converter intelligent control, and in particular to a DC-DC converter control method based on a TD3 reinforcement learning algorithm. Background Art

[0002] With the continuous development of power electronics and smart grid technologies, various distributed renewable energy systems have been integrated into existing power systems. Due to their low cost, simple structure, and high efficiency, DC-DC switching converters have attracted extensive research attention. They have been applied to DC microgrids, DC motor drives, and DC power supplies, and show great potential in future industrial, transportation, and military applications. However, converters are subject to various uncertainties during operation, including circuit parameter perturbations, load variations, and supply voltage fluctuations. Due to these factors, the converter's transient output voltage takes a long time to return to steady state, seriously degrading the power output quality. Therefore, when the system faces unavoidable disturbances, improving the dynamic response performance of DC-DC converters is considered a key issue.

[0003] Currently, reinforcement learning-based neural network controllers for DC-DC converters fall into two main categories. One uses a reinforcement learning-based neural network controller as an auxiliary module to modify the control parameters of a traditional controller to improve dynamic performance. However, this type of controller is complex in structure and has high hardware costs. The other uses a reinforcement learning-based neural network controller as the main controller to directly generate control signals. However, due to algorithmic limitations, this controller can only output discrete duty cycle signals for the DC-DC switching converter, resulting in limited control performance. Summary of the Invention

[0004] To solve the problems existing in the above-mentioned prior art, the present invention provides a DC-DC converter control method based on the TD3 reinforcement learning algorithm, comprising:

[0005] Obtaining state observation vector based on DC-DC converter;

[0006] Constructing a reinforcement learning controller model, wherein the reinforcement learning control model includes a reinforcement learning agent and a reward module, inputting the state observation vector into the reinforcement learning agent to obtain a behavior action and form a new state observation vector; based on the behavior action, the reward module obtains an immediate reward; and updating the reinforcement learning agent parameters based on the immediate reward until convergence, thereby obtaining a trained reinforcement learning controller model;

[0007] Based on the trained reinforcement learning controller model, control of the DC-DC converter is achieved.

[0008] The present invention has the following technical effects:

[0009] The present invention is suitable for model-free control, and can perform corresponding training and learning by simply treating the DC-DC converter as a black box system, eliminating the tedious converter modeling process;

[0010] This invention eliminates the need for separate control designs for operating conditions such as converter startup and system parameter changes, significantly simplifying the controller design process. By incorporating various disturbances and system parameter changes into the model environment during training, the controller can automatically adjust control parameters using a learning algorithm, reducing the impact of disturbance signals and system parameter changes on output voltage fluctuations and effectively improving the converter's dynamic response performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 The control architecture of the reinforcement learning controller in the embodiment of the present invention;

[0013] Figure 2 This is a control block diagram of a reinforcement learning system in an embodiment of the present invention;

[0014] Figure 3 is the strategy approximation function μ(s|θ based on the deep neural network in the embodiment of the present invention μ );

[0015] Figure 4 A reinforcement learning algorithm based on the actor-critic architecture in an embodiment of the present invention;

[0016] Figure 5 A diagram of the reinforcement learning training process in an embodiment of the present invention;

[0017] Figure 6 1 is the output voltage waveform of the synchronous buck converter during startup and steady state in an embodiment of the present invention;

[0018] Figure 7 : is the output voltage waveform of the synchronous BUCK converter in an embodiment of the present invention when the load changes step;

[0019] Figure 8 The output voltage waveform (load current I Load Change from 4A to 2A). DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] Example 1

[0022] This embodiment provides a DC-DC converter control method based on the TD3 reinforcement learning algorithm, which specifically includes:

[0023] 1. As Figure 1 As shown in Figure 2, the basic control architecture of the reinforcement learning controller is constructed. The reinforcement learning agent consists of two parts: strategy and reinforcement learning algorithm. The reinforcement learning strategy is a neural network with adjustable parameters, which is based on the state observation s from the environment. t Output action a t The learning algorithm of reinforcement learning continuously updates the policy parameters based on the action behavior, state observation and reward. Its goal is to find an optimal policy that maximizes the expected cumulative reward obtained during training.

[0024] In order to explain the proposed controller in more detail, in this embodiment, a DC-DC synchronous BUCK switching converter is taken as an example to illustrate the reinforcement learning controller. The system control block diagram is shown in FIG. Figure 2 As shown in the figure. The DC-DC converter as a black box environment has only two ports, input and output. The control signal a of the controller t As input, the voltage across the load v o As output. In addition to the reinforcement learning agent, the reinforcement learning controller also designs an observation module and a reward module to construct a Markov decision process (MDP). The observation module and reward module provide observation signals and reward signals to the reinforcement learning agent, respectively. After internal processing, the output behavior action is used as a control signal to control the converter. The specific workflow is as follows:

[0025] 1) The reinforcement learning controller collects the output voltage v of the DC-DC converter in real time o As a set of state observations, the error e between the output voltage and the reference voltage is calculated, and the error e and the integral of the error ∫e dt are used as the other two sets of state observations to finally generate the state observation vector s t [e,∫e dt,v o ] is passed to the state observation interface of the reinforcement learning agent.

[0026] 2) The reinforcement learning agent samples a state observation s each time t , will output the control signal a through the strategy function t The controlled object executes the control signal, changes the state of the environment, and generates a new set of state observations s at the next sampling. t+1 .

[0027] 3) After each sampling, the reinforcement learning controller obtains the state observation, and calculates the immediate reward R through the reward module t .

[0028]

[0029] Among them, R t where represents the instantaneous reward, e represents the error between the output voltage and the reference voltage, and v o Indicates the output voltage, v s Indicates the transformer input voltage.

[0030] Instant Reward R t It is used to evaluate the quality of the output actions of the reinforcement learning agent. At each sampling, the reinforcement learning algorithm updates the policy function so that the parameters of the policy function are iterated in the direction of maximizing the accumulated reward during training.

[0031] 4) Through continuous training iterations, the parameters of the policy function in the reinforcement learning agent gradually converge. At the end of each training session, the reinforcement learning agent achieves the maximum accumulated reward. At this point, the policy function represents the optimal mapping from environmental state observations to output behavioral actions. The reinforcement learning controller controls the transformer based on this mapping to improve system performance.

[0032] 2. Reinforcement learning agent design includes:

[0033] 1) Reinforcement Learning Agent Strategy Construction

[0034] Reinforcement learning strategy μ(s|θ μ ) is a function approximator with adjustable parameters, which is based on the state observation s from the environment t To output action a t This paper selects a two-layer deep neural network as the approximate function μ(s|θ μ ),like Figure 3 shown.

[0035] Policy function μ(s|θ μ ) is approximated by a two-layer deep neural network, and the reinforcement learning agent obtains the state observation vector s tAs the input of the neural network, the output is calculated by the middle layer as the control signal a of the DC-DC converter t The weights θ in the neural network μ w and bias θ μ b Parameters such as these are obtained through reinforcement learning algorithm training.

[0036] 2) Reinforcement Learning Agent Algorithm Architecture

[0037] During the reinforcement learning training process, the parameters of the policy approximation function are continuously updated by the reinforcement learning algorithm until they converge to the optimal parameters. This paper uses the double-delayed deep deterministic policy gradient (TD3) algorithm for training and learning to obtain the optimal policy. The actor-critic architecture used by the double-delayed deep deterministic policy gradient (TD3) algorithm is as follows: Figure 4 shown.

[0038] Figure 4 In the example, the behavior value function Q(s, a|θ Q ) and the policy function μ(s|θ μ ) are built using deep neural networks. The behavior value approximation function Q(s, a|θ Q ) is used to represent the gain G obtained by executing a specific action a on the current state s t expectations.

[0039] Q(s,a|θ Q )≈E[G t |s t =s,a t =a] (2)

[0040]

[0041] Harvest G t It is from a certain state s during the training process t Start sampling and continue until the end of the state, summing all rewards. γ is the discount factor used when calculating rewards. This factor is introduced so that the contribution of a subsequent state to the current state's reward is less than its reward.

[0042] Behavior value approximation function Q(s,a|θ Q ) constitutes the objective function J(θ μ ).

[0043]

[0044] During training, the strategy approximates the function μ(s|θ μ) parameter θ μ The gradient is updated in the direction of maximizing the objective function, that is, obtaining the maximum expected gain and the maximum accumulated reward.

[0045]

[0046] Behavior value approximation function Q(s,a|θ Q ) parameter θ Q It also needs to be updated iteratively through training. In order to make the approximate value function more accurately reflect the value of the state, the loss function L(θ Q ).

[0047]

[0048] y t =R t +γQ(s t+1 ,a t+1 |θ Q ) (7)

[0049] By minimizing the loss L(θ Q ) to update the parameter θ in the behavior value approximation function Q As reinforcement learning is trained, the parameters θ in the policy approximation function μ and the parameter θ in the behavior value approximation function Q Continuous updating, on the one hand, can evaluate and optimize the policy function based on the value function, on the other hand, the optimized policy function will make the value function more accurately reflect the value of the state, the two promote each other and ultimately obtain the optimal strategy.

[0050] 3) Reinforcement Learning Agent Algorithm Design

[0051] The Double-Delayed Deep Deterministic Policy Gradient (TD3) algorithm makes the following improvements to the Actor-Critic architecture described in the previous section, further improving the training efficiency of the reinforcement learning controller:

[0052] 1) The TD3 algorithm backs up a set of parameters for the neural network-based behavior value approximation function and the policy approximation function to calculate the expected value of the behavior value, thereby more stably improving the policy guidance level of the behavior value approximation function. The network using the backup parameters is called the target network, and the corresponding parameters are updated very small each time. The behavior value approximation function and policy approximation function corresponding to the other set of parameters are used to generate actual interactive behaviors and calculate the corresponding policy gradients. This set of parameters is updated once after each learning. The dual parameter setting can reduce the occurrence of non-convergence due to the guidance of approximate data;

[0053] 2) The TD3 algorithm uses mini-batch gradient descent to update the parameters of the policy approximation function and the action value approximation function. When training a neural network with a training set consisting of E training samples, the TD3 algorithm samples M (M << E) samples to update the parameter values. This configuration avoids the shortcomings of stochastic gradient descent and batch gradient descent, such as inaccurate gradient update direction and slow convergence.

[0054] 3) During the sampling phase, the TD3 algorithm adds a noise function to the behavior output by the policy approximation function, causing the reinforcement learning agent to conduct a small exploration around the deterministic behavior to search for the potential optimal policy. During the learning phase, the TD3 algorithm adds noise to the output actions of the policy approximation function's target network, reducing the probability of the policy exploring actions with high behavioral value evaluations and the possibility of generating suboptimal policies when the behavioral value function is overestimated.

[0055] 4) The TD3 algorithm learns two behavior-value approximation functions and uses the minimum value of the behavior-value function for estimation during policy updates. At the same time, the TD3 algorithm updates the policy function and objective function less frequently than it updates the behavior-value function, reducing the behavior-value function approximation error during training and avoiding overestimation of the behavior-value function, which causes the strategy to converge to a local optimal solution.

[0056] Based on the above improvements, the TD3 algorithm designed for DC-DC switching converter control is as follows:

[0057]

[0058]

[0059] 3. Verification and Analysis

[0060] A DC-DC synchronous buck converter model was built in MATLAB / Simulink. The parameters of the DC-DC synchronous buck converter model are shown in Table 1. The traditional PI control method was selected for comparison with the proposed reinforcement learning controller. The parameters of the PI controller and the TD3 algorithm are shown in Table 2.

[0061] Table 1

[0062]

[0063] Table 2

[0064]

[0065] The reinforcement learning controller is trained using the TD3 algorithm under the parameters shown in Table 2. The training process is as follows: Figure 5 shown.

[0066] Depend on Figure 5 As we know, after 720 training cycles, the control parameters of the reinforcement learning agent converged, and the accumulated rewards obtained from each training cycle remained essentially unchanged, at which point the reinforcement learning controller training was completed. To verify the dynamic performance of the reinforcement learning controller after training, simulations were performed during the converter startup phase and load step changes.

[0067] Figure 6 The output voltage startup waveforms of the converter are shown using a reinforcement learning controller and a PI controller. Compared to the PI controller, the reinforcement learning controller based on the TD3 algorithm has a faster startup time and smaller overshoot during the DC-DC converter startup.

[0068] Figure 7 and Figure 8 The dynamic response performance of the synchronous buck converter under load step changes is demonstrated. Compared to the PI controller, the TD3-based reinforcement learning controller achieves faster dynamic adjustment time under load step changes at the same overshoot. Detailed dynamic response data during startup and load step changes are shown in Table 3.

[0069] Table 3

[0070]

[0071] Table 4 shows a comparison with other related work. Due to the different switching frequencies, the settling time is represented by the adjustment period for a fair comparison. The comparison results show that the proposed TD3-based reinforcement learning controller has better dynamic adjustment performance than other controllers.

[0072] Table 4

[0073]

[0074] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A DC-DC converter control method based on TD3 reinforcement learning algorithm, characterized in that: include: Obtaining a state observation vector based on a DC-DC converter; wherein the process of obtaining the state observation vector comprises: obtaining a reference voltage, collecting an output voltage of the DC-DC converter in real time, obtaining a voltage error based on the reference voltage and the output voltage, and obtaining an error integral based on the voltage error; and obtaining a state observation vector based on the output voltage, the voltage error, and the error integral; A reinforcement learning controller model is constructed based on the TD3 reinforcement learning algorithm, wherein the reinforcement learning control model includes a reinforcement learning agent and a reward module. The state observation vector is input into the reinforcement learning agent to obtain a behavior action and form a new state observation vector. The reward module obtains an immediate reward based on the behavior action. The parameters of the reinforcement learning agent are updated based on the immediate reward until convergence, thereby obtaining a trained reinforcement learning controller model. The instant reward calculation formula is as follows: Among them, R t represents the instant reward, e represents the error between the output voltage and the reference voltage, v o Indicates the output voltage, v s Indicates the transformer input voltage; Based on the trained reinforcement learning controller model, control of the DC-DC converter is achieved.

2. The DC-DC converter control method based on the TD3 reinforcement learning algorithm according to claim 1, characterized in that: The reinforcement learning agent also includes a strategy module and a reinforcement learning algorithm module.

3. The DC-DC converter control method based on the TD3 reinforcement learning algorithm according to claim 2, characterized in that: The reinforcement learning controller model training process includes: Inputting the state observation vector into the strategy module, the strategy module outputting a behavior action, and the DC-DC converter generating a new state observation vector based on the behavior action; At the same time, the reward module obtains an immediate reward based on the behavioral action. Based on the immediate reward, the reinforcement learning algorithm module and the strategy module update parameters, and continuously iterate until convergence, thereby obtaining a trained reinforcement learning controller model; The strategy module includes strategy parameters, and the reinforcement learning algorithm module includes value parameters.

4. The DC-DC converter control method based on the TD3 reinforcement learning algorithm according to claim 3, characterized in that: The reinforcement learning controller model training method adopts a double-delayed deep deterministic policy gradient algorithm based on the actor-critic architecture.

5. The DC-DC converter control method based on the TD3 reinforcement learning algorithm according to claim 3, characterized in that: The method for updating the parameters of the reinforcement learning agent based on the immediate reward includes: updating the parameters of the reinforcement learning agent based on the dual parameters of the strategy parameter and the value parameter using a small block gradient descent algorithm.

Images