Digital dc-dc converter loop control method, system, device, and medium

Abstract

The application discloses a kind of digital DC-DC converter loop control method, system, equipment and medium, the method comprises the following steps: based on the digital DC-DC converter to be controlled, obtains observation value;The observation value obtained is input into the intelligent agent obtained by pre-training, and the zero-pole position is output;Based on the zero-pole position obtained, the updated transfer function is obtained, and the 3p3z controller is based on the updated transfer function and outputs PWM wave;The PWM wave is used to control the conduction state of power tube in DC-DC converter, and the output voltage value of DC-DC converter is constant.The application proposes a control method based on Q-learning reinforcement learning algorithm and 3p3z controller, which can dynamically adjust the position of zero-pole of 3p3z controller transfer function in real time according to the current state, and ultimately can achieve better dynamic characteristics under limited resource overhead.

Description

Technical Field

[0001] This invention belongs to the technical field of digital DC-DC converters, and specifically relates to a loop control method, system, device and medium for a digital DC-DC converter. Background Technology

[0002] Digital DC-DC converters, as a typical switching power supply, are widely used in digital power supplies, high-performance power adapters, microgrids and other fields. Research on digital DC-DC converters has important practical value.

[0003] Loop compensation is a key step in DC-DC converters, determining the dynamic response characteristics of digital DC-DC converters. In existing technologies, the loop design of classic digital DC-DC converters is manually adjusted by experts according to specific application scenarios. In addition, some studies have adopted dynamically adjustable loop control methods, such as fuzzy control and sliding mode control, to improve the dynamic response characteristics of converters.

[0004] Reinforcement learning is an important branch of machine learning. By training an agent and enabling it to act autonomously in a certain environment, the agent can continuously improve its behavior through interaction with the environment. For example, reinforcement learning algorithms have achieved remarkable results in fields such as Go, games, and control.

[0005] Research on reinforcement learning in the field of digital DC-DC converter control is still in its infancy. Currently, many scholars have adopted deep reinforcement learning methods, either directly controlling the loop or compensating for the loop output, to improve the loop's dynamic response. However, existing agent implementations using neural networks still have the following problems:

[0006] 1) A neural network is a black box function, and the relationship between input and output cannot be explicitly obtained. Therefore, the trained agent has a lack of interpretability, which is unacceptable in some high-reliability applications.

[0007] 2) The computational resources required for hardware deployment of neural networks are enormous, especially when they are used as loop controllers; for example, a shallow multilayer perceptron may consume hundreds of high-performance multipliers.

[0008] 3) Neural networks are slow to compute. Unless expensive high-performance computers are used, it is difficult for neural networks to achieve a forward propagation within a few microseconds. Therefore, this compensation method is not suitable for digital power supplies with high switching frequencies. Summary of the Invention

[0009] The purpose of this invention is to provide a loop control method, system, device, and medium for a digital DC-DC converter to solve one or more of the aforementioned technical problems. The technical solution provided by this invention proposes a control method based on Q-learning reinforcement learning algorithm and a 3p3z (3pole 3zero) controller. This method can dynamically adjust the positions of the zeros and poles of the 3p3z controller's transfer function in real time according to the current state, ultimately achieving better dynamic characteristics with limited resource overhead, thus ensuring a constant output voltage value for the DC-DC converter.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] This invention provides a loop control method for a digital DC-DC converter, comprising the following steps:

[0012] Based on the digital DC-DC converter to be controlled, the observed value e is obtained. t and e t-1 ; where e t e is the difference between the output voltage value and the reference voltage value at time t. t-1 The difference between the output voltage value and the reference voltage value at time t is the difference between the output voltage value at the previous time.

[0013] The obtained observation value e t and e t-1 The input is a pre-trained agent, and the output is used to adjust the zero and pole positions of the transfer function of the 3p3z controller.

[0014] The updated transfer function is obtained based on the zero and pole positions output by the intelligent agent, and the 3p3z controller outputs a PWM wave based on the updated transfer function; the PWM wave is used to control the conduction state of the power transistor in the DC-DC converter to achieve a constant output voltage value of the DC-DC converter.

[0015] The mathematical model of the intelligent agent is a Markov decision process, which includes a state space, an action space, and a reward function.

[0016] The state space S is expressed as S = [e t ,e t-1 ];

[0017] The actions of the agent in action space A are the zero and pole positions of the discrete 3p3z controller transfer function;

[0018] The expression for the reward function R is, In the formula, E is the desired ripple voltage magnitude, and n is the total number of steps in the trajectory;

[0019] The Q-Learning algorithm is used in the training process of the pre-trained agent.

[0020] A further improvement of the present invention is that, in the step of employing the Q-Learning algorithm, the Q-table used is represented as follows:

[0021] Obs1 Obs2 Obs3 … Obsn A1 Q(Obs1,A1) Q(Obs2,A1) Q(Obs3,A1) … Q(Obsn,A1) A2 Q(Obs1,A2) Q(Obs2,A2) Q(Obs3,A2) … Q(Obsn,A2) A3 Q(Obs1,A3) Q(Obs2,A3) Q(Obs3,A3) … Q(Obsn,A3) … … … … … … An Q(Obs1,An) Q(Obs2,An) Q(Obs3,An) … Q(Obsn,An)

[0022] In the table, Obs1, Obs2, Obs3 to Obsn in the horizontal column represent the state space corresponding to the Markov decision process; A1, A2, A3 to An in the vertical column represent the action space corresponding to the Markov decision process. A1, A2, A3 to An are transfer functions with different zero and pole locations, corresponding to different phase margins.

[0023] A further improvement of this invention is that, during the training process using the Q-Learning algorithm, the single trajectory for data collection through the interaction between the agent and the environment is...

[0024] From time 0 to T1, the input voltage linearly increases from 0 to V. in During this process, the loop control strategy of the digital DC-DC converter adopts a soft-start scheme; at time T1, the input voltage magnitude is V in The input voltage experiences a random step jump near its value, and at time T2, the input voltage magnitude steps back to V. in At time T3, the load current is in the range of I. out The current value experiences a random step jump once, and at time T4, the load current magnitude steps back to I. out The trajectory is completed and exited at time T5; the agent completes data collection within a single trajectory and performs training and policy improvement, with the desired goal of keeping the output voltage of the DC-DC converter stable at the reference voltage for as long as possible.

[0025] In the single trajectory, when the DC-DC switching frequency is k Hz, the trajectory includes a total of T5 / k steps.

[0026] This invention provides a loop control system for a digital DC-DC converter, comprising:

[0027] The observation acquisition module is used to acquire the observation value e based on the digital DC-DC converter to be controlled. t and e t-1 ; where e t e is the difference between the output voltage value and the reference voltage value at time t. t-1 The difference between the output voltage value and the reference voltage value at time t is the difference between the output voltage value at the previous time.

[0028] The zero-pole location acquisition module is used to obtain the acquired observation value e. t and e t-1The input is a pre-trained agent, and the output is used to adjust the zero and pole positions of the transfer function of the 3p3z controller.

[0029] The control module is used to obtain the updated transfer function based on the zero and pole positions output by the intelligent agent, and the 3p3z controller outputs a PWM wave based on the updated transfer function; the PWM wave is used to control the conduction state of the power transistor in the DC-DC converter to achieve a constant output voltage value of the DC-DC converter.

[0030] The mathematical model of the intelligent agent is a Markov decision process, which includes a state space, an action space, and a reward function.

[0031] The state space S is expressed as S = [e t ,e t-1 ];

[0032] The actions of the agent in action space A are the zero and pole positions of the discrete 3p3z controller transfer function;

[0033] The expression for the reward function R is, In the formula, E is the desired ripple voltage magnitude, and n is the total number of steps in the training trajectory;

[0034] The Q-Learning algorithm is used in the training process of the pre-trained agent.

[0035] A further improvement of the present invention is that, in the step of employing the Q-Learning algorithm, the Q-table used is represented as follows:

[0036] Obs1 Obs2 Obs3 … Obsn A1 Q(Obs1,A1) Q(Obs2,A1) Q(Obs3,A1) … Q(Obsn,A1) A2 Q(Obs1,A2) Q(Obs2,A2) Q(Obs3,A2) … Q(Obsn,A2) A3 Q(Obs1,A3) Q(Obs2,A3) Q(Obs3,A3) … Q(Obsn,A3) … … … … … … An Q(Obs1,An) Q(Obs2,An) Q(Obs3,An) … Q(Obsn,An)

[0037] In the table, Obs1, Obs2, Obs3 to Obsn in the horizontal column represent the state space corresponding to the Markov decision process; A1, A2, A3 to An in the vertical column represent the action space corresponding to the Markov decision process. A1, A2, A3 to An are transfer functions with different zero and pole locations, corresponding to different phase margins.

[0038] A further improvement of this invention is that, during the training process using the Q-Learning algorithm, the single trajectory for data collection through the interaction between the agent and the environment is...

[0039] From time 0 to T1, the input voltage linearly increases from 0 to V. in During this process, the loop control strategy of the digital DC-DC converter adopts a soft-start scheme; at time T1, the input voltage magnitude is V in The input voltage experiences a random step jump near its value, and at time T2, the input voltage magnitude steps back to V. inAt time T3, the load current is in the range of I. out The current value experiences a random step jump once, and at time T4, the load current magnitude steps back to I. out The trajectory is completed and exited at time T5; the agent completes data collection within a single trajectory and performs training and policy improvement, with the desired goal of keeping the output voltage of the DC-DC converter stable at the reference voltage for as long as possible.

[0040] In the single trajectory, when the DC-DC switching frequency is k Hz, the trajectory includes a total of T5 / k steps.

[0041] The present invention provides an electronic device comprising:

[0042] At least one processor; and,

[0043] A memory communicatively connected to the at least one processor; wherein,

[0044] The memory stores instructions that can be executed by the at least one processor, which, when executed, enable the at least one processor to perform any of the above-described digital DC-DC converter loop control methods of the present invention.

[0045] The present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the digital DC-DC converter loop control method described in any one of the present invention.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention discloses a loop control method for a digital DC-DC converter, which proposes a control method based on Q-learning reinforcement learning algorithm and 3p3z (3pole 3zero) controller. By acquiring the current state in real time and adjusting the position of the zeros and poles of the 3p3z controller transfer function based on the agent, better dynamic characteristics can be achieved with limited resource overhead, so that the output voltage value of the DC-DC converter remains constant.

[0048] Further explanation is provided: the technical solution of this invention employs a Q-learning algorithm based on lookup tables for training, which features strong interpretability and high reliability; the action space controls the zero and pole positions of the 3p3z transfer function, and compared with existing methods using neural networks, the behavior of the algorithm in this invention has better interpretability, avoiding the problem that the neural network may take dangerous actions during the exploration phase; the above features are very suitable for applications with high reliability, such as power supplies.

[0049] To further explain, existing control methods based on neural networks require enormous computational resources. This invention employs a lookup table-based algorithm, which is easier to implement and whose main computational resource overhead is relatively inexpensive ROM (Read-Only Memory) resources. Therefore, it is easier to achieve lightweight and miniaturization, and has the characteristics of low cost and high performance. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art are briefly introduced below; obviously, the drawings described below are some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0051] Figure 1 This is a schematic flowchart of a loop control method for a digital DC-DC converter disclosed in an embodiment of the present invention;

[0052] Figure 2 This is a schematic diagram of a 3p3z control scheme based on reinforcement learning in an embodiment of the present invention;

[0053] Figure 3 This is a schematic diagram of a single trajectory design in an embodiment of the present invention;

[0054] Figure 4 This is a schematic diagram of a Buck DC-DC converter in an embodiment of the present invention;

[0055] Figure 5 This is a schematic diagram comparing experimental results in an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] The present invention will now be described in further detail with reference to the accompanying drawings:

[0059] Please see Figure 1 The present invention provides a loop control method for a digital DC-DC converter, comprising the following steps:

[0060] Step 1: Based on the digital DC-DC converter to be controlled, obtain the observed value e. t and e t-1 ; where e t e is the difference between the output voltage value and the reference voltage value at time t. t-1 The difference between the output voltage value and the reference voltage value at time t is the difference between the output voltage value at the previous time.

[0061] Step 2, take the observation value e obtained in Step 1 t and e t-1 The input is a pre-trained agent, and the output is used to adjust the zero and pole positions of the transfer function of the 3p3z controller.

[0062] Step 3: Based on the zero and pole positions output by the agent in Step 2, obtain the updated transfer function. The 3p3z controller outputs a PWM (Pulse Width Modulation) wave based on the updated transfer function. The PWM wave is used to control the conduction state of the power transistor in the DC-DC converter to achieve a constant output voltage value of the DC-DC converter.

[0063] The technical solution disclosed in this invention is to use a reinforcement learning control method to achieve better dynamic characteristics.

[0064] Please see Figure 2 The control flow of the technical solution in the embodiments of the present invention includes:

[0065] The control object of the controlling agent is the 3p3z controller, and the agent controls the zero and pole positions of the transfer function of the 3p3z controller;

[0066] The 3P3Z controller implements the conversion of the transfer function from the frequency domain to the time domain, and its output is a PWM wave with a specific duty cycle, which is used to directly control the DC-DC converter.

[0067] The DC-DC converter is the final controlled object, and the desired output voltage value V of the DC-DC converter is... out It maintains a constant output and can quickly stabilize when the load current or input voltage of the DC-DC converter changes, meaning it has a faster dynamic response speed.

[0068] In the technical solution of this invention embodiment, at a certain moment during the training phase or after deployment, the agent determines the value e based on the observed value. t With e t-1 The zero and pole positions of the transfer function of the 3P3Z controller are adjusted; according to the new transfer function, the 3P3Z controller outputs the corresponding PWM wave to control the conduction state of the DC-DC power switch and achieve constant output voltage.

[0069] Among them, e t The output voltage value V at time t out With reference voltage value V out* The difference, e t-1 The output voltage value V at time t-1 (that is, the time before time t) out With reference voltage value V out* difference.

[0070] The following section will elaborate further on the construction of the model and training environment for reinforcement learning algorithms.

[0071] (1) The problem modeling process includes: The mathematical model of reinforcement learning is the Markov decision process. In this embodiment of the invention, it is necessary to design the state space S, action space A and reward function R in the decision process; among them, the state space A designed in this invention is one of the main innovations of the technical solution of this embodiment of the invention.

[0072] In this embodiment of the invention, the proposed state space S is defined by formula (1), and its expression is:

[0073] S = [e t , e t-1 ], (1)

[0074] In the formula, e t The output voltage value V at time t out With reference voltage value V out* The difference, e t-1 The output voltage V at the previous time (t-1) before time t. out With reference voltage value V out* difference.

[0075] In this embodiment of the invention, the action space proposed is the zero and pole positions of the discrete 3p3z controller for the actions of the agent, rather than compensating for the duty cycle of existing algorithms or directly outputting the duty cycle, thus achieving better interpretability; compared with the use of neural networks, the action space of this embodiment of the invention is discrete and can be described in tabular form.

[0076] In this embodiment of the invention, the proposed reward function R is defined by formula (2), and its expression is:

[0077]

[0078] In the formula, E is the desired ripple voltage magnitude, and n is the total number of steps in the training trajectory.

[0079] This invention embodiment is based on Q-Learning for training, and the corresponding Q-table is shown in Table 1. In the table, Obs1, Obs2, Obs3 to Obsn in the horizontal column are the state spaces corresponding to the Markov decision process, and A1, A2, A3 to An in the vertical column are the action spaces corresponding to the Markov decision process. The actions A1, A2, A3 to An are transfer functions with different zero and pole positions, and each corresponds to a different phase margin.

[0080] Table 1. Q-table settings in embodiments of the present invention

[0081] Obs1 Obs2 Obs3 … Obsn A1 Q(Obs1,A1) Q(Obs2,A1) Q(Obs3,A1) … Q(Obsn,A1) A2 Q(Obs1,A2) Q(Obs2,A2) Q(Obs3,A2) … Q(Obsn,A2) A3 Q(Obs1,A3) Q(Obs2,A3) Q(Obs3,A3) … Q(Obsn,A3) … … … … … … An Q(Obs1,An) Q(Obs2,An) Q(Obs3,An) … Q(Obsn,An)

[0082] The specific training algorithm for Q-Learning used in the embodiments of the present invention can be exemplarily exemplified by the method disclosed in the following literature: Venkateswaran R, Natesan B, Lee SR, et al. Maximum Power Extraction for PMVG-Based WECS Using Q-Learning MPPT Algorithm With Finite-Time Control Scheme[J].IEEE Transactions on Sustainable Energy,2022,14(1):516-524.

[0083] (2) Environment Design: In reinforcement learning algorithms, the agent interacts with the environment to collect data; this embodiment of the invention designs a single trajectory, such as... Figure 3 As shown, Figure 3 The horizontal axis represents the time scale;

[0084] From time 0 to T1, the input voltage linearly increases from 0 to V. in In this process, a traditional soft-start scheme can be adopted. For specific technical details, please refer to the following literature, which will not be elaborated here: Xue Longhui. Design and Implementation of Serialized Digital Power Supply [D]. Hangzhou Dianzi University, 2021. DOI:10.27075 / d.cnki.ghzdc.2021.000533.

[0085] At time T1, the input voltage magnitude is V in The input voltage experiences a random step jump near its value, and at time T2, the input voltage magnitude steps back to V. in ;

[0086] At time T3, the load current is I out The current value experiences a random step jump once, and at time T4, the load current magnitude steps back to I. out ;

[0087] The trajectory is completed and exited at time T5;

[0088] When the DC-DC switching frequency is k Hz, the trajectory includes a total of T5 / k steps.

[0089] The agent collects data and performs training and policy improvement within the aforementioned single trajectory, with the desired goal of stabilizing the DC-DC converter output at the desired voltage V for as long as possible. out* .

[0090] In this embodiment of the invention, the aim is to obtain an intelligent agent for the DC-DC converter to achieve the purpose of loop control of the DC-DC controller. The Buck-type DC-DC controller is used as an example below.

[0091] Please see Figure 4 , Figure 4 This is a Buck-type DC-DC circuit diagram. In the diagram, V... in For the varying input voltage source, M1 is a PMOS power transistor, M2 is an NMOS power transistor, R1 and R2 are resistors, L is an inductor whose parasitic resistance is denoted by L_ESR, C is a capacitor whose parasitic resistance is denoted by C_ESR, and RL is the load resistor. Figure 4 The parameters of each component are as follows: inductance L is 4.7uH, with a classic value of L_ESR of 10m ohms; C is 10uF, with a classic value of C_ESR of 5m ohms; M1 and M2 are Toshiba Electronics SSM3K333R and SSM3J332R, respectively.

[0092] The 3p3z controller outputs a PWM signal, which directly controls the gates of M1 and M2, controlling their on and off states. The 3p3z controller can increase the dead time to improve the output efficiency of the DC-DC converter, which will not be elaborated here.

[0093] V out The signal from the terminal is converted from analog to digital and then transmitted to the intelligent agent. Figure 2 Chinese V out* Subtract the signals to obtain e. t .

[0094] The agent collects data and performs training and policy improvement within the pre-defined trajectory, with the desired goal of keeping the DC-DC converter output stable at the desired voltage V for as long as possible. out* .

[0095] Please see Figure 5 The final experimental results are for example Figure 5 As shown, the black curve represents the algorithm disclosed in this embodiment of the invention, and the black dotted line represents the existing conventional 3p3z algorithm. The expected final output is within 1.8V ± 0.01V; therefore, the two gray dashed lines represent 1.81V and 1.79V respectively. At time step 20000, the input voltage jumps from 3.3V to 4.8V. At time step 40000, the input voltage jumps back from 4.8V to 3.3V. At time step 60000, the load current jumps from 1A to 0.5A. At time step 80000, it jumps back from 0.5A to 1A. It can be seen that the method proposed in this embodiment of the invention is significantly superior to the traditional fixed 3p3z control method in terms of both overshoot magnitude and recovery speed.

[0096] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the apparatus embodiments, please refer to the embodiments of the method of the present invention.

[0097] This invention provides a loop control system for a digital DC-DC converter, comprising:

[0098] The observation acquisition module is used to acquire the observation value e based on the digital DC-DC converter to be controlled. t and e t-1 ; where e t e is the difference between the output voltage value and the reference voltage value at time t. t-1 The difference between the output voltage value and the reference voltage value at time t is the difference between the output voltage value at the previous time.

[0099] The zero-pole location acquisition module is used to obtain the acquired observation value e. t and e t-1The input is a pre-trained agent, and the output is used to adjust the zero and pole positions of the transfer function of the 3p3z controller.

[0100] The control module is used to obtain the updated transfer function based on the zero and pole positions output by the intelligent agent, and the 3p3z controller outputs a PWM wave based on the updated transfer function; the PWM wave is used to control the conduction state of the power transistor in the DC-DC converter to achieve a constant output voltage value of the DC-DC converter.

[0101] The mathematical model of the intelligent agent is a Markov decision process, which includes a state space, an action space, and a reward function.

[0102] The state space S is expressed as S = [e t ,e t-1 ];

[0103] The actions of the agent in action space A are the zero and pole positions of the discrete 3p3z controller transfer function;

[0104] The expression for the reward function R is, In the formula, E is the desired ripple voltage magnitude, and n is the total number of steps in the training trajectory;

[0105] The Q-Learning algorithm is used in the training process of the pre-trained agent.

[0106] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a loop control method for a digital DC-DC converter.

[0107] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the digital DC-DC converter loop control method in the above embodiments.

[0108] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0111] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A loop control method for a digital DC-DC converter, characterized in that, Includes the following steps: Based on the digital DC-DC converter to be controlled, the observed value e is obtained. t and e t-1 ; Among them, e t e is the difference between the output voltage value and the reference voltage value at time t. t-1 The difference between the output voltage value and the reference voltage value at time t is the difference between the output voltage value at the previous time. The obtained observation value e t and e t-1 The input is a pre-trained agent, and the output is used to adjust the zero and pole positions of the transfer function of the 3p3z controller. The updated transfer function is obtained based on the zero and pole positions output by the intelligent agent, and the 3p3z controller outputs a PWM wave based on the updated transfer function; the PWM wave is used to control the conduction state of the power transistor in the DC-DC converter to achieve a constant output voltage value of the DC-DC converter. The mathematical model of the intelligent agent is a Markov decision process, which includes a state space, an action space, and a reward function. The state space S is expressed as S = [e t , e t-1 ]; The actions of the agent in action space A are the zero and pole positions of the discrete 3p3z controller transfer function; The expression for the reward function R is, In the formula, E is the desired ripple voltage magnitude, and n is the total number of steps in the trajectory. The Q-Learning algorithm is used in the training process of the pre-trained agent.

2. The loop control method for a digital DC-DC converter according to claim 1, characterized in that, In the step of employing the Q-Learning algorithm, the Q-table used is represented as follows: In the table, Obs1, Obs2, Obs3 to Obsn in the horizontal column represent the state space corresponding to the Markov decision process; A1, A2, A3 to An in the vertical column represent the action space corresponding to the Markov decision process. A1, A2, A3 to An are transfer functions with different zero and pole locations, corresponding to different phase margins.

3. The loop control method for a digital DC-DC converter according to claim 1, characterized in that, During training using the Q-Learning algorithm, the single trajectory for data collection through the agent's interaction with the environment is: From time 0 to T1, the input voltage linearly increases from 0 to V. in During this process, the loop control strategy of the digital DC-DC converter adopts a soft-start scheme; at time T1, the input voltage magnitude is V in The input voltage experiences a random step jump near its value, and at time T2, the input voltage magnitude steps back to V. in At time T3, the load current is in the range of I. out The current value experiences a random step jump once, and at time T4, the load current magnitude steps back to I. out The trajectory is completed and exits at time T5; The agent completes data collection and training within a single trajectory, and improves the policy. Its desired goal is to keep the output voltage of the DC-DC converter stable at the reference voltage for as long as possible. In the single trajectory, when the DC-DC switching frequency is k Hz, the trajectory includes a total of T5 / k steps.

4. A loop control system for a digital DC-DC converter, characterized in that, include: The observation acquisition module is used to acquire the observation value e based on the digital DC-DC converter to be controlled. t and e t-1 ; Among them, e t e is the difference between the output voltage value and the reference voltage value at time t. t-1 The difference between the output voltage value and the reference voltage value at time t is the difference between the output voltage value at the previous time. The zero-pole location acquisition module is used to obtain the acquired observation value e. t and e t-1 The input is a pre-trained agent, and the output is used to adjust the zero and pole positions of the transfer function of the 3p3z controller. The control module is used to obtain the updated transfer function based on the zero and pole positions output by the intelligent agent, and the 3p3z controller outputs a PWM wave based on the updated transfer function; the PWM wave is used to control the conduction state of the power transistor in the DC-DC converter to achieve a constant output voltage value of the DC-DC converter. The mathematical model of the intelligent agent is a Markov decision process, which includes a state space, an action space, and a reward function. The state space S is expressed as S = [e t , e t-1 ]; The actions of the agent in action space A are the zero and pole positions of the discrete 3p3z controller transfer function; The expression for the reward function R is, In the formula, E is the desired ripple voltage magnitude, and n is the total number of steps in the training trajectory. The Q-Learning algorithm is used in the training process of the pre-trained agent.

5. A digital DC-DC converter loop control system according to claim 4, characterized in that, In the step of employing the Q-Learning algorithm, the Q-table used is represented as follows: In the table, Obs1, Obs2, Obs3 to Obsn in the horizontal column represent the state space corresponding to the Markov decision process; A1, A2, A3 to An in the vertical column represent the action space corresponding to the Markov decision process. A1, A2, A3 to An are transfer functions with different zero and pole locations, corresponding to different phase margins.

6. A digital DC-DC converter loop control system according to claim 4, characterized in that, During training using the Q-Learning algorithm, the single trajectory for data collection through the agent's interaction with the environment is: From time 0 to T1, the input voltage linearly increases from 0 to V. in During this process, the loop control strategy of the digital DC-DC converter adopts a soft-start scheme; at time T1, the input voltage magnitude is V in The input voltage experiences a random step jump near its value, and at time T2, the input voltage magnitude steps back to V. in At time T3, the load current is in the range of I. out The current value experiences a random step jump once, and at time T4, the load current magnitude steps back to I. out The trajectory is completed and exits at time T5; The agent completes data collection and training within a single trajectory, and improves the policy. Its desired goal is to keep the output voltage of the DC-DC converter stable at the reference voltage for as long as possible. In the single trajectory, when the DC-DC switching frequency is k Hz, the trajectory includes a total of T5 / k steps.

7. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the digital DC-DC converter loop control method as described in any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the loop control method for the digital DC-DC converter as described in any one of claims 1 to 3.

Images