A modulation strategy design method and system for a DC-DC converter based on reinforcement learning

Abstract

The application discloses a modulation strategy design method and system of a DC-DC converter based on reinforcement learning, and belongs to the technical field of power electronics. The state vector X of the DC-DC converter is taken as input, the quality vector Q of all possible actions composed of the switching mode of the converter and the corresponding duty ratio is taken as output, a deep neural network is constructed as an intelligent agent, the modulation strategy design is regarded as a process of arranging and combining the switching mode of the converter and allocating the duty ratio, through reinforcement learning, the intelligent agent can summarize experience in trial and error without human intervention, finally generates a modulation strategy maximizing the reward, and the modulation strategy design efficiency of the DC-DC converter is improved.

Description

Technical Field

[0001] This invention belongs to the field of power electronics technology, and more specifically, relates to a method and system for designing modulation strategies for DC-DC converters based on reinforcement learning. Background Technology

[0002] Power electronics technology is a key technology for achieving efficient power conversion. DC-DC converters, which enable energy conversion between DC ports, are a core component of power electronics technology and are widely used in electric vehicles, DC microgrids, computer power supplies, and green home applications. DC-DC converters use multiple semiconductors (MOSFETs or diodes) to establish different current loops, thereby transferring power between different ports (input, output, and bidirectional ports). Because DC-DC converters have a large number of switching transistors, they have many switching modes. To achieve a specific energy management task, multiple switching modes need to be combined, and the duration (or duty cycle) of each switching mode needs to be specified to form a specific modulation strategy, making the design process quite complex.

[0003] Currently, the modulation strategy design of DC-DC converters mainly involves manually selecting several specific modes from all switching modes to form a mode sequence, then analyzing the circuit, calculating the duty cycle of each mode, and finally obtaining a specific modulation strategy. As the number of switches or ports increases, the possible combinations of switching states (i.e., switching modes) increase significantly, as do the effective permutations and combinations of switching modes (i.e., usable mode sequences). Furthermore, the duty cycle of each mode can also take different values. In addition, different applications have different requirements and specifications (such as minimizing inductor current ripple and semiconductor component losses), corresponding to different modulation strategies, resulting in a very large design space. Therefore, the traditional DC-DC converter modulation strategy design process is time-consuming and labor-intensive, highly dependent on expert knowledge and experience, leading to low design efficiency. Summary of the Invention

[0004] To address the shortcomings and improvement needs of existing technologies, this invention provides a method and system for designing modulation strategies for DC-DC converters based on reinforcement learning, with the aim of improving the efficiency of modulation strategy design for DC-DC converters.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a method for designing modulation strategies for a DC-DC converter based on reinforcement learning is provided, comprising:

[0006] S1. Construct a deep neural network as an agent. The input of the agent is the state vector X of the DC-DC converter, and the output is the quality vector Q of all possible actions in the agent's action space. The state vector X includes the voltage value V of each port of the DC-DC converter, the circuit switching mode and the corresponding duty cycle, and the initial state vector X0 = {V}. The agent's action space is the combination of all circuit switching modes and their corresponding duty cycles. The quality vector Q is used to characterize the probability of selecting each action to obtain the optimal modulation strategy.

[0007] S2, convert the state vector X at step t. t Input the agent and obtain the mass vector Q corresponding to the action. t ; and according to the mass vector Q t Select an action 'a' from the action space. t The state vector X at step t+1 is obtained. t+1 ; for the state vector X t+1 After evaluation, the reward value R is obtained. t Where t = 0, 1, 2…T, and T represents the upper limit of the number of switching modes allowed in one switching cycle;

[0008] S3. Repeat S2 to allow the agent to run one round, obtaining at most T groups of X. t a t R t and X t+1 ;

[0009] S4, using at most T groups X as described above t a t R t and X t+1 The agent is trained and learned in one round using a reinforcement learning algorithm to update the agent's weights and biases.

[0010] S5. Repeat S2-S4 until the preset number of training rounds or the loss of the agent converges, and then use the trained agent for the modulation strategy design of the actual DC-DC converter.

[0011] Furthermore, in S2, for the state vector X t+1 After evaluation, the reward value R is obtained. t ,include:

[0012] Using the state vector X t+1 The corresponding modulation strategy controls the DC-DC converter, and it is determined whether the control results of the DC-DC converter meet the first and second indicators.

[0013] If satisfied, the reward value R is... t If the result is positive, then stop this round of learning;

[0014] If only the first criterion is met, the reward value R is... t The negative number R SN And continue this round of learning;

[0015] If any of the first indicators is not met, the reward value R will be... t The negative number R BN And stop this round of learning, and proceed to the next round of learning; among them, negative numbers R SN Greater than negative number R BN ;

[0016] The first indicator includes at least one of the following:

[0017] The number of switching modes in one switching cycle is less than or equal to T;

[0018] The sum of the duty cycles satisfies the physical constraints of the circuit;

[0019] The power value of each port meets the constraints;

[0020] The second indicator is: the DC-DC converter can operate stably.

[0021] Furthermore, the first indicator also includes at least one of the following:

[0022] The current ripple satisfies the constraints;

[0023] The efficiency of the DC-DC converter meets the constraints;

[0024] The losses of semiconductor components meet the constraints.

[0025] Further, in S2, according to the mass vector Q t An action selection algorithm is used to select an action 'a' from the action space. t .

[0026] Furthermore, the action selection algorithm employs the ε-greedy algorithm to select the quality vector Q with a certain probability ε. t The action corresponding to the largest element value in the matrix is ​​randomly selected from the action space with probability 1-ε to obtain action a. t .

[0027] Furthermore, during training, the loss function of the agent is:

[0028]

[0029] In the formula, q k Let Q be the mass vector. t The value of the k-th element in the array. The target value is calculated using the DQN algorithm. The calculation formula is:

[0030]

[0031] In the formula, R t Let γ represent the reward after each action, γ represent the influence factor, and max(·) represent selecting the maximum element. Represents a deep neural network. This represents the agent's weights and bias parameters.

[0032] Furthermore, in S5, the trained agent is used for the modulation strategy design of the actual DC-DC converter, including:

[0033] The actual operating conditions of the DC-DC converter are input into the trained agent, wherein the actual operating conditions are the voltages of each port of the DC-DC converter;

[0034] For each step the agent takes, it outputs the corresponding mass vector Q. t According to the mass vector Q t Select the corresponding action a t The action a t Represents the mass vector Q t a t The switching modes and duty cycles corresponding to each element;

[0035] The agent executes one round and obtains up to T sets of switching modes and duty cycles corresponding to one switching cycle, the up to T sets of switching modes and duty cycles constituting a usable modulation strategy.

[0036] Furthermore, in S5, the loss convergence of the agent refers to the number of times the agent generates a usable modulation strategy reaches a set threshold.

[0037] According to a second aspect of the present invention, a modulation strategy design system for a DC-DC converter based on reinforcement learning is provided, including a computer-readable storage medium and a processor;

[0038] The computer-readable storage medium is used to store executable instructions;

[0039] The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method described in any of the first aspects.

[0040] According to a third aspect of the invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the first aspects.

[0041] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0042] (1) The modulation strategy design method of the DC-DC converter of the present invention regards the modulation strategy design as a process of arranging and combining the switching modes of the converter and allocating the duty cycle. A state vector of the DC-DC converter is constructed, which includes the voltage value V of each port of the converter, the circuit switching mode and the duty cycle corresponding to each circuit switching mode. The state vector is used to quantify the modulation strategy of the DC-DC converter under different operating conditions. The voltage value V of each port of the converter is used to characterize the operating condition of the converter. The state vector X is input to the agent. The output of the agent is used to characterize the possibility of selecting each circuit switching mode and the corresponding duty cycle to obtain the optimal modulation strategy. Deep reinforcement learning is used to efficiently search for the combination of switching modes and duty cycles, thereby automatically generating the optimal modulation strategy under the voltage condition, avoiding complex manual design and improving design efficiency.

[0043] (2) Further, during the training process of the agent, the circuit design rules and design requirements are converted into reinforcement learning rewards. Different rewards are given for satisfying different indicators. Positive rewards are used to train the deep neural network to increase the probability of corresponding actions when encountering the same input in the future. Negative rewards are used to train the deep neural network to reduce the probability of corresponding actions when encountering the same input in the future, so as to prompt the deep neural network to automatically explore and learn the modulation strategy that can maximize the reward.

[0044] In summary, the method of this invention uses a deep neural network as an agent to generate modulation strategies and employs a set of quantization rules to provide the rewards required for reinforcement learning, which are then used to train the agent. Through reinforcement learning, the agent can learn from trial and error without human intervention, ultimately generating a modulation strategy that maximizes rewards and improves design efficiency. Attached Figure Description

[0045] Figure 1 This is a flowchart of the modulation strategy design method for the DC-DC converter based on reinforcement learning according to the present invention.

[0046] Figure 2 This is a circuit diagram of a single-inductor multi-port DC-DC converter according to an embodiment of the present invention.

[0047] Figure 3(a) is a circuit diagram of the single-inductor multi-port DC-DC converter in mode m1 in an embodiment of the present invention.

[0048] Figure 3(b) is a circuit diagram of the single-inductor multi-port DC-DC converter in mode m1 in an embodiment of the present invention.

[0049] Figure 3(c) is a circuit diagram of the single-inductor multi-port DC-DC converter in mode m1 in an embodiment of the present invention.

[0050] Figure 3(d) is a circuit diagram of the single-inductor multi-port DC-DC converter in mode m1 in an embodiment of the present invention.

[0051] Figure 3(e) is a circuit diagram of the single-inductor multi-port DC-DC converter in mode m1 in an embodiment of the present invention.

[0052] Figure 4 The state vectors corresponding to different modulation strategies of the single-inductor multi-port DC-DC converter in the embodiments of the present invention are as follows: Figure 4 In the equation (a)-(d), the states X0, X1, X2, and X3 are respectively.

[0053] Figure 5(a) shows the optimal modulation strategy generated by the single-inductor multi-port DC-DC converter under one operating condition in an embodiment of the invention.

[0054] Figure 5(b) shows the optimal modulation strategy generated by the single-inductor multi-port DC-DC converter under another operating condition in the embodiment of the invention.

[0055] Figure 6(a) shows the simulated and experimental waveforms of the inductor current obtained using the modulation strategy generated in Figure 5(a) in the embodiment of the invention.

[0056] Figure 6(b) shows the simulated and experimental waveforms of the inductor current obtained using the modulation strategy generated in Figure 5(b) in the embodiment of the invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0058] In this invention, the terms "first," "second," etc., used in the invention and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0059] like Figure 1 As shown, the modulation strategy design method for a DC-DC converter based on reinforcement learning of the present invention mainly includes:

[0060] S1. Construct a deep neural network as an agent. The input of the agent is the state vector X of the DC-DC converter, and the output is the quality vector Q of all possible actions in the agent's action space. The state vector X quantifies the modulation strategy of the DC-DC converter under different operating conditions, including the voltage values ​​V at each port of the converter, the circuit switching modes, and the duty cycle corresponding to each circuit switching mode. The voltage values ​​V at each port of the converter characterize the operating conditions of the converter. During initialization, all elements except the converter port voltage values ​​are 0, i.e., the initial state vector X0 = {V}. The agent's action space is a combination of all circuit switching modes and the duty cycle corresponding to each circuit switching mode. Each element in the quality vector Q characterizes the probability of selecting each action to obtain the optimal modulation strategy; the larger the value, the greater the probability of obtaining the optimal modulation strategy. The optimal modulation strategy refers to the modulation strategy with the highest reward among the available strategies of the DC-DC converter.

[0061] S2, convert the state vector X at step t. t Input the agent and obtain the mass vector Q corresponding to the action. t ; and based on the mass vector Q t Select an action 'a' from the action space. t The state vector X at step t+1 is obtained. t+1 For the state vector X at step t+1 t+1 After evaluation, the reward value R is obtained. t Where t = 0, 1, 2…T; T represents the upper limit of the number of switching modes allowed in one switching cycle, and when t = 0, it is the initial state, corresponding to the initial state vector X0;

[0062] S3. Repeat S2 to allow the agent to run one round, that is, to allow the state vector of the DC-DC converter modulation strategy to complete one round from the initial state, obtaining at most T sets of X. t a t R t and X t+1 ;

[0063] S4, using at most T groups of X t a t R t and X t+1 The agent is trained and learned in one round using a reinforcement learning algorithm to update the agent's weights and biases.

[0064] S5. Repeat S2-S4 until the preset number of training rounds or the loss of the deep neural network converges, and then use the trained agent for the modulation strategy design of the actual DC-DC converter.

[0065] Specifically, in S1, the state vector X is:

[0066] X = {V,S1,S2,…,S} T ,d1,d2,…,d T}

[0067] Where V represents the voltage values ​​at each port of the converter, used to characterize the operating conditions of the converter, and is a 1×N vector, with N being the number of columns of the DC-DC converter; S j (j=1,2…T) represents different switching modes, which are the on / off combinations of all switches in the DC-DC converter. T is the upper limit of the number of switching modes allowed in one switching cycle, and S j It is a 1×M vector, where M is the number of columns and the elements are either 0 or 1, representing the on and off states of the switches, respectively; d j (j = 1, 2…T) is a scalar representing the duty cycle corresponding to each switching mode, 0 <d j <1.

[0068] That is, the state vector X t It is a 1×w vector, where w represents the number of elements in the state vector, i.e., the total number of columns, with a value of N+M*T+T. It is controlled by the port voltage state V and the switching mode S. j and its corresponding duty cycle d j It can represent the modulation strategy of a DC-DC converter under different operating conditions.

[0069] The mass vector Q is:

[0070] Q={q(m1,d1),q(m1,d2),…q(m1,d t ),q(m2,d1),…,q(m s ,d p )}

[0071] Where, m i (i = 1, 2…s) represent the switching modes of the DC-DC converter, s is the total number of switching modes in the converter, and d j (j=1,2…p) indicates that the duty cycle 0-1 is divided into p equal parts, each part being 1 / p. Each element value q(m) i ,d j ) indicates "selection switch mode m" i and duty cycle d j The action quality of the "combination"; the size of the quality vector Q is 1×n, and the value of the column number n is s*p.

[0072] All possible actions, or the action space, are all combinations of the switching modes of the DC-DC converter and the discrete duty cycles after the duty cycles are divided into 0-1 proportionally. The number of actions is s*p. The quality of an action is a numerical value; the higher the value, the better the action, and the more likely it is to generate the optimal modulation strategy.

[0073] Specifically, in S2, the state vector X at step t is... t The input layer of the deep neural network, through the deep neural network, X t It is mapped to a vector Q of size 1×n. t :

[0074]

[0075] The structure of a deep neural network needs to be appropriately selected based on the complexity of the design. In this embodiment of the invention, a fully connected neural network (FCNN) is selected as the deep neural network. In other embodiments, it can also be a convolutional neural network (CNN) or other deep neural networks.

[0076] Specifically, in S2, according to the mass vector Q t An action selection algorithm is used to select an action 'a' from the action space. t In this embodiment of the invention, the action selection algorithm employs the ε-greedy algorithm, selecting the quality vector Q with a certain probability ε. t Action a corresponding to the largest element value in the middle t Action a is randomly selected from the action space with probability 1-ε. t The action space contains n actions, and action a... t Represents the mass vector Q t a t The combination of switching modes and duty cycles corresponding to each element, a t Different values ​​represent different combinations of switching modes and duty cycles. In other embodiments, the action selection algorithm can also be a completely random algorithm, for example, based on the mass vector Q. t Select an action 'a' completely randomly from the action space. t Alternatively, it could be another action selection algorithm.

[0077] Action a t The corresponding switching modes and duty cycles are written into the state vector X. t From this, we can obtain the state vector X at step t+1. t+1 This is used to reflect the updated modulation strategy.

[0078] Specifically, in S2, for the state vector X at step t+1... t+1 After evaluation, the reward value R is obtained.t ,include:

[0079] The DC-DC converter is controlled using the updated modulation strategy. It is then determined whether the control results of the converter meet the specified first and second indicators. If they do, the reward value R is awarded. t R is a positive number P If only the first criterion is met, the reward value R is increased. t For a smaller negative number R SN Continuing this round of learning indicates that one switching mode has been used, encouraging the use of fewer switching modes to generate the optimal modulation strategy in the future; if any of the first indicators are not met, the reward value R is... t For a large negative number R BN That is, the negative number R SN Greater than negative number R BN At the same time, the current learning round is stopped, and the next learning round begins. The positive reward is used to train the deep neural network to learn from subsequent encounters with the same input X. t The probability of a corresponding action is increased; negative rewards are used to train deep neural networks to respond to the same input X later. t This reduces the likelihood of corresponding actions.

[0080] Specifically, the first indicator includes:

[0081] The number of switching modes in one switching cycle is less than or equal to T;

[0082] The sum of the duty cycles satisfies the physical constraints of the circuit. In this embodiment of the invention, the physical constraint satisfied by the sum of the duty cycles is: the sum of the duty cycles is less than or equal to 1. In other embodiments, the sum of the duty cycles may also be greater than 1, etc., depending on the specific circuit physical constraints.

[0083] The power value of each port meets the constraints.

[0084] The second indicator is that the DC-DC converter can operate stably.

[0085] In other embodiments, the metrics for measuring the reward are not fixed and can be adjusted based on the converter's operating metrics to meet design requirements. For example, in other embodiments, the first metric may also include: current ripple meeting constraints, converter efficiency meeting constraints, or semiconductor element losses meeting constraints, etc.

[0086] Specifically, in S3, when the agent completes one round and moves T steps, it can obtain T sets of X. t a t R t and X t+1If, in the current round, the agent has completed a certain step and found a usable modulation strategy—for example, using the updated modulation strategy to control the DC-DC converter, and the control results of the converter satisfy the specified first and second indicators—then X is obtained accordingly. t a t R t and X t+1 The number of groups is less than T.

[0087] In S4, the at most T groups X are used. t a t R t and X t+1 The agent is trained and learned using a reinforcement learning algorithm to update its weights and biases.

[0088] Specifically, in this embodiment of the invention, at each step of the intelligent agent's operation, the obtained X is... t a t R t X t+1 As a set of sample data, it is stored in the memory pool of the deep neural network. The memory pool also contains a large number of sample data before the end of the current step. When the sample data reaches the maximum capacity, the first-in-first-out principle is adopted to remove the sample data that entered the memory pool first.

[0089] A fixed number of samples are randomly drawn from all the sample data in the memory pool to train and learn the deep neural network in order to update the agent's weights and biases.

[0090] Specifically, in this embodiment of the invention, the DQN algorithm is used to train a deep neural network, and the training objective is to obtain a modulation strategy with the highest possible reward.

[0091] Based on reward value R t The target value was calculated using the DQN algorithm. for:

[0092]

[0093] Among them, R t The value represents the reward obtained after each action, max(·) represents selecting the maximum element; γ represents the influence factor; k represents the quality vector Q output by the deep neural network. t The k-th element in the array, i.e., a t This corresponds to the selected combination of switching modes and duty cycles.

[0094] The calculated target value With the real q k Perform loss training in between, and then adjust the weights and bias parameters of the deep neural network in reverse. Make q k Can approach Among them, the real q k The value given to the deep neural network, i.e., the quality vector Q. t The value of the k-th element in the array.

[0095] Specifically, the loss function Loss is:

[0096]

[0097] By calculating parameters The gradient of Loss is as follows:

[0098]

[0099] Here, α is the learning rate of the deep neural network. After training, the loss decreases, indicating that the deep neural network has updated its design strategy.

[0100] Specifically, in S5, the loss convergence of the deep neural network, that is, meeting the requirements for successful training of the deep neural network, corresponds to the number of times the neural network can generate a usable modulation strategy (satisfying the first and second indicators mentioned above) reaches a set threshold. If the preset number of training rounds is reached, the deep neural network is considered to have failed to train successfully and the program needs to be run again for the next learning cycle.

[0101] The trained agent is used for the modulation strategy design of a practical DC-DC converter, specifically including:

[0102] The actual operating conditions (i.e., port voltages) of the DC-DC converter are input into the trained agent. Each time the agent takes a step, it outputs a mass vector Q. t According to the mass vector Q t Select the corresponding action a t The action a t Represents the mass vector Q t a t Each element corresponds to a switching mode and its corresponding duty cycle;

[0103] The agent executes one round and obtains at most T sets of switching modes and duty cycles corresponding to one switching cycle. The at most T sets of switching modes and duty cycles corresponding to one switching cycle constitute a usable modulation strategy.

[0104] The present invention will be further described in detail below with reference to specific embodiments.

[0105] Single-Inductor Multi-Port (SIMP) DC-DC converters possess the characteristics of "high silicon content and low magnetism." They are small in size and easy to integrate, and are currently widely used in applications requiring multi-port connectivity, such as DC microgrids, electric vehicles, consumer electronics, and LED lighting. In this embodiment of the invention, a non-isolated single-inductor bidirectional-port (SIBP) DC-DC converter is used as an example to illustrate the technical solution of the invention in detail.

[0106] like Figure 2 The diagram shown is a topology of a non-isolated single-inductor bidirectional port DC-DC converter provided in an embodiment of the present invention. This SIBP converter includes one inductor, three switching transistors, and two diodes, and can realize input port V... i Battery port V b and load port V o The power flow between them. By properly controlling the three switches in this converter, the three ports can operate in Buck or Buck-Boost mode, with fewer port voltage limitations and a wide operating range. For this converter, V b >V o This is a condition that needs to be constantly satisfied; therefore, it can operate under three voltage relationships: V i >V b >V o V b >V i >V o V b >V o >V i .

[0107] This SIBP converter has 5 switching modes, such as Figures 3(a)-3(e) As shown in the figure, Figure 3(a) represents switching mode m1, with the corresponding switching states: S1 is on, S2 and S3 are off; Figure 3(b) represents switching mode m2, with the corresponding switching states: S1 and S2 are off, and S3 is on; Figure 3(c) represents switching mode m3, with the corresponding switching states: S1 is off, and S2 and S3 are on; Figure 3(d) represents switching mode m4, with the corresponding switching states: S1, S2, and S3 are off; Figure 3(e) represents switching mode m5, with the corresponding switching states: S1 and S3 are off, and S2 is on. For each switching mode, the inductor current i LEnergy flows through different ports, allowing it to be transferred from one part to another. It's important to note that the inductor is magnetized or demagnetized in each switching mode, but the volt-second balance condition must be met within a single switching cycle. Because of the numerous switching modes and combinations of switching modes and duty cycles, the modulation strategy design is complex. The following section describes the modulation strategy process of a single-inductor multi-port DC-DC converter under different operating conditions, using the method of this invention.

[0108] The main parameters of the SIBP converter in this embodiment are shown in Table 1. Its key parameters (port voltage, etc.) are input into a Python program for training a deep neural network. In this embodiment, the maximum number of steps in each working cycle (one round of agent operation) is T = 8, meaning that the maximum number of switching modes contained in one switching cycle is 8.

[0109] Table 1 Key parameters of the prototype

[0110]

[0111] The operating conditions (i.e., port voltage), the mode sequence, and the duty cycle corresponding to each switching mode are used as the modulation strategy state vector X. t The element, X t Inputting a deep neural network, as described in this embodiment of the invention, such as Figure 4 As shown:

[0112] (A) The operating conditions are determined by the three port voltages V of the SIBP converter. i V b V o Characteristic, such as Figure 4 As shown in (a), V in the initialized state vector X0 is the voltage of the three ports, providing a basis for the converter to operate under certain conditions.

[0113] (B) The modal sequence is characterized by the on / off states of switches S1, S2, and S3. j (j=1,2…8) are all 1×3 vectors, with the 3 columns representing the on / off states of the 3 switches, and their element values ​​being either 0 or 1. j The switching modes at a certain step are represented by a total of 8 steps, meaning the maximum number of switching modes within a switching cycle is 8. For example... Figure 4 As shown in (a), S j (j=1,2…8) are all 0, indicating that the switching mode has not been selected yet.

[0114] (C)d j (j=1,2…8) represents the duty cycle corresponding to each switching mode, that is, the time during which each switching mode is active, and the duty cycle is a value between 0 and 1.

[0115] (D) such as Figure 4 As shown in (a), except for the port voltage V i V b V o In X0, all other elements are 0, indicating that a switching mode has not yet been selected. At the start of the modulation strategy design, if the first step selects the switching mode m1 (S1 = {1,0,0}) and duty cycle d1, the state vector is updated to X1, as follows: Figure 4 As shown in (b) above. Based on X1, the second step of the modulation strategy selects the switching mode m2 (S2 = {0,0,1}) and the duty cycle d2, and the state vector is updated to X2, as follows. Figure 4 As shown in (c) above. Based on X2, the third step of the modulation strategy selects the switching mode m3 (S3 = {0,1,1}) and the duty cycle d3, and the state vector is updated to X3, as shown in (c). Figure 4 As shown in (d) in the figure.

[0116] (E) From (A)(B)(C)(D), the state vector X t It contains 35 elements, therefore the input layer of a deep neural network has 35 nodes. For the state vector X... t The first three elements in the data, namely the port voltage, are normalized, which makes it easier for the deep neural network to fit and improves the training speed.

[0117] The output of a deep neural network is a mass vector Q. t In this embodiment of the invention, vector Q t The element values ​​in the algorithm represent the action quality of "selecting each combination of switching modes and duty cycles", and the number of elements is the number of actions. Specifically, the reinforcement learning algorithm used is the DQN algorithm, whose action space is discrete. In order to transform the modulation strategy design problem into a discrete problem, in this embodiment of the invention, the duty cycle 0-1 is divided into 19 equal parts, that is, each part has a duty cycle of 0.05.

[0118] The action space encompasses all combinations of switching modes and discrete duty cycles. Since this SIBP converter offers 5 selectable switching modes and the duty cycle is divided into 19 parts, the action space size is n = 95, and the mass vector Q... t The size is 1×95. For example, q (5,1) Corresponding to the switching mode m5, the duty cycle is 0.05; q (1,19) Corresponding to the switching mode m1, the duty cycle is 0.95. Assume q... (5,1) =7 and q (1,19) =4 means that selecting the switching mode m5 and duty cycle 0.05 has better quality than selecting the switching mode m1 and duty cycle 0.95.

[0119] The output layer of a deep neural network has 95 nodes, each node representing the action quality of "selecting a certain combination of switching modes and duty cycles".

[0120] Based on the above inputs and outputs, the deep neural network in this embodiment of the invention has three layers: an input layer, a hidden layer, and an output layer. A fully connected neural network is chosen as the deep neural network, and the calculation formula for the deep neural network is:

[0121] X i+1 =g(w i ·X i +b i )

[0122] Among them, X i X i+1 These are the input and output, X. i+1 It also serves as the input for the next layer, and so on; g() is the activation function, w i b i These are the weight and bias matrices, representing the weights and biases of all layers in a deep neural network. i b i These are the internal parameters of a deep neural network.

[0123] When i equals 1, input X. i This is the input layer of the entire deep neural network, i.e., the state vector X. t There are 35 nodes in total. Output X i+1 This is a hidden layer of a deep neural network with 132 nodes, and g() is the ReLU activation function.

[0124] When i equals 2, input X. i This refers to the hidden layer of a deep neural network, which outputs X. i+1 This refers to the output layer of a deep neural network, also known as the mass vector Q. t That is, there are 95 nodes and no activation function g().

[0125] Based on the output quality vector Q of the deep neural network t The ε-greedy algorithm is applied, with a certain probability ε based on the quality vector Q. t Select action a for the highest element value t Action a is randomly selected with probability 1-ε. t This yields the switching mode and duty cycle for the next step. Action a t For the combination of switching mode and duty cycle, a t The value ranges from 0 to 94, representing 95 different combinations of switching modes and duty cycles.

[0126] The number of learning iterations m and the probability ε are related as follows:

[0127]

[0128] Where ε0 is the initial probability, 0≤η≤1 is the variable for adjusting the rate of change, and in this embodiment of the invention, ε0=0.7, η=1. m is the number of reinforcement learning iterations. max This is the upper limit of m. In this embodiment of the invention, let m be... max =400000.

[0129] Once the next switching mode and its corresponding duty cycle are obtained, this switching mode and duty cycle are added to the modulation strategy, and the modulation strategy is updated to X. t+1 The updated modulation strategy is evaluated according to the circuit rules to obtain a reward R. t Specifically, circuit simulation software is used to control the SIBP converter with the updated modulation strategy. Simulation is performed, and the current modulation strategy is evaluated based on the simulation results to obtain the reward R. t In this embodiment of the invention, the first indicator in the evaluation criteria includes:

[0130] 1) The number of switching modes within a switching cycle is less than or equal to the upper limit T, i.e., 8;

[0131] 2) The sum of duty cycles is not greater than 1;

[0132] 3) The power input to the photovoltaic port shall not exceed the rated power value of 240W;

[0133] The second indicator is that the inductance L satisfies the volt-second balance.

[0134] When both the first and second indicators mentioned above are met, it indicates that the modulation strategy is feasible and will yield a large positive reward R. P =600, and stop this round of learning; when only the first indicator is satisfied, a small negative reward R will be obtained. SN = -10; If any indicator in the first metric is not met, a large negative reward R will be given. BN = -300, and this round of learning ends, and the next round begins.

[0135] Store the four values ​​X at step t. t a t R t X t+1 In this embodiment of the invention, after each step the agent takes, 250 sets of X are randomly selected from the memory pool. t a t R t X t+1 The sample data is used to train and learn the deep neural network's internal parameters through a reinforcement learning algorithm. An update is performed. In other embodiments, the agent may complete a round and utilize multiple sets of X randomly drawn from the memory pool. t a t R t X t+1 The sample data is used to train the neural network. In this embodiment of the invention, the specific training and learning process is as follows:

[0136] Based on the following DQN algorithm formula, R0 is calculated for 250 sets of sample data. t and X t+1 Processing:

[0137]

[0138] Adjusting the weights and bias parameters of a deep neural network Make q k Can approach Adjust by the following formula

[0139]

[0140] Here, 0.001 represents the learning rate of the deep neural network. After training, the loss decreases, indicating that the deep neural network has updated its design strategy.

[0141] The overall process in the above embodiments can be summarized as follows:

[0142] (1) State vector X t (2) Input into NN; (3) Based on the output mass vector Q t The ε-greedy algorithm takes action a t (3) After updating the modulation strategy, evaluate the action and give a reward R. t (4) Update q according to the DQN algorithm formula. * (5) Fit q ​​using NN * Calculate the loss and adjust the NN parameters in reverse.

[0143] In this embodiment of the invention, when a negative reward R is obtained... BN = -300 or positive reward R P When the value is 600, one round of the modulation strategy design process ends, and then the next round starts again from the initial state X0 of the modulation strategy.

[0144] When the number of training iterations reaches the maximum value m max (i.e., 400,000), or the training is considered complete when a feasible modulation strategy is found 400 times.

[0145] In this embodiment of the invention, when the port voltage V i=20V, V b =30V, V o When the voltage is 24V, the deep neural network can be trained in about 10 minutes. The modulation strategy automatically generated by the trained deep neural network is shown in Figure 5(a).

[0146] When the port voltage is changed to V i =33V, V b =37V, V o When the voltage is 24V, the deep neural network needs to be retrained. The modulation strategy automatically generated by the retrained deep neural network is shown in Figure 5(b).

[0147] The generated modulation strategy is used for actual SIBP converter control when the port voltage V i =20V, V b =30V, V o When the voltage is 24V, using the modulation strategy shown in Figure 5(a), the simulated and experimental waveforms of the inductor current are shown in Figure 6(a), where the left side shows the simulated waveforms of the drive signal and inductor current, and the right side shows the experimental waveforms of the inductor current. When the port voltage V... i =33V, V b =37V, V o When the voltage is 24V, using the modulation strategy shown in Figure 5(b), the simulated and experimental waveforms of the inductor current are shown in Figure 6(b). The experimental and simulated waveforms of the inductor current are basically consistent, which verifies the feasibility and reliability of the modulation strategy design method for the DC-DC converter based on reinforcement learning proposed in this paper.

[0148] The modulation strategy design method for DC-DC converters of the present invention views modulation strategy design as a process of arranging and combining the switching modes of the converter and allocating duty cycles. The modulation strategy (circuit switching modes and the corresponding duty cycle for each circuit switching mode) of the DC-DC converter under different operating conditions is input as a state vector X to the agent. The output of the agent is used to characterize the probability of selecting each circuit switching mode and its corresponding duty cycle to obtain the optimal modulation strategy. Deep reinforcement learning is used to efficiently search for combinations of switching modes and duty cycles, thereby automatically generating the optimal modulation strategy under the given voltage condition, avoiding complex manual design and improving design efficiency.

[0149] During the training of the agent, the circuit design rules and design requirements are converted into reinforcement learning rewards. Different rewards are given for meeting different indicators. Positive rewards are used to train the deep neural network to increase the probability of the corresponding action when encountering the same input in the future. Negative rewards are used to train the deep neural network to reduce the probability of the corresponding action when encountering the same input in the future, so as to prompt the deep neural network to automatically explore and learn the modulation strategy that can maximize the reward.

[0150] The DQN algorithm is used to train a deep neural network, which transforms the modulation strategy design problem into a reinforcement learning problem of finding the maximum sum of rewards. Reinforcement learning is used to continuously explore the action space and learn from experience during the trial and error process, and finally find the optimal solution efficiently. The optimal modulation strategy under different operating conditions can be generated without human intervention. It has strong adaptability and saves storage space.

[0151] The present invention also provides a modulation strategy design system for a DC-DC converter based on reinforcement learning, comprising a computer-readable storage medium and a processor; the computer-readable storage medium is used to store executable instructions; the processor is used to read the executable instructions stored in the computer-readable storage medium and execute the modulation strategy design method for the DC-DC converter based on reinforcement learning in the above embodiments.

[0152] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the modulation strategy design method for a DC-DC converter based on reinforcement learning as described in the above embodiments.

[0153] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A modulation strategy design method for a DC-DC converter based on reinforcement learning, characterized in that, include: S1. Construct a deep neural network as an agent. The input of the agent is the state vector X of the DC-DC converter, and the output is the mass vector Q of all possible actions in the agent's action space. The state vector X includes the voltage values ​​V at each port of the DC-DC converter, the circuit switching modes and their corresponding duty cycles, and the initial state vector. The agent's action space is a combination of all circuit switching modes and their corresponding duty cycles; the quality vector Q is used to characterize the probability of selecting each action to obtain the optimal modulation strategy. S2, the first t The state vector X of the step t Input the agent and obtain the mass vector Q corresponding to the action. t ; and according to the mass vector Q t Select an action from the agent's action space. a t , obtained the t +1 step state vector X t+1 For the state vector X t+1 After evaluation, a reward value is obtained. R t ;in, t =0,1,2… T , T This indicates the upper limit of the number of circuit switching modes allowed within a single switching cycle; S3. Repeat S2, allowing the agent to run one round and obtain at most... T Group X t , a t , R t and X t+1 ; S4, utilizing the maximum T Group X t , a t , R t and X t+1 The agent is trained and learned in one round using a reinforcement learning algorithm to update the agent's weights and biases. S5. Repeat S2-S4 until the preset number of training rounds or the loss of the agent converges, and use the trained agent for the modulation strategy design of the actual DC-DC converter. In S2, the state vector X t+1 After evaluation, a reward value is obtained. R t ,include: Using the state vector X t+1 The corresponding modulation strategy controls the DC-DC converter, and it is determined whether the control results of the DC-DC converter meet the first and second indicators. If satisfied, the reward value is... R t If the result is positive, then stop this round of learning; If only the first criterion is met, the reward value is... R t negative number R SN And continue this round of learning; If any of the first indicators is not met, the reward value will be... R t negative number R BN And stop this round of learning, and proceed to the next round; among them, negative numbers R SN Greater than negative numbers R BN ; The first indicator includes at least one of the following: The number of switching modes in a circuit within one switching cycle is less than or equal to T ; The sum of the duty cycles satisfies the physical constraints of the circuit; The power value of each port meets the constraints; The second indicator is: the DC-DC converter can operate stably.

2. The method according to claim 1, characterized in that, The first indicator also includes at least one of the following: The current ripple satisfies the constraints; The efficiency of the DC-DC converter meets the constraints; The losses of semiconductor components meet the constraints.

3. The method according to claim 1, characterized in that, In S2, according to the mass vector Q t An action selection algorithm is used to select an action from the agent's action space. a t .

4. The method according to claim 3, characterized in that, The action selection algorithm adopts ε-greedy Algorithm, with a certain probability ε Select the mass vector Q t The action corresponding to the largest element value in the middle is given with a probability of 1. ε The action is obtained by randomly selecting an action from the agent's action space. a t .

5. The method according to claim 1, characterized in that, During training, the loss function of the agent for: In the formula, q k Let Q be the mass vector. t The first in k Each element value The target value is calculated using the DQN algorithm. The calculation formula is: In the formula, R t This represents the reward received after each action. Represents the impact factor, max( () indicates selecting the largest element. f φ ( ) represents a deep neural network, and φ represents the weights and bias parameters of the agent.

6. The method according to claim 1, characterized in that, In S5, the trained agent is used for the modulation strategy design of the actual DC-DC converter, including: The actual operating conditions of the DC-DC converter are input into the trained agent, wherein the actual operating conditions are the voltage values ​​V of each port of the DC-DC converter; For each step the agent takes, it outputs the corresponding mass vector Q. t According to the mass vector Q t Select the corresponding action a t The action a t Represents the mass vector Q t The Middle a t The circuit switching modes and duty cycles corresponding to each element; The agent executes one round and obtains at most the corresponding values ​​within one switching cycle. T The switching modes and duty cycles of the circuit group, the most T The switching modes and duty cycles of the circuit group constitute the available modulation strategies.

7. The method according to claim 1, characterized in that, In S5, the loss convergence of the agent refers to the number of times the agent generates a usable modulation strategy reaches a set threshold.

8. A modulation strategy design system for a DC-DC converter based on reinforcement learning, characterized in that, Includes computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is configured to read executable instructions stored in the computer-readable storage medium and execute the method according to any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images