Grid-connected inverter control method based on DDPG algorithm and inverter
Through the grid-connected inverter control method based on DDPG algorithm, the problem of slow dynamic response and insufficient adaptability of PI control method is solved, and more refined control and higher system stability and adaptability are achieved, which is suitable for complex scenarios Inverter system.
Patent Information
- Application Number
- CN202410858994.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2024-11-12
AI Technical Summary
The PI control method of existing grid-connected inverters has slow dynamic response, and there are limitations to adapt to changing system conditions, and parameter adjustment depends on experience, which is suitable for complex scenarios inadequate performance.
Using the control method based on the DDPG algorithm, the current electrical parameters of the grid-connected inverter are obtained and converted into a state space representation. The action signal is generated using the DDPG algorithm model, and the control signal of the switch tube is generated through the PWM generator, using the Critic network and The update mechanism of the Actor network adapts to different working conditions.
It improves the continuity and smoothness of control, enhances the stability and adaptability of the system, reduces the correlation in the learning process, improves learning efficiency, and enables the inverter to respond quickly and adapt to changing environments.
Smart Images

Figure CN118944408A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of power inverters, and in particular relates to a grid-connected inverter control method based on a DDPG algorithm and an inverter. Background Art
[0002] At present, the control method of grid-connected inverter mainly uses traditional PI control. The PI control method of grid-connected inverter first establishes the model of the inverter circuit; then abstracts and approximates the transfer function of the inverter circuit; adds PI control to the circuit model, calculates the frequency response characteristics of the transfer function; roughly selects PI parameters according to the frequency response characteristics and actual needs, and then observes the effect of PI control in the actual circuit or simulation environment, and fine-tunes the parameters based on experience. PI parameters are related to the hardware parameters of the system. The development of new inverters requires re-tuning parameters, and parameter tuning requires algorithm engineers to have rich experience. In addition, when PI controllers are used in inverter systems, the dynamic response is slow and there are limitations in adapting to changing system conditions.
[0003] Specifically, the PI controller first approximates and simplifies the circuit model, and then adds the PI controller to the circuit, so there is an error in the calculated transfer function. The PI controller has few parameters and limited room for adjustment. For complex scenarios or scenarios with higher performance requirements, a more complex controller is required. In addition, since the switch tube has a delay in the inverter circuit and a dead zone is set, errors will also be introduced in the circuit model. Summary of the invention
[0004] To solve the above problems, the purpose of the present invention is to provide a grid-connected inverter control method and inverter based on the DDPG algorithm, which can accurately capture the dynamic characteristics of the system by obtaining the current electrical parameters of the grid-connected inverter and converting them into state space representation. This representation method makes state estimation more accurate and provides a reliable basis for subsequent control.
[0005] The technical solution provided by the present invention is: a grid-connected inverter control method based on the DDPG algorithm, comprising the following steps:
[0006] Obtain the current electrical parameters of the grid-connected inverter, and obtain the current state space (id) corresponding to the current electrical parameters based on the current electrical parameters (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) );
[0007] Taking the target electrical parameters of the grid-connected inverter as the target, generating the current moment action through the DDPG algorithm model based on the current moment state space, and obtaining the current moment modulated wave signal through the dq inverse transformation of the current moment action;
[0008] Generate a control signal for the switch tube through a PWM generator based on the modulation wave signal at the current moment;
[0009] Among them, the Critic network in the DDPG algorithm model also obtains the state space and reward at the next moment, and stores the current state space, action, next state space, and reward as a set of data in the experience playback area to update the actor network and Critic network in the DDPG algorithm model.
[0010] Preferably, based on the current electrical parameter, the current state space (id (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ) further comprises:
[0011] Get the current electrical parameter i of the grid-connected inverter and convert it into the corresponding d-axis component id in the dq coordinate system through dq transformation (t) and the q-axis component iq (t) ;
[0012] Calculate the difference ed between the current electrical parameter and the target electrical parameter d-axis component, and calculate the difference q between the current electrical parameter and the target electrical parameter (t) The difference of the axis components eq (t) ;
[0013] Calculate the time integral value ∫ed of the difference between the current electrical parameter and the target electrical parameter d-axis component (t) , calculate the time integral value ∫eq of the difference between the current electrical parameter and the target electrical parameter q-axis component (t) .
[0014] Preferably, updating the Critic network in the DDPG algorithm model further includes:
[0015] Use the TD algorithm to update parameters. The specific update method is as follows:
[0016] action next =U(ω,state next ),
[0017] q t=Q(ω,state,action),
[0018] q t+1 =Q(ω,state next , action next ),
[0019] δ=q t+1 -(λ*q t +r),
[0020]
[0021] ω=ω-α*δ*d ω ,
[0022] Among them, the actor network in the DDPG algorithm model is denoted as U (θ) , the critic network is denoted as Q (ω) , θ and ω are the parameters of the neural network, state represents the current state space, state next Indicates the spatial state at the next moment, action indicates the action at the current moment, action next The next action, δ represents the time difference error, r represents the reward, and λ represents the coefficient.
[0023] Preferably, the Actor network in the DDPG algorithm model is updated using stochastic gradient ascent to update parameters:
[0024]
[0025] θ=θ+β*δ*d θ
[0026] Preferably, before obtaining the current electrical parameters of the grid-connected inverter, the method further includes:
[0027] A DDPG algorithm model including an actor network and a critic network is established. The input of the Actor network is the current parameters and voltage parameters of the grid-connected inverter, and the output is the actions that the grid-connected inverter can perform. The input of the Critic network is the current parameters, voltage parameters of the grid-connected inverter in the environment, and the actions that the grid-connected inverter can perform.
[0028] Preferably, the method further comprises:
[0029] After building the DDPG algorithm model, set the total number of training times to episodes_num, and each episode to train step_num steps;
[0030] Before deploying the DDPG algorithm model, the grid-connected inverter environment is reset and the current parameters and voltage parameters of the grid-connected inverter are obtained.
[0031] Preferably, the method further comprises, when training the DDPG algorithm model:
[0032] If the grid-connected inverter environment triggers the termination condition, the actor network is stored when the training episode exceeds episode_num;
[0033] If the grid-connected inverter environment does not trigger the termination condition, the actor network is stored when the training step exceeds step_num and the training episode exceeds episode_num.
[0034] Based on the same concept, the present invention also provides a grid-connected inverter based on the DDPG algorithm, comprising:
[0035] The state acquisition module is used to obtain the current electrical parameters of the grid-connected inverter, and obtain the current state space (id) corresponding to the current electrical parameters based on the current electrical parameters. (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) );
[0036] A modulation wave calculation module is used to generate a current moment action based on the current moment state space through a DDPG algorithm model with the target electrical parameters of the grid-connected inverter as the target, and obtain a current moment modulation wave signal by performing a dq inverse transformation on the current moment action;
[0037] A control signal generating module, used to generate a control signal of a switch tube through a PWM generator based on the modulation wave signal at the current moment;
[0038] Among them, the Critic network in the DDPG algorithm model also obtains the state space and reward at the next moment, and stores the current state space, action, next state space, and reward as a set of data in the experience playback area to update the actor network and Critic network in the DDPG algorithm model.
[0039] Preferably, the state acquisition module obtains the current state space (id) corresponding to the current electrical parameter based on the current electrical parameter. (t) , iq (t) ,ed (t) , eq (t) ,∫ed(t) , ∫eq (t) , vd (t) , vq (t) ) further comprises:
[0040] Get the current electrical parameter i of the grid-connected inverter and convert it into the corresponding d-axis component id in the dq coordinate system through dq transformation (t) and the q-axis component iq (t) ;
[0041] Calculate the difference ed between the current electrical parameter and the target electrical parameter d-axis component, and calculate the difference q between the current electrical parameter and the target electrical parameter (t) The difference of the axis components eq (t) ;
[0042] Calculate the time integral value ∫ed of the difference between the current electrical parameter and the target electrical parameter d-axis component (t) , calculate the time integral value ∫eq of the difference between the current electrical parameter and the target electrical parameter q-axis component (t) .
[0043] Based on the same concept, the present invention also provides a readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, any one of the above-mentioned grid-connected inverter control methods based on the DDPG algorithm is implemented.
[0044] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art:
[0045] 1. The present invention obtains the current electrical parameters of the grid-connected inverter and converts them into state space representation. This solution can accurately capture the dynamic characteristics of the system. This representation method makes the state estimation more accurate and provides a reliable basis for subsequent control. The DDPG algorithm is used to generate the action at the current moment, that is, the modulation wave signal, and then the control signal of the switch tube is generated through the PWM generator. This process ensures the continuity and smoothness of the control. The advantage of the DDPG algorithm in processing continuous action space makes the control more precise and improves the stability of the system.
[0046] 2. The Critic network in the DDPG algorithm model of the present invention not only evaluates the value of the current strategy, but also learns state transitions and reward changes by obtaining the state and reward at the next moment. This learning method enables the algorithm to adapt to changes in different working conditions, improving the adaptability and robustness of the system. The application of the experience replay mechanism enables the algorithm to learn from historical experience, reduces the correlation in the learning process, and improves learning efficiency. This is especially important for systems such as inverters that require fast response, because it can quickly learn and adapt in a constantly changing environment. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings, wherein:
[0048] Figure 1 It is a schematic diagram of the interaction between the DDPG algorithm model of the present invention and the inverter environment;
[0049] Figure 2 This is a schematic diagram of the internal architecture of the DDPG algorithm model of the present invention;
[0050] Figure 3 Schematic diagram of the implementation process of the DDPG algorithm model of the present invention. DETAILED DESCRIPTION
[0051] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in a very simplified form and use non-precise ratios, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.
[0052] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.
[0053] At present, when the system changes and PI parameters need to be re-tuned, they are tuned by experienced algorithm engineers. For defects in PI parameter performance, algorithm engineers usually make some algorithm compensation based on experience in specific scenarios to improve the performance of the PI controller.
[0054] Reinforcement learning, especially deep reinforcement learning, is a relatively new discipline and has not yet been used in the field of inverter control. In addition, reinforcement learning solutions have certain requirements for the computing performance of the system. Traditional control chips cannot meet the computing power requirements and cannot be truly implemented. In recent years, many DSP chips have integrated AI instruction sets, NPU and other heterogeneous acceleration units, making it possible to deploy deep reinforcement learning solutions in control DSPs.
[0055] For the inverter system, the abstracted model makes many approximations, and some factors cannot be accurately expressed in the model. In addition, the PI parameters are small, the algorithm is too simple, and the performance upper limit of the algorithm is limited. When the system changes are more complex, PI control is difficult to meet the needs.
[0056] Based on the above factors, the inventor of the present invention thought of increasing the parameters of the controller. The more parameters a neural network has, the more complex tasks it can handle, adapt to more environmental changes, and provide more precise control.
[0057] Since the data of reinforcement learning is obtained through the interaction between the algorithm model and the environment, some factors in the environment that are difficult to quantify and calculate, such as the delay of the switch tube, dead zone, inductor resistance, line resistance, and complex transfer functions of the system will be truly reflected in the data. The solution learned by the algorithm model will take these factors into account to achieve better performance. In addition, the machine learning solution only requires the algorithm engineer to master the process. When the inverter hardware parameters change, the algorithm model can be re-learned according to the fixed process. Engineers do not need to master the details of the algorithm, which is more convenient for the development and iteration of new products.
[0058] Since the action space of inverter control is continuous, the present invention selects the DDPG (deep deterministic policy gradient) algorithm in reinforcement learning as the control algorithm of the inverter. The DDPG algorithm is a reinforcement learning algorithm that adopts an actor-critic architecture and is suitable for scenarios where the action space is a continuous domain.
[0059] First embodiment
[0060] The DDPG algorithm used in the present invention contains two neural networks, an actor network, denoted as U(θ), and a critic network, denoted as Q(ω), where θ and ω are parameters of the neural network. The actor network is responsible for calculating the action action according to the state of the environment, and the critic network is responsible for judging the quality of the action action output by the actor network.
[0061] The dynamic space is (Ed, Eq), Ed and Eq are the expected values of the voltage in the dq coordinate system. These two values are transformed through dq inverse transformation to obtain the modulation wave of the three phases abc, and then the control signal of the switch tube is obtained through the PWM generator.
[0062] Taking the current parameters as an example below (the voltage parameters are similar and will not be described in detail here), the state space is (id(t), iq(t), ed(t), eq(t), ∫ed(t), ∫eq(t), vd(t), vq(t)), where id represents the d-axis component of the current in the dq coordinate system, iq represents the q-axis component of the current in the dq coordinate system, ed represents the difference between the d-axis current target value and the actual value, eq represents the difference between the q-axis current target value and the actual value, ∫ed(t) represents the time-integrated value of ed, and ∫eq(t) represents the time-integrated value of eq.
[0063] The reward is -(ed*ed+eq*eq).
[0064] See also Figure 1 , the DDPG algorithm model interacts with the inverter environment, obtains the state (state), the actor network calculates the action action based on the state, the environment executes the action action, the algorithm model obtains the state (state_next) and reward (reward) of the environment at the next moment, and stores (state, action, state_next, reward) as a set of data in the experience area of the algorithm model, and uses these data to update the actor network and critic network.
[0065] Furthermore, the Critic network uses the TD algorithm to update parameters. The specific update method is as follows:
[0066] action next =u(ω,state next )
[0067] q t =Q(ω,state,action)
[0068] q t+1 =Q(ω,state next , action next )
[0069] δ=q t+1 -(λ * q t +r)
[0070]
[0071] ω=ω-α * δ * d ω
[0072] The Actor network uses stochastic gradient ascent to update parameters as follows:
[0073]
[0074] θ=θ+β*δ*d θ
[0075] After the network parameters are updated, they interact with the environment to generate new data and then update the network parameters. After multiple rounds of this cycle, the network parameters and rewards will converge to better values and the training is completed.
[0076] After training is completed, the actor network is deployed to the actual environment or simulation environment, and then the entire system is put into operation.
[0077] Therefore, the technical solution of this embodiment is: a grid-connected inverter control method based on the DDPG algorithm, comprising the following steps:
[0078] Obtain the current electrical parameters of the grid-connected inverter, and obtain the current state space (id) corresponding to the current electrical parameters based on the current electrical parameters (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) );
[0079] Taking the target electrical parameters of the grid-connected inverter as the target, generating the current moment action through the DDPG algorithm model based on the current moment state space, and obtaining the current moment modulated wave signal through the dq inverse transformation of the current moment action;
[0080] Generate a control signal for the switch tube through a PWM generator based on the modulation wave signal at the current moment;
[0081] Among them, the Critic network in the DDPG algorithm model also obtains the state space and reward at the next moment, and stores the current state space, action, next state space, and reward as a set of data in the experience playback area to update the actor network and Critic network in the DDPG algorithm model.
[0082] The technical solution of this embodiment obtains the current electrical parameters of the grid-connected inverter and converts them into state space representation. The solution can accurately capture the dynamic characteristics of the system. This representation method makes the state estimation more accurate and provides a reliable basis for subsequent control. The DDPG algorithm is used to generate the action at the current moment, that is, the modulation wave signal, and then the control signal of the switch tube is generated by the PWM generator. This process ensures the continuity and smoothness of the control. The advantage of the DDPG algorithm in processing continuous action space makes the control more refined and the stability of the system is improved. In addition, the Critic network in the DDPG algorithm model not only evaluates the value of the current strategy, but also learns the state transition and reward change by obtaining the state and reward at the next moment. This learning method enables the algorithm to adapt to changes in different working conditions and improves the adaptability and robustness of the system. The application of the experience replay mechanism enables the algorithm to learn from historical experience, reduces the correlation in the learning process, and improves the learning efficiency. This is particularly important for systems such as inverters that require fast response, because it can quickly learn and adapt in a changing environment.
[0083] Specifically, based on the current electrical parameter, the current state space (id (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ) further comprises:
[0084] Get the current electrical parameter i of the grid-connected inverter and convert it into the corresponding d-axis component id in the dq coordinate system through dq transformation (t) and the q-axis component iq (t) ;
[0085] Calculate the difference ed between the current electrical parameter and the target electrical parameter d-axis component, and calculate the difference q between the current electrical parameter and the target electrical parameter (t) The difference of the axis components eq (t) ;
[0086] Calculate the time integral value ∫ed of the difference between the current electrical parameter and the target electrical parameter d-axis component (t) , calculate the time integral value ∫eq of the difference between the current electrical parameter and the target electrical parameter q-axis component (t) .
[0087] Preferably, updating the Critic network in the DDPG algorithm model further includes:
[0088] Use the TD algorithm to update parameters. The specific update method is as follows:
[0089] action next =U(ω,state next ),
[0090] q t =Q(ω,state,action),
[0091] q t+1 =Q(ω,state next , action next ),
[0092] δ=q t+1 -(λ * q t +r),
[0093]
[0094] ω=ω-α*δ*d ω ,
[0095] Among them, the actor network in the DDPG algorithm model is denoted as U (θ) , the critic network is denoted as Q (ω) , θ and ω are the parameters of the neural network, state represents the current state space, state next Indicates the spatial state at the next moment, action indicates the action at the current moment, action next The next action, δ represents the time difference error, r represents the reward, and λ represents the coefficient.
[0096] Preferably, the Actor network in the DDPG algorithm model is updated using stochastic gradient ascent to update parameters:
[0097]
[0098] θ=θ+β*δ*d θ
[0099] Preferably, before obtaining the current electrical parameters of the grid-connected inverter, the method further includes:
[0100] A DDPG algorithm model including an actor network and a critic network is established. The input of the Actor network is the current parameters and voltage parameters of the grid-connected inverter, and the output is the actions that the grid-connected inverter can perform. The input of the Critic network is the current parameters, voltage parameters of the grid-connected inverter in the environment, and the actions that the grid-connected inverter can perform.
[0101] Preferably, the method further comprises:
[0102] After building the DDPG algorithm model, set the total number of training times to episodes_num, and each episode to train step_num steps;
[0103] Before deploying the DDPG algorithm model, the grid-connected inverter environment is reset and the current parameters and voltage parameters of the grid-connected inverter are obtained.
[0104] Preferably, the method further comprises, when training the DDPG algorithm model:
[0105] If the grid-connected inverter environment triggers the termination condition, the actor network is stored when the training episode exceeds episode_num;
[0106] If the grid-connected inverter environment does not trigger the termination condition, the actor network is stored when the training step exceeds step_num and the training episode exceeds episode_num.
[0107] Based on the same concept, the present invention also provides a grid-connected inverter based on the DDPG algorithm, comprising:
[0108] The state acquisition module is used to obtain the current electrical parameters of the grid-connected inverter, and obtain the current state space (id) corresponding to the current electrical parameters based on the current electrical parameters. (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) );
[0109] A modulation wave calculation module is used to generate a current moment action based on the current moment state space through a DDPG algorithm model with the target electrical parameters of the grid-connected inverter as the target, and obtain a current moment modulation wave signal by performing a dq inverse transformation on the current moment action;
[0110] A control signal generating module, used to generate a control signal of a switch tube through a PWM generator based on the modulation wave signal at the current moment;
[0111] Among them, the Critic network in the DDPG algorithm model also obtains the state space and reward at the next moment, and stores the current state space, action, next state space, and reward as a set of data in the experience playback area to update the actor network and Critic network in the DDPG algorithm model.
[0112] The technical solution of this embodiment obtains the current electrical parameters of the grid-connected inverter and converts them into state space representation. The solution can accurately capture the dynamic characteristics of the system. This representation method makes the state estimation more accurate and provides a reliable basis for subsequent control. The DDPG algorithm is used to generate the action at the current moment, that is, the modulation wave signal, and then the control signal of the switch tube is generated by the PWM generator. This process ensures the continuity and smoothness of the control. The advantage of the DDPG algorithm in processing continuous action space makes the control more refined and the stability of the system is improved. In addition, the Critic network in the DDPG algorithm model not only evaluates the value of the current strategy, but also learns the state transition and reward change by obtaining the state and reward at the next moment. This learning method enables the algorithm to adapt to changes in different working conditions and improves the adaptability and robustness of the system. The application of the experience replay mechanism enables the algorithm to learn from historical experience, reduces the correlation in the learning process, and improves the learning efficiency. This is particularly important for systems such as inverters that require fast response, because it can quickly learn and adapt in a changing environment.
[0113] Furthermore, the state acquisition module obtains the current state space (id (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ) further comprises:
[0114] Get the current electrical parameter i of the grid-connected inverter and convert it into the corresponding d-axis component id in the dq coordinate system through dq transformation (t) and the q-axis component iq (t) ;
[0115] Calculate the difference ed between the current electrical parameter and the target electrical parameter d-axis component, and calculate the difference q between the current electrical parameter and the target electrical parameter (t) The difference of the axis components eq (t) ;
[0116] Calculate the time integral value ∫ed of the difference between the current electrical parameter and the target electrical parameter d-axis component (t) , calculate the time integral value ∫eq of the difference between the current electrical parameter and the target electrical parameter q-axis component (t) .
[0117] The implementation process of the above technical solution is introduced below. First, the application scenario is explained. The environment in which the DDPG algorithm model is applied is a three-phase inverter grid-connected circuit. The state space used in the DDPG algorithm is (id(t), iq(t), ed(t), eq(t), ∫ed(t), ∫eq(t), vd(t), vq(t)). Action represents the action space (ed, eq). After inverting the action space dq, it can be used as a modulation wave to generate a PWM signal to control the inverter circuit. Experience replay area: a cache for storing the interaction between the environment and the algorithm model. When the data exceeds the size of the cache, the oldest data is replaced. Episode: a training session starting from the initial state of the environment. Step: the number of times the algorithm model interacts with the environment.
[0118] See also Figure 3 , introduce the specific steps:
[0119] Step 1: Initialize the algorithm model and build an actor network and a critic network. The input of the actor network is the state of the environment, and the output is the action that the environment can perform. The input of the critic network is the state of the environment and the action, and the output is a score. The higher the score, the better the action. Set some algorithm hyperparameters. Set the total number of training to episodes_num, and each episode trains step_num steps.
[0120] Step 2: Reset the environment.
[0121] Step 3: Get the status of the environment.
[0122] Step 4: Use the actor network to calculate the action.
[0123] Step 5: The environment runs the action.
[0124] Step 6: Obtain the environmental reward and the state of the next moment.
[0125] Step 7: Store (state, action, reward, next_state) as a set of data in the experience playback area.
[0126] Step 8: Use the data from the experience replay area to update the actor and critic networks. The critic network is updated using the TD algorithm, and the actor network is updated using the stochastic gradient ascent algorithm.
[0127] Step 9: Does the environment trigger the termination condition? If yes, go to step 11; if no, go to step 10.
[0128] Step 10: Check whether the training step exceeds step_num. If yes, proceed to step 11, otherwise return to step 4.
[0129] Step 11: Does the training episode exceed episode_num? If yes, go to step 12; if no, go to step 2.
[0130] Step 12: Store the actor network.
[0131] Step 13: Load the actor network into the three-phase inverter grid-connected circuit.
[0132] Step 14: The actor network continues to interact with the environment, and the environment runs.
[0133] Based on the same concept, the present invention also provides an electronic device, comprising: a memory, the memory being used to store a processing program; and a processor, the processor implementing any one of the above-mentioned grid-connected inverter control methods based on the DDPG algorithm when executing the processing program.
[0134] Based on the same concept, the present invention also provides a readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, any one of the above-mentioned grid-connected inverter control methods based on the DDPG algorithm is implemented.
[0135] If the grid-connected inverter control method based on the DDPG algorithm is implemented in the form of program instructions and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of software, and the computer software is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.
[0136] Those skilled in the art can clearly understand that, for the sake of convenience and brevity in description, the identification content specifically executed by the above-described system and device can refer to the corresponding process in the aforementioned method embodiment.
[0137] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they still fall within the protection scope of the present invention.
Claims
1. A grid-connected inverter control method based on DDPG algorithm, It is characterized in that The steps include: Obtain the current electrical parameters of the grid-connected inverter, and obtain the current state space (id) corresponding to the current electrical parameters based on the current electrical parameters (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ); Taking the target electrical parameters of the grid-connected inverter as the target, generating the current moment action through the DDPG algorithm model based on the current moment state space, and obtaining the current moment modulated wave signal through the dq inverse transformation of the current moment action; Generate a control signal for the switch tube through a PWM generator based on the modulation wave signal at the current moment; Among them, the Critic network in the DDPG algorithm model also obtains the state space and reward at the next moment, and stores the current state space, action, next state space, and reward as a set of data in the experience playback area to update the actor network and Critic network in the DDPG algorithm model.
2. The grid-connected inverter control method based on the DDPG algorithm according to claim 1, It is characterized in that Based on the current electrical parameter, the current state space (id (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ) further comprises: Get the current electrical parameter i of the grid-connected inverter and convert it into the corresponding d-axis component id in the dq coordinate system through dq transformation (t) and the q-axis component iq (t) ; Calculate the difference ed between the current electrical parameter and the target electrical parameter d-axis component, and calculate the difference q between the current electrical parameter and the target electrical parameter (t) The difference of the axis components eq (t) ; Calculate the time integral value ∫ed of the difference between the current electrical parameter and the target electrical parameter d-axis component (t) , calculate the time integral value ∫eq of the difference between the current electrical parameter and the target electrical parameter q-axis component (t) .
3. The grid-connected inverter control method based on the DDPG algorithm according to claim 2, It is characterized in that The Critic network in the updated DDPG algorithm model further includes: Use the TD algorithm to update parameters. The specific update method is as follows: action next =U(ω,state next ), q t =Q(ω,state,action), qt +1 =Q(ω,statenext,actionnext), δ=qt +1 -(λ*qt+r), ω=ω-α*δ*d ω , Among them, the actor network in the DDPG algorithm model is denoted as U (θ) , the critic network is denoted as Q (ω) , θ and ω are the parameters of the neural network, state represents the current state space, statenex t Indicates the spatial state at the next moment, action indicates the action at the current moment, actionnex t The next action, δ represents the time difference error, r represents the reward, and λ represents the coefficient.
4. The grid-connected inverter control method based on the DDPG algorithm according to claim 2, It is characterized in that Update the Actor network in the DDPG algorithm model using stochastic gradient ascent to update parameters: θ=θ+β*δ*d θ 5. The grid-connected inverter control method based on the DDPG algorithm according to claim 1, It is characterized in that Before obtaining the current electrical parameters of the grid-connected inverter, the method further includes: A DDPG algorithm model including an actor network and a critic network is established. The input of the Actor network is the current parameters and voltage parameters of the grid-connected inverter, and the output is the actions that the grid-connected inverter can perform. The input of the Critic network is the current parameters, voltage parameters of the grid-connected inverter in the environment, and the actions that the grid-connected inverter can perform.
6. The grid-connected inverter control method based on the DDPG algorithm according to claim 5, It is characterized in that The method further comprises: After building the DDPG algorithm model, set the total number of training times to episodes_num, and each episode to train step_num steps; Before deploying the DDPG algorithm model, the grid-connected inverter environment is reset and the current parameters and voltage parameters of the grid-connected inverter are obtained.
7. The grid-connected inverter control method based on the DDPG algorithm according to claim 6, It is characterized in that The method further includes, when training the DDPG algorithm model: If the grid-connected inverter environment triggers the termination condition, the actor network is stored when the training episode exceeds episode_num; If the grid-connected inverter environment does not trigger the termination condition, the actor network is stored when the training step exceeds step_num and the training episode exceeds episode_num.
8. A grid-connected inverter based on DDPG algorithm, It is characterized in that include: The state acquisition module is used to obtain the current electrical parameters of the grid-connected inverter, and obtain the current state space (id) corresponding to the current electrical parameters based on the current electrical parameters. (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ); A modulation wave calculation module is used to generate a current moment action based on the current moment state space through a DDPG algorithm model with the target electrical parameters of the grid-connected inverter as the target, and obtain a current moment modulation wave signal by performing a dq inverse transformation on the current moment action; A control signal generating module, used to generate a control signal of a switch tube through a PWM generator based on the modulation wave signal at the current moment; Among them, the Critic network in the DDPG algorithm model also obtains the state space and reward at the next moment, and stores the current state space, action, next state space, and reward as a set of data in the experience playback area to update the actor network and Critic network in the DDPG algorithm model.
9. The grid-connected inverter based on the DDPG algorithm according to claim 8, It is characterized in that The state acquisition module obtains the current state space (id) corresponding to the current electrical parameter based on the current electrical parameter. (t) , iq (t) ,ed (t) , eq (t) ,∫ed (t) , ∫eq (t) , vd (t) , vq (t) ) further comprises: Get the current electrical parameter i of the grid-connected inverter and convert it into the corresponding d-axis component id in the dq coordinate system through dq transformation (t) and the q-axis component iq (t) ; Calculate the difference ed between the current electrical parameter and the target electrical parameter d-axis component, and calculate the difference q between the current electrical parameter and the target electrical parameter (t) The difference of the axis components eq (t) ; Calculate the time integral value ∫ed of the difference between the current electrical parameter and the target electrical parameter d-axis component (t) , calculate the time integral value ∫eq of the difference between the current electrical parameter and the target electrical parameter q-axis component (t) .
10. A readable storage medium, It is characterized in that The readable storage medium stores a processing program, and when the processing program is executed by the processor, the grid-connected inverter control method based on the DDPG algorithm described in any one of claims 1 to 7 is implemented.
