Direct current micro-grid control method based on imitation learning

Abstract

The invention relates to a direct current micro-grid control method based on imitation learning. The method comprises the following steps: S1, building a Buck converter model; s2, a voltage and current double-closed-loop PID controller is selected and applied to the Buck converter model, input and output values of the PID controller are used for fitting an uninitialized neural network, a pre-trained neural network is obtained, the input of the pre-trained neural network is state information of the system, and the output of the pre-trained neural network is a control signal; s3, an initial agent is constructed, the initial agent adopts input and output of the pre-trained neural network as an initial control strategy, and the agent is trained to obtain a micro-grid control agent; and S4, acquiring actual state information, inputting the actual state information into the micro-grid control agent, and controlling the Buck converter by the output of the micro-grid control agent. Compared with the prior art, the method has the advantages that the deep model training time in direct-current micro-grid control is shortened, and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of microgrid control, and in particular to a direct current microgrid control method based on imitation learning. Background Art

[0002] As a new type of energy management system, DC microgrids have garnered widespread attention in recent years. Driven particularly by the integration of renewable energy and smart grid technologies, they have become a crucial platform for achieving efficient, flexible, and green energy utilization. The greatest advantage of DC microgrids lies in their ability to effectively connect multiple renewable energy sources (such as solar and wind) with energy storage devices (such as lithium batteries and supercapacitors), and to achieve real-time scheduling and optimized management of electrical energy through intelligent control systems. Compared to traditional AC microgrids, DC microgrids not only offer higher energy conversion efficiency but also effectively reduce energy losses and simplify the power electronics conversion process, thereby improving overall system performance.

[0003] However, as the scale of DC microgrids expands and the complexity of the operating environment increases, how to accurately control key parameters such as current, voltage, and frequency within the DC microgrid to ensure system stability and reliability has become a major technical challenge. The existing control methods mainly include:

[0004] Traditional control methods, such as patents (CN 117096846A, "Complementary Sliding Mode Control Method for Bus Voltage of a Photovoltaic Storage System Based on a Finite-Time Observer," November 21, 2023) and (CN 116169661A, "Comprehensive Control Method for Bus Voltage of a DC Microgrid," May 26, 2023), offer advantages such as simplicity and real-time performance. However, most methods exhibit limitations in complex systems and dynamic environments, particularly requiring high system modeling accuracy. Furthermore, they suffer from insufficient computational overhead and adaptability in large-scale systems.

[0005] Intelligent control methods: For example, patents (CN 118263842A, Coordinated control method for DC microgrids based on multi-agent systems and deep reinforcement learning, June 28, 2024), patents (CN 118944032A, Control method and related device for DC microgrid systems, November 12, 2024), and patents (CN 110601246A, Current balancing method for DC microgrids based on radial basis function neural network prediction, December 20, 2019). These methods have strong adaptability and complex system modeling capabilities, and can optimize control strategies in dynamic environments. However, they often require large amounts of training data and computing resources, have long training cycles, and may face the risk of instability or overfitting. Summary of the Invention

[0006] The purpose of the present invention is to provide a DC microgrid control method based on imitation learning in order to reduce the deep model training time in DC microgrid control.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A DC microgrid control method based on imitation learning, the method comprising the following steps:

[0009] S1. Build the Buck converter model;

[0010] S2. Select a voltage-current dual closed-loop PID controller and apply it to the Buck converter model. Use the input and output values of the PID controller to fit an uninitialized neural network to obtain a pre-trained neural network. The input of the pre-trained neural network is the system state information, and the output is the control signal.

[0011] S3. Construct an initial intelligent agent. The initial intelligent agent uses the input and output of the pre-trained neural network as the initial control strategy, and trains the intelligent agent to obtain a microgrid control intelligent agent.

[0012] S4. Acquire actual state information, input the actual state information into the microgrid control intelligent agent, and the output of the microgrid control intelligent agent controls the Buck converter.

[0013] Furthermore, the neural network is a feedforward neural network.

[0014] Furthermore, the feedforward neural network includes an input layer, an output layer and a hidden layer.

[0015] Furthermore, RMSE is used as the loss function for training the feedforward neural network.

[0016] Furthermore, the specific steps of S3 are:

[0017] S31. Construct an initial intelligent agent. The initial intelligent agent uses the input and output of the pre-trained neural network as the initial control strategy, loads the initial control strategy into the neural network of the initial intelligent agent, and defines the state space, action space, and reward function.

[0018] S32, initialize the weight θ of the Q network and the weight θ of the target network target , initialize the experience replay pool;

[0019] S33, at each time step t, according to the current state s t , using a greedy strategy to select an action a t , then perform action a t , observe the next state s t+1 and reward r t , will st ,a t ,r t ,s t+1 Stored in the experience replay pool, s t+1 The state of the t+1 time step, after repeating S33 a certain number of times, enters S34;

[0020] S34. Sample a mini-batch of experience from the experience replay pool, calculate the target Q value and the current Q value, calculate the mean square error loss function based on the target Q value and the current Q value, and iteratively update the Q network weights, returning to S33 until the training converges or reaches the predetermined number of training times.

[0021] Furthermore, the state space is:

[0022] S t =[v0,v 0_delay ,e,e delay ,i L ] T

[0023] in, Represent the output voltage and the output voltage at the previous moment, e, e respectively. delay Represent the tracking error and the previous moment tracking error respectively, i L Represents the inductor current.

[0024] Furthermore, the reward function is:

[0025]

[0026] Among them, R t is the reward value after executing the action, e(t) is the tracking error between the output voltage and the bus reference voltage, τ 1～3 They represent the critical range of the decomposed subtasks. The subtasks refer to the intervals divided based on the tracking error |e(t)|. Different error intervals set different strengths of reward items. 1～2 They represent the reward items of the corresponding subtasks, and β represents the penalty item of each subtask.

[0027] Furthermore, the action space is discretized into a set of specific duty cycle values.

[0028] Furthermore, the status information of the system includes output voltage and output current.

[0029] Furthermore, the control signal is a duty cycle.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] When training is performed directly in reinforcement learning, the agent usually needs a lot of interaction and trial and error to learn the optimal strategy from the environment. Especially in complex systems, this process may require hundreds or even thousands of training steps to achieve ideal performance. At this time, the convergence speed of the reward function is usually slow. Especially when the system is dynamically complex and the state space is huge, the training process may be in the "exploration" stage for a long time and cannot quickly approach the optimal solution. The present invention uses imitation learning in advance to significantly accelerate this process. Imitation learning provides an initial and more reasonable strategy for reinforcement learning by allowing the agent to learn the behavior of experts. This means that the agent can start training based on a strategy that is relatively close to the optimal strategy, thereby avoiding the slow convergence process starting from a random strategy. In addition, since it is relatively close to the optimal strategy from the beginning, the initial training instability problem of the DRL deep reinforcement learning model can be effectively avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the PID control block diagram of the present invention;

[0033] Figure 2 Graph of the fitting process for imitation learning;

[0034] Figure 3 is the reference duty cycle diagram;

[0035] Figure 4 The reward function curve is not used for imitation learning;

[0036] Figure 5 To use imitation learning reward function curve;

[0037] Figure 6 This is the bus voltage tracking situation without using imitation learning;

[0038] Figure 7 Figure 3. Bus voltage tracking using imitation learning. DETAILED DESCRIPTION

[0039] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0040] In order to solve the problems of low training efficiency and poor control stability in existing methods, the present invention proposes a DC microgrid control method based on imitation learning. By introducing the pre-training stage of imitation learning into deep reinforcement learning, the training process is effectively accelerated, and the stability and efficiency of control are improved, thereby providing a new solution for the intelligent control of DC microgrids. The present invention proposes a DC microgrid control method based on imitation learning, which includes the following steps:

[0041] S1. Build the Buck converter model;

[0042] S2. Select a voltage-current dual closed-loop PID controller and apply it to the Buck converter model. Use the input and output values of the PID controller to fit an uninitialized neural network to obtain a pre-trained neural network. The input of the pre-trained neural network is the system state information, and the output is the control signal.

[0043] S3. Construct an initial intelligent agent. The initial intelligent agent uses the input and output of the pre-trained neural network as the initial control strategy, and trains the intelligent agent to obtain a microgrid control intelligent agent.

[0044] S4. Acquire actual state information, input the actual state information into the microgrid control intelligent agent, and the output of the microgrid control intelligent agent controls the Buck converter.

[0045] The specific steps of S3 are:

[0046] S31. Construct an initial intelligent agent. The initial intelligent agent uses the input and output of the pre-trained neural network as the initial control strategy, loads the initial control strategy into the neural network of the initial intelligent agent, and defines the state space, action space, and reward function.

[0047] S32, initialize the weight θ of the Q network and the weight θ of the target network target , initialize the experience replay pool;

[0048] S33, at each time step t, according to the current state s t , using a greedy strategy to select an action a t , then perform action a t , observe the next state s t+1 and reward r t , will s t ,a t ,r t ,s t+1 Stored in the experience replay pool, s t+1 The state of the t+1 time step, after repeating S33 a certain number of times, enters S34;

[0049] S34. Sample a mini-batch of experience from the experience replay pool, calculate the target Q value and the current Q value, calculate the mean square error loss function based on the target Q value and the current Q value, and iteratively update the Q network weights, returning to S33 until the training converges or reaches the predetermined number of training times.

[0050] To address the numerous issues with existing deep reinforcement learning controllers for DC microgrids, this paper proposes a deep reinforcement learning control strategy based on imitation learning. Combining its deep learning and reinforcement learning toolboxes, this approach enables precise modeling and control strategy development for DC microgrids. First, a simulation model of the power electronic converter is constructed. Second, the imitation learning phase is designed. Third, a deep reinforcement learning controller is designed, constructing the state space, action space, and reward function. Finally, deep reinforcement learning training is performed to achieve superior control performance.

[0051] In a specific example, the parameter specifications of the DC-DC Buck converter are shown in Table 1:

[0052] Table 1 Buck converter specifications

[0053]

[0054] Step 1): A Buck converter model was built using power electronic devices, including key components such as the input power supply, switching devices, diodes, inductors, capacitors, and loads. By properly configuring parameters, the feasibility of the model in actual circuits was ensured.

[0055] Step 2): In the neural network simulation stage, first select the voltage and current dual closed-loop PID controller and apply it to the Buck converter. The PID control block diagram is as follows: Figure 1 As shown. An imitation learning framework is then designed to learn the output response of the PID controller through a neural network. The input of the neural network model is the system state information (including output voltage, current, and other information that can be used for reinforcement learning), and the output is the control signal (duty cycle). The imitation learning framework can be divided into the following steps:

[0056] (1) Load data, perform data cleaning, remove abnormal data, and prevent learning abnormal values;

[0057] (2) Normalize all data and divide them into training set, validation set and test set;

[0058] (3) Define a relatively simple feedforward neural network. The network consists of seven layers, including an input layer, an output layer, three fully connected layers, and two hidden layers. The output layer is responsible for outputting the final result. The two hidden layers each contain 128 neurons and use rectified linear units as the activation function.

[0059] (4) Set the hyperparameters of imitation learning and perform data fitting. The hyperparameters are shown in Table 2. The line graph of RMSE and loss during training is shown in Figure 2 As shown;

[0060] Table 2 Imitation learning hyperparameters

[0061]

[0062] (5) Make predictions on the test set.

[0063] Step 3): Implement the deep reinforcement learning DQN algorithm. The main steps of the algorithm are as follows:

[0064] (1) Initialize the weights θ of the Q network and the weights θ of the target network target , initialize the experience replay pool;

[0065] (2) Interaction with the environment:

[0066] At each time step t, according to the current state s t , select an action a t The epsilon-greedy strategy is used: an action is chosen randomly with probability ε, and otherwise the best action evaluated by the current Q network is chosen:

[0067] a t =argmax a Q θ (s t , a)

[0068] Where θ is the parameter of the current Q network, s′ is the next state, and a represents the action taken in this state;

[0069] Then perform action a t , observe the next state a t+1 and reward r t , will s t ,a t ,r t ,s t+1 Stored in the experience replay pool.

[0070] (3) Sampling from the experience replay pool:

[0071] First, randomly sample a mini-batch of experiences s,a,r,s′ from the replay pool;

[0072] Calculate the target Q value:

[0073]

[0074] Where r represents the reward at the current moment, which is the immediate reward obtained by the agent after performing an action in the state; γ represents the discount factor, a constant between 0 and 1, which is used to measure the importance of future rewards; The target network is used to estimate the maximum future reward over all possible actions in the next state. y is the target Q value, which is the expected future reward after performing an action in the state.

[0075] Calculate the current Q value Q θ (s,a);

[0076] (4) Update the Q network weights using the mean square error loss function (MSE):

[0077]

[0078] Where N is the batch size.

[0079] (5) Update the target network regularly and repeat the above steps until the training converges or the predetermined number of training times is reached.

[0080] Step 4): Define the state space:

[0081] S t =[v0,v 0_delay ,e,e delay ,i L ] T

[0082] Where, Represent the output voltage and the output voltage at the previous moment, e, e respectively. delay Represent the tracking error and the previous moment tracking error respectively, i L Represents the inductor current.

[0083] Define the reward function:

[0084]

[0085] Where R t is the reward value after executing the action, e(t) is the tracking error between the output voltage and the bus reference voltage, τ 1～3 Represent the critical range of the decomposed subtasks, α 1～2 Represent the reward items of the corresponding subtasks, and β represents the penalty items of each subtask. t is the reward value after executing the action, e(t) is the tracking error between the output voltage and the bus reference voltage, τ 1～3They represent the critical range of the decomposed subtasks. The subtasks refer to the intervals divided based on the tracking error |e(t)|. Different error intervals set reward items of different strengths. The purpose is to provide more subtle feedback signals to the agent. 1～2 Represent the reward for the corresponding subtask, and β represents the penalty for each subtask. Rewards and penalties are rarely set in one go; they typically require repeated experimentation and adjustment in a simulation environment. This paper employs a trial-and-error approach combined with binary search. First, a rough parameter range is determined using trial and error. Then, the median value is taken, gradually narrowing the range until the desired range is met.

[0086] Define the action space:

[0087] The action space is discretized into a set of specific duty cycle values to simplify the balance between exploration and exploitation. Referring to the output value of the double closed-loop PI controller, the action space is set to a discrete value between 0.4 and 0.6 with a step size of 0.01, so that the system can better imitate the PI controller for effective control and regulation. Figure 3 shown.

[0088] Define the neural network:

[0089] The neural network refers to the network structure defined in imitation learning.

[0090] Step 5): Perform hyperparameter adjustment. The specific parameters are shown in Table 3.

[0091] Table 3. Deep reinforcement learning hyperparameters

[0092]

[0093] Step 6): By using the output of the imitation learning network as the initial policy for deep reinforcement learning, DRL deep reinforcement learning is quickly guided into the convergence region. Imitation learning can help DRL deep reinforcement learning learn some basic control strategies in advance, reducing training time during the exploration process.

[0094] Compared with the existing technology, the beneficial effects of this technology are as follows:

[0095] 1. Improve training efficiency: By pre-training the neural network through imitation learning, the time consumption in the DRL deep reinforcement learning model training process is reduced.

[0096] 2. Improve training stability: The initial strategy based on the PID controller can effectively avoid the initial training instability problem of the DRL deep reinforcement learning model.

[0097] When training directly in reinforcement learning, agents typically require extensive interaction and trial-and-error to learn the optimal policy from the environment. Especially in complex systems, this process can require hundreds or even thousands of training steps to achieve ideal performance. In this case, the reward function often converges slowly, especially in complex system dynamics and large state spaces. Training can remain in the "exploration" phase for extended periods, preventing rapid progress toward the optimal solution.

[0098] In contrast, using imitation learning upfront can significantly accelerate this process. By allowing the agent to imitate the behavior of an expert, imitation learning provides an initial, relatively reasonable policy for reinforcement learning. This means that the agent can begin training based on a policy that is relatively close to the optimal one, thus avoiding the slow convergence process of starting with a random policy.

[0099] like Figure 4 and Figure 5 As shown in , compared with direct reinforcement learning training, using imitation learning in advance can accelerate the convergence of the reward function. Figure 6 and Figure 7 As shown in the figure, the constant power load is initially set to 200W. At 0.14 seconds, the constant power load rises to 500W and then returns to 200W. When other training conditions and hyperparameters are the same, the use of imitation learning can achieve the effect of using reinforcement learning alone. In fact, at 0-0.14 seconds, the effect is even better than the control effect of reinforcement learning alone.

[0100] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A DC microgrid control method based on imitation learning, characterized in that: The method comprises the following steps: S1. Build the Buck converter model; S2. Select a voltage-current dual closed-loop PID controller and apply it to the Buck converter model. Use the input and output values of the PID controller to fit an uninitialized neural network to obtain a pre-trained neural network. The input of the pre-trained neural network is the system state information, and the output is the control signal. S3. Construct an initial intelligent agent. The initial intelligent agent uses the input and output of the pre-trained neural network as the initial control strategy, and trains the intelligent agent to obtain a microgrid control intelligent agent. S4. Acquire actual state information, input the actual state information into the microgrid control intelligent agent, and the output of the microgrid control intelligent agent controls the Buck converter.

2. A DC microgrid control method based on imitation learning according to claim 1, characterized in that: The neural network is a feedforward neural network.

3. A DC microgrid control method based on imitation learning according to claim 2, characterized in that: The feedforward neural network includes an input layer, an output layer and a hidden layer.

4. A DC microgrid control method based on imitation learning according to claim 3, characterized in that: The RMSE is used as the loss function for training feedforward neural networks.

5. The DC microgrid control method based on imitation learning according to claim 1, characterized in that: The specific steps of S3 are: S31. Construct an initial intelligent agent. The initial intelligent agent uses the input and output of the pre-trained neural network as the initial control strategy, loads the initial control strategy into the neural network of the initial intelligent agent, and defines the state space, action space, and reward function. S32, initialize the weight θ of the Q network and the weight θ of the target network target , initialize the experience replay pool; S33, at each time step t, according to the current state s t , using a greedy strategy to select an action a t , then perform action a t , observe the next state s t+1 and reward r t , will s t ,a t ,r t ,s t+1 Stored in the experience replay pool, s t+1 The state of the t+1 time step, after repeating S33 a certain number of times, enters S34; S34. Sample a mini-batch of experience from the experience replay pool, calculate the target Q value and the current Q value, calculate the mean square error loss function based on the target Q value and the current Q value, and iteratively update the Q network weights, returning to S33 until the training converges or reaches the predetermined number of training times.

6. A DC microgrid control method based on imitation learning according to claim 5, characterized in that: The state space is: S t =[v0,v 0_delay ,he,he delay ,i L ] T Among them, v0, Represent the output voltage and the output voltage at the previous moment, e, e respectively. delay Represent the tracking error and the previous moment tracking error respectively, i L Represents the inductor current.

7. A DC microgrid control method based on imitation learning according to claim 6, characterized in that: The reward function is: Among them, R t is the reward value after executing the action, e(t) is the tracking error between the output voltage and the bus reference voltage, τ 1～3 They represent the critical range of the decomposed subtasks. The subtasks refer to the intervals divided based on the tracking error |e(t)|. Different error intervals set different strengths of reward items. 1～2 They represent the reward items of the corresponding subtasks, and β represents the penalty item of each subtask.

8. A DC microgrid control method based on imitation learning according to claim 7, characterized in that: The action space is discretized into a set of specific duty cycle values.

9. The DC microgrid control method based on imitation learning according to claim 1, characterized in that: The system status information includes output voltage and output current.

10. A DC microgrid control method based on imitation learning according to claim 1, characterized in that: The control signal is the duty cycle.

Images