A construction method and application of microgrid energy storage model based on reinforcement learning

Abstract

The present invention discloses a construction method and application of a microgrid energy storage model based on reinforcement learning. The method trains a preset model based on a Q-learning algorithm, including: obtaining microgrid parameter information and initializing a Q table; defining the state, action, and reward function of the Q-learning algorithm; obtaining the current state of the microgrid at a preset time interval, determining the current action of the microgrid according to the state, inputting the action into the reward function to obtain a corresponding reward value, iteratively training the Q table through the reward value, and obtaining a trained microgrid energy storage model. The method can learn the best strategy based on real-time data of different time scales and system interaction, and can realize short-time rolling optimization or long-term scheduling planning according to demand, thereby meeting the requirements of different application scenarios.

Description

Technical Field

[0001] The present invention relates to the field of microgrid energy storage technology, and specifically to a construction method and application of a microgrid energy storage model based on reinforcement learning. Background Art

[0002] Renewable energy generation technology continues to develop, and microgrid systems composed of distributed energy, energy storage devices, and loads are widely used. In order to improve the energy utilization efficiency of microgrids, reduce operating costs, meet the needs of large power grids, reduce power grid shocks, and improve power quality, a large number of studies have been conducted on the optimized operation of microgrids. At present, researchers have given a variety of solutions to the control and scheduling problems of energy storage operation in grid-connected microgrids. There are mainly three types of classical optimization algorithms based on planning, heuristic algorithms, and artificial intelligence methods. Among them, classical optimization algorithms can be divided into linear programming, nonlinear programming, hybrid linear programming and other planning-based algorithms. This type of traditional control method is based on artificial prior knowledge and requires detailed and accurate modeling of the power grid environment. The actual application effect is difficult to guarantee. Heuristic methods include genetic algorithms, particle swarm algorithms, and distribution estimation algorithms. These methods have low prediction accuracy for renewable energy generation and load demand. Artificial intelligence methods use neural network prediction to solve the uncertainty problem in microgrids, and the accuracy of control strategies depends on the recognition accuracy of artificial intelligence algorithms. Although the accuracy of prediction algorithms has been continuously improved in recent years, there is still a certain error between the predicted load demand and the efficiency of renewable power generation and the actual situation. The existing optimization scheduling algorithm relies on the accuracy of prediction data, and it is difficult to obtain the best operation strategy due to the influence of random factors in the microgrid environment. Summary of the invention

[0003] (I) Purpose of the invention

[0004] The purpose of the present invention is to provide a method for constructing a microgrid energy storage model based on reinforcement learning and its application. The method uses a Q-learning algorithm to minimize the cost of inputting electricity from an external power grid. It can learn the optimal strategy based on real-time data and system interaction at different time scales, so that the model can maximize rewards in a complex and uncertain power grid environment, avoiding the problem of difficulty in obtaining the optimal operating strategy due to the influence of random factors in the microgrid environment.

[0005] (II) Technical solution

[0006] To solve the above problems, the first aspect of the present invention provides a method for constructing a microgrid energy storage model based on reinforcement learning, which trains a preset model based on a Q-learning algorithm, comprising:

[0007] Obtain microgrid parameter information and initialize the Q table;

[0008] Define the state, action, and reward function of the Q-learning algorithm;

[0009] Acquire the current state of the microgrid at a preset time interval, determine the current action of the microgrid according to the state, input the action into the reward function to obtain a corresponding reward value, and iteratively train the Q table through the reward value;

[0010] When the iterative training meets the preset convergence conditions, a trained microgrid energy storage model is obtained.

[0011] Further, determining the current action of the microgrid according to the state includes: determining the execution action at the current moment according to a preset strategy according to the current state of the microgrid, wherein the preset strategy is an ε-greedy strategy.

[0012] Further, the iterative training of the Q table by using the reward value includes: continuously updating the microgrid environment by using the reward value to obtain an action value function, and iteratively training the Q table by using the action value function.

[0013] Furthermore, the state includes: energy storage SOC, load demand power and photovoltaic power generation power.

[0014] Furthermore, the actions include: charging, discharging and idling.

[0015] Furthermore, the calculation formula of the reward function is:

[0016]

[0017] In the formula, t is the current time, a is t is the action at the current moment, s t is the current state, r(s t ,a t ) is the reward function value at the current moment, C in represents the cost of obtaining electricity from the external power grid, C out represents the revenue from selling electricity to the grid, represents the power of the power grid, and Δt represents the time interval for collecting data.

[0018] Furthermore, the calculation formula of the action value function is:

[0019] Q(s t ,a t )=Q(s t ,a t )+α[r(s t ,a t )+γmaxQ(s t+1 ,a t+1)-Q(s t ,a t )];

[0020] In the formula, Q(s t ,a t ) is the value of the action value function at the current moment, γ represents the attenuation factor, α represents the learning rate, maxQ(s t+1 ,a t+1 ) is the maximum value of the action value function at time t+1.

[0021] Furthermore, determining the execution action at the current moment according to the preset strategy includes: selecting the action with the largest Q value at the current moment as the execution action with a probability of 1-∈ and randomly selecting any action at the current moment as the execution action with a probability of ∈, where ∈ represents the exploration probability.

[0022] Furthermore, the preset time is set to 25 to 35 minutes.

[0023] In addition, a second aspect of the present invention provides a method for optimizing and scheduling microgrid energy storage using a microgrid energy storage model, wherein the method utilizes the microgrid energy storage model to perform real-time optimization scheduling on the microgrid, and the microgrid energy storage model is trained using a method for constructing a microgrid energy storage model based on reinforcement learning as described in any one of the above descriptions.

[0024] (III) Beneficial effects

[0025] The above technical scheme of the present invention has the following beneficial technical effects: the present invention provides a method and application for constructing a microgrid energy storage model based on reinforcement learning. The method uses the Q-learning algorithm to minimize the cost of inputting electricity from the external power grid. It can learn the best strategy based on real-time data and system interaction at different time scales, and can realize short-term rolling optimization (such as hourly level) or long-term scheduling (such as weekly, monthly) planning according to demand, so as to meet the requirements of different application scenarios. The specific process is: the present invention trains the preset model through the Q-learning algorithm, first defines the state, action and reward function of the Q-learning algorithm, then obtains the current state of the microgrid at a preset time interval, determines the current action of the microgrid according to the state, inputs the action into the reward function to obtain the corresponding reward value, and continuously optimizes and updates the Q table through the reward value, so as to iterate the training and obtain the trained model. In reinforcement learning, the intelligent agent takes a certain action according to the current state, and then evaluates the quality of the action through the reward signal fed back by the environment, and optimizes its own behavior strategy through continuous attempts to achieve a specific purpose. Reinforcement learning is based on an interactive learning model. It collects empirical data through the interaction between the intelligent agent and the environment, formulates the best strategy for correct actions, and continuously adjusts the best strategy to obtain a trained model. The present invention can make online real-time optimal decisions based on reinforcement learning, and solves the linear timing control problem of coordinated control of microgrid energy storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a microgrid energy storage optimization scheduling method based on reinforcement learning of the present invention;

[0027] Figure 2 It is a diagram of the reinforcement learning algorithm architecture (Q-learning algorithm) of the present invention;

[0028] Figure 3 is a microgrid topology diagram of the present invention;

[0029] Figure 4 It is a flow chart of a microgrid energy storage optimization scheduling method according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0031] In recent years, reinforcement learning has been applied to solve the optimal problem of coordinated control of energy storage in microgrids. As a machine learning method, reinforcement learning can be used to solve a class of problems that are difficult to solve with traditional supervised learning or unsupervised learning methods. It aims to enable the agent to learn the optimal behavior strategy through trial and error and interaction in the environment. In reinforcement learning, the agent takes a certain action based on the current state, and then evaluates the quality of this action through the reward signal fed back by the environment, and optimizes its own behavior strategy through continuous attempts to achieve a specific goal. Reinforcement learning is based on an interactive learning model. It collects empirical data through the interaction between the agent and the environment, formulates the best strategy for correct action, and can make online real-time decisions. For linear timing control problems such as coordinated control of energy storage in microgrids, reinforcement learning can make the best decision. In view of the fact that the existing optimization scheduling algorithm relies on the accuracy of prediction data, it is difficult to obtain the best operating strategy due to the influence of random factors in the microgrid environment. The present invention provides a microgrid energy storage optimization scheduling method and system based on reinforcement learning, which are as follows:

[0032] like Figure 1 As shown, the present invention provides a method for constructing a microgrid energy storage model based on reinforcement learning. The method trains a preset model based on the Q-learning algorithm. The method uses the Q-learning algorithm to minimize the cost of external grid input power. It can learn the best strategy based on real-time data and system interaction at different time scales. It can realize short-term rolling optimization (such as hourly level) or long-term scheduling (such as weekly, monthly) planning according to demand, so as to meet the requirements of different application scenarios. The Q-learning algorithm architecture is shown in Figure 2 As shown. The present invention optimizes scheduling for a 24-hour operation cycle, for example, so that the model can maximize rewards in a complex and uncertain power grid environment. Figure 4 The specific instructions are as follows:

[0033] S1, obtain microgrid parameter information and initialize the Q-table. The Q-table is the core of the Q-learning algorithm, which is used to store the value estimates of each action taken by the intelligent agent in different states. The expected benefits of taking a certain action in a specific state are represented by continuously updating a table called the Q value. According to the microgrid environment, the microgrid parameter (state) data (including power generation power, load demand, energy storage capacity, on-grid electricity price, off-grid electricity price, charging and discharging power, sampling time interval, control command) is obtained. In the present invention, the Q-table is initialized to a short-term future reward, and the convergence time is shortened by setting the algorithm hyperparameter γ (γ represents the attenuation factor) to 0.

[0034] S2, define the state, action and reward function of the Q-learning algorithm. Q-learning is a value-based reinforcement learning algorithm used to learn how to choose the best action in a given environment to maximize the accumulated reward. In this step, the strategy parameters, state (state space), and action (action space) of the reinforcement learning algorithm Q-learning are constructed. Set the reward function and explore the equation. The state includes: energy storage SOC, load demand power, and photovoltaic power generation power, such as Figure 3 As shown. Photovoltaic power generation is used to meet load demand. If it cannot be met, power is supplied by the energy storage system or the external power grid. The energy storage system is charged by photovoltaic power generation, and can also be charged or discharged from the external power grid as needed. In the present invention, automatic decision-making and control of energy storage charging and discharging are implemented according to the reinforcement learning algorithm. The decision-making form is charging, discharging or idling of energy storage. The environmental factors considered include photovoltaic power generation power, load demand, and energy storage SOC. The relationship between the three is as follows:

[0035] 1) State Space

[0036]

[0037]

[0038] In formula (1), SOC represents the state of the energy storage battery. Represents the difference between load demand power and photovoltaic power generation power, s t represents the state at time t, and S represents the state set. In formula (2),

[0039] Indicates the load demand power, Represents photovoltaic power generation power.

[0040] 2) Action Space

[0041] The actions include: charging, discharging and idle. The "charging" and "discharging" operations can be further refined into different power levels, such as "low-speed charging / discharging", "high-speed charging / discharging", etc., to adapt to different load requirements and energy storage device characteristics. The action space is represented as follows:

[0042] A={a|(charging, discharging, idle)} (3)

[0043] In formula (3), A represents the action set and a represents the action.

[0044] 3) Reward Function

[0045] In the present invention, the reward function is selected to minimize the operating cost of importing electricity from the grid and maximize the revenue from selling electricity to the grid. To achieve this goal, the reward function is designed as follows:

[0046]

[0047] In formula (4), t is the current time, a t is the action at the current moment, s t is the current state, r(s t ,a t ) is the reward function value at the current moment, C in represents the cost of obtaining electricity from the external power grid, C out represents the revenue from selling electricity to the grid, represents the power of the power grid, and Δt represents the time interval for collecting data. The calculation formula is as follows:

[0048]

[0049] In formula (5), Indicates the energy storage charging power.

[0050] S3, obtain the current state of the microgrid at a preset time interval, determine the current action of the microgrid according to the state, input the action into the reward function to obtain the corresponding reward value, and iteratively train the Q table through the reward value. During the training process, an agent (intelligent agent) is abstracted to sample environmental information, obtain real-time load demand, photovoltaic power generation, energy storage SOC and other state information in the microgrid, the model selects the next action according to the current state, and the environment changes the current state information and corresponding rewards according to the currently selected action. The microgrid energy storage optimization scheduling problem based on reinforcement learning can be described as a given microgrid environmental state. The agent will execute the state transfer from the predetermined action set (action space) according to the strategy, where the predetermined action set refers to the specific operations that can be performed by the microgrid energy storage system, including but not limited to "charging, discharging, and idle". By repeating the above steps and updating the strategy for selecting actions, the reward is maximized. The agent obtains the current state of the microgrid every 25 to 35 minutes (preset time). For example, the agent is set to obtain microgrid related information every 30 minutes, including energy storage SOC, load demand power, and photovoltaic power generation power.

[0051] S4, when the iterative training meets the preset convergence conditions, a trained microgrid energy storage model is obtained.

[0052] Further, the step S3 of determining the current action of the microgrid according to the state includes: determining the execution action at the current moment according to the preset strategy according to the current state of the microgrid. In the Q-learning algorithm, the strategy is the rule for the agent to select the action according to the current state, and the common ones include random strategy, ε-greedy strategy, Softmax strategy, etc. Among them, the preset strategy is the ε-greedy strategy. In the present invention, the agent adopts the ε-greedy strategy to select the current execution action. This strategy balances the contradiction between "trying new strategies" and "using known best strategies" through a simple probability mechanism to avoid falling into the local optimal solution too early. The execution action at the current moment determined according to the preset strategy includes: selecting the action with the largest Q value at the current moment as the execution action with a probability of 1-ε and randomly selecting any action at the current moment as the execution action with a probability of ε. Specifically: in the ε-greedy strategy, ε represents the exploration probability, and the agent selects the action in two ways. One is to select the action with the largest current Q value with a probability of 1-ε, and execute it by selecting the action with the largest Q value in the Q table under the current state. The other is to randomly select an action in the current state with a probability of ε to explore potential better strategies.

[0053] Furthermore, the iterative training of the Q table by the reward value in step S3 includes: continuously updating the microgrid environment by the reward value to obtain the action value function, and iteratively training the Q table by the action value function. According to the execution of the action, the microgrid environment is updated. The reward obtained by executing the action in the current state is calculated according to the reward function, and the Q table is updated by the action value function. The domain of the action value function Q(s,a) is the state-action pair, and the action a comes from the action space A. It represents the expected value of the cumulative reward that the agent can obtain in the future after selecting action a in state s, as shown in formula (7). After a certain number of iterations, the discount factor γ is modified so that the model can learn from long-term experience. The backbone of the Q-learning algorithm is the two components described below:

[0054]

[0055] This formula mainly contains two parts. The first part before the plus sign represents the immediate reward of the current action, and the second part after the plus sign is the delayed reward at time t under a given strategy π. The strategy π refers to selecting the action with the largest Q value in any state. It represents the expectation of the cumulative reward value that can be obtained at time t. In Q-learning, the strategy is learned implicitly and can be identified as not requiring any prior knowledge. The Bellman equation is used to repeatedly update the Q value through experience to approximate the action value function. The calculation formula of the action value function is:

[0056] Q(st ,a t )=Q(s t ,a t )+α[r(s t ,a t )+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )] ; (7)

[0057] In formula (7), Q(s t ,a t ) is the value of the action value function at the current moment, γ represents the attenuation factor, γ∈[0,1], α represents the learning rate, which indicates the importance of the information obtained during the training process, α∈[0,1]. When α=0, it means ignoring the new information during the training process, and when α=1, it means only using the new information obtained during the training. maxQ(s t+1 ,a t+1 ) is the maximum value of the action value function at time t+1.

[0058] The following is a detailed introduction to the real-time optimization and scheduling method for microgrid energy storage based on reinforcement learning in conjunction with the embodiments, but the implementation of the present invention is not limited to this. This method is based on the interaction between the Q-learning algorithm and the real-time microgrid information, with the goal of minimizing the external grid input cost. The model training is completed by continuously iterative training and updating the Q table, so that the model can select the action with the largest Q value through the microgrid state information after the training is completed. Before the Q-learning algorithm starts training, the Q table needs to be initialized. The algorithm state space and action space are constructed based on the current environmental information of the microgrid, and the reward function of the algorithm is designed to optimize the execution strategy of the intelligent agent. During the model training process, the microgrid environmental state information is input, and the intelligent agent selects an action to execute according to the current state based on the ε-greedy strategy, calculates the reward for the next state, and updates the Q table through the Q value update formula. After continuous training iterations, the model can learn the optimal strategy. The specific usage steps are as follows:

[0059] Step 1:

[0060] Microgrid environment information statistics and algorithm initialization: Statistics of actual grid environment related parameters, and initialization of Q table. Construct algorithm state space, including energy storage battery state, load demand power and photovoltaic power generation power, as shown in formula (1). Construct algorithm action space, and issue relevant instructions to the energy storage system, including charging, discharging and idle, as shown in formula (3). Collect microgrid environment status every 30 minutes. Implement reward function, and construct reward function according to online and offline electricity costs, external grid power, and energy storage charging and discharging power. Among them, the offline electricity cost sets the peak and valley flat electricity price according to the electricity usage time, which is 1.09, 0.29, and 0.66 yuan respectively. The online electricity price is set to 0.37 yuan, and the reward function is shown in formula (4). In the present invention, the reward function is to minimize the operating cost of inputting electricity from the grid and maximize the income of selling electricity to the grid, which represents the negative value of online cost or offline cost. The online cost and offline cost in the time interval are calculated as the reward of the algorithm.

[0061] Step 2:

[0062] Algorithm model parameter setting and training: Input real-time microgrid state data to train the model, where the model iteration number is set to 10000, the time interval t is 30min, the learning rate α is 0.5, the decay factor γ is set to 0.5, and ε is set to 0.6. The model is iteratively trained based on this parameter setting. During the training process, the agent selects the current action according to the ε-greedy strategy, and the control center issues instructions to control the energy storage equipment to charge and discharge. The energy storage capacity is 12000KWh, and the charging and discharging power is 1150KW. The upper and lower limits of the energy storage SOC are 0.85 and 0.19. When executing action instructions, the actual constraints must also be met. During the training process, by inputting microgrid environment data into the model, the agent selects the action instruction of the current state through the ε-greedy strategy, and selects the action with the largest current Q value with a probability of 1-ε, and randomly selects an action in the current state with a probability of ε to explore a better strategy. The grid power for executing the current action is calculated according to the action instruction, and the Q table is updated by the reward obtained by calculating the reward function. Since the calculation of the reward function is the negative value of the cost of going online or going offline, if the current action results in more cost of going online, the reward value will be smaller, and the probability of the subsequent model choosing the changed action will be lower. With the continuous iteration of the training process, the Q-learning algorithm can gradually learn the optimal execution strategy under different states, reducing the cost of inputting electricity from the external power grid. Realize real-time and efficient decision-making for microgrids.

[0063] Step 3:

[0064] Microgrid real-time dispatch: After the reinforcement learning algorithm is trained, real-time decision-making is completed based on the input microgrid environment information. Compared with the traditional decision-making model trained based on prediction data, the model performance of this model will be affected by the data prediction error. The present invention trains the model with real-time microgrid data to avoid decision-making errors caused by prediction errors, and can obtain better decision-making results.

[0065] In addition, the second aspect of the present invention provides a method for optimizing the scheduling of microgrid energy storage using a microgrid energy storage model, wherein the method uses the microgrid energy storage model to optimize the scheduling of the microgrid in real time, and the microgrid energy storage model is trained by any one of the methods for constructing a microgrid energy storage model based on reinforcement learning described above. In this method, the trained model is used to make real-time decisions on the microgrid energy storage to achieve optimal scheduling. After the training is completed, the reinforcement learning model can make real-time decisions on the microgrid and issue correct charging and discharging instructions to the energy storage system based on environmental status information.

[0066] The present invention discloses a construction method and application of a microgrid energy storage model based on reinforcement learning. The method uses a Q-learning algorithm to minimize the cost of 24h external grid input power as the goal, and learns the best strategy based on real real-time data and system interaction, so that the model can maximize rewards in a complex and uncertain grid environment. Compared with the existing optimization scheduling algorithm that relies on the accuracy of prediction data, it is difficult to obtain the best operation strategy due to the influence of random factors in the microgrid environment. This method is trained based on real-time data of the microgrid, and can obtain more accurate and efficient decision results, thereby meeting the requirements of different application scenarios.

[0067] It should be understood that the above specific embodiments of the present invention are only used to illustrate or explain the principles of the present invention, and do not constitute a limitation of the present invention. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present invention should be included in the protection scope of the present invention. In addition, the claims attached to the present invention are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries. The present invention is described above with reference to the embodiments of the present invention. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, a person skilled in the art can make a variety of substitutions and modifications, which should fall within the scope of the present invention. For ordinary technicians in the field, other different forms of changes or modifications can be made on the basis of the above description. It is not necessary and impossible to list all the embodiments here. And the obvious changes or changes derived from this are still within the protection scope of the present invention. The present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1The steps of the functions specified in one or more boxes. Those of ordinary skill in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes the following processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (Read-Only Memory, referred to as ROM) or a random access memory (Random Access Memory, referred to as RAM), etc. The steps in the method of the embodiment of the present invention can be adjusted in sequence, merged and deleted according to actual needs. The module units in the system terminal or device of the embodiment of the present invention can be merged, divided and deleted according to actual needs.

Claims

1. A method for constructing a microgrid energy storage model based on reinforcement learning, characterized in that: The method trains a preset model based on a Q-learning algorithm, including: Obtain microgrid parameter information and initialize the Q table; Define the state, action, and reward function of the Q-learning algorithm; Acquire the current state of the microgrid at a preset time interval, determine the current action of the microgrid according to the state, input the action into the reward function to obtain a corresponding reward value, and iteratively train the Q table through the reward value; When the iterative training meets the preset convergence conditions, a trained microgrid energy storage model is obtained.

2. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 1, characterized in that: Determining the current action of the microgrid according to the state includes: determining the execution action at the current moment according to a preset strategy based on the current state of the microgrid, wherein the preset strategy is an ε-greedy strategy.

3. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 2, characterized in that: The iterative training of the Q table by using the reward value includes: continuously updating the microgrid environment by using the reward value to obtain an action value function, and iteratively training the Q table by using the action value function.

4. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 1, characterized in that: The states include: energy storage SOC, load demand power and photovoltaic power generation power.

5. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 1, characterized in that: The actions include: charging, discharging and idling.

6. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 1, characterized in that: The calculation formula of the reward function is: In the formula, t is the current time, a is t is the action at the current moment, s t is the current state, r(s t ,a t ) is the reward function value at the current moment, C in represents the cost of obtaining electricity from the external power grid, C out represents the revenue from selling electricity to the grid, represents the power of the power grid, and Δt represents the time interval for collecting data.

7. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 3, characterized in that: The calculation formula of the action value function is: Q(s t ,a t )=Q(s t ,a t )+α[r(s t ,a t )+γmaxQ(s t+1 ,a t+1 )-Q(s t ,a t )]; In the formula, Q(s t ,a t ) is the value of the action value function at the current moment, γ represents the attenuation factor, α represents the learning rate, maxQ(s t+1 ,a t+1 ) is the maximum value of the action value function at time t+1.

8. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 7, characterized in that: Determining the execution action at the current moment according to the preset strategy includes: selecting the action with the largest Q value at the current moment as the execution action with a probability of 1-∈ and randomly selecting any action at the current moment as the execution action with a probability of ∈, where ∈ is the exploration probability.

9. The method for constructing a microgrid energy storage model based on reinforcement learning according to claim 1, characterized in that: The preset time is set to 25 to 35 minutes.

10. A method for optimizing the scheduling of microgrid energy storage using a microgrid energy storage model, characterized in that: The method uses a microgrid energy storage model to perform real-time optimization scheduling on the microgrid, and the microgrid energy storage model is trained by the reinforcement learning-based microgrid energy storage model construction method as described in any one of claims 1-9.

Images