Distributed energy storage regulation method and system for microgrid based on deep reinforcement learning

Abstract

The application discloses a distributed energy storage regulation method and system for a micro-grid based on deep reinforcement learning, which comprises the following steps: establishing a response model of each component in the micro-grid according to the obtained micro-grid data and considering demand response; setting a state space, an action space and a reward function of the micro-grid according to the state of the micro-grid and energy storage action decision; constructing an advantage neural network model by using an adversarial network structure according to the response model, the state space, the action space and the reward function; combining deep reinforcement learning with a WHO algorithm, and determining the neural network weight of the advantage neural network model through algorithm iteration optimization; and determining the current energy storage regulation decision according to the obtained advantage neural network model and the state of the micro-grid at each moment. The scheme can improve the ability of energy storage regulation to respond to randomness, the regulation effect is stable, and the overall efficiency of the micro-grid can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of new energy micro-grid, and particularly relates to a distributed energy storage regulation method and system for a micro-grid based on deep reinforcement learning. BACKGROUND

[0002] A micro-grid is an energy network system in which local distributed renewable energy systems, energy storage systems and loads are integrated together, which can be operated in parallel with a power grid and can trade electric energy with a power market, and can also utilize local distributed renewable energy generation and energy storage. As an important component of a micro-grid, an energy storage system effectively suppresses the randomness and volatility of renewable energy generation in a micro-grid by storing and releasing excess energy, improves the economic benefits of a micro-grid, and is an important means of economic dispatching of a micro-grid.

[0003] An energy management system in a micro-grid affects the overall efficiency of the micro-grid and can also schedule local resources. Existing micro-grid energy storage coordination optimization problems only use historical wind speed and light intensity data of each period, and do not take into account the uncertainty of wind power and photovoltaic power. Due to the randomness, volatility, uncertainty and intermittency of distributed wind and light resources in a micro-grid, the uncertainty of load demand and the uncertainty of dynamic prices of a power market, the existing scheme has poor ability to cope with randomness, unstable regulation effect, and low overall efficiency of a micro-grid. SUMMARY

[0004] To overcome the above-mentioned defects, the present application provides a distributed energy storage regulation method and system for a micro-grid based on deep reinforcement learning, which can improve the ability of energy storage regulation to cope with randomness, has stable regulation effect, and can improve the overall efficiency of a micro-grid.

[0005] The embodiment of the present application provides a distributed energy storage regulation method for a micro-grid based on deep reinforcement learning, comprising:

[0006] establishing a response model of each component in the micro-grid according to the obtained micro-grid data and considering demand response;

[0007] setting a state space, an action space and a reward function of the micro-grid according to the state of the micro-grid and energy storage action decision;

[0008] constructing an advantage neural network model using an adversarial network structure according to the response model, the state space, the action space and the reward function;

[0009] combining deep reinforcement learning with a WHO algorithm, determining neural network weights of the advantage neural network model through algorithm iteration optimization, and determining a current energy storage regulation decision according to the advantage neural network model and the state of the micro-grid at each time.

[0010] Preferably, a response model of each component inside the micro-grid is established according to the obtained micro-grid data, including:

[0011] According to the obtained historical wind speed data and illumination intensity data of each period, a wind-solar power generation uncertainty model containing wind speed and illumination intensity prediction is constructed;

[0012] According to the preset electricity cost of the peak period and the valley period, a demand response model of the household price response load is constructed.

[0013] Further, the wind-solar power generation uncertainty model includes , ;

[0014] The demand response model of the household price response load includes ;

[0015] Wherein, are the actual values of wind power generation and photovoltaic power generation at time t respectively; are the predicted values of wind power generation and photovoltaic power generation at time t respectively; are random variables subject to Gaussian distribution, representing the wind power prediction error and the photovoltaic prediction error at time t respectively; L i t is the household price response load at time t, is the basic load at time t; is the transfer load of the household price response load i at time t; is the load transferred from the time before time t to time t of the household price response load i.

[0016] Preferably, the state space, the action space and the reward function of the micro-grid are set according to the state of the micro-grid and the energy storage action decision, including:

[0017] The state information of the internal components of the micro-grid and the external grid information are obtained to construct the state space of the micro-grid at each period;

[0018] According to the state of charge of the energy storage of the micro-grid and the real-time electricity price, the action when the energy storage has surplus and the action when the energy storage has shortage are determined to construct the energy storage action space;

[0019] According to the load and renewable energy generation data at the current time, combined with demand response, the pricing operation action is set by adjusting the electricity price inside the micro-grid to indirectly control the price response load, and the energy management action space is determined;

[0020] The reward function is constructed to maximize the energy income of the micro-grid.

[0021] Further, the state space includes ;

[0022] The energy management action space is ;

[0023] The reward function is ;

[0024] Wherein, s t is the state space of the micro-grid at time t, is the state of charge of the energy storage i at time t; is the market electricity price at time t, is the price of the micro-grid selling electricity to the grid; is the wind power prediction value at time t; is the photovoltaic power prediction value at time t; is the basic load at time t, t is the time information; a t is the action at time t, A represents the action space, is the energy storage power surplus action, is the energy storage power shortage action, A P is the pricing operation action; R t is the reward at time t, L i t is the household price response load at time t, and are the electricity sold to and bought from the external grid, respectively; is the price set by the micro-grid for the household response load.

[0025] Preferably, according to the response model, the state space, the action space and the reward function, an advantage neural network model is constructed using an adversarial network structure, comprising:

[0026] According to the response model, the state space and the reward function, an evaluation network for evaluating the value of the action and a target network for making the network training process more stable are constructed;

[0027] According to the evaluation network and the target network, the action value output by the adversarial network is split into the sum of the state value and the action advantage, and a decentralized action advantage function is constructed;

[0028] Noise is added to the parameters of the neural network of the action advantage function during training, to obtain an advantage neural network model based on the adversarial network and considering noise.

[0029] Preferably, the deep reinforcement learning is combined with the WHO algorithm, neural network weights of the advantage neural network model are determined through algorithm iteration optimization, current energy storage adjustment decisions are determined according to the advantage neural network model and the microgrid state at each moment, and the current energy storage adjustment decisions comprise:

[0030] The WHO algorithm is initialized, and initial parameters of the algorithm are set, the initial parameters comprising a population size N, a proportion of stallions, a population cross rate and a maximum iteration number of the algorithm;

[0031] The population is randomly grouped and initialized, the position information of each individual in the population is randomly initialized, and a preset fitness function is used to calculate the fitness of each individual;

[0032] When the fitness of the global optimal solution individual is not less than a preset threshold, the position information of the population individual is updated according to the behavior of the wild horse population, and the fitness is recalculated after the position information is updated to obtain the updated fitness;

[0033] When the fitness of the global optimal solution individual is less than the threshold, or the iteration number reaches the maximum iteration number of the algorithm, the optimal solution of the current individual is output as the neural network weight of the advantage neural network model;

[0034] The current energy storage adjustment decisions are determined according to the determined advantage neural network model and the microgrid state at each moment.

[0035] The embodiment of the application also provides a distributed energy storage adjustment system of a microgrid based on deep reinforcement learning, the system comprising:

[0036] A response module is configured to establish a response model of each component in the microgrid according to the obtained microgrid data and considering demand response;

[0037] A state module is configured to set a state space, an action space and a reward function of the microgrid according to the state of the microgrid and energy storage action decisions;

[0038] A reward module is configured to construct an advantage neural network model using an adversarial network structure according to the response model, the state space, the action space and the reward function;

[0039] A decision module is configured to combine the deep reinforcement learning with the WHO algorithm, determine neural network weights of the advantage neural network model through algorithm iteration optimization, and determine current energy storage adjustment decisions according to the obtained advantage neural network model and the microgrid state at each moment.

[0040] Preferably, the response module is specifically configured to:

[0041] According to the acquired historical wind speed data and illumination intensity data of each period, a wind-solar power generation uncertainty model containing wind speed and illumination intensity prediction is constructed;

[0042] According to the preset peak period and valley period electricity cost, a demand response model of the household price response load is constructed.

[0043] Further, the wind-solar power generation uncertainty model comprises 、 ;

[0044] The demand response model of the household price response load comprises ;

[0045] Wherein, respectively are actual values of wind power generation and photovoltaic power generation at t moment; respectively are wind power prediction values and photovoltaic prediction values at t moment; are random variables subject to Gaussian distribution, representing wind power prediction error and photovoltaic prediction error at t moment respectively; L i t is the household price response load at t moment, is the basic load at t moment; is the transfer load of the household price response load i at t moment; is the load of the household price response load i transferred from previous time to t moment.

[0046] Preferably, the state module is specifically used for:

[0047] Acquiring state information of internal components of the microgrid and external grid information, and constructing a state space of the microgrid at each period;

[0048] According to the state of charge of the microgrid energy storage and the real-time electricity price, determining the action when the energy storage power is in surplus and the action when the energy storage power is in shortage, and constructing an energy storage action space;

[0049] According to the current load and renewable energy generation data, combined with demand response, indirectly controlling the price response load by adjusting the electricity price in the microgrid, setting a pricing operation action, and determining an energy management action space;

[0050] An reward function is constructed to maximize the energy income of the microgrid.

[0051] Further, the state space comprises ;

[0052] The energy management action space is ;

[0053] The reward function is ;

[0054] wherein s t is the state space of the microgrid at time t, is the state of charge of the energy storage i at time t; is the market electricity price at time t, is the price of the electricity sold by the microgrid to the grid; is the wind power prediction value at time t; is the photovoltaic power prediction value at time t; is the basic load at time t, t is time information; a t is the action at time t, A represents an action space, is the energy storage surplus action, is the energy storage deficit action, A P is the pricing operation action; R t is the reward at time t, L i t is the household price response load at time t, and are the electricity quantities sold to and bought from the external grid, respectively; is the price set by the microgrid for the household price response load.

[0055] Preferably, the reward module is specifically used for:

[0056] building an evaluation network for evaluating the value of the action and a target network for stabilizing the network training process according to the response model, the state space and the reward function;

[0057] splitting the action value output by the adversarial network into the sum of the state value and the action advantage according to the evaluation network and the target network, and building a decentralized action advantage function;

[0058] adding noise to the parameters of the neural network of the action advantage function during training to obtain an advantage neural network model based on the adversarial network and considering the noise.

[0059] Preferably, the decision module is specifically used for:

[0060] initializing the WHO algorithm, setting the initial parameters of the algorithm, the initial parameters including the population size N, the proportion of stallions, the population crossing rate and the maximum number of iterations of the algorithm;

[0061] randomly grouping the population for initialization, randomly initializing the position information of each individual in the population, and calculating the fitness of each individual by using a preset fitness function;

[0062] When the fitness of the individual of the global optimal solution is not less than a preset threshold, the position information of the population individual is updated according to the behavior of a wild horse population, and the fitness is recalculated after the position information is updated to obtain an updated fitness;

[0063] When the fitness of the individual of the global optimal solution is less than the threshold, or the number of iterations reaches the maximum number of iterations of the algorithm, the current individual optimal solution is output as the neural network weight of the advantage neural network model.

[0064] The current energy storage adjustment decision is determined according to the determined advantage neural network model and the state of the microgrid at each moment.

[0065] The microgrid data obtained is used to establish a response model of each component in the microgrid by considering demand response; the state space, the action space and the reward function of the microgrid are set according to the state of the microgrid and the energy storage action decision; the advantage neural network model is constructed by using the adversarial network structure according to the response model, the state space, the action space and the reward function; the deep reinforcement learning is combined with the WHO algorithm, the neural network weight of the advantage neural network model is determined by algorithm iteration optimization, and the current energy storage adjustment decision is determined according to the obtained advantage neural network model and the state of the microgrid at each moment. The scheme can improve the ability of energy storage adjustment to respond to randomness, the adjustment effect is stable, and the overall efficiency of the microgrid can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 is a flowchart of a microgrid distributed energy storage adjustment method based on deep reinforcement learning provided by an embodiment of the application;

[0067] Figure 2 is a structural diagram of a microgrid provided by an embodiment of the application;

[0068] Figure 3 is a wind power output curve diagram provided by an embodiment of the application;

[0069] Figure 4 is a photovoltaic output curve diagram provided by an embodiment of the application;

[0070] Figure 5 is a structural diagram of a microgrid distributed energy storage adjustment system based on deep reinforcement learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0071] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0072] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application. Figure 1 , is a flowchart of a distributed energy storage regulation method of a microgrid based on deep reinforcement learning provided by the embodiments of the present application, and the method comprises steps S1-S4:

[0073] In step S1, a response model of each component in the microgrid is established according to the obtained microgrid data and considering demand response.

[0074] In step S2, a state space, an action space and a reward function of the microgrid are set according to the state of the microgrid and energy storage action decision.

[0075] In step S3, an advantage neural network model is constructed by using an adversarial network structure according to the response model and the state space.

[0076] In step S4, deep reinforcement learning is combined with the WHO algorithm, the neural network weight of the advantage neural network model is determined through algorithm iteration optimization, and the current energy storage regulation decision is determined according to the obtained advantage neural network model and the state of the microgrid at each time.

[0077] In the specific implementation of the embodiments, the response model of each component in the microgrid is established according to the obtained microgrid data, including an energy storage model, a price response load model and a wind and light uncertainty model.

[0078] The state space, the action space and the reward function of the microgrid are set according to the state of the microgrid. That is, the state space of the microgrid at each time is set, which includes the predicted value of wind speed and light intensity, load and energy storage state of charge; the action space of the microgrid is set, which includes the action of energy storage and the internal price segment of the microgrid considering demand response; and the reward function of the model is set.

[0079] The advantage neural network model is constructed by using an adversarial network structure according to the response model and the state space. In order to solve the overestimation problem in traditional Q-Learning and DQN, two networks with the same structure, a target network and an evaluation network, are used. The action with the maximum Q value is selected through the evaluation network, and the value of the action is determined through the target network.

[0080] The deep reinforcement learning is combined with a WHO (Wild Horse Optimizer) algorithm, that is, the deep reinforcement learning is combined with the WHO wild horse optimization algorithm, the WHO algorithm is used to replace a traditional optimizer, neural network weights of the advantage neural network model are determined through algorithm iteration optimization, and energy storage adjustment decisions are determined according to the advantage neural network model and the microgrid state at each moment.

[0081] The scheme provided in the application considers the uncertainty of wind and light power generation and demand response, autonomously makes decisions by an energy management system agent according to real-time states and data of the microgrid, optimizes the energy storage charging and discharging strategy, indirectly regulates and controls the price response load by changing the electricity price in the microgrid, and optimizes the weight parameters of the neural network of the agent through the WHO algorithm optimization process, which is beneficial to fully consume the distributed renewable energy in the microgrid, such as wind and light, and maximize the benefits of the microgrid.

[0082] In another embodiment provided in the application, the step S1 specifically comprises the following steps:

[0083] Referring to Figure 2 is a structural schematic diagram of the microgrid provided in the embodiment of the application. The microgrid structure comprises a microgrid energy management system for energy management of the distributed renewable energy, the external power grid and the energy storage.

[0084] The control information flow of the energy storage adjustment is generated through the information flow of the distributed renewable energy, the external power supply and the energy storage, and then the control of the energy flow is realized, the indirect control of the household price response load is considered in the energy storage adjustment, and the maximum benefit of the microgrid is realized.

[0085] Based on the historical wind speed and light intensity data of each period, a wind and light power generation uncertainty model is constructed, in which the wind speed and light intensity prediction error satisfies normal distribution;

[0086] Referring to Figure 3 is a wind power output curve diagram provided in the embodiment of the application. Referring to Figure 4 is a photovoltaic power output curve diagram provided in the embodiment of the application.

[0087] The electricity price in the microgrid is adjusted within a certain range near the market electricity price, the household price response load is indirectly controlled, the load at the electricity peak period can be shifted to the load at the electricity valley period to a certain extent, and a response model of the household price response load is constructed according to the electricity cost at the preset peak period and valley period.

[0088] The energy management intelligent agent can only observe the predicted wind power and photovoltaic power data when making decisions; and the actual power generation of wind and light needs to be settled in the settlement single moment, and the wind and light abandonment and load loss caused by the decision problem will be charged with a certain penalty cost, which requires the intelligent agent to consider the uncertainty of wind and light in decision-making, thereby improving the ability of the energy management system intelligent agent to handle new energy prediction error.

[0089] In another embodiment provided by the application, the wind and light power generation uncertainty model constructed including wind speed and light intensity prediction comprises ; ;

[0090] Among them, respectively, the actual value of wind power generation and the actual value of photovoltaic power generation at t moment; respectively, the predicted value of wind power generation and the predicted value of photovoltaic power generation at t moment; are random variables subject to Gaussian distribution, representing the wind power prediction error and the photovoltaic prediction error at t moment respectively;

[0091] Assuming that the household price response load in the micro-grid has a variable component affected by the price, the price in the micro-grid can be adjusted within a certain range around the market price, so as to indirectly control the household price response load, so that the load can shift the electricity use in the load peak period to the load valley period to a certain extent, thereby reducing the load peak-valley difference of the micro-grid and bringing greater benefits. The price response load has two characteristic parameters, which are and ; wherein is the price sensitivity factor of the household price response load i, and , which represents the percentage of load that can be reduced or increased under the condition of price rise or fall; is the patience parameter of the household price response load i, indicating that the load shifted should be repaid within how long.

[0092] The demand symbol load model of the household price response load includes: ;

[0093] Among them, L i t is the household price response load at t moment, is the basic load at t moment; is the shifted load of the household price response load i at t moment; is the load shifted by the household price response load i from the time before t moment to t moment.

[0094] ;

[0095] ;

[0096] where, is the price level at time t, and is the amount of load that household i shifts in response to the price is positive because when the microgrid pricing is higher, in order to reduce cost, it will try to consume this part of load at later time when the price is lower; vice versa when is negative. is a 0-1 decision variable that indicates whether the load shifted away from time period j is to be consumed at the current time, which value depends on a random variable that depends on the current price level and the time experienced since time step j, which is closer to the patience parameter , the probability of the load to be executed at the current time is higher. The random variable is calculated as

[0097] where,

[0098] In a further embodiment provided by the application, the step S2 specifically comprises:

[0099] The state space of each time period of the microgrid should contain the microgrid state information needed for making action decisions and the external grid information that can be obtained, specifically: the predicted wind speed and light intensity value at this time, market electricity price information, load information, state of charge, climbing rate and maximum charging and discharging power of each energy storage.

[0100] Therefore, the state information of the microgrid and the external grid information are obtained, and the state space of each time period of the microgrid is constructed.

[0101] The action space includes two parts of energy storage action and microgrid internal pricing action. Among them, the action of the energy storage needs to be determined according to the size of the net load (wind and light power generation minus load power) and the real-time electricity price, so the action space of the energy storage is set to two dimensions, which are the action when the power surplus (net load is greater than 0) and the action when the power shortage (net load is less than 0), and both are 0-1 variables, which are used for the priority of the energy storage and the grid. If the net load level is greater than 0, at this time, if the power surplus action is 1, the excess power is preferentially charged to the energy storage, and if there is still power left after the energy storage is fully charged, the remaining part is sold to the grid; if is 0, the excess power is directly sold to the grid. If the net load level is less than 0, at this time, if the power shortage action ​​​​If 1, the shortage between distributed energy and load is preferentially made up by energy storage discharge, and the insufficient part is made up by purchasing electricity from the power grid; if 0, the net load shortage part is directly made up by purchasing electricity from the power grid. If 0, the net load shortage part is directly made up by purchasing electricity from the power grid.

[0102] For the internal pricing action part in the action space of the micro-grid, considering that some resident loads in the micro-grid have a variable component affected by the electricity price, according to the load and renewable energy generation at the current moment, the demand response is considered, the price response load condition is indirectly controlled by adjusting the high and low of the electricity price in the micro-grid. The pricing operation is set to consist of five possible actions, which represent different electricity price segments, and is represented as According to the load and renewable energy generation data at the current moment, combined with the demand response, the price response load condition is indirectly controlled by adjusting the electricity price in the micro-grid, the pricing operation action is set, and the energy management action space is determined;

[0103] In the micro-grid optimization scheduling model, the reward function is constructed to maximize the overall income of the micro-grid.

[0104] In another embodiment provided by the application, the state space comprises ;

[0105] Wherein, s t is the state at time t, is the state of charge of the energy storage i at time t; is the electricity price of market electricity sales at time t, is the price of electricity sold to the power grid by the micro-grid; is the wind power prediction value at time t; is the photovoltaic power generation prediction value at time t; is the basic load at time t, and t is the time information.

[0106] The energy management action space is ;

[0107] Wherein, A represents the action space, is the electricity surplus action, is the electricity shortage action, is the pricing operation action;

[0108] The reward function is ;

[0109] Wherein, R t is the reward at time t, L i t is the household price response load at time t, and are the electricity sold and bought from the external power grid, respectively; A price for a price-responsive load in a microgrid.

[0110] In yet another embodiment provided by the present application, the step S3 specifically comprises the following steps:

[0111] To solve the overestimation problem in traditional Q-Learning and DQN, two networks with the same structure are adopted: a target network and an evaluation network. The action with the maximum Q value is selected by the evaluation network, and the value of the action is determined by the target network. That is, the evaluation network for selecting actions and the target network for determining the value of actions are constructed according to the response model and the state space.

[0112] Therefore, the calculation of the target value is:

[0113] ;

[0114] wherein, is the target reward value at time t; is the reward value at time t; is a state-action value function, fitted by a neural network, denotes the parameters of the neural network; and are the parameters of the evaluation network and the target network, respectively; , and denote the Q values obtained by the evaluation network and the target network, respectively, when the action is taken in the state . The formula indicates that the policy that can make the Q value reach the maximum value is selected by the evaluation network , and the Q value of taking this policy in the state is evaluated by the target network.

[0115] Both the target network and the evaluation network adopt the structure of an adversarial network. Unlike the direct output of the Q value in the past, the output is split into the sum of the state value and the action advantage. According to the evaluation network and the target network, the output of the reward function is split into the sum of the state value and the action advantage, that is:

[0116]

[0117] wherein, is a state-action value function; is a state value function, used to evaluate the value in a specific state; is an action advantage function, used to evaluate the relative degree of importance of each action in a specific state; is a network parameter thereof; denotes the number of selectable actions, ​​represents the average of all the action advantage, and the average of all the action advantage is subtracted to decentralize the action advantage function.

[0118] The parameters of the neural network of the action advantage function are added with noise during training to obtain an advantage neural network model based on an adversarial network and considering noise.

[0119] To optimize the exploration of the optimal strategy and the robustness of the training during the training, the parameters of the neural network are added with noise. Therefore, the neural network full connection layer expression considering noise is:

[0120] ;

[0121] wherein, is a parameter that needs to be learned from experience; the matrix and the vector Each element of the matrix and the vector is independently drawn from a normal distribution

[0122] , indicating noise.

[0123] wherein, is a parameter, is noise drawn from a normal distribution .

[0124] The noise has randomness, and after the training is completed, the noise is no longer needed when making decisions, and the parameter is set to 0, and only the parameter is retained. Therefore, the neural network considering the adversarial network structure and added with noise can be represented as:

[0125]

[0126] It should be noted that the scheme of the present application also sets a priority experience replay mechanism, specifically:

[0127] When the agent interacts with the environment each time, the current state, the action taken, the reward obtained, and the next state transferred after the action is taken, i.e. , all these information will be packaged and stored in the memory bank. The memory bank capacity is set to Memory_size, and when the existing experience data in the memory bank reaches a certain amount, the neural network will be trained each time by randomly sampling batch_size experience bars from the memory bank for training, so as to break the correlation between samples and improve the sample efficiency.

[0128] However, the importance of all experience samples is not consistent, and the samples with larger errors and larger upward space should be learned preferentially to improve the training efficiency. ;

[0129] wherein, is the Q value generated by the target network, is the Q value generated by the evaluation network, and the experience sample with a larger difference between the two needs to be learned preferentially, thereby evaluating the priority of the experience sample. The agent continuously interacts with the environment and learns past experience, updates iteration, and finally learns a neural network that can replace the most valuable function, thereby comprehensively considering demand response and forming a storage coordination strategy that can maximize the profit of the micro-grid.

[0130] In another embodiment provided by the application, the step S4 specifically comprises the following steps:

[0131] The training of reinforcement learning is essentially to find the optimal state value function, and deep reinforcement learning fits the state value function through a neural network, continuously interacts with the environment, and tries and errors, so as to find a set of optimal neural network weight parameters. In order to accelerate and optimize the process of finding optimal network weights of the deep reinforcement learning neural network, the wild horse optimization algorithm WHO is used to solve the process.

[0132] A set of weights of the evaluation network and the target network is defined as a horse in the wild horse optimization algorithm, and the position of the horse represents the size of the neural network weight parameter. The percentage of stallions in the population is set to , and the population is initially divided into groups according to this, the number of stallions is G, and the number of the remaining horses is . The wild horse optimization algorithm simulates the behaviors of the wild horse population in the optimization process:

[0133] Grazing behavior: the stallion leads the entire population to move, and the remaining horses graze around the stallion to search for the optimal solution. The position update formula of each horse is:

[0134] wherein, is the current position of the remaining individual, is the updated position of the remaining individual; is the current position of the stallion, is a random number of to ensure that the remaining individuals graze around the population. Z is an adaptive parameter for adjusting the distance between each position update and the stallion individual.

[0135] Hybridization behavior: When foals in a population reach maturity, they will leave the population to mate with foals from other populations to produce new individuals. The location information of hybrid individuals is updated in the following way: ;

[0136] in, The position of the new individual within the group after mating; This provides the positional information for the two parent individuals, i and j.

[0137] Population movement in search of waterholes: To find better waterholes (optimal solutions), stallions will lead the herd onward, with the stallion's location information replacing the location information of the herd it leads. The location of an individual stallion is updated as follows:

[0138] ;

[0139] In the formula, The location of the waterhole is also the global optimal solution that needs to be found in this algorithm; For population The current location of the stallion; For population The updated location of the stallion; for The random number determines the clockwise or counterclockwise direction of the population's movement; Z is the aforementioned adaptive parameter for adjusting the position and updating the distance. for Random numbers.

[0140] Stallion selection behavior within the population: Initially, stallions are randomly selected. As the algorithm progresses, individual stallions are selected based on the fitness information of the population. The selection mechanism is as follows:

[0141] ;

[0142] In the formula, The position of a group member; For individuals The fitness function, in this case, is the loss function for individual i in this set of parameters. ,Right now:

[0143]

[0144] in, , They are and The reward discount factor for each step; They are the first The state of the step; These are the parameters of the i-th target network and the evaluation network, respectively; is the return of the step; respectively, the action strategy taken by the first step; parameters of the target network, is the evaluation network parameter.

[0145] The specific steps for updating the WHO-based evaluation network and the target network parameters are as follows:

[0146] Before each training begins, the WHO algorithm is initialized, and the initial parameters of the algorithm are set, including the population size N, the proportion of stallion individuals , the population cross rate , and the maximum number of iterations of the algorithm ;

[0147] As a preferred solution, see Table 1, which is an example of a set of initial parameters provided.

[0148] Table 1 Initial parameter table

[0149]

[0150] The population is randomly initialized, and the position information of each individual in the population is randomly initialized, and the corresponding fitness value is calculated .

[0151] The optimization algorithm performs grazing, crossing, population movement to find water pits, and selection of stallion individuals within the population according to the behavior of the wild horse population; after each behavior, the position information of the population individuals is substituted into the fitness function (the above loss function) to calculate the fitness, and the new fitness is returned to the population individuals;

[0152] The above steps are repeatedly performed until the fitness function value of the global optimal solution is less than or the number of iterations reaches the maximum number of iterations , the optimization is stopped.

[0153] The current individual optimal solution is output as the neural network weight of the dominant neural network model;

[0154] According to the determined dominant neural network model and the state of the microgrid at each time, the current energy storage regulation decision is determined.

[0155] The application scheme is based on the deep reinforcement learning Rainbow+WHO algorithm, and in a data-driven manner, by considering the uncertainty of wind and light power generation and demand response, the energy management system agent makes autonomous decision according to the real-time state and data of the microgrid, optimizes the charging and discharging strategy of the energy storage, changes the electricity price in the microgrid to indirectly regulate and control the price response load, and optimizes the weight parameter optimization process of the neural network of the agent through the WHO algorithm, which is beneficial to fully consume the distributed renewable energy such as wind and light in the microgrid, and realizes the maximum benefit of the microgrid.

[0156] The embodiment of the application also provides a distributed energy storage regulation system of a microgrid based on deep reinforcement learning, referring to Figure 5 It is a structural schematic diagram of the distributed energy storage regulation system of the microgrid based on deep reinforcement learning provided by the embodiment of the application, and the system comprises:

[0157] A response module is used to establish a response model of each component in the microgrid according to the obtained microgrid data and considering demand response;

[0158] A state module is used to set the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision;

[0159] A reward module is used to construct an advantage neural network model by using an adversarial network structure according to the response model, the state space, the action space and the reward function;

[0160] A decision module is used to combine deep reinforcement learning with the WHO algorithm, to determine the neural network weight of the advantage neural network model through algorithm iteration optimization, and to make energy storage regulation decisions according to the advantage neural network model and the state of the microgrid at each moment.

[0161] It should be noted that the distributed energy storage regulation system of the microgrid based on deep reinforcement learning provided by the embodiment of the application can perform the distributed energy storage regulation method of the microgrid based on deep reinforcement learning described in any of the above embodiments, and the specific functions of the distributed energy storage regulation system of the microgrid based on deep reinforcement learning will not be repeated here.

[0162] The above is the preferred embodiment of the application, and it should be noted that for ordinary skilled persons in the technical field, several improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements are also considered within the protection scope of the application.

Claims

1. A distributed energy storage regulation method for a microgrid based on deep reinforcement learning, characterized in that, The method comprises: According to the obtained micro-grid data and considering demand response, a response model of each component in the micro-grid is established; According to the state of the micro-grid and the action decision of the energy storage, a state space, an action space and a reward function of the micro-grid are set; According to the response model, the state space, the action space and the reward function, an advantage neural network model is constructed by using an adversarial network structure; Deep reinforcement learning is combined with the WHO algorithm, and the neural network weight of the advantage neural network model is determined through algorithm iteration optimization, and the current energy storage adjustment decision is determined according to the obtained advantage neural network model and the state of the micro-grid at each moment; According to the response model, the state space, the action space and the reward function, an advantage neural network model is constructed by using an adversarial network structure, comprising: According to the response model, the state space and the reward function, an evaluation network for evaluating the value of action and a target network for making the network training process more stable are constructed; According to the evaluation network and the target network, the action value output by the adversarial network is split into the sum of state value and action advantage, and a decentralized action advantage function is constructed; Noise is added to the parameters of the neural network of the action advantage function during training, and an advantage neural network model based on the adversarial network and considering noise is obtained; Deep reinforcement learning is combined with the WHO algorithm, and the neural network weight of the advantage neural network model is determined through algorithm iteration optimization, and the current energy storage adjustment decision is determined according to the obtained advantage neural network model and the state of the micro-grid at each moment, comprising: The WHO algorithm is initialized, and the initial parameters of the algorithm are set, including population size N, stallion individual proportion, population crossing rate and maximum iteration number of the algorithm; The population is randomly grouped and initialized, and the position information of each individual in the population is randomly initialized, and a preset fitness function is used to calculate the fitness of each individual; When the fitness of the global optimal solution individual is not less than a preset threshold, the position information of the population individual is updated according to the behavior of the stallion population, and the fitness is recalculated after the position information is updated to obtain the updated fitness; When the fitness of the global optimal solution individual is less than the threshold, or the iteration number reaches the maximum iteration number of the algorithm, the optimal solution of the current individual is output as the neural network weight of the advantage neural network model; According to the determined advantage neural network model and the state of the micro-grid at each moment, the current energy storage adjustment decision is determined; According to the obtained micro-grid data and considering demand response, a response model of each component in the micro-grid is established, comprising: According to the obtained historical wind speed data and light intensity data of each period, a wind and light power generation uncertainty model containing wind speed and light intensity prediction is constructed; According to the preset peak period and valley period electricity cost, a demand response model of household price response load is constructed; The wind-solar power generation uncertain model comprises , ; The household price response load demand response model includes ; wherein, are the actual wind power generation and the actual photovoltaic power generation at time t, respectively; are the predicted wind power and the predicted photovoltaic power at time t, respectively; are the random variables subject to Gaussian distribution, representing the predicted wind power error and the predicted photovoltaic power error at time t, respectively; i t is the household price-responsive load at time t, is the basic load at time t; is the shifting load of the household price-responsive load i at time t; is the load shifted from time before time t to time t of the household price-responsive load i. According to the state of the micro-grid and the action decision of the energy storage, a state space, an action space and a reward function of the micro-grid are set, comprising: Obtain state information of components inside the micro-grid and external grid information, and construct a state space of the micro-grid at each time period; Determine actions when the energy storage is in surplus and actions when the energy storage is in deficit according to the state of charge of the energy storage of the micro-grid and real-time electricity prices, and construct an energy storage action space; According to the current load and renewable energy generation data, combined with demand response, indirectly control the price response load by adjusting the price inside the micro-grid, set the pricing operation action, and determine the energy management action space; Maximize the energy revenue of the micro-grid as the target to construct a reward function.

2. The method of claim 1, wherein, The state space comprises ; The energy management action space is ; The reward function is ; wherein s t is the state space of the microgrid at time t, is the state of charge of the energy storage i at time t; is the market price of electricity sold at time t, is the price at which the microgrid sells electricity to the grid; is the wind power prediction at time t; is the photovoltaic power prediction at time t; is the basic load at time t, t being the time information; a t is the action at time t, A denoting the action space, is the energy storage surplus action, is the energy storage deficit action, A P is the pricing action; R t is the reward at time t, L i t is the household price-responsive load at time t, and are the amounts of electricity sold to and bought from the external grid, respectively; is the price set by the microgrid for the household price-responsive load.

3. A distributed energy storage regulation system for a microgrid based on deep reinforcement learning, characterized in that, The system comprises: a response module configured to establish a response model of each component inside the micro-grid according to the obtained micro-grid data and considering demand response; a state module configured to set a state space, an action space and a reward function of the micro-grid according to the state of the micro-grid and the energy storage action decision; a reward module configured to construct an advantage neural network model using an adversarial network structure according to the response model, the state space, the action space and the reward function; a decision module configured to combine deep reinforcement learning with WHO algorithm, determine neural network weights of the advantage neural network model through algorithm iteration optimization, and determine a current energy storage adjustment decision according to the obtained advantage neural network model and the state of the micro-grid at each time; The reward module is specifically configured to: construct an evaluation network for evaluating the value of actions and a target network for making the network training process more stable according to the response model, the state space and the reward function; split the action value output by the adversarial network into the sum of state value and action advantage according to the evaluation network and the target network, and construct a decentralized action advantage function; add noise to the parameters of the neural network of the action advantage function during training to obtain an advantage neural network model based on the adversarial network and considering noise; The decision module is specifically configured to: initialize the WHO algorithm, set initial parameters of the algorithm, and the initial parameters include population size N, stallion individual proportion, population crossing rate and maximum iteration number of the algorithm; randomly group the population for initialization, randomly initialize the position information of each individual in the population, and calculate the fitness of each individual using a preset fitness function; when the fitness of the global optimal solution is not less than a preset threshold, update the position information of the population individuals according to the behavior of the stallion population, and calculate the updated fitness after updating the position information; when the fitness of the global optimal solution is less than the threshold or the iteration number reaches the maximum iteration number of the algorithm, output the optimal solution of the current individual as the neural network weights of the advantage neural network model; determine a current energy storage adjustment decision according to the determined advantage neural network model and the state of the micro-grid at each time; establish a response model of each component inside the micro-grid according to the obtained micro-grid data and considering demand response, comprising: construct a wind and light power generation uncertainty model containing wind speed and light intensity prediction according to the obtained historical wind speed data and light intensity data at each time period; According to the preset peak period and valley period electricity cost, a demand response model of household price response load is constructed; The wind-solar power generation uncertain model comprises , ; The household price response load demand response model includes ; wherein, respectively are the actual wind power generation value and the actual photovoltaic power generation value at time t; respectively are the predicted wind power value and the predicted photovoltaic value at time t; are random variables obeying Gaussian distribution, respectively representing the predicted wind power error and the predicted photovoltaic error at time t; L i t is the household price response load at time t, is the basic load at time t; is the transfer load of the household price response load i at time t; is the load transferred from the time before time t to time t of the household price response load i. The state module is specifically used for: Obtaining state information of internal components of the microgrid and external grid information, and constructing a state space of the microgrid in each period; According to the state of charge of the energy storage of the microgrid and the real-time electricity price, the action when the energy storage has surplus power and the action when the energy storage has power shortage are determined, and an action space of the energy storage is constructed; According to the current load and renewable energy generation data, combined with demand response, the price response load is indirectly controlled by adjusting the electricity price in the microgrid, the pricing operation action is set, and the energy management action space is determined; An incentive function is constructed to maximize the energy income of the microgrid.

Images