Distributed energy storage adjustment method and system of micro-grid based on deep reinforcement learning

Abstract

The invention discloses a distributed energy storage adjustment method and system of a micro-grid based on deep reinforcement learning. The method comprises the following steps: establishing a response model of each component in the micro-grid according to acquired micro-grid data and by considering demand response; setting a state space, an action space and a reward function of the micro-grid according to the state of the micro-grid and an energy storage action decision; constructing a dominant neural network model by adopting an adversarial network structure according to the response model, the state space, the action space and the reward function; and combining deep reinforcement learning with a WHO algorithm, determining a neural network weight of the dominant neural network model through algorithm iterative optimization, and determining a current energy storage adjustment decision according to the obtained dominant neural network model and the microgrid state at each moment. According to the scheme, the capability of energy storage adjustment coping with randomness can be improved, the adjustment effect is stable, and the overall efficiency of the microgrid can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of new energy microgrids, and in particular to a distributed energy storage regulation method and system for a microgrid based on deep reinforcement learning. Background Art

[0002] A microgrid is an energy network system that integrates local distributed renewable energy systems, energy storage systems, and loads. It can operate in parallel with the power grid and trade electricity with the power market, and can also use local distributed renewable energy for power generation and storage. As an important component of a microgrid, the energy storage system effectively smooths out the randomness and volatility of renewable energy generation in the microgrid by storing and releasing excess energy, thereby improving the economic benefits of the microgrid and is an important means of economic dispatch for the microgrid.

[0003] The energy management system in the microgrid affects the overall efficiency of the microgrid and can also perform local resource scheduling. The existing microgrid energy storage coordination optimization problem only uses the wind speed and light intensity data of each historical period, and does not take into account the uncertainty of wind power and photovoltaics. Due to the randomness, volatility, uncertainty and intermittency of distributed wind and solar power generation in the microgrid, the uncertainty of load demand and the uncertainty of dynamic prices in the electricity market, the existing scheme has poor ability to deal with randomness in regulation decisions, and the regulation effect is unstable, resulting in low overall efficiency of the microgrid. Summary of the invention

[0004] In response to the above-mentioned defects, the present invention provides a distributed energy storage regulation method and system for a microgrid based on deep reinforcement learning, which can improve the ability of energy storage regulation to cope with randomness, has a stable regulation effect, and can improve the overall efficiency of the microgrid.

[0005] The embodiment of the present invention provides a distributed energy storage regulation method for a microgrid based on deep reinforcement learning, comprising: Establish a response model of each component within the microgrid based on the acquired microgrid data and considering demand response; Setting the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision; According to the response model, the state space, the action space and the reward function, a dominant neural network model is constructed using an adversarial network structure; Deep reinforcement learning is combined with the WHO algorithm, and the neural network weights of the dominant neural network model are determined through algorithm iteration and optimization. The current energy storage adjustment decision is determined based on the dominant neural network model and the state of the microgrid at each moment.

[0006] Preferably, a response model of each component within the microgrid is established based on the acquired microgrid data, including: Based on the historical wind speed data and light intensity data obtained in each period, a wind and solar power generation uncertainty model including wind speed and light intensity prediction is constructed; A demand response model for household price-responsive loads is constructed based on the preset electricity costs during peak and off-peak hours.

[0007] Furthermore, the wind and solar power generation uncertainty model includes , ; The demand response model of household price responsive load includes ; in, are the actual values ​​of wind power generation and photovoltaic power generation at time t respectively; They are the wind power forecast value and photovoltaic forecast value at time t respectively; are random variables that obey Gaussian distribution, representing the wind power prediction error and photovoltaic prediction error at time t respectively; L i t is the household price response load at time t, is the basic load at time t; is the shifted load of household price response load i at time t; is the load of household price response load i transferred from time before time t to time t.

[0008] Preferably, the state space, action space and reward function of the microgrid are set according to the state of the microgrid and the energy storage action decision, including: Obtain the status information of the internal components of the microgrid and the external grid information, and construct the state space of the microgrid in each period; According to the charge state of the microgrid energy storage and the real-time electricity price, determine the actions when the energy storage power is surplus and the actions when the energy storage power is insufficient, and build the energy storage action space; According to the current load and renewable energy generation data, combined with demand response, the price response load is indirectly controlled by adjusting the electricity price within the microgrid, setting pricing operation actions and determining the energy management action space; The reward function is constructed with the goal of maximizing the energy benefit of the microgrid.

[0009] Furthermore, the state space includes ; The energy management action space is ; The reward function is ; Among them, s t is the state space of the microgrid at time t, is the state of charge of energy storage i at time t; is the market electricity price at time t, The price at which the microgrid sells electricity to the grid; is the wind power forecast value at time t; is the predicted value of photovoltaic power generation at time t; is the basic load at time t, t is the time information; a t is the action at time t, A represents the action space, For energy storage power surplus action, A is the energy storage power shortage action. P is the pricing operation; R t is the reward at time t, L i t is the household price response load at time t, and The electricity sold to and purchased from the external grid respectively; The electricity price set for household responsive loads within the microgrid.

[0010] Preferably, according to the response model, the state space, the action space and the reward function, an adversarial network structure is used to construct a dominant neural network model, including: Constructing an evaluation network for evaluating action value and a target network for making the network training process more stable according to the response model, the state space and the reward function; According to the evaluation network and the target network, the action value output by the adversarial network is split into the sum of the state value and the action advantage, and a decentralized action advantage function is constructed; During training, noise is added to the parameters of the neural network of the action advantage function to obtain an advantage neural network model based on an adversarial network and taking noise into consideration.

[0011] Preferably, deep reinforcement learning is combined with the WHO algorithm, and the neural network weights of the dominant neural network model are determined by algorithm iteration and optimization. The current energy storage regulation decision is determined according to the dominant neural network model and the microgrid state at each moment, including: Initialize the WHO algorithm and set the initial parameters of the algorithm, including the population size N, the proportion of stallion individuals, the population hybridization rate, and the maximum number of iterations of the algorithm; The population is randomly divided into groups and initialized, the position information of each individual in the population is randomly initialized, and the fitness of each individual is calculated using a preset fitness function; When the fitness of the individual of the global optimal solution is not less than the preset threshold, the position information of the individual of the population is updated according to the behavior of the wild horse population, and the fitness is recalculated after the position information is updated to obtain the updated fitness; When the fitness of the individual of the global optimal solution is less than the threshold, or the number of iterations reaches the maximum number of iterations of the algorithm, output the current individual optimal solution as the neural network weight of the dominant neural network model; The current energy storage regulation decision is determined based on the determined dominant neural network model and the microgrid status at each moment.

[0012] An embodiment of the present invention further provides a distributed energy storage regulation system for a microgrid based on deep reinforcement learning, the system comprising: A response module is used to establish a response model of each component within the microgrid based on the acquired microgrid data and taking into account the demand response; A state module, used to set the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision; A reward module, configured to construct a dominant neural network model using an adversarial network structure according to the response model, the state space, the action space, and the reward function; The decision-making module is used to combine deep reinforcement learning with the WHO algorithm, determine the neural network weights of the dominant neural network model through algorithm iteration and optimization, and determine the current energy storage regulation decision based on the obtained dominant neural network model and the microgrid status at each moment.

[0013] Preferably, the response module is specifically used for: Based on the historical wind speed data and light intensity data obtained in each period, a wind and solar power generation uncertainty model including wind speed and light intensity prediction is constructed; A demand response model for household price-responsive loads is constructed based on the preset electricity costs during peak and off-peak hours.

[0014] Furthermore, the wind and solar power generation uncertainty model includes , ; The demand response model of household price responsive load includes ; in, are the actual values ​​of wind power generation and photovoltaic power generation at time t respectively; They are the wind power forecast value and photovoltaic forecast value at time t respectively; are random variables that obey Gaussian distribution, representing the wind power prediction error and photovoltaic prediction error at time t respectively; L i t is the household price response load at time t, is the basic load at time t; is the shifted load of household price response load i at time t; is the load of household price response load i transferred from time before time t to time t.

[0015] Preferably, the status module is specifically used for: Obtain the status information of the internal components of the microgrid and the external grid information, and construct the state space of the microgrid in each period; According to the charge state of the microgrid energy storage and the real-time electricity price, determine the actions when the energy storage power is surplus and the actions when the energy storage power is insufficient, and build the energy storage action space; According to the current load and renewable energy generation data, combined with demand response, the price response load is indirectly controlled by adjusting the electricity price within the microgrid, setting pricing operation actions and determining the energy management action space; The reward function is constructed with the goal of maximizing the energy benefit of the microgrid.

[0016] Furthermore, the state space includes ; The energy management action space is ; The reward function is ; Among them, s t is the state space of the microgrid at time t, is the state of charge of energy storage i at time t; is the market electricity price at time t, The price at which the microgrid sells electricity to the grid; is the wind power forecast value at time t; is the predicted value of photovoltaic power generation at time t; is the basic load at time t, t is the time information; a t is the action at time t, A represents the action space, For energy storage power surplus action, A is the energy storage power shortage action. P is the pricing operation; R t is the reward at time t, L i t is the household price response load at time t, and The electricity sold to and purchased from the external grid respectively; The electricity price set for household price-responsive loads within the microgrid.

[0017] Preferably, the reward module is specifically used for: Constructing an evaluation network for evaluating action value and a target network for making the network training process more stable according to the response model, the state space and the reward function; According to the evaluation network and the target network, the action value output by the adversarial network is split into the sum of the state value and the action advantage, and a decentralized action advantage function is constructed; During training, noise is added to the parameters of the neural network of the action advantage function to obtain an advantage neural network model based on an adversarial network and taking noise into consideration.

[0018] Preferably, the decision module is specifically used for: Initialize the WHO algorithm and set the initial parameters of the algorithm, including the population size N, the proportion of stallion individuals, the population hybridization rate, and the maximum number of iterations of the algorithm; The population is randomly divided into groups and initialized, the position information of each individual in the population is randomly initialized, and the fitness of each individual is calculated using a preset fitness function; When the fitness of the individual of the global optimal solution is not less than the preset threshold, the position information of the individual of the population is updated according to the behavior of the wild horse population, and the fitness is recalculated after the position information is updated to obtain the updated fitness; When the fitness of the individual of the global optimal solution is less than the threshold, or the number of iterations reaches the maximum number of iterations of the algorithm, output the current individual optimal solution as the neural network weight of the dominant neural network model; The current energy storage regulation decision is determined based on the determined dominant neural network model and the microgrid status at each moment.

[0019] The distributed energy storage regulation method and system of a microgrid based on deep reinforcement learning provided by the present invention establishes a response model of each component within the microgrid according to the acquired microgrid data and taking into account the demand response; sets the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision; constructs a dominant neural network model using an adversarial network structure according to the response model, the state space, the action space and the reward function; combines deep reinforcement learning with the WHO algorithm, determines the neural network weights of the dominant neural network model through algorithm iteration and optimization, and determines the current energy storage regulation decision based on the obtained dominant neural network model and the state of the microgrid at each moment. This solution can improve the ability of energy storage regulation to cope with randomness, has a stable regulation effect, and can improve the overall efficiency of the microgrid. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a distributed energy storage regulation method of a microgrid based on deep reinforcement learning provided by an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a microgrid provided by an embodiment of the present invention; Figure 3is a wind power output curve diagram provided by an embodiment of the present invention; Figure 4 is a photovoltaic output curve diagram provided by an embodiment of the present invention; Figure 5 It is a structural schematic diagram of a distributed energy storage regulation system for a microgrid based on deep reinforcement learning provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 , is a flow chart of a distributed energy storage regulation method for a microgrid based on deep reinforcement learning provided by an embodiment of the present invention, the method comprising steps S1 to S4: Step S1, establishing a response model of each component within the microgrid based on the acquired microgrid data and taking into account demand response; Step S2, setting the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision; Step S3, constructing a dominant neural network model using an adversarial network structure according to the response model and the state space; Step S4, combining deep reinforcement learning with the WHO algorithm, determining the neural network weights of the dominant neural network model through algorithm iteration and optimization, and determining the current energy storage adjustment decision based on the obtained dominant neural network model and the microgrid status at each moment.

[0023] During the specific implementation of this embodiment, response models of various components within the microgrid are established based on the acquired microgrid data, including energy storage, price response load and wind and solar uncertainty models.

[0024] The state space, action space and reward function of the microgrid are set according to the state of the microgrid. That is, the state space of the microgrid at each moment, including the predicted values ​​of wind speed and light intensity, load and energy storage charge state, etc., is set; the action space of the microgrid is set, which includes the action of energy storage and the internal electricity price segment of the microgrid taking into account demand response; and the reward function of the model is set.

[0025] According to the response model and the state space, an adversarial network structure is used to construct a dominant neural network model; to solve the overestimation problem in traditional Q-Learning and DQN, two networks with the same structure are used: a target network and an evaluation network. The evaluation network selects the action that maximizes the Q value, and the target network determines the value of the action; Deep reinforcement learning is combined with the WHO (Wild Horse Optimizer) algorithm, that is, deep reinforcement learning is combined with the WHO wild horse optimization algorithm, the WHO algorithm is used to replace the traditional optimizer, and the neural network weights of the dominant neural network model are determined through algorithm iteration and optimization. The energy storage regulation decision is determined according to the dominant neural network model and the microgrid status at each moment.

[0026] This application scheme takes into account the uncertainty of wind and solar power generation and demand response, and makes autonomous decisions based on the real-time status and data of the microgrid through the energy management system intelligent body, optimizes the energy storage charging and discharging strategy, changes the electricity price in the microgrid to indirectly regulate the price response load, and optimizes the weight parameter optimization process of the intelligent body neural network through the WHO algorithm, which is conducive to fully absorbing distributed renewable energy such as wind and solar power within the microgrid and maximizing the benefits of the microgrid. It can improve the ability of energy storage regulation to cope with randomness, the regulation effect is stable, and the overall efficiency of the microgrid can be improved.

[0027] In another embodiment provided by the present invention, the step S1 specifically includes the following steps: See also Figure 2 , is a schematic diagram of the structure of a microgrid provided by an embodiment of the present invention. The microgrid structure includes a microgrid energy management system for energy management of distributed renewable energy, external power grid and energy storage.

[0028] Through the information flow with distributed renewable energy, external power supply and energy storage, the control information flow of energy storage regulation is generated, thereby realizing the control of energy flow. When adjusting the energy storage, the indirect control of household price response load is taken into account to maximize the benefits of the microgrid.

[0029] Based on the wind speed and light intensity data of each historical period, a wind and solar power generation uncertainty model is constructed in which the wind speed and light intensity prediction errors satisfy the normal distribution; See also Figure 3 , is a wind power output curve diagram provided by an embodiment of the present invention. Figure 4 , is a photovoltaic output curve diagram provided by an embodiment of the present invention.

[0030] By adjusting the electricity price within the microgrid within a certain range near the market electricity price, the household price-responsive load is indirectly controlled so that it can transfer electricity consumption during peak load periods to low load periods to a certain extent. A response model of the household price-responsive load is constructed based on the preset electricity costs during peak and low load periods.

[0031] The energy management agent can only observe the predicted wind power and photovoltaic power generation data when making decisions; and at the settlement time, the microgrid's income must be settled according to the actual power generation of wind and solar power. The wind and solar power abandonment and load loss caused by decision-making problems will be included in a certain penalty cost. This requires the agent to take the uncertainty of wind and solar power into account when making decisions, thereby improving the ability of the energy management system agent to deal with new energy prediction errors.

[0032] In another embodiment of the present invention, the wind-solar power generation uncertainty model including wind speed and light intensity prediction includes: ; ; in, are the actual values ​​of wind power generation and photovoltaic power generation at time t respectively; They are the wind power forecast value and photovoltaic forecast value at time t respectively; are random variables that obey Gaussian distribution, representing the wind power prediction error and photovoltaic prediction error at time t respectively; Assuming that the household price-responsive load within the microgrid has a variable component affected by price, the household price-responsive load can be indirectly controlled by adjusting the electricity price within the microgrid within a certain range near the market electricity price, so that it can transfer electricity consumption during peak load periods to low load periods to a certain extent, thereby reducing the peak-to-valley difference of the microgrid load and bringing greater benefits. The price-responsive load has two characteristic parameters, namely and ;in is the price sensitivity factor of household price response load i, and , which represents the percentage of load that can be reduced or increased if the price increases or decreases; is the patience parameter of household price response load i, indicating the time limit within which the transferred load should be repaid.

[0033] The demand-side load model for household price-responsive loads includes: ; Among them, L i t is the household price response load at time t, is the basic load at time t; is the shifted load of household price response load i at time t; is the load of household price response load i transferred from time before time t to time t.

[0034] ; ; in, is the price level at time t, when The amount of load transfer of household price response load i is positive, because when the microgrid is priced higher, in order to reduce costs, it will try to consume this part of the load when the electricity price is lower later; on the contrary, when hour is negative; A 0-1 decision variable indicating whether the load transferred from time period j should be consumed at the current time. Its value depends on a random variable that depends on the current price level. and the time elapsed from time step j, which is the distance from the patience parameter The closer, the load is executed at the current moment The greater the probability. The random variable is calculated as ; in, .

[0035] In another embodiment provided by the present invention, the step S2 specifically includes: The state space of the microgrid in each time period should contain the microgrid state information required for making action decisions and the external grid information that can be obtained, specifically: the predicted wind speed and light intensity values ​​at that moment, market electricity price information, load information, charge state of each energy storage, climbing rate and maximum charge and discharge power.

[0036] Therefore, the state information of the microgrid and the external grid information are obtained to construct the state space of the microgrid in each time period.

[0037] The action space includes two parts: energy storage action and microgrid internal pricing action. Among them, the action of energy storage needs to be determined according to the size of net load (wind and solar power generation power minus load power) and real-time electricity price. Therefore, its action space is set to two dimensions, namely, the action when there is a surplus of electricity (net load greater than 0) and the action when there is a surplus of electricity (net load greater than 0). Action when power shortage (net load is less than 0) , and both are 0-1 variables, used for the priority of energy storage and power grid. If the net load level is greater than 0, then if there is a surplus of electricity, If it is 1, the excess power will be used to charge the energy storage first. If there is any power left after the energy storage is fully charged, the remaining power will be sold to the grid. If the net load level is less than 0, the excess electricity will be sold directly to the grid. If it is 1, the gap between distributed energy and load will be made up by energy storage discharge first, and the insufficient part will be made up by purchasing electricity from the grid; if If it is 0, the net load shortfall will be met directly through purchasing electricity from the grid.

[0038] For the microgrid internal pricing action part in the action space, considering that some residential loads in the microgrid have variable components affected by electricity prices, according to the current load and renewable energy generation, considering demand response, the price response load is indirectly controlled by adjusting the electricity price inside the microgrid. The pricing operation consists of five possible actions, representing different electricity price segments, expressed as According to the current load and renewable energy generation data, combined with demand response, the price response load is indirectly controlled by adjusting the electricity price within the microgrid, setting pricing operation actions, and determining the energy management action space; In this microgrid optimization scheduling model, a reward function is constructed with the goal of maximizing the overall benefit of the microgrid.

[0039] In another embodiment provided by the present invention, the state space includes ; Among them, s t is the state at time t, is the state of charge of energy storage i at time t; is the market electricity price at time t, The price at which the microgrid sells electricity to the grid; is the wind power forecast value at time t; is the predicted value of photovoltaic power generation at time t; is the basic load at time t, and t is the time information.

[0040] The energy management action space is ; Among them, A represents the action space, For power surplus action, For power shortage action, To perform pricing operations; The reward function is ; Among them, R t is the reward at time t, L i t is the household price response load at time t, and are the electricity sold and purchased from the external power grid; The electricity price set for household price-responsive loads within the microgrid.

[0041] In another embodiment provided by the present invention, the step S3 specifically includes the following steps: To solve the overestimation problem in traditional Q-Learning and DQN, two networks with the same structure are used: the target network and the evaluation network. The evaluation network selects the action that maximizes the Q value, and the target network determines the value of the action. That is, the evaluation network for selecting the action and the target network for determining the value of the action are constructed according to the response model and the state space.

[0042] Therefore, the target value is calculated as: ; in, is the target reward value at time t; is the reward value at time t; is the state-action-value function, fitted by a neural network, represents the parameters of the neural network; are the parameters of the evaluation network and the target network respectively; , Respectively represent the state output by the evaluation network and the target network Take action The Q value obtained. This formula indicates that the strategy that can maximize the Q value is selected by evaluating the network. , and is evaluated by the target network This strategy is adopted in the state Q value.

[0043] Both the target network and the evaluation network adopt the structure of adversarial network. Different from the previous direct output of Q value, the output is split into the sum of state value and action advantage. According to the evaluation network and the target network, the output of the reward function is split into the sum of state value and action advantage, that is: in, is the state-action-value function; is the state value function, which is used to evaluate the value of being in a specific state; is the action advantage function, which is used to evaluate the relative importance of each action in a specific state; Its network parameters; Indicates the number of optional actions. represents the average advantage of all optional actions. Subtracting the average advantage of all optional actions is to decentralized the action advantage function.

[0044] During training, noise is added to the parameters of the neural network of the action advantage function to obtain an advantage neural network model based on an adversarial network and taking noise into consideration.

[0045] In order to optimize the exploration of the optimal strategy during the training process and the robustness of the training, noise is added to the parameters of the neural network during training. Therefore, the expression of the fully connected layer of the neural network considering noise is: ; in, are parameters that need to be learned from experience; the matrix and vector Each element of is independently drawn from a normal distribution The extracted one represents the noise.

[0046] The neural network with noise added can be expressed as: ; in, is a parameter, From the normal distribution The noise extracted from .

[0047] Noise has a certain degree of randomness. After training, noise is no longer needed when making decisions. Set to 0, only keep the parameters Therefore, the neural network with noise added and considering the adversarial network structure can be expressed as: It should be noted that this application scheme also sets a priority experience playback mechanism, specifically: Each time the agent interacts with the environment, the current state, the action taken, the reward obtained, and the next state transferred to after taking the action, that is, , all this information will be packed and stored in the memory. Set the memory capacity to Memory_size. When the existing experience data in the memory reaches a certain amount, each neural network training will randomly sample batch_size experience data from the memory for training, which can break the correlation between samples and improve the efficiency of sample use.

[0048] However, not all experience bar samples are equally important. Prioritizing learning samples with large errors and large room for improvement can effectively improve training efficiency. Introducing TD error as an indicator for evaluating the priority of experience bars: ; in, is the Q value generated by the target network, To evaluate the Q value generated by the network, the experience bar with the greater difference between the two needs to be learned first, and the priority of the experience bar is evaluated accordingly. The intelligent agent is allowed to continuously interact with the environment and learn from past experience, update and iterate, and finally learn a neural network that can replace the most valuable function, thereby comprehensively considering demand response and forming a set of energy storage coordination strategies that can maximize the profit of the microgrid.

[0049] In another embodiment provided by the present invention, the step S4 specifically includes the following steps: The essence of reinforcement learning training is to find the optimal state value function, while deep reinforcement learning fits the state value function through neural networks, constantly interacts with the environment, and trial and error to find a set of optimal neural network weight parameters. In order to accelerate and optimize the process of deep reinforcement learning neural networks finding the optimal network weights, the Mustang optimization algorithm WHO is used to solve this process.

[0050] The weights of the evaluation network and the target network are defined as a horse in the wild horse optimization algorithm. The position of the horse represents the size of the neural network weight parameter. The percentage of stallions in the population is set to , and based on this, the group is initially divided into population, the number of stallions is G, and the number of other horses is The wild horse optimization algorithm simulates the optimization process as the behaviors of a wild horse population: Herding behavior: The stallion leads the entire herd to move, and the other horses graze around the stallion to search for the optimal solution. The position update formula for each horse is: In the formula, is the current position of the remaining individuals, Updated positions for the remaining individuals; For the stallion's current position, for A random number to ensure that the remaining individuals revolve around the population Z is an adaptive parameter used to adjust the distance between general individuals and stallion individuals at each position update.

[0051] Hybridization behavior: When a population's foals mature, they will leave the population and mate with foals from other populations to produce new individuals. The location information of the hybrid individuals is updated as follows: ; in, The position of the new individual in the group after mating; is the position information of the two individuals i and j in the parent generation.

[0052] The behavior of the population moving to find a water hole: In order to find a better habitat (a better solution), the stallion will lead the population to move, and the location information of the stallion will replace the location information of the population led by the stallion. The position update method of the stallion individual is: ; In the formula, is the location of the habitat puddle, which is also the global optimal solution that needs to be found in the algorithm; For population The stallion's current location; For population The updated position of the stallion; for The random number determines the clockwise or counterclockwise direction of the population's advance; Z is the adaptive parameter for adjusting the position update distance mentioned above; for A random number.

[0053] Selection behavior of stallions within the population: Initially, stallions are randomly selected. As the algorithm progresses, stallion individuals are selected based on the fitness information of the individuals in the population. The selection mechanism is: ; In the formula, for the location of group members; For individuals The fitness function is the loss function corresponding to the individual i in this group of parameters. ,Right now: in, , They are and The reward discount factor for the step; They are The status of the step; are the parameters of the i-th group of target network and evaluation network respectively; yes Step return; are the action strategies taken in the first step; are the parameters of the target network, is to evaluate the network parameters.

[0054] The specific steps for updating the evaluation network and target network parameters based on WHO are as follows: Before each training, the WHO algorithm is initialized and the initial parameters of the algorithm are set, including the population size N, the proportion of stallions , population hybridization rate , the maximum number of iterations of the algorithm ; As a preferred solution, see Table 1, which is an example of a set of initial parameters provided.

[0055] Table 1 Initial parameter table The population is randomly divided and initialized, the position information of each individual in the population is randomly initialized, and the corresponding fitness value is calculated .

[0056] The optimization algorithm performs grazing, hybridization, population movement to find water holes, and individual stallion selection behaviors according to the behavior of the wild horse population; after each behavior, the position information of the population individual is substituted into the fitness function (the above loss function) for fitness calculation, and the new fitness is returned to the population individual; Repeat the above steps continuously. When the fitness function value of the global optimal solution is less than Or the number of iterations reaches the maximum number of iterations When , stop searching for the optimal solution.

[0057] Outputting the current individual optimal solution as the neural network weight of the dominant neural network model; The current energy storage regulation decision is determined based on the determined dominant neural network model and the microgrid status at each moment.

[0058] This application scheme is based on the deep reinforcement learning Rainbow+WHO algorithm. It uses a data-driven approach and takes into account the uncertainty of wind and solar power generation and demand response. The energy management system intelligent body makes autonomous decisions based on the real-time status and data of the microgrid, optimizes the energy storage charging and discharging strategy, changes the electricity price within the microgrid to indirectly regulate the price response load, and optimizes the weight parameter optimization process of the intelligent body neural network through the WHO algorithm, which is conducive to fully absorbing distributed renewable energy such as wind and solar power within the microgrid and maximizing the benefits of the microgrid.

[0059] The embodiment of the present invention also provides a distributed energy storage regulation system for a microgrid based on deep reinforcement learning, see Figure 5 , is a schematic diagram of the structure of a distributed energy storage regulation system for a microgrid based on deep reinforcement learning provided by an embodiment of the present invention, the system comprising: A response module is used to establish a response model of each component within the microgrid based on the acquired microgrid data and taking into account the demand response; A state module, used to determine the state space, action space and reward function of the microgrid according to the state setting of the microgrid and the energy storage action decision; A reward module, configured to construct a dominant neural network model using an adversarial network structure according to the response model, the state space, the action space, and the reward function; The decision-making module is used to combine deep reinforcement learning with the WHO algorithm, determine the neural network weights of the dominant neural network model through algorithm iteration and optimization, and make energy storage adjustment decisions based on the dominant neural network model and the microgrid status at each moment.

[0060] It should be noted that the distributed energy storage regulation system for a microgrid based on deep reinforcement learning provided in an embodiment of the present invention can execute the distributed energy storage regulation method for a microgrid based on deep reinforcement learning described in any of the above embodiments, and the specific functions of the distributed energy storage regulation system for a microgrid based on deep reinforcement learning will not be described here.

[0061] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A distributed energy storage regulation method for a microgrid based on deep reinforcement learning, characterized in that: The method comprises: Establish a response model of each component within the microgrid based on the acquired microgrid data and considering demand response; Setting the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision; According to the response model, the state space, the action space and the reward function, a dominant neural network model is constructed using an adversarial network structure; Deep reinforcement learning is combined with the WHO algorithm, and the neural network weights of the dominant neural network model are determined through algorithm iterative optimization. The current energy storage adjustment decision is determined based on the obtained dominant neural network model and the state of the microgrid at each moment.

2. The distributed energy storage regulation method of a microgrid based on deep reinforcement learning as claimed in claim 1, characterized in that: Based on the acquired microgrid data and considering the demand response, the response model of each component within the microgrid is established, including: Based on the historical wind speed data and light intensity data obtained in each period, a wind and solar power generation uncertainty model including wind speed and light intensity prediction is constructed; A demand response model for household price-responsive loads is constructed based on the preset electricity costs during peak and off-peak hours.

3. The distributed energy storage regulation method of a microgrid based on deep reinforcement learning as claimed in claim 2, characterized in that: The wind and solar power generation uncertainty model includes , ; The demand response model of household price responsive load includes ; in, are the actual values ​​of wind power generation and photovoltaic power generation at time t respectively; They are the wind power forecast value and photovoltaic forecast value at time t respectively; are random variables that obey Gaussian distribution, representing the wind power prediction error and photovoltaic prediction error at time t respectively; L i t is the household price response load at time t, is the basic load at time t; is the shifted load of household price response load i at time t; is the load of household price response load i transferred from time before time t to time t.

4. The distributed energy storage regulation method of a microgrid based on deep reinforcement learning as claimed in claim 1, characterized in that: The state space, action space and reward function of the microgrid are set according to the state of the microgrid and the energy storage action decision, including: Obtain the status information of the internal components of the microgrid and the external grid information, and construct the state space of the microgrid in each period; According to the charge state of the microgrid energy storage and the real-time electricity price, determine the actions when the energy storage power is surplus and the actions when the energy storage power is insufficient, and build the energy storage action space; According to the current load and renewable energy generation data, combined with demand response, the price response load is indirectly controlled by adjusting the electricity price within the microgrid, setting pricing operation actions and determining the energy management action space; The reward function is constructed with the goal of maximizing the energy benefit of the microgrid.

5. The distributed energy storage regulation method of a microgrid based on deep reinforcement learning as claimed in claim 4, characterized in that: The state space includes ; The energy management action space is ; The reward function is ; Among them, s t is the state space of the microgrid at time t, is the state of charge of energy storage i at time t; is the market electricity price at time t, The price at which the microgrid sells electricity to the grid; is the wind power forecast value at time t; is the predicted value of photovoltaic power generation at time t; is the basic load at time t, t is the time information; a t is the action at time t, A represents the action space, For energy storage power surplus action, A is the energy storage power shortage action. P is the pricing operation; R t is the reward at time t, L i t is the household price response load at time t, and The electricity sold to and purchased from the external grid respectively; The electricity price set for household price-responsive loads within the microgrid.

6. The distributed energy storage regulation method of a microgrid based on deep reinforcement learning as claimed in claim 1, characterized in that: According to the response model, the state space, the action space and the reward function, a dominant neural network model is constructed using an adversarial network structure, including: Constructing an evaluation network for evaluating action value and a target network for making the network training process more stable according to the response model, the state space and the reward function; According to the evaluation network and the target network, the action value output by the adversarial network is split into the sum of the state value and the action advantage, and a decentralized action advantage function is constructed; During training, noise is added to the parameters of the neural network of the action advantage function to obtain an advantage neural network model based on an adversarial network and taking noise into consideration.

7. The distributed energy storage regulation method of a microgrid based on deep reinforcement learning as claimed in claim 1, characterized in that: The deep reinforcement learning is combined with the WHO algorithm, and the neural network weight of the dominant neural network model is determined through algorithm iteration and optimization. The current energy storage regulation decision is determined according to the obtained dominant neural network model and the microgrid state at each moment, including: Initialize the WHO algorithm and set the initial parameters of the algorithm, including the population size N, the proportion of stallion individuals, the population hybridization rate, and the maximum number of iterations of the algorithm; The population is randomly divided into groups and initialized, the position information of each individual in the population is randomly initialized, and the fitness of each individual is calculated using a preset fitness function; When the fitness of the individual of the global optimal solution is not less than the preset threshold, the position information of the individual of the population is updated according to the behavior of the wild horse population, and the fitness is recalculated after the position information is updated to obtain the updated fitness; When the fitness of the individual of the global optimal solution is less than the threshold, or the number of iterations reaches the maximum number of iterations of the algorithm, output the current individual optimal solution as the neural network weight of the dominant neural network model; The current energy storage regulation decision is determined based on the determined dominant neural network model and the microgrid status at each moment.

8. A distributed energy storage regulation system for a microgrid based on deep reinforcement learning, characterized in that: The system comprises: A response module is used to establish a response model of each component within the microgrid based on the acquired microgrid data and taking into account the demand response; A state module, used to set the state space, action space and reward function of the microgrid according to the state of the microgrid and the energy storage action decision; A reward module, configured to construct a dominant neural network model using an adversarial network structure according to the response model, the state space, the action space, and the reward function; The decision-making module is used to combine deep reinforcement learning with the WHO algorithm, determine the neural network weights of the dominant neural network model through algorithm iteration and optimization, and determine the current energy storage regulation decision based on the obtained dominant neural network model and the microgrid status at each moment.

9. The distributed energy storage regulation system of a microgrid based on deep reinforcement learning as claimed in claim 8, characterized in that: The state module is specifically used for: Obtain the status information of the internal components of the microgrid and the external grid information, and construct the state space of the microgrid in each period; According to the charge state of the microgrid energy storage and the real-time electricity price, determine the actions when the energy storage power is surplus and the actions when the energy storage power is insufficient, and build the action space of the energy storage; According to the current load and renewable energy generation data, combined with demand response, the price response load is indirectly controlled by adjusting the electricity price within the microgrid, setting pricing operation actions and determining the energy management action space; The reward function is constructed with the goal of maximizing the energy benefit of the microgrid.

10. The distributed energy storage regulation system of a microgrid based on deep reinforcement learning as claimed in claim 8, characterized in that: The decision module is specifically used for: Initialize the WHO algorithm and set the initial parameters of the algorithm, including the population size N, the proportion of stallion individuals, the population hybridization rate, and the maximum number of iterations of the algorithm; The population is randomly divided into groups and initialized, the position information of each individual in the population is randomly initialized, and the fitness of each individual is calculated using a preset fitness function; When the fitness of the individual of the global optimal solution is not less than the preset threshold, the position information of the individual of the population is updated according to the behavior of the wild horse population, and the fitness is recalculated after the position information is updated to obtain the updated fitness; When the fitness of the individual of the global optimal solution is less than the threshold, or the number of iterations reaches the maximum number of iterations of the algorithm, output the current individual optimal solution as the neural network weight of the dominant neural network model; The current energy storage regulation decision is determined based on the determined dominant neural network model and the microgrid status at each moment.

Images