Regional building group source network load storage and utilization system energy management method based on GRU and A3C

Abstract

The invention relates to the field of electric power, in particular to a regional building group source network load storage and utilization system energy management method based on GRU and A3C, and the method comprises the following steps: S1, obtaining the electric power information data of a regional building group, a plurality of variables influencing the nitrogen electric power information data, and parameter values of the variables; s2, performing preprocessing according to a plurality of parameter values of the electric power information data to obtain preprocessed data; s3, constructing a GRU neural network according to the preprocessed data; s4, an Actor-Critic algorithm is introduced into the GRU neural network, and a GRU-A3C neural network is obtained; s5, constructing a training set according to the preprocessed data, and training the GRU-A3C neural network; and S6, parameters of the GRU-A3C neural network are optimized according to an optimization algorithm, and a maximum expected cumulative reward value is obtained. The method has the advantages that an effective energy optimization management scheme is designed and realized based on the GRU and the deep reinforcement learning algorithm A3C, convergence can be faster, reward value accumulation is carried out, and cost reduction of energy optimization management is realized.

Description

[Technical field]

[0001] The present invention relates to the field of electric power, and in particular to an energy management method for a regional building group source-grid-load-storage-use system based on GRU and A3C. [Background technology]

[0002] With the improvement of economic level and the continuous high temperature in recent years, the power load has continued to rise. Power load forecasting is one of the core links of demand response. Accurate power load forecasting is of great significance to improving the stability of the power system. Considering the uncertainty brought by demand response plans, reactive loads and renewable energy, and in order to further achieve effective smoothing of seasonal peak-valley differences, enable the power grid to respond more actively to various emergencies, and make energy conservation and consumption reduction work enter a virtuous circle, building a reasonable power load forecasting model for different regions has become a task that power workers need to solve urgently.

[0003] By considering the demand response of thermal load, the optimization model of energy management has been improved. For example, a two-stage microgrid optimization scheduling method predicts photovoltaic power generation in the first stage and obtains prediction data in the form of probability density function; in the second stage, a random optimization method is proposed to minimize the operating cost and load transfer of the isolated microgrid. However, with the increasing number of factors affecting the rapid development of society, new demands have been put forward for energy management models, requiring operators to meet the higher electricity load demands of the masses while saving investment costs.

[0004] Therefore, a regional building complex source-grid-load-storage system energy management method based on GRU and A3C is proposed. An effective energy optimization management scheme is designed based on the gated recurrent unit GRU and the deep reinforcement learning algorithm A3C. The optimization target is designed, which can converge faster during the offline training of the equipment, accumulate reward values, and reduce the cost of energy optimization management. [Summary of the invention]

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an energy management method for a regional building complex source-grid-load-storage system based on GRU and A3C. An effective energy optimization management solution is designed based on the gated recurrent unit GRU and the deep reinforcement learning algorithm A3C. The optimization target is designed, and it can converge faster during the offline training of the equipment, accumulate reward values, and reduce the cost of energy optimization management.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C, comprising the following steps:

[0008] S1. Obtaining power information data of regional buildings, several variables affecting power information data and their parameter values;

[0009] S2. Preprocessing is performed according to several parameter values ​​of the power information data to obtain preprocessed data;

[0010] S3. Construct a GRU neural network based on the preprocessed data;

[0011] S4. Introduce the Actor-Critic algorithm into the GRU neural network to obtain the GRU-A3C neural network;

[0012] S5. Construct a training set based on the preprocessed data and train the GRU-A3C neural network;

[0013] S6. Optimize the parameters of the GRU-A3C neural network according to the optimization algorithm to obtain the maximum expected cumulative reward value.

[0014] Preferably, S3 specifically includes introducing a reset gate and an update gate into the GRU neural network to perform calculations to select timing information.

[0015] As a preferred method, the GRU neural network calculation formula is as follows:

[0016] R t =σ(X t W xr +H t-1 W hr +b r )

[0017] Z t =σ(X t W xz +H t-1 W hz +b z )

[0018] Among them, t is a given time, X t is the input unit, and X t ∈i n*d , n is the number of samples, d is the number of inputs, H t-1 is a hidden unit, and H t-1 ∈i n*h , h is the number of hidden units, R t To reset the gate, Z t is the update gate, W xr is a weight function, and W xz ∈i d*h , b z is the impact factor.

[0019] Preferably, S4 specifically includes:

[0020] By introducing advantage functions and asynchronous updates, the Actor network is used to output actions according to different environmental states, and the Critic network is used to estimate the value of the actions selected for each interaction. The Actor network and the Critic network output the optimal learning parameters during the training process.

[0021] As a preferred option, the Actor network fitting strategy π(a|s; θ π ) completes the mapping of state s to action a, where θ π is the network parameter, and the expected cumulative reward R is maximized t : Among them, f is the objective function of reinforcement learning and E is the mathematical expectation.

[0022] Preferably, a value function is introduced to update the optimization function to reduce data errors.

[0023] As a preferred advantage function: A(a t |s t θ v )=R t -V(s t θ v ), where V(s;θ v ) is the value function.

[0024] Preferably, a Markov decision process is introduced to observe the state of the microgrid, adjust the operating state of the power controllable devices in the secondary grid and the amount of power traded between the secondary grid and the main grid.

[0025] As a preferred embodiment, the Actor network fitting strategy function can be expressed as:

[0026] π(a|s;θ π )=N(μ(s),σ(s))

[0027] Where N is the Gaussian distribution function, θ π is the network parameter, action a t Generated by random sampling of the policy function.

[0028] Preferably, the Actor network uses a four-layer network structure, and the Critic network uses a four-layer network structure.

[0029] The beneficial effects of the present invention are as follows:

[0030] 1. Based on the gated recurrent unit GRU and the deep reinforcement learning algorithm A3C, an effective energy optimization management solution is designed. First, when connected to the main power grid, photovoltaic power generation, wind power generation, micro gas turbines, energy storage equipment, and user loads in the microgrid are taken into account, and the corresponding mathematical model is established. Based on the cost function and boundary conditions in the mathematical model, the optimization target is designed;

[0031] 2. Then, the optimization problem is transformed into a Markov decision process, and the energy optimization management is solved as quickly as possible using the reinforcement learning method. The rewards are vectorized to help the neural network approach the optimal strategy.

[0032] 3. Finally, in order to solve the problem that the multi-dimensional action space makes it difficult for the network to converge to the optimal strategy, the GRVA3C algorithm is used and the reward function is decomposed and vectorized to correspond to each dimension of the action. Then, the neural network can make more accurate updates for each dimension of action to avoid unnecessary exploration of the action space, making it easier for the network to converge to the optimal strategy, thereby producing better optimization effects. It enables faster convergence during offline training, accumulates the reward values, calculates the highest cumulative reward value, and enables users to understand the data faster, achieving the lowest cost microgrid energy optimization management.

[0033] These features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings.

Brief Description of the Drawings

[0034] The present invention will be further described below in conjunction with the accompanying drawings:

[0035] Figure 1 This is an overall schematic diagram of the oil bottle cover of the first embodiment of the present invention being installed on the bottle body;

[0036] Figure 2 for Figure 1 Schematic diagram of the connecting components in FIG.

[0037] Figure 3 The diagram is an exploded view of the oil bottle cover in the first embodiment of the present invention. [Specific implementation method]

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] The following first describes the concepts involved in this application in conjunction with the accompanying drawings. It should be noted that the following description of each concept is only to make the content of this application easier to understand and does not limit the scope of protection of this application.

[0040] Embodiment 1:

[0041] A method for energy management of regional building cluster source-grid-load-storage system based on GRU and A3C. Figure 1 As shown, the following steps are included:

[0042] S1. Obtaining power information data of regional buildings, several variables affecting power information data and their parameter values;

[0043] S2. Preprocessing is performed according to several parameter values ​​of the power information data to obtain preprocessed data;

[0044] S3. Construct a GRU neural network based on the preprocessed data;

[0045] S3 specifically includes that the GRU neural network introduces a reset gate and an update gate to perform calculations to select timing information.

[0046] S4. Introduce the Actor-Critic algorithm into the GRU neural network to obtain the GRU-A3C neural network;

[0047] like Figure 2 As shown, the calculation formulas for the reset gate and update gate in the GRU neural network are as follows:

[0048] R t =σ(X t W xr +H t-1 W hr +b r )

[0049] Z t =σ(X t W xz +H t-1 W hz +b z )

[0050] Among them, t is a given time, X t is the input unit, and X t ∈i n*d , n is the number of samples, d is the number of inputs, H t-1 is a hidden unit, and H t-1 ∈i n*h , h is the number of hidden units, R t To reset the gate, Z t is the update gate, W xris the weight function, and W xz ∈i d*h , b z is the impact factor.

[0051] S4 specifically includes:

[0052] By introducing advantage functions and asynchronous updates, the Actor network is used to output actions according to different environmental states, and the Critic network is used to estimate the value of the actions selected for each interaction. The Actor network and the Critic network output the optimal learning parameters during the training process.

[0053] The Actor-Critic framework combines the advantages of value-based and policy-based algorithms. The Actor network is used to output actions based on different environmental states.

[0054] The Critic network is used to estimate the value of the action selected for each interaction. The two networks depend on each other during the training process and gradually learn the optimal parameters.

[0055] The asynchronous actor-critic algorithm (Asynchronous Advantage Actor-Critic, A3C) used in the present invention is an improved algorithm based on the Actor-Critic framework, and the improvement is mainly reflected in the introduction of the advantage function (Advantage Function) and the asynchronous update (Asynchronous Update) method.

[0056] (1) Asynchronous update:

[0057] Another improvement of A3C over the basic Actor-Critic algorithm is the introduction of the idea of ​​asynchronous updates. When training the network, multiple threads are started to interact with the environment separately. Each thread periodically updates the network parameters of the global network, and then pulls the latest network parameters from the global network. This can make full use of CPU resources, explore a large amount of action space in a short time, reduce the correlation between data, save training time, and be more conducive to network convergence.

[0058] S5. Construct a training set based on the preprocessed data and train the GRU-A3C neural network;

[0059] S6. Optimize the parameters of the GRU-A3C neural network according to the optimization algorithm to obtain the maximum expected cumulative reward value.

[0060] The design optimization objective function is a constrained optimization problem. The optimization goal is to minimize the total cost of system operation, and the constraints are the electrical boundary conditions of the equipment.

[0061] In order to evaluate the performance of the current network, the Critic network introduces a value function V π (s t ):S→R, specifically expressed as: V π (s t )=E π [R t |s t ], E π is the mathematical expectation, V π (s t ) Return status s t The long-term value of t is the current state, and only considers how much reward this state can get.

[0062] Actor network fitting strategy π(a|s;θ π ) completes the mapping of state s to action a, where θ π is the network parameter, and the expected cumulative reward R is maximized t :f(π)=Ε π [R0], calculate the cumulative reward Among them, f is the objective function of reinforcement learning, E is the mathematical expectation, and γ is the discount coefficient.

[0063] like Figure 3 As shown in the figure, in the tth round of energy optimization management, the microgrid state s t The input gated recurrent unit GRU extracts time series features and then feeds them into the A3C network. A3C includes an Actor network and a Critic network, both of which use a four-layer network structure. The first layer is the input layer, followed by two hidden layers, and finally the output layer. A3C uses the Actor network to fit the policy function in the MDP. MDP is an extension of the Markov chain and provides a mathematical framework for modeling decision scenarios, which can be expressed as:

[0064] π(a|s;θ π )=N(μ(s),σ(s));

[0065] Where N is the Gaussian distribution function, θ π is the network parameter, action subset a t Generated by random sampling of the strategy function, the action value function formula is a t =[P t MT ,P t MG ];

[0066] And according to the power balance constraint calculation Perform homeostatic correction;

[0067] Then, the optimization problem is transformed into a Markov decision process so that it can be solved using deep reinforcement learning; then, the reward is vectorized and decomposed; finally, the overall process of the designed optimization management algorithm is given.

[0068] The preprocessed data is processed by GRU and A3C to construct a Markov decision process. The agent observes the state S of the microgrid. i ∈S, select an action a according to the action selection strategy π:S→A t ∈A is used to adjust the operating state of the power controllable devices in the microgrid and the amount of power traded with the main grid. A is the action set, and then transfers to the next state s t+1 ∈S, and obtain the corresponding immediate reward r t , the discount factor γ controls the importance of immediate rewards and future rewards, and vectorizes the rewards obtained to obtain r t =[r t MT ,r t MG ];

[0069] The purpose of reward vectorization is to allow the agent to optimize each dimension of action separately, reduce unnecessary exploration of action space and produce a more stable control effect. The reward function is vectorized and the reward is decomposed into vectors corresponding to the action.

[0070] The design based on gated recurrent unit GRU and deep reinforcement learning algorithm A3C effectively implements an effective energy optimization management solution.

[0071] Firstly, when connected to the main grid, the photovoltaic power generation, wind power generation, micro gas turbine, energy storage equipment and user load in the microgrid are taken into account, and the corresponding mathematical model is established. The optimization objective is designed according to the cost function and boundary conditions in the mathematical model.

[0072] Then, the optimization problem was transformed into a Markov decision process, and the energy optimization management was solved as quickly as possible using reinforcement learning methods. The rewards were vectorized to help the neural network approach the optimal strategy.

[0073] Finally, in order to solve the problem that the multi-dimensional action space makes it difficult for the network to converge to the optimal strategy, the GRVA3C algorithm is used and the reward function is decomposed and vectorized to correspond to each dimension of the action. Then, the neural network can make more accurate updates for each dimension of action to avoid unnecessary exploration of the action space, making it easier for the network to converge to the optimal strategy, thereby producing better optimization effects. It enables faster convergence during offline training, accumulates the reward values, calculates the highest cumulative reward value, and allows users to observe and understand more quickly, ultimately achieving microgrid energy optimization management at the lowest cost.

[0074] (2) Advantage function:

[0075] A(a t |s t θ v )=R t -V(s t θ v );

[0076] Among them, s t is the state of the microgrid. In the Actor-Critic algorithm, the Actor network fits the strategy π(a|s; θ π ) completes the mapping of state s to action a, (where θ π is the network parameter), the goal of reinforcement learning is to enable the network to learn the optimal strategy π to maximize the expected cumulative reward R t :f(π)=Ε π [R0], where f is the objective function of reinforcement learning and E is the mathematical expectation.

[0077] Those of ordinary skill in the art will appreciate that the units (or modules, steps, etc., the same below) of the examples described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition of each example has been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0078] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored, etc.

[0079] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0080] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-0nly Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc., which can store program code.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for energy management of a regional building cluster source-grid-load-storage system based on GRU and A3C, characterized in that: The following steps are involved: S1. Obtaining power information data of regional buildings, several variables affecting power information data and their parameter values; S2. Preprocessing is performed according to the several parameter values ​​of the power information data to obtain preprocessed data; S3. constructing a GRU neural network based on the preprocessed data; S4. Introduce the Actor-Critic algorithm into the GRU neural network to obtain a GRU-A3C neural network; S5. construct a training set based on the preprocessed data and train the GRU-A3C neural network; S6. Optimize the parameters of the GRU-A3C neural network according to the optimization algorithm to obtain the maximized expected cumulative reward value.

2. A method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 1, characterized in that: The S3 specifically includes that the GRU neural network introduces a reset gate and an update gate to perform calculations to select timing information.

3. The energy management method of the regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 1 is characterized in that: The GRU neural network calculation formula is as follows: R t =σ(X t W xr +H t-1 W hr +b r ) Z t =σ(X t W xz +H t-1 W hz +b z ) Among them, t is a given time, X t is the input unit, and X t ∈i n*d , n is the number of samples, d is the number of inputs, H t-1 is a hidden unit, and H t-1 ∈i n*h , h is the number of hidden units, R t To reset the gate, Z t is the update gate, W xr is a weight function, and W xz ∈i d*h , b z is the impact factor.

4. The method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 1, characterized in that: The S4 specifically includes: By introducing advantage functions and asynchronous updates, the Actor network is used to output actions according to different environmental states, and the Critic network is used to estimate the value of the action selected for each interaction. The Actor network and the Critic network output optimal learning parameters during the training process.

5. The method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 4, characterized in that: The Actor network fitting strategy π(a|s; θ π ) completes the mapping of state s to action a, where θ π is the network parameter, and the expected cumulative reward R is maximized t : Among them, f is the objective function of reinforcement learning and E is the mathematical expectation.

6. A method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 4, characterized in that: The value function is introduced to update the optimization function to reduce data errors.

7. The method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 4, characterized in that: The advantage function is: A(a t |s t θ v )=R t -V(s t θ v ), where V(s;θ v ) is the value function.

8. The method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 4, characterized in that: The Markov decision process is introduced to observe the state of the microgrid, adjust the operating state of the power controllable devices in the subgrid and the amount of power traded between the subgrid and the main grid.

9. The method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 5, characterized in that: The Actor network fitting strategy function can be expressed as: π(a|s;θ π )=N(μ(s),σ(s)) Where N is the Gaussian distribution function, θ π is the network parameter, action a t Generated by random sampling of the policy function.

10. The method for managing energy of a regional building group source-grid-load-storage system based on GRU and A3C as claimed in claim 4, characterized in that: The Actor network uses a four-layer network structure, and the Critic network uses a four-layer network structure.

Images