Energy distribution methods and systems for regional energy systems

Abstract

This disclosure relates to an energy allocation method and system for a regional energy system. The regional energy system includes a main grid and multiple distributed microgrids connected to the main grid. The energy allocation method includes: predicting a first non-decision variable for each microgrid during a predetermined period based on a first learning network. The first non-decision variable includes load parameters, environmental energy generation-related parameters, the unit cost of introducing energy from the main grid, and the unit cost of supplying energy to the main grid; predicting a second non-decision variable during the predetermined period based on a second learning network. The second non-decision variable includes a first expenditure parameter for energy allocation between the microgrid and other microgrids and a credit parameter after performing energy allocation during the predetermined period; modeling the decision-making process of the decision variables of each microgrid as a decision-making process based on multi-agent Q-learning, with each microgrid correspondingly modeled as an agent, and using the multi-agent Q-learning-based decision-making process to optimize the decision variables of each microgrid.

Description

Technical Field

[0001] This disclosure pertains to the field of energy internet system operation, and specifically relates to an energy distribution method and system for a regional energy system. Background Technology

[0002] In existing energy internet systems, with the integration of new energy forms such as photovoltaics and wind power, the scale of the energy internet and regional energy systems, represented by park energy systems, is continuously expanding, and their system models are also evolving towards high-dimensional, nonlinear, and complex networks. The increased complexity of high-dimensional problems and networks poses significant challenges to traditional modeling and solving methods, especially centralized optimization methods. The communication and computational burdens of centralized solutions, along with their inherent single points of error and high maintenance costs, make them unsuitable for situations with highly complex network models.

[0003] On the other hand, with the development of technologies such as blockchain, more and more energy internet systems have introduced credit mechanisms, and there is a coupling relationship between credit mechanisms and energy allocation. Existing energy allocation methods cannot accurately model credit mechanisms, such as those based on blockchain technology. Therefore, credit mechanisms cannot be accurately taken into account when allocating energy. Summary of the Invention

[0004] This disclosure is provided to address the aforementioned problems existing in the prior art.

[0005] There is a need for an energy allocation method that comprehensively considers the cost and credit score of energy allocation in a regional energy system, while addressing the problems of high computational load and difficulty in solving centralized energy allocation methods. This disclosure provides an energy allocation method for regional energy systems that combines machine learning networks with distributed multi-agent cooperative Q-learning. Using a composite optimization index that considers the cost and credit score of energy allocation as the optimization objective, the microgrid in the regional energy system is modeled as a multi-agent system. Multiple machine learning networks that have learned the intrinsic mechanism of energy allocation are used to predict relevant data of the system. Decentralized distributed cooperative multi-agent Q-learning is then used to optimize and solve the energy allocation scheme. This addresses the shortcomings of existing technologies that do not consider the role of credit score in energy allocation, as well as the problems of single point of error, high maintenance costs, high computational load, and difficulty in solving problems caused by centralized energy allocation methods.

[0006] According to a first aspect of this disclosure, an energy allocation method for a regional energy system is provided. The regional energy system includes a main grid and multiple distributed microgrids connected to the main grid. The energy allocation method includes: predicting a first non-decision variable for each microgrid during a predetermined time period based on a first learning network. The first non-decision variable includes load parameters, environmental energy generation-related parameters, the unit cost of introducing energy from the main grid, and the unit cost of supplying energy to the main grid; predicting a second non-decision variable for the microgrid during the predetermined time period based on a second learning network. The second non-decision variable includes a first expenditure parameter for energy allocation between the microgrid and other microgrids and a credit parameter after performing energy allocation during the predetermined time period; and modeling the decision-making process of the decision variables of each microgrid as a decision-making process based on multi-agent Q-learning, wherein each microgrid is correspondingly modeled as an agent, and in the multi-agent Q-learning decision-making process, for Each agent: An action is defined as a decision variable to be optimized for the predetermined time period of the microgrid corresponding to that agent; a state is defined as the energy allocation state of the microgrid corresponding to that agent during the predetermined time period, wherein the energy allocation state includes the first non-decision variable and the second non-decision variable predicted for the microgrid corresponding to that agent; a reward function is determined based on the first non-decision variable predicted for the microgrid corresponding to that agent, the second non-decision variable predicted for the microgrid corresponding to that agent, and the second expenditure parameter of the microgrid corresponding to that agent relative to the main grid; and the value of the reward function is passed to another agent exchanging information with that agent; the value of the agent's collaborative reward function is determined based on the value of the agent's reward function and the values ​​of the reward functions passed to the agent by other agents; and for each microgrid, the decision variables of each microgrid are optimized using the decision process based on multi-agent Q-learning.

[0007] According to a second aspect of this disclosure, a regional energy system is provided, the regional energy system including a main power grid and multiple microgrids connected to and distributed with the main power grid, each microgrid using an energy distribution method according to various embodiments of this disclosure to achieve energy distribution between itself and other microgrids and the main power grid.

[0008] By utilizing the energy allocation method and system of the regional energy system according to various embodiments of the present disclosure, and taking into account both the cost and credit score factors of energy allocation in the regional energy system, the method combines machine learning networks with cooperative Q-learning based on distributed multi-agents to solve the problems of single point of error, high maintenance cost, large computational load and difficulty in solving existing centralized energy allocation methods, and provides a faster, less communication and computational burden and more accurate energy allocation method. Attached Figure Description

[0009] In drawings that are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. The drawings generally illustrate various embodiments by way of example rather than limitation and are used, together with the description and claims, to illustrate the disclosed embodiments. Where appropriate, the same reference numerals are used in all drawings to refer to the same or similar parts. Such embodiments are illustrative and not intended to be exhaustive or exclusive embodiments of the apparatus or method.

[0010] Figure 1 A schematic diagram showing the composition of a regional energy system according to an embodiment of the present disclosure is shown;

[0011] Figure 2 A flowchart illustrating an energy distribution method for a regional energy system according to an embodiment of the present disclosure is shown.

[0012] Figure 3 A schematic diagram of a first learning network according to an embodiment of the present disclosure is shown;

[0013] Figure 4 A schematic diagram of a second learning network according to an embodiment of the present disclosure is shown; and

[0014] Figure 5 A schematic diagram illustrating the optimization solution process of multi-agent Q-learning according to an embodiment of the present disclosure is shown. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, the disclosure will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and specific examples, but these are not intended to limit the scope of this disclosure.

[0016] Figure 1 A schematic diagram illustrating the composition of a regional energy system according to an embodiment of the present disclosure is shown. As an example, such as... Figure 1 The regional energy system 1 shown according to an embodiment of this disclosure includes an energy management center 10, a main power grid 11, microgrids 12, 13, and 14 as energy consumers, and microgrids 15 and 16 as energy prosumers. Microgrids 12, 13, and 14, as energy consumers, only have load devices that consume energy, while microgrids 15 and 16, as energy prosumers, in addition to load devices, also have environmentally friendly energy generation devices that generate energy and energy storage devices for storing the generated environmentally friendly energy. In some embodiments, the environmentally friendly energy generation devices may be rooftop photovoltaic power generation equipment, etc. As an example, Figure 1The diagram schematically illustrates three microgrids 12, 13, and 14 as consumers, and two microgrids 15 and 16 as prosumers. However, it should be understood that the number of microgrids of each type is not limited to this; in some embodiments, the regional energy system 1 may include only one or more microgrids acting as prosumers. Figure 1 In the diagram, solid lines represent energy flows, and dashed lines represent information transmission connections. In some embodiments, the main grid 11 is an energy network capable of exchanging energy with various microgrids in the regional energy system 1. More specifically, each microgrid 12, 13, and 14, as an energy consumer, can draw energy from the main grid 11, and each microgrid 15 and 16, as an energy producer-consumer, can both draw energy from the main grid 11 and supply its generated energy to the main grid 11. Figure 1 In the regional energy system 1 shown, the microgrids may or may not have information transmission connections depending on the physical implementation or logical settings. Two microgrids with information transmission connections are referred to as "neighboring" microgrids below. In some embodiments, the regional energy system 1 may include a residential park energy system, with each resident user in the residential park acting as a microgrid.

[0017] In embodiments according to this disclosure, as an example, the energy allocation, including energy introduction and supply, for each microgrid in the regional energy system 1 within an energy allocation cycle (e.g., one day, 24 hours) needs to be determined one day in advance. That is, the actual execution time of the energy allocation scheme matched by the energy management center 10 is the day after it is determined. Each energy allocation cycle has an energy allocation scheme submission stage and an energy allocation scheme matching stage. In the energy allocation scheme submission stage, each microgrid in the regional energy system 1 submits its desired energy allocation scheme. In the subsequent energy allocation scheme matching stage, the energy management center 10 matches the desired energy allocation schemes submitted by each microgrid. In other embodiments, when each microgrid submits its desired energy allocation scheme, the energy allocation cycle is divided into multiple time periods, such as including but not limited to dividing 24 hours into 1-hour time periods. The same energy allocation scheme exists within the same time period, while different time periods may have different energy allocation schemes.

[0018] In some embodiments, the energy allocation schemes submitted by each microgrid may include, within a predetermined time period (e.g., one hour in 24 hours), the energy that the microgrid expects to draw from other microgrids in the system and the unit cost of the energy expected to be supplied by other microgrids, the energy supplied to other microgrids, the unit cost of energy drawn from other microgrids, the energy drawn from the main grid 11, and the energy supplied to the main grid 11. Similarly, during the energy allocation scheme matching phase, the energy management center 10 may match the energy allocation within the predetermined time period according to the energy allocation schemes submitted by each microgrid and following certain built-in rules, generating matching values ​​corresponding to each expected value. In some embodiments, it is possible that the expected energy allocation schemes between the microgrids cannot be matched. For example, the first microgrid may want to draw energy E from the second microgrid during the period of 9:00-10:00, but the second microgrid can provide less than (2 / 3)E during that period. In this case, the energy management center 10 can match the unmatched portion (1 / 3)E of energy with the main grid 11, with the unit cost being a uniform unit cost of drawing energy from the main grid 11 and the unit cost of supplying energy to the main grid 11. It is worth noting that once the energy allocation scheme matching phase begins, the energy allocation schemes submitted by each microgrid cannot be modified and can only be submitted again during the energy allocation scheme submission phase of the next energy allocation cycle.

[0019] The following will describe it in detail. Figure 2 A flowchart illustrating an energy distribution method for a regional energy system according to an embodiment of the present disclosure is shown.

[0020] In step S201, the regional energy system according to this embodiment of the present disclosure can predict the first non-decision variables of each microgrid in a predetermined k-period based on a first learning network. In some embodiments, the first non-decision variables include load parameters, environmental energy generation-related parameters, the unit cost of introducing energy from the main grid 11, and the unit cost of supplying energy to the main grid 11. The load parameters may include, for example, the power of the load, and the environmental energy generation-related parameters may include, for example, the power generation capacity of the environmental energy. In some embodiments, the first non-decision variables may have different values ​​in each time period. Taking an energy allocation cycle of 24 hours and each time period fixed at 1 hour as an example, the variables of the first non-decision variables of the i-th microgrid in the predetermined k-period can be expressed as follows:

[0021] The power L of the load as a load parameter i (k);

[0022] The power generation capacity of environmentally friendly energy sources is a relevant parameter for environmentally friendly energy power generation.

[0023] The unit cost p of introducing energy from the main gridg (k);

[0024] The unit cost q of supplying energy to the main grid g (k).

[0025] Then, in step S202, the regional energy system according to embodiments of this disclosure can predict the second non-decision variables of each microgrid during a predetermined k-period based on a second learning network. In some embodiments, the second non-decision variable of microgrid i during the predetermined k-period may include a first expenditure parameter C for energy allocation between microgrid i and other microgrids. i (k) and the credit parameter R after the predetermined energy allocation for k time periods. i (k). In some embodiments, assuming there are n microgrids in the regional energy system of this embodiment, the credit parameter of microgrid i after performing energy allocation for a predetermined period of k is denoted as R. i (k). In some embodiments, the credit parameter is a credit score of each microgrid in the regional energy system based on a certain inherent credit mechanism. Specifically, R i (k) represents the credit score updated by the system for microgrid i after it has fulfilled the energy allocation plan matched for it by the system for k time periods. Generally, fulfilling the energy allocation plan will increase the credit score, and vice versa. In other embodiments, the microgrid's credit score will have a related impact on the energy allocation plan matched for it by the system. In some embodiments, the first expenditure parameter C for the predetermined k time periods... i (k) For example, it can be calculated from the following variables associated with microgrid i:

[0026]

[0027] The meanings of the variables in the above formula are as follows:

[0028] Energy is introduced from microgrid j to microgrid i;

[0029] p i,j (k) represents the unit cost of energy supplied by microgrid j to microgrid i;

[0030] For microgrid i to supply energy to microgrid j;

[0031] q i,j (k) represents the unit cost of energy introduced by microgrid j from microgrid i.

[0032] Next, in step S203, the decision-making process of the decision variables of each micronet is modeled as a decision-making process based on multi-agent Q-learning, wherein each micronet is modeled as an agent. For the agent serving as the model of each micronet, its Q-learning can be defined as the following Markov process:

[0033] <X i U i ,r i ,π i >

[0034] Among them, X i For the state space, in some embodiments, the state x of agent i during a predetermined time period k is... i (k)∈X i This can be defined as the energy allocation state of microgrid i corresponding to agent i over a predetermined k-period, and this energy allocation state includes the first non-decision variable and the second non-decision variable predicted by microgrid i corresponding to agent i. More specifically, during the predetermined k-period, x i (k) can be written as: Formula (2)

[0035] in:

[0036] For the energy stored in the microgrid i-energy storage battery, assuming and When considering the charging and discharging efficiencies of energy storage devices, and It has the following relationship:

[0037]

[0038] ΔE i (k) represents the energy deviation of microgrid i, which will be described in detail below;

[0039] The charging power of the energy storage devices in the microgrid i;

[0040] T is the duration of the predetermined k-time period;

[0041] L i (k) represents the power of the load in microgrid i;

[0042] The power generation capacity of microgrid i-environmental energy;

[0043] p g (k) represents the unit cost of microgrid i drawing energy from the main grid;

[0044] q g(k) represents the unit cost of microgrid i supplying energy to the main grid;

[0045] C i (k) is the first expenditure parameter of microgrid i;

[0046] R i (k) represents the credit parameter of microgrid i after performing the predetermined energy allocation for time period k.

[0047] The above U i In the action space, the actions of agent i during a predetermined k-period are the decision variables to be optimized for the corresponding microgrid i during that predetermined k-period. In some embodiments, these may include the charging and discharging parameters of the energy storage device of microgrid i, the energy introduced from other microgrids, the unit cost of energy supplied by other microgrids, the energy supplied to other microgrids, the unit cost of energy introduced from other microgrids, the energy introduced from the main grid, and the energy supplied to the main grid. Specifically, in some embodiments, each action during the predetermined k-period, i.e., the decision variable u... i (k)∈U i It can be recorded as:

[0048]

[0049] The variables in the above formula are as follows:

[0050] The discharge power of the energy storage device in microgrid i;

[0051] The charging power of the energy storage devices in the microgrid i;

[0052] Energy is drawn from the main grid for the microgrid i;

[0053] The energy supplied by the microgrid i to the main grid;

[0054] To introduce energy from other microgrids for microgrid i;

[0055] The unit cost of energy supplied by other microgrids to microgrid i;

[0056] The energy supplied by microgrid i to other microgrids;

[0057] The unit cost for other microgrids to draw energy from microgrid i.

[0058] In some embodiments, during a predetermined time period k, the reward function r of agent i... i It can be recorded as:

[0059] For the reward function, as an example, the reward function of agent i based on the state space and decision space can be based on a composite optimization index. To determine, for example, assuming r i (x i (k),u i (k) is:

[0060]

[0061] For each agent i, when the decision vector u i (k) Achieved composite optimization index The better, the higher the reward function r i The larger the value of r, the better. Taking an energy distribution cycle of 24 hours as an example, to make r... i To maximize, the optimization objective for each agent i can be set as a composite optimization index for each time period in the next 24 hours. The sum of the smallest possible values ​​is denoted as:

[0062]

[0063] in,

[0064] Constraint C1 refers to the energy deviation ΔE of microgrid i during a predetermined time period k. i (k) and energy introduced from the main power grid and the energy supplied to the main power grid Maintaining a balance between them, that is:

[0065]

[0066] in, For microgrid i to draw energy from the main grid, The energy supplied by microgrid i to the main grid, and ΔE i (k) can be further expressed as:

[0067]

[0068] in:

[0069] The power generation capacity of microgrid i-environmental energy;

[0070] The discharge power of the energy storage device in the microgrid i The charging power of the energy storage devices in the microgrid i;

[0071] The energy supplied by microgrid i to other microgrids. The energy introduced by microgrid i from other microgrids can be expressed as follows:

[0072]

[0073]

[0074] L i (k) represents the power of the load in microgrid i;

[0075] χ i The characteristic coefficients can be defined as follows:

[0076]

[0077] Constraint C2 states that for microgrid i, the energy introduced from any microgrid j is equal to the energy supplied from microgrid j to microgrid i. In other words, the energy introduction and supply satisfy the following symmetry:

[0078]

[0079] Constraint C3 refers to the fact that the energy storage device of microgrid i has a maximum power. The limitation is that the charging power within a predetermined k-period is... and discharge power Between 0 and maximum power Between, that is:

[0080]

[0081] Constraint C4 refers to the charging state of the energy storage device of microgrid i during a predetermined k-period (SoC) period. i (k) is within the first charging state threshold Second charging state threshold Between, among which, the charging state SoC i (k) represents the energy stored by the energy storage device during a predetermined time period k, and the capacity Cap of the energy storage device. i The ratio, that is:

[0082]

[0083] in,

[0084]

[0085] Here, the first charging state threshold Second charging state threshold All values ​​are within the range of [0,1], but to avoid damage to energy storage devices, such as energy storage batteries, the charging state SoC is generally set. i(k) Restricted to a smaller range, for example, we can let

[0086] Constraint C5 means that the unit cost of energy supplied by other microgrids to microgrid i and the unit cost of energy introduced by other microgrids from microgrid i should both be below the first supply cost threshold p. min (k) and the second supply cost threshold p max Between (k), energy demands outside this range will not be matched:

[0087]

[0088] Under the constraints of constraints 1-5 defined by formulas (5) to (16) above, in some embodiments, the composite optimization index Further optimization can be based on multiple optimization indicators, such as the first optimization indicator characterizing energy costs. And the second optimization index characterizing the credit parameters after energy allocation is performed according to the predetermined k-period. The fusion is used for calculation, as an example, when considering and In this case, The calculation can be performed using the following formula:

[0089]

[0090] in

[0091]

[0092]

[0093] The value of the coefficient β∈[0,1] can be used to adjust the composite optimization index. For the first optimization index Second optimization index The emphasis, for example, when β > 0.5, indicates that in In the calculation, the first optimization index The second optimization metric has a larger weight, and conversely, the second optimization metric has a smaller weight. To influence More important metrics are calculated. In some embodiments, when β = 1, it is equivalent to considering only... right The influence of β is considered when β = 0, which means only considering the effect of β. right The impact. In embodiments according to this disclosure, in the composite optimization index In terms of configuration, the main objectives of each microgrid in the regional energy system when submitting its desired energy allocation plan are comprehensively considered, as well as the inherent mechanism of the actual system in energy allocation. That is, each microgrid expects the energy allocation plan to have the minimum cost / maximum benefit. Such a requirement is addressed in the above... This is reflected in, and The setting of β takes into account that when the regional energy system matches the energy quantity and unit cost of energy for each microgrid, it may refer to the credit rating of the microgrid. Generally, the better the credit rating, the more favorable the energy allocation scheme will be for the microgrid. The importance of the above two types of optimization indicators, as well as the degree of correlation between them, can be fully approximated by the characteristics of the actual system through the reasonable setting of the value of β.

[0094] From this, we can further conclude that:

[0095] r i (x i (k),u i (k))=-β(C i (k)+Cg i (k))-(1-β)R i (k) Formula (20)

[0096] Among them, Cg i (k) conforms to the following relationship:

[0097]

[0098] From the above formulas and expressions, it can be seen that the reward function r of agent i is... i It can be based on the first non-decision variable predicted for the microgrid i corresponding to agent i, the second non-decision variable predicted for the microgrid i corresponding to agent i, and the second expenditure parameter Cg of the microgrid i corresponding to agent i relative to the main grid. i (k) is used to determine this. Specifically, the first optimization metric is... This includes the first expenditure parameter C of the energy allocation between microgrid i corresponding to agent i and other microgrids during time period k. i (k) and the second expenditure parameter Cg of microgrid i corresponding to agent i in time period k relative to the main grid. i The sum of (k), where the first expenditure parameter C i (k) can utilize p in the first non-decision variable. i,j (k), q i,j (k), and the corresponding parameters of the two parameters mentioned above. and The second optimization index is calculated according to formula (1) above. This directly utilizes the credit parameter R from the second non-decision variable, which corresponds to the microgrid i corresponding to agent i, after performing energy allocation for k time periods. i (k).

[0099] In some embodiments, π can be set i For the decision variable u of microgrid i i (k) During optimization, the state transition strategy of microgrid i is used to define the probability distribution of state transitions. As an example, an ε-greedy strategy can be chosen as the state transition strategy, that is, for a fixed ε∈[0,1], u is randomly selected. i (k)∈U i The probability is ε, and the locally optimal action is chosen. The probability of finding the optimal action is 1-ε. This ε-greedy strategy ensures that every action in the action space has a probability of being selected, theoretically allowing for the solution to reach the global optimum of the (state-action) tuple. In other embodiments, algorithms such as UCB (Upper Confidence Bound) can be used as state transition strategies to further optimize the probability of the agent discovering the optimal action. In some embodiments, the locally optimal action is defined as:

[0100]

[0101] The Q function is used to measure the accumulated reward value, and it can be iteratively updated using the following formula:

[0102]

[0103] Where α∈(0,1] is the learning efficiency, γ∈(0,1] is the discount factor, and cr i (k) is the Collaborative Reward function, x′ i To take action u′ i The state reached later. In some embodiments, the learning speed of the agent can be adjusted by adjusting the values ​​of the α and γ parameters. At the same time, the consideration of reward values ​​in the time domain can also be adjusted. For example, when the value of γ is closer to 0, it means that it only cares about the current reward value, and when the value of γ is 1, it means that it considers all future reward values.

[0104] In the above formula, the collaborative reward function cr iThe calculation of (k) is based on a mechanism that enables information exchange between neighboring agents. In various embodiments of this disclosure, when each micronet is modeled as an agent, the information transmission links between micronets are also modeled. Therefore, agents corresponding to micronets with information transmission links are topologically neighboring agents. During multi-agent-based optimization, neighboring agents can locally exchange relevant data, such as reward function values. In some embodiments, agent i can transfer the calculated reward function r... i The value of the reward function is passed to the neighboring agent i that is exchanging information with agent i; that is, the neighboring agent. Correspondingly, agent i will also receive the reward function values ​​from the other neighboring agents. Agent i can then use its own reward function r... i The value of the reward function cr is used to determine the joint reward function cr of agent i, along with the values ​​of the reward functions passed to other agents i. i The value of (k). This distributed optimization computation method based on multi-agent cooperation, because it has a certain degree of access to near-global information, can effectively avoid the communication burden, computational load, and speed defects that may arise from centralized optimization computation in a central network without significantly reducing optimization performance. Especially when the scale of the regional energy system is large, the size of the multi-agent state space after modeling grows exponentially with the increase of the number of features, thus causing the "curse of dimensionality" and even making it difficult to solve. The multi-agent distributed optimization method can avoid the "curse of dimensionality" and has advantages such as low communication burden and fast solution speed. Furthermore, in some other embodiments, especially when the regional energy system is large and the topology is complex, some agents can be selected from neighboring agents, instead of all of them, to exchange reward function values. In this way, the communication and computational burden can be further reduced without sacrificing optimization performance. In some embodiments, the cr of agent i i (k) For example, it can be calculated using the following formula:

[0105]

[0106] Where δ∈(0,1) is a constant.

[0107]

[0108] θ > 0 is a very small constant used to prevent the denominator from being equal to 0.

[0109] In some embodiments, the calculation of the agent cooperative reward function can enhance the agent's ability to automatically adapt to the dynamic changes of the system environment and other agents, thereby helping the agent learn better decision variables and enabling the distributed optimization solution of each agent to reach a global optimal solution close to that of centralized solution.

[0110] After modeling each microgrid as an agent, in step S204, each microgrid i modeled as agent i, for each predetermined k time period (k = 1, ..., 24) within the energy allocation cycle (e.g., 24 hours), iteratively optimizes the decision variable u of microgrid i according to the aforementioned agent Q-learning method, utilizing the predicted outputs of relevant parameters from the first and second learning networks, as well as the interaction of reward function value information between neighboring agents. i (k). When If convergence occurs, the current decision vector u will be... i (k) is the optimal decision for time period k. In this way, the optimal decision for each microgrid in each time period of the energy allocation cycle is calculated as its expected energy allocation scheme. During the energy allocation scheme submission stage, the scheme is submitted to the energy management center, which then performs the final matching of energy allocation.

[0111] Below, in conjunction with Figure 3 and Figure 4 Describe the functions and specific inputs and outputs of the first and second learning networks.

[0112] In some embodiments, the microgrid i modeled as an agent utilizes, for example... Figure 3 The first learning network 300 shown is used to predict the first non-decision variable L of microgrid i for each predetermined k-period within an energy distribution cycle. i (k+h) p g (k+h) and q g (k+h), more specifically, based on the historical data L of the first non-decision variable of microgrid i in the past time period before the predetermined k-time period. i (kh′), p g (kh′) and q g (kh′) is used for prediction.

[0113] As an example, the past time period could be set to the past 24 hours, that is... Figure 3In the first learning network 300 shown, the value of h′ is h′ = 1, 2, ..., 24. In other embodiments, the range of h′ values ​​can also be set according to the actual system requirements for the first learning network 300, such as prediction accuracy, computational load, and computational speed. In some embodiments, when an energy cycle is 24 hours and each hour is divided into a time period, the range of parameter h is h = 0, 1, ..., 23. In some embodiments, the first learning network 300 can be implemented using various types of neural networks. In other embodiments, the first learning network 300 can also use machine learning frameworks such as A2C (Advantage Actor-Critic). By introducing the first learning network 300, training data can be used to perform machine learning, avoiding the use of traditional theories and solution methods, to achieve prediction of variables such as load parameters, environmental energy generation-related parameters, unit cost of energy introduced from the main grid, and unit cost of energy supplied to the main grid based on artificial intelligence networks, obtaining more accurate optimal solutions with lower computational cost and faster computation time.

[0114] exist Figure 4 In the second learning network 400, the first expenditure parameter C, which includes the energy allocation between microgrid i and other microgrids, is used to predict the energy allocation of microgrid i as an agent for each predetermined k-period within an energy allocation cycle. i (k+h) and the credit parameter R after performing the predetermined energy allocation for k time periods. i The second non-decision variable, including (k+h), more specifically, is based on the historical data of the second non-decision variable of microgrid i in the past time period before the predetermined k-time period, i.e., C. i (kh′) and R i (kh′), and some of the decision variables, namely, the energy introduced from other microgrids. The unit cost of energy supplied by other microgrids Energy supplied to other microgrids The unit cost of energy introduced by other microgrids To predict the second non-decision variable of the micronetwork during the predetermined k-period. Figure 3 The parameter settings for the first learning network 300 are similar, in Figure 4In the second learning network, as an example, the ranges of h and h′ can be set as: h = 0, 1, ..., 23, h′ = 1, 2, ..., 24. In some embodiments, the second learning network 400 can be implemented using various types of neural networks. In other embodiments, the second learning network 400 can also employ machine learning frameworks such as A2C. In some embodiments, the second learning network 400 can replace traditional solution methods, utilize training data, and through machine learning, effectively simulate the rules and mechanisms for matching energy allocation schemes submitted by each microgrid and performed by the energy management center, including the credit parameter R of each microgrid after energy allocation is executed. i The update mechanism can assist each microgrid in predicting more accurate second non-decision variable data based on historical data and related inputs during the energy allocation scheme submission stage, thereby facilitating the provision of optimized decision vectors for each microgrid modeled as an agent.

[0115] The following is combined with Figure 5 This document details the optimization solution process using multi-agent Q-learning according to embodiments of the present disclosure. As an example, in this embodiment, it is still assumed that an energy allocation cycle is 24 hours. However, when using agent Q-learning for optimization, the 24 hours are divided into 24 time periods, each lasting one hour.

[0116] The optimization solution for Q-learning of multi-agent i according to embodiments of this disclosure begins at step S501. In step S501, the first non-decision variable is predicted using a first learning network based on historical data of the first non-decision variable. In some embodiments, the first learning network may be a machine learning network that has been pre-trained using a large amount of actual system operation data or simulated data of actual system operation and has achieved certain performance indicators.

[0117] In step S502, it is determined whether the Q function has not been assigned a value. If so, an arbitrary initial value is assigned; otherwise, the already assigned value is used. When performing the optimization calculation for the first time, the Q function has not yet been initialized. In this case, an arbitrary initial value is assigned to the Q function, and the assigned initial value has no effect on the subsequent iterative optimization process.

[0118] In step S503, decision variables are selected according to the agent's current state using an ε-greedy strategy. In this embodiment, by appropriately setting the value of ε, it is ensured that the decision variables in the action space can be fully explored, and the optimal solution for the decision variables can be obtained at a relatively fast speed.

[0119] In step S504, based on historical data of the second non-decision variable and some decision variables, a trained second learning network is used to predict the second non-decision variable. In some embodiments, the second learning network may be a machine learning network that has been pre-trained using a large amount of actual system operation data or simulated data of actual system operation and has achieved certain performance indicators.

[0120] In step S505, the reward function value of a single agent is calculated and sent to other neighboring agents. In some embodiments, the agent's own reward function is determined based on a modeled composite optimization index. For example, a first optimization index representing the energy cost can be used. And the second optimization index characterizing the credit parameters after energy allocation is implemented. The reward function is calculated by weighted fusion. In some embodiments, reward function values ​​are exchanged between specific neighboring agents so that each agent has access to certain global information. The exchanged reward function values ​​are used to calculate the cooperative reward function value to achieve multi-agent cooperative optimization.

[0121] In step S506, after receiving the reward function values ​​of other neighboring agents, a cooperative reward function value is calculated. In some embodiments, by setting the parameters in the cooperative reward function, the agent's perception ability of the system environment and other agents is adjusted, making it easier for the agent to obtain the optimal solution.

[0122] In step S507, the value of the Q function is updated. In some embodiments, the convergence speed and probability of convergence to the optimal solution of the agent's Q-learning can be controlled by adjusting the learning efficiency parameter and discount factor parameter of the Q function. At the same time, different values ​​of the parameters can be used to control the emphasis on the reward value in the time domain.

[0123] In step S508, it is determined whether the value of the Q function has converged. If the result is "yes", the process proceeds to step S509; otherwise, it returns to step S503 and executes steps S503-S508 again until the Q function converges, and then proceeds to step S509.

[0124] In step S509, the decision variable that makes the Q function value converge is output as the optimal decision for the current time period.

[0125] In step S510, it is determined whether the optimization solution for the decision variables of all time periods in the energy allocation cycle has been completed. If not, the process returns to step S501, and the optimization solution process from steps S501 to S510 is executed for the next time period. In this embodiment, when the optimization solution for all 24 time periods in an energy allocation cycle is completed, the process proceeds to step S511.

[0126] In step S511, based on the optimal decision variables for each time period obtained in the above steps, the expected energy allocation scheme for microgrid i corresponding to agent i in each time period of the energy allocation cycle is generated, and the energy allocation scheme submission is completed. The energy allocation scheme submission stage ends here.

[0127] In step S512, the energy allocation scheme submission stage transitions to the energy allocation scheme matching stage. In step S512, the energy management center matches the energy allocation schemes submitted by each microgrid in step S511. In some embodiments, the matched energy allocation schemes generated in step S512 can be provided to each microgrid for execution in subsequent energy allocation cycles. In other embodiments, the matched energy allocation schemes can also be used as training data, provided to the first and second learning networks used by each microgrid for machine learning.

[0128] Furthermore, although exemplary embodiments have been described herein, their scope includes any and all embodiments based on this disclosure that have equivalent elements, modifications, omissions, combinations (e.g., schemes involving intersections of various embodiments), adaptations, or alterations. Elements in the claims will be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the implementation of this application, and such examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered illustrative only, and the true scope and spirit are indicated by the full scope of the claims and their equivalents.

[0129] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more thereof) can be used in combination with each other. Other embodiments may be used by those skilled in the art upon reading the above description. Furthermore, in the above detailed description, various features may be grouped together to simplify this disclosure. This should not be construed as an intention that a disclosed feature, which is not claimed, is necessary for any claim. Rather, the subject matter of the invention may be less than all the features of a particular disclosed embodiment. Thus, the following claims are incorporated herein by reference as examples or embodiments, wherein each claim is independently considered as a separate embodiment, and these embodiments are contemplated as being possible in various combinations or arrangements. The scope of the invention should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. An energy distribution method for a regional energy system, wherein the regional energy system includes a main power grid and multiple distributed microgrids connected to the main power grid, characterized in that, The energy distribution method includes: The first non-decision variables of each microgrid are predicted based on the first learning network during a predetermined period. The first non-decision variables include load parameters, power generation parameters related to environmental energy, unit cost of introducing energy from the main grid, and unit cost of supplying energy to the main grid. A second non-decision variable for the microgrid during the predetermined time period is predicted based on a second learning network. This second non-decision variable includes a first expenditure parameter for energy allocation between the microgrid and other microgrids, and a credit parameter after the energy allocation for the predetermined time period. The decision-making process of the decision variables of each micronet is modeled as a decision-making process based on multi-agent Q-learning, wherein each micronet is correspondingly modeled as an agent. In the multi-agent Q-learning decision-making process, for each agent: The action is defined as the decision variable to be optimized for the predetermined time period in the microgrid corresponding to the agent. The state is defined as the energy allocation state of the microgrid corresponding to the agent during the predetermined time period, and the energy allocation state includes the first non-decision variable and the second non-decision variable predicted by the microgrid corresponding to the agent. The reward function is determined based on the first non-decision variable predicted for the microgrid corresponding to the agent, the second non-decision variable predicted for the microgrid corresponding to the agent, and the second expenditure parameter of the microgrid corresponding to the agent relative to the main grid, and the value of the reward function is passed to another agent that exchanges information with the agent. The value of the agent's collaborative reward function is determined based on the value of the agent's reward function and the values ​​of the reward functions of other agents passed to the agent; and For each micronet, the decision variables of each micronet are optimized using the decision process based on multi-agent Q-learning.

2. The energy distribution method according to claim 1, characterized in that, The regional energy system includes a residential park energy system, with each resident user acting as a microgrid.

3. The energy distribution method according to claim 1, characterized in that, Another agent that exchanges information with each agent is the neighboring agent of each agent.

4. The energy distribution method according to claim 1, characterized in that, The decision variables for each microgrid include the charging and discharging parameters of the energy storage device, the energy introduced from other microgrids, the unit cost of energy supplied by other microgrids, the energy supplied to other microgrids, the unit cost of energy introduced from other microgrids, the energy introduced from the main grid, and the energy supplied to the main grid.

5. The energy distribution method according to claim 1, characterized in that, The energy distribution status of each microgrid during the predetermined time period further includes: the energy stored by the energy storage device of the microgrid during the predetermined time period and the energy deviation of the microgrid during the predetermined time period.

6. The energy distribution method according to claim 1, characterized in that, The decision-making process based on multi-agent Q-learning uses an ε-greedy strategy as the state transition strategy.

7. The energy distribution method according to claim 1, characterized in that, For each micronetwork, optimizing the decision variables of each micronetwork using the multi-agent Q-learning-based decision process specifically includes: Determine the value of the cooperative reward function for the agents corresponding to the micronetwork; The Q-function value is determined based on the value of the established collaborative reward function; and When the Q-function value satisfies the convergence condition, the current decision vector is used as the optimized decision vector.

8. The energy distribution method according to claim 1, characterized in that, The reward function is determined based on a composite optimization index that integrates the first and second optimization indices. The better the composite optimization index achieved by the decision vector, the larger the value of the reward function. The first optimization index includes the sum of a first expenditure parameter and a second expenditure parameter, wherein the first parameter represents the energy allocation cost between the microgrid corresponding to the agent and other microgrids during the predetermined time period, and the second expenditure parameter represents the energy allocation cost of the microgrid corresponding to the agent relative to the main grid during the predetermined time period. The second optimization metric is the credit parameter of the microgrid corresponding to the agent after it performs the energy allocation for the predetermined time period.

9. The energy distribution method according to claim 1, characterized in that, Predicting the first non-decision variable of each micronet during a predetermined period based on the first learning network specifically includes: predicting the first non-decision variable of each micronet during the predetermined period based on historical data of the first non-decision variable of each micronet in past periods before the predetermined period.

10. The energy distribution method according to claim 1, characterized in that, Predicting the second non-decision variable of the microgrid during the predetermined period based on the second learning network specifically includes: predicting the second non-decision variable of the microgrid during the predetermined period based on historical data of the second non-decision variable in past periods prior to the predetermined period, as well as energy introduced from other microgrids, the unit cost of energy supplied by other microgrids, energy supplied to other microgrids, and the unit cost of energy introduced from other microgrids.

11. The energy distribution method according to claim 1, characterized in that, For each micronetwork, optimizing the decision variables of each micronetwork using the multi-agent Q-learning-based decision process further includes performing optimization under constraints, which include: The energy deviation of each microgrid during the predetermined time period is balanced with the energy introduced from the main grid and the energy supplied to the main grid. For any first microgrid and second microgrid in each microgrid, the energy introduced by the first microgrid from the second microgrid is equal to the energy supplied by the second microgrid to the first microgrid; The charging and discharging power of the energy storage devices in each microgrid are between 0 and the maximum power. The charging state of the energy storage devices of each microgrid during the predetermined period is between a first charging state threshold and a second charging state threshold, wherein the charging state is the ratio of the energy stored by the energy storage device to the capacity of the energy storage device during the predetermined period. The unit cost of energy supplied by the other microgrids is between a first supply cost threshold and a second supply cost threshold; and The unit cost of energy introduced by the other microgrids is between a first introduction cost threshold and a second introduction cost threshold.

12. A regional energy system, the regional energy system comprising a main power grid and a plurality of microgrids connected to and distributed with the main power grid, characterized in that, Each microgrid uses the energy distribution method according to any one of claims 1-11 to achieve energy distribution between itself and other microgrids and the main grid.

Images