MADDPG-based power grid enterprise electricity purchasing and selling joint strategy optimization method and system

Abstract

The invention discloses a power grid enterprise power purchase and sale joint strategy optimization method and system based on MADDPG, and the method comprises the steps: constructing a power purchase decision model of a power grid company in a spot market and a supply model of the power grid company at a user side, and considering the power supply deviation loss of the power grid company, and obtaining a joint decision objective function of the power grid company; and on the premise that the intelligent agents are trained through a multi-intelligent-agent algorithm, optimal strategies for power purchase and load prediction are given respectively, and a reinforcement learning algorithm MADDPG is adopted, so that the objective fact that a power grid company is difficult to obtain all user information is met, and meanwhile, a large amount of calculation power consumed by modeling prediction is avoided; through mutual strategy approximation of the two agents, the dynamic stability of the agent training process is ensured, and the algorithm efficiency and the model solving effect are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power market, and relates to a MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method and system. Background Art

[0002] In the context of power system reform, power grid companies need to make full use of market tools and maintain the safety and stability of the entire network through proxy power purchases, inter-provincial power purchases, etc., to achieve high-quality regional power supply. Under the current market rules, power grid companies need to participate in both medium- and long-term market and spot market transactions. In the medium- and long-term market, power grid companies need to determine the amount of power purchased for medium- and long-term transactions based on annual, quarterly, monthly or multi-day forecasts of the load end; in the spot market, power grid companies need to decompose the load curve, determine the real-time power demand at the load end, and determine the power purchase decision in the spot market based on the gap between supply and demand. In addition, due to the responsibility of maintaining the safety and stability of the power grid, on the user side, power grid companies also need to guide users to transfer or reduce loads through demand response or other means to maintain load stability. However, at this stage, when power grid companies participate in the spot market to determine short-term power purchase decisions, they have not yet considered the impact of factors such as deviations between supply and actual load, user utility, and power purchase volume on power purchase decisions. Summary of the invention

[0003] The purpose of the present invention is to solve the problem in the prior art that when power grid companies make short-term power purchase decisions, factors such as the deviation between supply and actual load, user utility and power purchase amount affect power purchase decisions, and to provide a method and system for optimizing the joint power purchase and sales strategy of power grid enterprises based on MADDPG.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The MADDPG-based optimization method for the power grid enterprise's power purchase and sales joint strategy includes:

[0006] Consider the marginal cost of the generator set and obtain the bid decision of the generator;

[0007] Considering the diminishing marginal utility of power generators, obtain the bidding decision of power demanders;

[0008] Based on the bidding decisions of power generators and power demanders, obtain the power purchase decision model of power grid companies in the spot market;

[0009] Based on the initial load of users in each period and considering the impact of the interaction between the power grid and users on the actual load, a supply model of the power grid company at the user end is constructed;

[0010] Based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company, the joint decision-making objective function of the power grid company is constructed;

[0011] Based on the MADDPG algorithm, the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company are optimized respectively to obtain the optimal joint decision objective function of the power grid company.

[0012] A further improvement of the present invention is:

[0013] Furthermore, considering the marginal cost of the generator set, the bidding decision of the generator is obtained, which is as follows:

[0014]

[0015] Among them, Θ i (P i,t ) represents the cost function related to power of the i-th generator set in period t; P i,t represents the output of the i-th generator set in period t; a i , b i 、c i are the coefficient terms in the quadratic cost function respectively;

[0016] By taking the derivative of formula (1), we can obtain the marginal cost function of the generator set, which is:

[0017] θ i (P i,t )=2a i P i,t +b i (2)

[0018] Among them, θ i (P i,t ) represents the power output P of the generator set i and the period i,t The relevant marginal costs.

[0019] Furthermore, considering the diminishing marginal utility of the generator set, the bidding decision of the power demander is obtained. Specifically, for the power grid company and other power purchase and sales entities, they are all power demanders in the spot market. Considering the diminishing marginal utility, the bidding decision of the power demander is:

[0020]

[0021] Among them, U j (P j,t ) represents the utility function of the jth power demander related to power; P j,t represents the power demanded by the jth power demander in period t; α j , βj are the coefficient terms in the quadratic utility function respectively;

[0022] Deriving formula (4), we can obtain the marginal utility of power demander j as

[0023] u j (P j,t )=-2α j P j,t +β j (4)

[0024] Among them, u j (P j,t ) represents the marginal utility of demander j related to the power demanded in the time period.

[0025] Furthermore, based on the bidding decisions of the power generators and the power demanders, the power purchase decision model of the power grid company in the spot market is obtained, which is as follows:

[0026] The power generators make their bids based on marginal costs, while the power demanders make their bids based on their overall benefit goals for power purchase and sales. The bidding strategies of the power generators and power demanders are as follows:

[0027]

[0028] Among them, i,t represents the power supply quotation given by generator set i in period t; Ω j,t represents the power purchase quotation given by power demander j in period t; k i,t and j,t The multiplier used by generators and demanders when making bids based on marginal cost / marginal utility;

[0029] When the generators and electricity demanders bid in the spot market respectively according to the above behavior mechanism, the market clearing model can be expressed as:

[0030]

[0031] Among them, {(P i,t ,Π i,t ),(P j,t ,Ω j,t )} represents the order book set of the spot market; N represents the total number of demand parties; M represents the total number of generators; when the market is cleared, the supply and demand balance of the order set, the power balance of the entire network and the flow balance should be satisfied at the same time, so the constraint condition of formula (6) is

[0032]

[0033] The first part of the formula represents the supply and demand balance condition for market clearing; the second part represents the power constraint condition for unit output; the third part represents the power flow constraint condition. represents the upper limit of power flow safety of line l, s represents the node, S represents the total number of nodes, ψ l,t represents the power transfer distribution factor of line l during period t.

[0034] Furthermore, based on the initial load of users in each period and considering the impact of the interaction between the power grid and users on the actual load, a supply model of the power grid company at the user end is constructed. Specifically, the power purchase amount in the power grid company's power purchase decision is based on the load forecast on the user side. The load forecast includes not only the forecast of the initial load of users in each period, but also the impact of the interaction between the power grid and users on the actual load. When the social welfare utility is maximized, the utility of the power grid company is

[0035]

[0036] Among them, f t represents the supply objective function of the power grid company in period t; c represents the user; C represents the set of users; represents the utility function of user c related to load; represents the initial load demand of user c in period t; L c,t Indicates the actual load level of the user due to load transfer; represents the unit price of subsidy for effective demand response determined according to the policy; t is a dummy variable. When the user provides effective demand response in period t, the value is 1, otherwise it is 0; ρ t Indicates the unit price of electricity used by the user;

[0037] The actual load level and initial load demand of the user are expressed as

[0038]

[0039] in, represents the proportion of user c transferring the load demand in period t to the next period; ε c,t is a random variable;

[0040] The user's load level utility function is:

[0041]

[0042] Among them, α c and β c are the coefficient items in the quadratic utility function respectively.

[0043] Furthermore, based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company, the joint decision objective function of the power grid company is constructed, which is as follows:

[0044]

[0045] Among them, (Ω j,t ,λ t ) represents the set of control variables; r represents the discount rate; λ t represents the spot market clearing price in period t, which can be replaced by the node marginal electricity price LMP; L t represents the actual total load level of all users of the power grid company in period t; X t (L t ,P j,t ) represents the loss incurred by the power grid company due to the deviation between the power purchased and the actual total load;

[0046] Deviation loss of power supply from the power grid company X t (L t ,P j,t ) is expressed as

[0047] X t (L t ,P j,t )=ξ|L t -P j,t | (12)

[0048] Where ξ represents the deviation penalty coefficient in the dual settlement market.

[0049] Furthermore, based on the MADDPG algorithm, the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company are optimized respectively, as follows:

[0050] Define two agents as in represents the electricity purchasing agent, represents the load forecasting agent; the state space represents the variable space observable by the agent, which is represented by sample data. The state space is The specific sequence of two agents in the state space is expressed as:

[0051]

[0052] Among them, P represents the sample sequence of the power grid company's time-period power purchase; Ω represents the sample sequence of the power purchase quotation of the power sales company; λ represents the sample sequence of the time-sharing clearing electricity price of the spot market; represents the grid company's forecast of the initial load demand of users in each time period; L represents the sample sequence of retail users' load level in each time period; the length of the above sample sequence is recorded as n, that is, the first n historical data of the current time period are taken as the sample sequence;

[0053] Define the set of actions for the agent Indicates that the electricity purchasing agent changes from Select a data. Indicates that the load forecasting agent changes from Select a data from the table, the specific expression is

[0054]

[0055] Among them, Ω t Indicates that the electricity purchasing agent The best electricity purchase quotation given in; Indicates the value range of the quote status sequence; Represents the load forecasting agent from The forecast of initial load demand for the period given in ; Indicates the value range of the initial load demand forecast; L t It represents the prediction of the actual load of users in each period by the load forecasting agent; Indicates the value range of the prediction of the actual user load level;

[0056] In MDP, the strategy represents the probability of the agent choosing a certain action when the state value is given, expressed as

[0057]

[0058] in, Indicates a given Agent Selection Strategy of action; Indicates the function mapping relationship between the two; Indicates a given Agent Selection Strategy of action; Represents the function mapping relationship between the two; in this mapping relationship, the definition Respectively represent the parameters of the two agents when adopting a given strategy;

[0059] According to the characteristics of the model, the global reward function of the agent is designed as

[0060] R t =(ρ t -λ t )L t -x t (Lt ,P j,t ) (16)

[0061] Taking the electricity purchasing agent as the research object, without considering the maximum iteration step, its cumulative expected reward is expressed as

[0062]

[0063] in, represents the cumulative expected reward of the electricity purchasing agent; ρ π represents the probability of giving strategy π; γ t Represents the reward discount coefficient at the t-th iteration;

[0064] For the above cumulative expected reward, the policy gradient is

[0065]

[0066] in, Indicated in the parameter Get the gradient down; It represents the Q learning value given by the electricity purchasing agent network under given state and action;

[0067] The loss function in the critic network update in MADDPG is expressed as

[0068]

[0069] in, The expected reward representing the variance between the Q-learned value and the true value; x ` represents other values ​​that are mutually exclusive with the agent's value in the state set; y represents the true value, which is fitted by the Q learning value of the target network. The specific expression is:

[0070]

[0071] in, represents the Q learning value given by the target network when taking other values ​​in the strategy space given π; π ` Indicates other strategies that are mutually exclusive with the current strategy; Indicates the action selected by the critic network Other mutually exclusive actions in the subsequent action set; Represents the load forecasting agent To get the value of the condition;

[0072] MADDPG estimates the strategies of other agents through the information entropy of the strategy. The details are as follows:

[0073]

[0074] in, Representing an Agent Estimating the agent by approximation The loss function of the policy; Represents the communication parameters between agents; Representing an Agent For Agents Strategy estimation; τ represents the weight of information entropy; Represents the information entropy function between agents; minimize formula (21) to obtain the approximation strategy of other agents.

[0075] Furthermore, the MADDPG algorithm also includes: both agents are iteratively updated based on each other's information. In order to make the optimal strategies trained by the two agents consistent with the overall optimal strategy of the set, the strategy of a single agent is split into k parts, and only the sub-strategy is used in each training cycle; the strategy of the electricity purchasing agent is used as an example. It consists of k sub-strategies; in each training cycle, the overall reward of its strategy set is maximized,

[0076]

[0077] Where e represents the length of the training cycle episode; unif(1,K) represents uniform random value selection in the interval (1,K); K is a constant, indicating the maximum upper limit of k; represents the kth sub-strategy;

[0078] At this time, the gradient update for each sub-strategy is

[0079]

[0080] in, For each sub-strategy The storage location of the build; Indicates that according to the conditions Updated Q value.

[0081] The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization system includes:

[0082] A first acquisition module, wherein the first acquisition module considers the marginal cost of the power generation unit and acquires the bidding decision of the power generation party;

[0083] A second acquisition module, wherein the second acquisition module considers the diminishing marginal utility of the power generation unit and acquires the bidding decision of the power demander;

[0084] A third acquisition module, wherein the third acquisition module acquires a power purchase decision model of the power grid company in the spot market based on the bidding decision of the power generator and the bidding decision of the power demander;

[0085] A first construction module, wherein the first construction module constructs a supply model of the power grid company at the user end based on the initial load of the user in each time period and considering the impact of the interaction between the power grid and the user on the actual load;

[0086] A second building block, wherein the second building block builds a joint decision-making objective function of the power grid company based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company;

[0087] The optimization module optimizes the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company based on the MADDPG algorithm to obtain the optimized joint decision objective function of the power grid company.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] The present invention constructs a power purchase decision model of a power grid company in the spot market and a supply model of the power grid company at the user end, and considers the deviation loss of the power supply of the power grid company to obtain the joint decision-making objective function of the power grid company; and trains the intelligent agents through a multi-agent algorithm to respectively give the optimal strategies for power purchase and load forecasting under the premise of consistent overall goals, and adopts the reinforcement learning algorithm MADDPG, which not only conforms to the objective fact that it is difficult for power grid companies to obtain all user information, but also avoids the large amount of computing power consumed by modeling and prediction. Through the mutual strategy approximation of the two intelligent agents, the dynamic stability of the intelligent agent training process is guaranteed, and the efficiency of the algorithm and the model solving effect are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0091] Figure 1 It is a flow chart of the MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method of the present invention;

[0092] Figure 2 It is a structural schematic diagram of the MADDPG-based power grid enterprise power purchase and sales joint strategy optimization system of the present invention. DETAILED DESCRIPTION

[0093] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0094] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0095] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0096] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0097] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0098] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0099] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0100] See also Figure 1 The present invention discloses a MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method, including:

[0101] S101: Considering the marginal cost of the power generation unit, obtaining the bidding decision of the power generation party;

[0102] To simplify the analysis, we make the following assumptions about the model:

[0103] (1) The impact of medium- and long-term agreement prices on the decision-making of spot market participants is not considered, and the participants in the spot market, whether supply or demand, have the same market position, that is, they have the same market information and are all price takers;

[0104] (2) Demand parties that have market access conditions (industrial users that do not directly supply electricity and commercial users that can directly participate in the spot market) have rigid electricity demand and will not reduce or shift their necessary loads due to the high or low time-of-use electricity prices;

[0105] (3) Load aggregators or other new market players have the same business model and nature of electricity purchase and sales as power grid companies, and their decision-making goal is to maximize the overall benefits of electricity sales.

[0106] (4) In the spot market, power suppliers make sales quotations based on their marginal cost function, and demanders make purchase quotations based on the marginal utility function of power.

[0107] Based on the above settings, considering the bidding process of the generator set, the bidding decision of the generator set is based on the marginal cost function. For its cost function, the specific expression is:

[0108]

[0109] Among them, Θ i (P i,t ) represents the cost function related to power of the i-th generator set in period t; P i,t represents the output of the i-th generator set in period t; a i , b i 、c i are the coefficient terms in the quadratic cost function respectively;

[0110] By taking the derivative of formula (1), we can obtain the marginal cost function of the generator set, which is:

[0111] θ i (P i,t )=2a i P i,t +b i (2)

[0112] Among them, θi (P i,t ) represents the power output P of the generator set i and the period i,t The relevant marginal costs.

[0113] S102: Considering the diminishing marginal utility of the power generating set, obtaining the bidding decision of the power demander;

[0114] For power grid companies and other electricity buyers and sellers, they are all power demanders in the spot market. Considering the diminishing marginal utility, the bidding decision of power demanders is:

[0115]

[0116] Among them, U j (P j,t ) represents the utility function of the jth power demander related to power; P j,t represents the power demanded by the jth power demander in period t; α j , β j are the coefficient terms in the quadratic utility function respectively;

[0117] Deriving formula (4), we can obtain the marginal utility of power demander j as

[0118] u j (P j,t )=-2α j P j,t +β j (4)

[0119] Among them, u j (P j,t ) represents the marginal utility of demander j related to the power demanded in the time period.

[0120] S103: Based on the bidding decisions of the power generators and the power demanders, a power purchase decision model of the power grid company in the spot market is obtained;

[0121] The power generators make their bids based on marginal costs, while the power demanders make their bids based on their overall benefit goals for power purchase and sales. When making market bidding decisions, both suppliers and demanders in the market are motivated to adopt economic and physical retention behaviors. The bidding strategies of power generators and power demanders are:

[0122]

[0123] Among them, i,t represents the power supply quotation given by generator set i in period t; Ω j,t represents the power purchase price given by power demander j in period t; κ i,t and j,tThe multiplier used by generators and demanders when making bids based on marginal cost / marginal utility;

[0124] When the generators and electricity demanders bid in the spot market respectively according to the above behavior mechanism, the market clearing model can be expressed as:

[0125]

[0126] Among them, {(P i,t ,Π i,t ),(P j,t ,Ω j,t )} represents the order book set of the spot market; N represents the total number of demand parties; M represents the total number of generators; when the market is cleared, the supply and demand balance of the order set, the power balance of the entire network and the flow balance should be satisfied at the same time, so the constraint condition of formula (6) is

[0127]

[0129] The first part of the formula represents the supply and demand balance condition for market clearing; the second part represents the power constraint condition for unit output; the third part represents the power flow constraint condition. represents the upper limit of power flow safety of line l, s represents the node, S represents the total number of nodes, ψ l,t represents the power transfer distribution factor of line l during period t.

[0130] S104: Based on the initial load of the user in each period and considering the impact of the interaction between the power grid and the user on the actual load, a supply model of the power grid company at the user end is constructed;

[0131] The amount of electricity purchased in the power grid company's power purchasing decision is based on the load forecast on the user side. The load forecast includes not only the prediction of the user's initial load in each time period, but also the impact of the interaction between the power grid and the user on the actual load. This impact can be determined by constructing an objective function that maximizes utility to determine the boundary conditions.

[0132] Consider a system that only includes users and power grid companies. The decision-making goal of the power grid company on the user side is to determine the best prediction of the user's load level to maximize the social welfare utility of the power grid company's power supply. Since the system only includes two entities, the power grid company and the user, the price of power supply provided by the power grid company to the user is fixed. Therefore, when the social welfare utility is maximized, the utility of the power grid company should be the remaining part of the power supply revenue and expenditure minus the user load utility, so the objective function on the supply side is expressed as

[0133]

[0134] Among them, f t represents the supply objective function of the power grid company in period t; c represents the user; C represents the set of users; represents the utility function of user c related to load; represents the initial load demand of user c in period t; L c,t Indicates the actual load level of the user due to load transfer; represents the unit price of subsidy for effective demand response determined according to the policy; t is a dummy variable. When the user provides effective demand response in period t, the value is 1, otherwise it is 0; ρ t Indicates the unit price of electricity used by the user;

[0135] The actual load level and initial load demand of the user are expressed as

[0136]

[0137] in, represents the proportion of user c transferring the load demand in period t to the next period; ε c,t is a random variable;

[0138] The user's load level utility function is:

[0139]

[0140] Among them, α c and β c are the coefficient items in the quadratic utility function respectively.

[0141] S105: Based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company, a joint decision objective function of the power grid company is constructed;

[0142] After giving the short-term power purchase model of the power grid company in the spot market and the load forecast model of the user end, in addition to considering the decision-making objectives of each link, the power grid company's power purchase balance should be maximized under the deviation settlement. Therefore, the joint decision-making objective function of the power grid company is expressed as

[0143]

[0144] Among them, (Ω j,t ,λ t ) represents the set of control variables; r represents the discount rate; λ t represents the spot market clearing price in period t, which can be replaced by the node marginal electricity price LMP; L t represents the actual total load level of all users of the power grid company in period t; t (Lt ,P j,t ) represents the loss incurred by the power grid company due to the deviation between the power purchased and the actual total load;

[0145] Deviation loss of power supply from the power grid company χ t (L t ,P j,t ) is expressed as

[0146] χ t (L t ,P j,t )=ξ|L t -P j,t |(12)

[0147] Where ξ represents the deviation penalty coefficient in the dual settlement market.

[0148] After completing the above model, the decision variables to be solved include the power P during the power purchase phase. j,t And the corresponding quotation strategy Ω j,t , where Ω j,t It also includes the marginal electricity price λ of the node in the time period t ; The decision variables in the user-side model include the actual load level variable L for each time period of the user t The best prediction.

[0149] S106: Based on the MADDPG algorithm, the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company are optimized respectively to obtain the optimized joint decision objective function of the power grid company.

[0150] Since it is difficult for power grid companies to grasp the demand information of all users at all times on the user side, the statistical model prediction method is not efficient. When we seek to use artificial intelligence methods to solve strategies, model-free deep learning methods are the first option considered.

[0151] Since the problem to be solved is in the form of a two-stage decision-making goal coupling, a multi-agent algorithm is used to train the agents to give the optimal strategy for purchasing electricity in the spot market and the joint optimal strategy for the optimal load forecast on the user side under a given overall goal. At the same time, considering that the power grid company only has local information about user demand information, the MADDPG algorithm is used to achieve the consistency between the local optimal strategy of each agent and the goal of optimal global return.

[0152] Define two agents as in represents the electricity purchasing agent, represents the load forecasting agent; since the model-free deep learning method is actually a Markov decision process MDP, first define the state parameters and action parameters under the MDP; the state space represents the variable space observable by the agent, represented by sample data, and the state space is denoted as The specific sequence of two agents in the state space is expressed as:

[0153]

[0154] Among them, P represents the sample sequence of the power grid company's time-period power purchase; Ω represents the sample sequence of the power purchase quotation of the power sales company; λ represents the sample sequence of the time-sharing clearing electricity price of the spot market; represents the grid company's forecast of the initial load demand of users in each time period; L represents the sample sequence of retail users' load level in each time period; the length of the above sample sequence is recorded as n, that is, the first n historical data of the current time period are taken as the sample sequence;

[0155] Define the set of actions for the agent Indicates that the electricity purchasing agent changes from Select a data. Indicates that the load forecasting agent changes from Select a data from the table, the specific expression is

[0156]

[0157] Among them, Ω t Indicates that the electricity purchasing agent The best electricity purchase quotation given in; Indicates the value range of the quote status sequence; Represents the load forecasting agent from The forecast of initial load demand for the period given in ; Indicates the value range of the initial load demand forecast; L t It represents the prediction of the actual load of users in each period by the load forecasting agent; Indicates the value range of the prediction of the actual user load level;

[0158] In MDP, the strategy represents the probability of the agent choosing a certain action when the state value is given, expressed as

[0159]

[0160] in, Indicates a given Agent Selection Strategy of action; Indicates the function mapping relationship between the two; Indicates a given Agent Selection Strategy of action; Represents the function mapping relationship between the two; in this mapping relationship, the definition Respectively represent the parameters of the two agents when adopting a given strategy;

[0161] According to the characteristics of the model, the global reward function of the agent is designed as

[0162] R t =(ρ t -λ t )L t -x t (L t ,P j,t ) (16)

[0163] Taking the electricity purchasing agent as the research object, without considering the maximum iteration step, its cumulative expected reward is expressed as

[0164]

[0165] in, represents the cumulative expected reward of the electricity purchasing agent; ρ π represents the probability of giving strategy π; γ t Represents the reward discount coefficient at the t-th iteration;

[0166] For the above cumulative expected reward, the policy gradient is

[0167]

[0168] in, Indicated in the parameter Get the gradient down; It represents the Q learning value given by the electricity purchasing agent network under given state and action;

[0169] The loss function in the critic network update in MADDPG is expressed as

[0170]

[0171] in, The expected reward representing the variance between the Q-learned value and the true value; x ` represents other values ​​that are mutually exclusive with the agent's value in the state set; y represents the true value, which is fitted by the Q learning value of the target network. The specific expression is:

[0172]

[0173] in, represents the Q learning value given by the target network when taking other values ​​in the strategy space given π; π ` Indicates other strategies that are mutually exclusive with the current strategy; Indicates the action selected by the critic network Other mutually exclusive actions in the subsequent action set; Represents the load forecasting agent To get the value of the condition;

[0174] Note that the updates of formulas (19) and (20) use the information x of other agents. ` as well as This requires constant communication and consumes a lot of computing resources. MADDPG estimates the strategies of other agents through the information entropy of the strategy. The details are as follows:

[0175]

[0176] in, Representing an Agent Estimating the agent by approximation The loss function of the policy; Represents the communication parameters between agents; Representing an Agent For Agents Strategy estimation; τ represents the weight of information entropy; represents the information entropy function between agents; minimize formula (21) to obtain the approximation strategy of other agents;

[0177] At this time, formula (20) is replaced by

[0178]

[0179] in, It means that when the state space takes x i When the target network is divided by x i Other status values ​​and given communication parameters other than Estimate the strategies of other agents; This means that the approximation process is repeated according to the variable length of the state space.

[0180] The above process of agents estimating each other's strategies and states can be applied to both game and competition goals, as well as cooperation or given team goals to determine the optimal independent reward allocation problem. However, since the two agents are iteratively updating based on each other's information, the above process is prone to dynamic instability in the environment. In order to make the optimal strategies trained by the two agents consistent with the overall optimal strategy of the set, the strategy of a single agent is split into k parts, and only sub-strategies are used in each training cycle; the power purchasing agent is used as an example, and its strategy It consists of k sub-strategies; in each training cycle, the overall reward of its strategy set is maximized,

[0181]

[0182] Where e represents the length of the training cycle episode; unif(1,K) represents uniform random value selection in the interval (1,K); K is a constant, indicating the maximum upper limit of k; represents the kth sub-strategy;

[0183] At this time, the gradient update for each sub-strategy is

[0184]

[0185] in, For each sub-strategy The storage location of the build; Indicates that according to the conditions Updated Q value.

[0186] Through the above steps, the sample data is supplied to the MADDPG neural network, and the two agents can give the optimal strategies for their respective goals under the target condition of optimal team return. The local optimal strategy meets the decision coupling conditions of the power grid company in the interaction between the spot market and the user end, and is consistent with the goal of maximizing the overall utility of the power grid company.

[0187] See also Figure 2 The present invention discloses a MADDPG-based power grid enterprise power purchase and sales joint strategy optimization system, including:

[0188] A first acquisition module, wherein the first acquisition module considers the marginal cost of the power generation unit and acquires the bidding decision of the power generation party;

[0189] A second acquisition module, wherein the second acquisition module considers the diminishing marginal utility of the power generation unit and acquires the bidding decision of the power demander;

[0190] A third acquisition module, wherein the third acquisition module acquires a power purchase decision model of the power grid company in the spot market based on the bidding decision of the power generator and the bidding decision of the power demander;

[0191] A first construction module, wherein the first construction module constructs a supply model of the power grid company at the user end based on the initial load of the user in each time period and considering the impact of the interaction between the power grid and the user on the actual load;

[0192] A second building block, wherein the second building block builds a joint decision-making objective function of the power grid company based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company;

[0193] The optimization module optimizes the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company based on the MADDPG algorithm to obtain the optimized joint decision objective function of the power grid company.

[0194] The terminal device provided in an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0195] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0196] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0197] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0198] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0199] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0200] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method is characterized by: include: Consider the marginal cost of the generator set and obtain the bid decision of the generator; Considering the diminishing marginal utility of power generators, obtain the bidding decision of power demanders; Based on the bidding decisions of power generators and power demanders, obtain the power purchase decision model of power grid companies in the spot market; Based on the initial load of users in each period and considering the impact of the interaction between the power grid and users on the actual load, a supply model of the power grid company at the user end is constructed; Based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company, the joint decision-making objective function of the power grid company is constructed; Based on the MADDPG algorithm, the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company are optimized respectively to obtain the optimal joint decision objective function of the power grid company.

2. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 1 is characterized in that: The marginal cost of the generator set is considered to obtain the bid decision of the generator, which is specifically: Among them, Θ i (P i,t ) represents the cost function related to power of the i-th generator set in period t; P i,t represents the output of the i-th generator set in period t; a i 、b i 、c i are the coefficient terms in the quadratic cost function respectively; By taking the derivative of formula (1), we can obtain the marginal cost function of the generator set, which is: θ i (P i,t )=2a i P i,t +b i (2) Among them, θ i (P i,t ) represents the power generation unit i and the output P of the period i,t The relevant marginal costs.

3. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 2 is characterized in that: The consideration of the diminishing marginal utility of the generator set to obtain the quotation decision of the power demander is as follows: for the power grid company and other power purchase and sales entities, they are all power demanders in the spot market. Considering the diminishing marginal utility, the quotation decision of the power demander is: Among them, U j (P j,t ) represents the utility function of the jth power demander related to power; P j,t represents the power demand of the jth power demander in period t; α j , β j are the coefficient terms in the quadratic utility function respectively; Deriving formula (4), we can obtain the marginal utility of power demander j as u j (P j,t )=-2α j P j,t +β j (4) Among them, u j (P j,t ) represents the marginal utility of demander j related to the power demanded in the time period.

4. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 3 is characterized in that: The power purchase decision model of the power grid company in the spot market is obtained based on the bidding decision of the power generation party and the bidding decision of the power demand party, which is specifically: The power generators make their bids based on marginal costs, while the power demanders make their bids based on their overall benefit goals for power purchase and sales. The bidding strategies of the power generators and power demanders are as follows: Among them, i,t represents the power supply quotation given by generator set i in period t; Ω j,t represents the power purchase quotation given by power demander j in period t; k i,t and l j,t The multiplier used by generators and demanders when making bids based on marginal cost / marginal utility; When the generators and electricity demanders bid in the spot market respectively according to the above behavior mechanism, the market clearing model can be expressed as: Among them, {(P i,t ,Π i,t ),(P j,t ,Ω j,t )} represents the order book set of the spot market; N represents the total number of demand parties; M represents the total number of generators; when the market is cleared, the supply and demand balance of the order set, the power balance of the entire network and the flow balance should be satisfied at the same time, so the constraint condition of formula (6) is The first part of the formula represents the supply and demand balance condition for market clearing; the second part represents the power constraint condition for unit output; the third part represents the power flow constraint condition. represents the upper limit of power flow safety of line l, s represents the node, S represents the total number of nodes, ψ l,t represents the power transfer distribution factor of line l during period t.

5. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 4 is characterized in that: The supply model of the power grid company at the user end is constructed based on the initial load of the user in each time period and the impact of the interaction between the power grid and the user on the actual load. Specifically, the power purchase amount in the power grid company's power purchase decision is based on the load forecast on the user side. The load forecast includes not only the forecast of the initial load of the user in each time period, but also the impact of the interaction between the power grid and the user on the actual load. When the social welfare utility is maximized, the utility of the power grid company is Among them, f t represents the supply objective function of the power grid company in period t; c represents the user; C represents the set of users; represents the utility function of user c related to load; represents the initial load demand of user c in period t; L c,t Indicates the actual load level of the user due to load transfer; represents the unit price of subsidy for effective demand response determined according to the policy; t is a dummy variable. When the user provides effective demand response in period t, the value is 1, otherwise it is 0; ρ t Indicates the unit price of electricity used by the user; The actual load level and initial load demand of the user are expressed as Among them, θ c,t represents the proportion of user c transferring the load demand in period t to the next period; ε c,t is a random variable; The user's load level utility function is: Among them, α c and β c are the coefficient items in the quadratic utility function respectively.

6. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 5 is characterized in that: The joint decision-making objective function of the power grid company is constructed based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company. Specifically, it is: Among them, (Ω j,t ,λ t ) represents the set of control variables; r represents the discount rate; λ t represents the spot market clearing price in period t, which can be replaced by the node marginal electricity price LMP; L t represents the actual total load level of all users of the power grid company in period t; t (L t ,P j,t ) represents the loss incurred by the power grid company due to the deviation between the power purchased and the actual total load; Deviation loss of power supply from the power grid company χ t (L t ,P j,t ) is expressed as χ t (L t ,P j,t )=ξ|L t -P j,t | (12) Where ξ represents the deviation penalty coefficient in the dual settlement market.

7. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 6 is characterized in that: The MADDPG algorithm is used to optimize the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company, specifically: Define two agents as in represents the electricity purchasing agent, represents the load forecasting agent; the state space represents the variable space observable by the agent, which is represented by sample data. The state space is The specific sequence of two agents in the state space is expressed as: Among them, P represents the sample sequence of the power grid company's time-period power purchase; Ω represents the sample sequence of the power purchase quotation of the power sales company; λ represents the sample sequence of the time-sharing clearing electricity price of the spot market; represents the grid company's forecast of the initial load demand of users in each time period; L represents the sample sequence of retail users' load level in each time period; the length of the above sample sequence is recorded as n, that is, the first n historical data of the current time period are taken as the sample sequence; Define the set of actions for the agent Indicates that the electricity purchasing agent changes from Select a data. Indicates that the load forecasting agent changes from Select a data from the table, the specific expression is Among them, Ω t Indicates that the electricity purchasing agent The best electricity purchase quotation given in; Indicates the value range of the quote status sequence; Represents the load forecasting agent from The forecast of initial load demand for the period given in ; Indicates the value range of the initial load demand forecast; L t It represents the prediction of the actual load of users in each period by the load forecasting agent; Indicates the value range of the prediction of the actual user load level; In MDP, the strategy represents the probability of the agent choosing a certain action when the state value is given, expressed as in, Indicates a given Agent Selection Strategy of action; Indicates the function mapping relationship between the two; Indicates a given Agent Selection Strategy of action; Represents the function mapping relationship between the two; in this mapping relationship, the definition Respectively represent the parameters of the two agents when adopting a given strategy; According to the characteristics of the model, the global reward function of the agent is designed as R t =(ρ t -l t )L t -x t (L t ,P j,t ) (16) Taking the electricity purchasing agent as the research object, without considering the maximum iteration step, its cumulative expected reward is expressed as in, represents the cumulative expected reward of the electricity purchasing agent; ρ π represents the probability of giving strategy V; γ t Represents the reward discount coefficient at the t-th iteration; For the above cumulative expected reward, the policy gradient is in, Indicates that the parameter η b Get the gradient down; It represents the Q learning value given by the electricity purchasing agent network under given state and action; The loss function in the critic network update in MADDPG is expressed as in, represents the expected reward of the variance between the Q-learning value and the true value; x` represents other values ​​that are mutually exclusive with the agent value in the state set; y represents the true value, which is fitted by the Q-learning value of the target network. The specific expression is in, represents the Q learning value given by the target network when taking other values ​​in the strategy space when V is given; V` represents other strategies that are mutually exclusive with the current strategy; Indicates the action selected by the critic network Other mutually exclusive actions in the subsequent action set; represents the load forecasting agent To get the value of the condition; MADDPG estimates the strategies of other agents through the information entropy of the strategy. The details are as follows: in, Representing an Agent Estimating the agent by approximation The loss function of the policy; Represents the communication parameters between agents; Representing an Agent For Agents Strategy estimation; τ represents the weight of information entropy; Represents the information entropy function between agents; minimize formula (21) to obtain the approximation strategy of other agents.

8. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization method according to claim 7 is characterized in that: The MADDPG algorithm also includes: both agents are iteratively updated based on each other's information. In order to make the optimal strategies trained by the two agents consistent with the overall optimal strategy of the set, the strategy of a single agent is split into k parts, and only the sub-strategy is used in each training cycle; the strategy of the electricity purchasing agent is used as an example. It consists of k sub-strategies; in each training cycle, the overall reward of its strategy set is maximized, Where e represents the length of the training cycle episode; unif(1,K) represents uniform random value selection in the interval (1,K); K is a constant, indicating the maximum upper limit of k; represents the kth sub-strategy; At this time, the gradient update for each sub-strategy is in, For each sub-strategy The storage location of the build; Indicates that according to the conditions Updated Q value.

9. The MADDPG-based power grid enterprise power purchase and sales joint strategy optimization system is characterized by: include: A first acquisition module, wherein the first acquisition module considers the marginal cost of the power generation unit and acquires the bidding decision of the power generation party; A second acquisition module, wherein the second acquisition module considers the diminishing marginal utility of the power generation unit and acquires the bidding decision of the power demander; A third acquisition module, wherein the third acquisition module acquires a power purchase decision model of the power grid company in the spot market based on the bidding decision of the power generator and the bidding decision of the power demander; A first construction module, wherein the first construction module constructs a supply model of the power grid company at the user end based on the initial load of the user in each time period and considering the impact of the interaction between the power grid and the user on the actual load; A second building block, wherein the second building block builds a joint decision-making objective function of the power grid company based on the power purchase decision model of the power grid company in the spot market, the supply model of the power grid company at the user end, and considering the deviation loss of the power supply of the power grid company; The optimization module optimizes the power purchase decision model of the power grid company in the spot market and the supply objective function of the power grid company based on the MADDPG algorithm to obtain the optimized joint decision objective function of the power grid company.

Images