User increment incentive method and bidding and demand response decision asynchronous optimization method

Abstract

The invention belongs to the technical field of power systems, and relates to a user increment incentive method and a bidding and demand response decision asynchronous optimization method. The user increment incentive method comprises the following steps that: a user purchases power required in each time period from a system operator according to a power demand, and participates in demand response to improve earnings. In the bidding and demand response decision asynchronous optimization method, a system operator optimizes a bidding strategy in an electric power wholesale market and a demand response strategy in an electric power retail market, and an asynchronous double-interaction DRL algorithm based on a depth deterministic strategy gradient is adopted to solve the coupling of bidding and demand response decisions. The coupling problem of the bidding and demand response decision is described as a Markov process. According to the invention, a price design problem under the condition of one retailer and many consumers is considered, and a flexibility mining method for demand-side difference users is provided; the asynchronous double-interaction decision-making method based on deep reinforcement learning is provided to solve the coupling problem of bidding and demand response decision-making.

Description

Technical Field

[0001] The present invention belongs to the technical field of power systems, and in particular, relates to a user incremental incentive method and an asynchronous optimization method for bidding and demand response decisions. Background Art

[0002] Demand response has become an important part of the smart grid, which aims to encourage users to adjust their electricity consumption and improve energy efficiency. Generally speaking, large industrial and commercial users are considered to be good candidates for demand response programs because each user can provide considerable response and the small number of users is conducive to the implementation of demand response. Residential users with large demand-side flexibility account for about 50% of peak load, but the utilization rate of their flexibility potential is less than 2%. Since each residential user can only provide limited response, it is necessary to aggregate a large number of consumers to form a considerable response capacity.

[0003] The power system operator has limited management capabilities for a large number of entities, and relatively small consumers cannot directly participate in the wholesale electricity market. Therefore, residential users must participate in the wholesale electricity market through the system operator. Generally speaking, in order to provide energy supply services to consumers in the electricity retail market, the system operator obtains electricity from the wholesale electricity market through bidding. In demand response, the system operator guides consumers to change their load consumption through incentives. When the cost incurred by consumers due to response can be matched, consumers participate in the response. According to the consumer behavior theory in microeconomics, the marginal cost of consumers will increase with the accumulation of responses, and the marginal costs of different consumers vary greatly. In order to meet the electricity needs of users, the system operator also needs to formulate a bidding strategy. In practice, bidding is usually carried out in the day-ahead market, while demand response is implemented during the day. The bidding strategy and demand response strategy affect each other, but in the day-ahead market bidding, the system operator usually cannot know the demand response incentive strategy in advance, so the system operator needs to make bidding decisions in an environment with incomplete information.

[0004] Existing technologies have deficiencies in both incentive mechanisms and optimization strategies. In terms of incentive mechanisms, the contradiction between making full use of consumer differences and computational efficiency in demand response has not been well resolved. Since the incremental incentive cost in the existing incentive mechanism is positively correlated with the total response of the consumer, the response depth of highly elastic consumers is limited. In addition, the uneven distribution of consumer surplus will also reduce the profits of system operators. In terms of optimization strategies, since the intraday demand response strategy and the day-ahead bidding strategy influence each other and are executed at different times in actual applications, it is necessary to formulate an optimization strategy with incomplete information for day-ahead bidding. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a user incremental incentive method and an asynchronous optimization method for bidding and demand response decision-making.

[0006] In a first aspect, the present invention provides a user incremental incentive method, comprising:

[0007] Consumers receive income based on the increment of demand response. Let the income received by consumers based on the increment of demand response be U i,t (R i,t ), the response of the i-th user at time t is R i,t , f(x) is the incremental activation function, then:

[0008]

[0009] Users purchase the electricity they need in each period from the system operator according to their electricity demand, and participate in demand response to increase their revenue. The system operator calculates the user's incremental incentive cost based on the i-th user's electricity purchase cost in time period t, the i-th user's comfort cost in time period t, and the user's incremental incentive cost, including:

[0010] Assume that the actual load of the i-th user at time t is The basic load of the i-th user at time t is The response of the i-th user at time t is R i,t ,but:

[0011]

[0012] set up is the load rebound of the ith consumer, ξ i,j is the load correlation coefficient of the ith consumer. The load correlation coefficient indicates the relationship between the load reduction in the previous time period and the load rebound in the next time period. The load consumption behavior of consumers is cyclical. The load rebound amount of consumers is expressed as:

[0013]

[0014] By analogy with the function of the income obtained by consumers according to the increment of demand response, we can get the function of the electricity purchase cost of the ith user in time period t. Let l i (R i,t ) is the comfort function of the ith consumer. The comfort function is a monotonically increasing function. The comfort cost of the ith user in time period t is but:

[0015]

[0016] set up is the time-of-use electricity price determined in advance by the system operator. The electricity purchase cost of the i-th user in time period t is but:

[0017]

[0018] The system operator specifies the demand response strategy based on the user's incremental incentive cost, including: i,t is the income of the ith consumer participating in demand response in each period, and the optimal action of the user at time t is converted into The solution of

[0019] In a second aspect, the present invention provides an asynchronous optimization method for bidding and demand response decision-making, wherein the system operator optimizes the bidding strategy in the wholesale power market and the demand response strategy in the retail power market according to the user incremental incentive cost, including:

[0020] In the wholesale electricity market, system operators purchase electricity through bidding. The bidding strategy is represented by a monotonically increasing function. Let the price of the purchased electricity be p t , L t is the system operator's expected output p in period t t The amount of electricity purchased at the price, α t represents the maximum purchase price of the system operator, β t is the slope of a monotonically increasing function, then:

[0021] p t =α t +β t L t ;

[0022] After the wholesale electricity market is cleared, electricity with a unit price higher than or equal to the wholesale electricity market clearing price will be traded. is the settlement price in the wholesale electricity market, and the cost of the system operator in the wholesale electricity market period t is In a competitive electricity market, where the clearing price in the wholesale electricity market is not affected by the bids of a single system operator, then:

[0023]

[0024] If the transaction volume If the power demand of users cannot be met, the system operator pays a penalty to the independent system operator responsible for compensating the load imbalance. Let C S,t For fines paid by system operators, then:

[0025]

[0026] Let L tS is the amount of electricity that the independent system operator replenishes after the system operator pays the penalty, and the power balance constraint is:

[0027]

[0028] Assuming there are M users in the system, the total load of users in any period should not exceed the maximum capacity of the transmission line, which is L. max ,Right now:

[0029]

[0030] Assume that the comprehensive profit obtained by the system operator from the wholesale electricity market and the retail electricity market in period t is U LSE,t , U i,t is the benefit obtained by the i-th consumer through participating in demand response in the t-th period, then:

[0031]

[0032] The optimization goal of the system operator is to maximize the cumulative profit over multiple periods throughout the day, namely:

[0033]

[0034] There is no negative incentive in the electricity retail market, f(R) is positive and monotonically increasing, and satisfies f(R)≥0,df(R≥0; in the electricity wholesale market, the bidding curve submitted by the system operator is monotonically decreasing, α t ≥0, β t ≤0;

[0035] The asynchronous dual-interaction DRL algorithm based on deep deterministic policy gradient is used to solve the coupling of bidding and demand response decisions. The coupling problem of bidding and demand response decisions is described as a Markov process. Let S be the state space, A be the action space, T be the transition probability between states, R be the reward function, γ be the discount factor, and the form of the Markov process is defined as a five-tuple M = (S, A, T, R, γ);

[0036] Since the wholesale electricity market is a day-ahead market, the information related to the wholesale electricity market obtained by the system operator includes the forecast price in the wholesale electricity market, the time-of-use electricity price, and the forecast load without demand response the next day. Suppose the forecast price in the wholesale electricity market is The time-of-use electricity price is The predicted load demand is The forecast load without demand response the next day is The bidding action is a W,t , the demand response action is a R,t , S W,trepresents the set of information available to the system operator before executing a bidding action in the wholesale electricity market, then:

[0037]

[0038] Demand response is implemented during the day, and the information related to the electricity retail market that the system operator can obtain includes the actual load and price in the electricity wholesale market before the current period, and the predicted load demand. Settlement prices in wholesale electricity markets Time-of-use electricity prices Let S R,t Represents the system operator's implementation of demand response actions in the electricity retail market. R,t The previous available formation set is expressed as:

[0039]

[0040] The system operator needs to decide on the bidding strategy in the wholesale electricity market, that is, to decide α t and β t The value of X t Represents all parameters of the incremental activation function, a t The action is defined as:

[0041] a t =(α t ,β t ,X t );

[0042] The reward r at time t t Defined as the profit earned by the system operator, then:

[0043]

[0044] After bidding and demand response actions are completed, the network is trained offline. Let G W Represents a sub-agent in the wholesale electricity market, G R Represents a subagent in the electricity retail market, with a bidding action of W,t , the reward value is r t , the market bidding strategy network is π w (S w，t ), the demand response strategy network is π R (S R，t ), in the day-ahead market, G W First, according to the market bidding strategy network πw(Sw, t), the bidding action a is executed in the wholesale electricity market. W,t , at time t-1, G R According to the demand response strategy network π R (S R, t) Implementing demand response actions in the electricity retail marketa R,t , calculate the reward value r t ;

[0045] Update parameters of the market bidding strategy network and the demand response strategy network according to the reward value;

[0046] The system operator executes the bidding strategy in the wholesale electricity market and the demand response strategy in the retail electricity market according to the updated market bidding strategy network and the demand response strategy network.

[0047] The beneficial effects of the present invention are as follows: the present invention considers the price design problem in the case of one retailer and many consumers, and proposes a flexibility mining method for differentiated users on the demand side; the present invention simultaneously considers the incremental demand response incentive mechanism of the wholesale electricity market and the retail electricity market, and proposes an asynchronous dual-interaction decision-making method based on deep reinforcement learning to solve the coupling problem of bidding and demand response decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the master-slave game optimization model between system operators and users;

[0049] Figure 2 Schematic diagram of the timeline of different entities participating in market decision-making. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Example 1

[0052] As an embodiment, in order to solve the above technical problems, this embodiment provides a user incremental incentive method, including:

[0053] In the wholesale electricity market, the system operator provides energy supply services and demand response programs to multiple users and bids for the required electricity from the wholesale electricity market, such as Figure 1 Schematic diagram of the master-slave game optimization model between the system operator and the user; each day is divided into T time periods, each time point is represented as t, the system operator provides power supply services to consumers and bids for the required electricity from the wholesale electricity market.

[0054] When the settlement price in the wholesale electricity market is higher than the set threshold, the system operator implements demand response to increase profits, and the user's electricity demand must be met in any case, otherwise the system operator will pay a penalty for the power imbalance;

[0055] Assume that the price response elasticity of consumers in demand response is E, R is the response made by the user, and I is the price of the incentive, then:

[0056]

[0057] Consumers receive income based on the increment of demand response. Let the income received by consumers based on the increment of response be U i,t (R i,t ), the response of the i-th user at time t is R i,t , f(x) is the incremental activation function, then:

[0058]

[0059] f(x) is monotonically increasing, so the unit benefit obtained by the consumer increases with the increase of the response depth. It should be noted that any function that meets the above requirements can be used as the incremental incentive function in the proposed mechanism, such as linear function, polynomial function and exponential function.

[0060] Users purchase the electricity they need in each period from the system operator according to their electricity demand, and participate in demand response to increase their revenue. The system operator calculates the user's incremental incentive cost based on the i-th user's electricity purchase cost in time period t, the i-th user's comfort cost in time period t, and the user's incremental incentive cost, including:

[0061] Assume that the actual load of the i-th user at time t is The basic load of the i-th user at time t is The base load of the ith user at time t is the baseline for calculating the user's demand response. The response of the ith user at time t is R i,t ,but:

[0062]

[0063] When users participate in demand response at time t, part of the load will be transferred to the subsequent time due to the influence of transferable loads (such as temperature control loads and electric vehicles). Since consumers' load consumption behavior is cyclical, load rebound has similar characteristics in the same period;

[0064] set up is the load rebound of the ith consumer, ξ i,jis the load correlation coefficient of the ith consumer. The load correlation coefficient indicates the relationship between the load reduction in the previous time period and the load rebound in the next time period. The load consumption behavior of consumers is cyclical. The load rebound amount of consumers is expressed as:

[0065]

[0066] In practical applications, the rebound load cannot be directly obtained For optimization, only the actual load including rebound load needs to be known.

[0067] Assume l i (R i,t ) is the comfort function of the ith consumer. When the load demand decreases, the marginal cost (i.e., the comfort loss of the consumer's unit response) increases. Therefore, l i (R i,t ) is a monotonically increasing function, whose form and parameters vary with the characteristics of the consumer. The electricity purchase cost of the i-th user in time period t is but:

[0068]

[0069] set up is the time-of-use electricity price determined in advance by the system operator. The electricity purchase cost of the i-th user in time period t is but:

[0070]

[0071] Let U i,t is the income of the ith consumer participating in demand response in each period, and the optimal action of the user at time t is converted into The solution of

[0072] The present invention considers the price design problem under the condition of one retailer and many consumers, and proposes a flexibility mining method for differentiated users on the demand side.

[0073] Example 2

[0074] Based on the same principle as the method shown in Embodiment 1 of the present invention, an asynchronous optimization method for bidding and demand response decision is also provided in the embodiment of the present invention, and the system operator optimizes the bidding strategy in the wholesale power market and the demand response strategy in the retail power market according to the user incremental incentive cost, including:

[0075] In the wholesale electricity market, system operators purchase electricity through bidding. The bidding strategy is represented by a monotonically increasing function. Let the price of the purchased electricity be p t , L tis the system operator's expected output p in period t t The amount of electricity purchased at the price, α t represents the maximum purchase price of the system operator, β t is the slope of a monotonically increasing function, then:

[0076] p t =α t +β t L t ;

[0077] After the wholesale electricity market is cleared, electricity with a unit price higher than or equal to the wholesale electricity market clearing price will be traded. is the settlement price in the wholesale electricity market, and the cost of the system operator in the wholesale electricity market period t is In a competitive electricity market, where the clearing price in the wholesale electricity market is not affected by the bids of a single system operator, then:

[0078]

[0079] If the transaction volume If the power demand of users cannot be met, the system operator pays a penalty to the independent system operator responsible for compensating the load imbalance. Let C S,t For fines paid by system operators, then:

[0080]

[0081] set up is the amount of electricity that the independent system operator replenishes after the system operator pays the penalty, and the power balance constraint is:

[0082]

[0083] Assuming there are M users in the system, the total load of users in any period should not exceed the maximum capacity of the transmission line, which is L. max ,Right now:

[0084]

[0085] Assume that the comprehensive profit obtained by the system operator from the wholesale electricity market and the retail electricity market in period t is U LSE,t , U i,t is the benefit obtained by the i-th consumer through participating in demand response in the t-th period, then:

[0086]

[0087] The optimization goal of the system operator is to maximize the cumulative profit over multiple periods throughout the day, namely:

[0088]

[0089] There is no negative incentive in the electricity retail market, f(R) is positive and monotonically increasing, and satisfies f(R)≥0,df(R≥0; in the electricity wholesale market, the bidding curve submitted by the system operator is monotonically decreasing, α t ≥0, β t ≤0;

[0090] Although bidding and demand response decisions are interrelated, in practice they are executed at different times. Figure 2 As shown, bidding decisions are made in the day-ahead market, while demand response is executed intraday.

[0091] In the day-ahead market, the system operator submits bids for each period of the next day to the independent system operator, and then the independent system operator clears the wholesale electricity market based on the bids / offers of buyers and sellers to derive wholesale electricity prices and the trading volume of each participant. Before the tth period of the day, the system operator announces the incentives for demand response in the next period to consumers, and consumers respond accordingly.

[0092] The coupling problem of bidding and demand response can be transformed into a two-layer optimization problem, where the upper layer is the bidding problem, and the settlement price and electricity are used as inputs to the lower layer demand response problem. In addition, the optimization results of the lower layer will also affect the decisions of the upper layer. However, the parameters of the comfort loss function for each consumer are difficult to obtain, so the optimization decision cannot be directly calculated.

[0093] Reinforcement learning can find the optimal solution in a model-free environment. Bidding and demand response are asynchronous decision-making processes. The training process of the deep reinforcement learning algorithm is offline. The network can be trained using complete historical information after both bidding and demand response are completed. In addition, thanks to the learning and memory capabilities of neural networks, deep reinforcement learning can infer the possible optimal results of demand response when information is incomplete, and make the optimal bidding decision based on this. Considering that the action space of bidding and demand response is continuous, this patent uses an asynchronous dual-interaction DRL (Deep Reinforcement Learning) algorithm based on deep deterministic policy gradient (DDPG) to solve the coupling problem.

[0094] The asynchronous dual-interaction DRL algorithm based on deep deterministic policy gradient is used to solve the coupling of bidding and demand response decisions. The coupling problem of bidding and demand response decisions is described as a Markov process. Let S be the state space, A be the action space, T be the transition probability between states, R be the reward function, γ be the discount factor, and the form of the Markov process is defined as a five-tuple M = (S, A, T, R, γ);

[0095] Since the wholesale electricity market is a day-ahead market, the information related to the wholesale electricity market obtained by the system operator includes the forecast price in the wholesale electricity market, the time-of-use electricity price, and the forecast load without demand response the next day. Suppose the forecast price in the wholesale electricity market is The time-of-use electricity price is The predicted load demand is The forecast load without demand response the next day is The bidding action is a W,t , the demand response action is a R,t , S W,t represents the set of information available to the system operator before executing a bidding action in the wholesale electricity market, then:

[0096]

[0097] Demand response is implemented during the day, and the information related to the electricity retail market that the system operator can obtain includes the actual load and price in the electricity wholesale market before the current period, and the predicted load demand. Settlement prices in wholesale electricity markets Time-of-use electricity prices Let S R,t Represents the system operator's implementation of demand response actions in the electricity retail market. R,t The previous available formation set is expressed as:

[0098]

[0099] The system operator needs to decide on the bidding strategy in the wholesale electricity market, that is, to decide α t and β t The value of X t Represents all parameters of the incremental activation function, a t The action is defined as:

[0100] a t =(α t ,β t ,X t );

[0101] The reward r at time t t Defined as the profit earned by the system operator, then:

[0102]

[0103] Although bidding and demand response are executed sequentially, these two problems are coupled. The formulation of bidding strategies requires the optimization results of demand response, which cannot be known in advance. In addition, the reward is calculated jointly by bidding and demand response, that is, the reward cannot be obtained immediately after the bidding behavior is completed. In order to solve the above problems, the present invention proposes a solution based on asynchronous and deep reinforcement learning algorithms, that is, the network is trained offline after both the bidding action and the demand response action are completed.

[0104] After bidding and demand response actions are completed, the network is trained offline. Let G W Represents a sub-agent in the wholesale electricity market, G R Represents a subagent in the electricity retail market, with a bidding action of W,t , the reward value is r t , the market bidding strategy network is π w (S w，t ), the demand response strategy network is π R (S R，t ), in the day-ahead market, G W First, according to the market bidding strategy network πw(Sw, t), the bidding action a is executed in the wholesale electricity market. W,t , at time t-1, G R According to the demand response strategy network π R (S R , t) Implementing demand response actions in the electricity retail marketa R,t , calculate the reward value r t ;

[0105] Update parameters of the market bidding strategy network and the demand response strategy network according to the reward value;

[0106] The system operator executes the bidding strategy in the wholesale electricity market and the demand response strategy in the retail electricity market according to the updated market bidding strategy network and the demand response strategy network.

[0107] In the deep reinforcement learning algorithm, the interactions between sub-agents in the wholesale electricity market and sub-agents in the retail electricity market and the environment are coupled, the network training is independent, and the actual load is affected by the demand response. The demand response agent is trained first, and then the demand response agent is used as the environment to participate in the training of the bidding agent.

[0108] The present invention simultaneously considers the incremental demand response incentive mechanism of the electricity wholesale market and the electricity retail market, and proposes an asynchronous dual-interaction decision-making method based on deep reinforcement learning to solve the coupling problem of bidding and demand response decision-making.

[0109] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A user incremental incentive method, characterized in that: include: Consumers receive income based on the increment of demand response. Let the income received by consumers based on the increment of demand response be U i,t (R i,t ), the response of the i-th user at time t is R i,t , f(x) is the incremental activation function, then: Users purchase the electricity they need in each period from the system operator according to their electricity demand, and participate in demand response to increase their revenue. The system operator calculates the user's incremental incentive cost based on the i-th user's electricity purchase cost in time period t, the i-th user's comfort cost in time period t, and the user's incremental incentive cost, including: Assume that the actual load of the i-th user at time t is The basic load of the i-th user at time t is The response of the i-th user at time t is R i,t ,but: set up is the load rebound of the ith consumer, ξ i,j is the load correlation coefficient of the ith consumer. The load correlation coefficient indicates the relationship between the load reduction in the previous time period and the load rebound in the next time period. The load consumption behavior of consumers is cyclical. The load rebound amount of consumers is expressed as: By analogy with the function of the income obtained by consumers according to the increment of demand response, we can get the function of the electricity purchase cost of the ith user in time period t. Let l i (R i,t ) is the comfort function of the ith consumer. The comfort function is a monotonically increasing function. The comfort cost of the ith user in time period t is but: set up is the time-of-use electricity price determined in advance by the system operator. The electricity purchase cost of the i-th user in time period t is but: The system operator specifies the demand response strategy based on the user's incremental incentive cost, including: i,t is the income of the ith consumer participating in demand response in each period, and the optimal action of the user at time t is converted into The solution of 2. Asynchronous optimization method for bidding and demand response decision making, characterized in that: System operators optimize bidding strategies in wholesale electricity markets and demand response strategies in retail electricity markets based on incremental incentive costs for users, including: In the wholesale electricity market, system operators purchase electricity through bidding. The bidding strategy is represented by a monotonically increasing function. Let the price of the purchased electricity be p t , L t is the system operator's expected output p in period t t The amount of electricity purchased at the price, α t represents the maximum purchase price of the system operator, β t is the slope of a monotonically increasing function, then: p t =a t +b t L t ; After the wholesale electricity market is cleared, electricity with a unit price higher than or equal to the wholesale electricity market clearing price will be traded. is the settlement price in the wholesale electricity market, and the cost of the system operator in the wholesale electricity market period t is In a competitive electricity market, where the clearing price in the wholesale electricity market is not affected by the bids of a single system operator, then: If the transaction volume If the power demand of users cannot be met, the system operator pays a penalty to the independent system operator responsible for compensating the load imbalance. Let C S,t For fines paid by system operators, then: set up is the amount of electricity that the independent system operator replenishes after the system operator pays the penalty, and the power balance constraint is: Assuming there are M users in the system, the total load of users in any period should not exceed the maximum capacity of the transmission line, which is L. max ,Right now: Assume that the comprehensive profit obtained by the system operator from the wholesale electricity market and the retail electricity market in period t is U LSE,t , U i,t is the benefit obtained by the i-th consumer through participating in demand response in the t-th period, then: The optimization goal of the system operator is to maximize the cumulative profit over multiple periods throughout the day, namely: There is no negative incentive in the electricity retail market, f(R) is positive and monotonically increasing, and satisfies f(R)≥0,df(R≥0; in the electricity wholesale market, the bidding curve submitted by the system operator is monotonically decreasing, α t ≥0, β t ≤0; The asynchronous dual-interaction DRL algorithm based on deep deterministic policy gradient is used to solve the coupling of bidding and demand response decisions. The coupling problem of bidding and demand response decisions is described as a Markov process. Let S be the state space, A be the action space, T be the transition probability between states, R be the reward function, γ be the discount factor, and the form of the Markov process is defined as a five-tuple M = (S, A, T, R, γ); Since the wholesale electricity market is a day-ahead market, the information related to the wholesale electricity market obtained by the system operator includes the forecast price in the wholesale electricity market, the time-of-use electricity price, and the forecast load without demand response the next day. Suppose the forecast price in the wholesale electricity market is The time-of-use electricity price is The predicted load demand is The forecast load without demand response the next day is The bidding action is a W,t , the demand response action is a R,t , S W,t represents the set of information available to the system operator before executing a bidding action in the wholesale electricity market, then: Demand response is implemented during the day, and the information related to the electricity retail market obtained by the system operator includes the actual load and price in the electricity wholesale market before the current period, and the predicted load demand. Settlement prices in wholesale electricity markets Time-of-use electricity prices Let S R,t Represents the system operator's implementation of demand response actions in the electricity retail market. R,t The previously available set of formations is represented as: The system operator needs to decide on the bidding strategy in the wholesale electricity market, that is, to decide α t and β t The value of X t Represents all parameters of the incremental activation function, a t The action is defined as: a t =(a t ,b t ,X t ); The reward r at time t t Defined as the profit earned by the system operator, then: After bidding and demand response actions are completed, the network is trained offline. Let G W Represents a sub-agent in the wholesale electricity market, G R Represents a subagent in the electricity retail market, with a bidding action of W,t , the reward value is r t , the market bidding strategy network is π w (S w，t ), the demand response strategy network is π R (S R，t ), in the day-ahead market, G W First, according to the market bidding strategy network πw(Sw, t), the bidding action a is executed in the wholesale electricity market. W,t , at time t-1, G R According to the demand response strategy network π R (S R , t) Implementing demand response actions in the electricity retail marketa R,t , calculate the reward value r t ; Update parameters of the market bidding strategy network and the demand response strategy network according to the reward value; The system operator executes the bidding strategy in the wholesale electricity market and the demand response strategy in the retail electricity market according to the updated market bidding strategy network and the demand response strategy network.

Images