Generator bidding decision-making method based on improved MA-DDQN

Abstract

The invention relates to the technical field of electric power system decision making, in particular to a power generator bidding decision making method based on improved MA-DDQN, and aims to optimize a bidding strategy of a power generator by modeling the power generator into a plurality of agents and utilizing an MA-DDQN model so as to maximize income. And the MA-DDQN model is used to optimize the bidding strategy so as to maximize the income. Secondly, through interaction with an independent system operator (ISO), each agent submits a quoted price and adjusts a strategy according to feedback, and the model continuously learns in a simulated bidding process; and finally, based on the method, the power generator can realize intelligent decision making in the dynamic power market, so that the market competitiveness of the power generator is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system decision-making, and in particular to a power producer bidding decision method based on improved MA-DDQN. Background Art

[0002] In the electricity market, the bidding decision of power generators is a key link that affects market efficiency and power generator revenue. Traditional bidding decision methods for power generators are usually based on the following:

[0003] Bidding decision-making method based on game theory: Under the framework of game theory, power generators formulate their own bidding strategies by analyzing the strategies and market status of other market participants. Classic game models include Nash equilibrium, Stark game, etc. However, game theory models usually assume that participants are completely rational and can obtain each other's decision-making information. In the actual power market, information is incomplete and asymmetric, and the market environment is complex, resulting in certain limitations in the practical application of game models, making it difficult to effectively deal with the uncertainty and dynamism of the market.

[0004] Optimization methods based on linear programming or mixed integer programming: This method establishes a mathematical optimization model, takes the maximization of power generators' revenue or minimization of costs as the goal, considers constraints such as power generation costs, market prices, and load demand, and solves the optimal bidding solution. However, since the power market is a highly dynamic and uncertain environment, traditional optimization methods have difficulty dealing with complex constraints and nonlinear characteristics in the market. At the same time, as the market size increases, the computational complexity increases rapidly, making it difficult to obtain the optimal solution within a reasonable time.

[0005] Bidding strategies based on machine learning: In recent years, with the development of artificial intelligence technology, machine learning methods have gradually been applied to bidding decisions of power producers. For example, reinforcement learning methods such as Q-learning and deep Q network (DQN) can gradually learn the optimal bidding strategy through continuous interaction with the environment. However, traditional DQN has limitations in multi-agent scenarios and it is difficult to effectively handle the collaboration and competition among multiple power producers. At the same time, the training process of DQN responds slowly to environmental changes, and the strategy has poor adaptability when facing a dynamic market environment.

[0006] In summary, when dealing with power market bidding decisions, existing technologies either fail to cope with market uncertainty and nonlinearity or have difficulty in dealing with multi-agent competition and collaboration issues, resulting in less than ideal bidding strategy results. Summary of the invention

[0007] In view of this, the purpose of the present invention is to propose a bidding decision method for power producers based on improved MA-DDQN to solve the problem that the bidding strategy given by the existing method is not ideal.

[0008] Based on the above purpose, the present invention provides a bidding decision method for power producers based on improved MA-DDQN, comprising the following steps:

[0009] S1. Model multiple power generators as multiple independent agents and construct a multi-agent DDQN model. The multi-agent DDQN model includes agents, environments, states, and actions.

[0010] S2. Build the Online Network and Target Network in the multi-agent DDQN model and initialize the network state;

[0011] S3, input the current state s, and calculate the Q value corresponding to each action;

[0012] S4. Calculate the dynamic probability ε and determine whether it is greater than or equal to the random number random. If so, select action a with the largest Q value, otherwise randomly select action a;

[0013] S5. Submit the action to the independent system operator and calculate the current reward r based on action a;

[0014] S6. Execute action a, update state s, obtain the next state s', and store (s, a, r, s') in the experience pool;

[0015] S7, repeat steps S3-S6 for a preset M times;

[0016] S8. Randomly extract samples from the experience pool to train the DDQN network, calculate the loss function, and perform gradient descent, updating the TargetNetwork every N times;

[0017] S9, repeat steps S3-S8 until the upper limit of the training times is reached, and obtain the final selected action, that is, obtain the power generator quotation result.

[0018] Preferably, in a multi-agent DDQN model;

[0019] The environment is the entire power market, and ISO is the feedback provider of the environment. ISO clears the market based on the bidding curves submitted by all power generators, and provides feedback to all power generators on the total system load level, the power generation volume that the power generator won the bid for, and the node electricity price.

[0020] The state is that when calculating the bid coefficient of the power generator in a certain period, based on the historical data of the previous bidding, the system load and node electricity price of the previous period, the system load and node electricity price of the current period, a total of four known quantities;

[0021] Action is the three-stage bidding coefficient k selected by generator i in time period h hin , where n is 1 or 2 or 3;

[0022] The reward is the profit f calculated by clearing the generator Gi ;

[0023]

[0024] Among them, f Gi is the revenue of generator i, C i is the fuel cost function of generator i, λ h is the clearing price of G generators at time h, P h Gi is the electricity cleared by generator i at time h, is the electricity reported by generator i in the nth period at time h, where a i , b i are the coefficients of the quadratic and linear terms of the cost of generator i.

[0025] Preferably, in step S2, initializing the network state includes:

[0026] Initialize two network parameters, the network parameter of Online Network is recorded as θ, and the network parameter of TargetNetwork is recorded as θ'. Initialize the state s = (0, 0, λ 0 , P 0 Gi ), set the system load and node electricity price of the previous period, the system load and node electricity price of the current period to the initial values, where λ 0 is the clearing price of G generators at time 0, P 0 Gi is the power cleared by generator i at time 0.

[0027] Preferably, in step S3, inputting the current state s, and calculating the Q value corresponding to each action includes:

[0028] Input the current state s and calculate the Q value Q corresponding to each action a at time n n (s,a|θ).

[0029] Preferably, in step S4, the dynamic probability ε is calculated according to the following formula:

[0030]

[0031] T k =δ k *T0

[0032] Q(s,a r ) is the action a selected according to the uniform random strategy rThe corresponding Q value, Q(s,a g ) is the maximum Q value in the current environment, T k is the temperature corresponding to the kth simulated annealing process, T0 is the initial temperature, and δ is the cooling coefficient.

[0033] Preferably, in step S6, executing action a, updating state s, and obtaining the next moment state s' includes: updating h to the next moment, and the ISO calculates the current node electricity price λ according to the clearing model h and the clearing power P h Gi

[0034] s'=(λ h-1 , P h-1 Gi ,λ h , P h Gi ).

[0035] Preferably, in step S8, the formula for calculating the loss function is:

[0036] L(θ)=([r n +γmaxQ n (s n+1 ,a′|θ′)]-Q n (s,a|θ) 2 .

[0037] Among them, L(θ) represents the loss function, r n represents the reward at the nth moment, γ represents the discount factor, θ represents the network parameter of OnlineNetwork, θ' represents the network parameter of TargetNetwork, Q n (s,a|θ) is the Q value corresponding to action a at time n, Q n (s n+1 ,a'|θ') is at time n+1, when the state is s n+1 Next, select the Q value of the TargetNetwork network corresponding to action a'.

[0038] Beneficial effects of the present invention: The present invention can help power generators learn the optimal bidding strategy by simulating multiple rounds of bidding processes through the application of the MA-DDQN model in power generator bidding decision scenarios. By using the power generator's profit function as a reward mechanism, MA-DDQN can gradually adjust the bidding behavior of each intelligent agent to improve its profit in a complex market environment. At the same time, the introduction of the dual network structure (Online Network and TargetNetwork) can further improve the stability of the model and reduce oscillation and over-estimation problems during the strategy update process. This MA-DDQN-based power generator bidding decision method provides power generators with an intelligent bidding tool, which is expected to be widely used in future power markets. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of a bidding decision method for power producers based on improved MA-DDQN according to an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of the interaction between the intelligent agent and ISO according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0043] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0044] like Figure 1 As shown, the embodiment of this specification provides a bidding decision method for power producers based on improved MA-DDQN, including the following steps:

[0045] S1. Model multiple power generators as multiple independent agents and construct a multi-agent DDQN model (Multi-agent Double Deep Q-Network, MA-DDQN). The multi-agent DDQN model includes agents, environments, states, and actions.

[0046] S2. Build the Online Network and Target Network in the multi-agent DDQN model and initialize the network state;

[0047] S3, input the current state s, and calculate the Q value corresponding to each action;

[0048] S4. Calculate the dynamic probability ε and determine whether it is greater than or equal to the random number random. If so, select action a with the largest Q value, otherwise randomly select action a;

[0049] S5. Submit the action to the Independent System Operator (ISO) and calculate the current reward r based on action a.

[0050] S6. Execute action a, update state s, obtain the next state s', and store (s, a, r, s') in the experience pool;

[0051] S7, repeat steps S3-S6 for a preset M times;

[0052] S8. Randomly extract samples from the experience pool to train the DDQN network, calculate the loss function, and perform gradient descent, updating the TargetNetwork every N times;

[0053] S9. Repeat steps S3-S8 until the upper limit of the training times is reached, and obtain the final selected action, that is, obtain the power generator's quotation result, where the quotation result refers to the quotation coefficient.

[0054] For step S1: MA-DDQN model includes agent, environment, state and action

[0055] Agent: Multiple power generators in the market are modeled as multiple independent agents. In this example, there are four power generators, and their information is shown in the following table:

[0056] Table 1 Power generator information

[0057]

[0058] The first-order coefficients of units 1 and 4 are small, and they are low-cost generators. The first-order coefficients of units 2 and 3 are large. Power generators 1, 2, 3, and 4 are selected as intelligent agents.

[0059] Environment: The entire power market can be considered as the external environment, and ISO is the feedback provider of the environment. ISO clears the market based on the bidding curves submitted by all power generators, and provides feedback to all power generators on the total system load level, the power generation volume won by the power generator, and the node electricity price.

[0060] Status: When calculating the bid coefficient of a power producer in a certain period, based on the historical data of the previous bidding, the system load and node electricity price in the previous period, the system load and node electricity price in the current period, a total of four known quantities are taken as the current status;

[0061] Action: Set the three-stage quotation coefficient k selected by generator i in period h h in As an action, n takes 1, 2, or 3;

[0062] Reward: The purpose of the power generator's strategic bidding is to increase its profit, so the profit f calculated by the power generator clearing is considered Gi To represent rewards.

[0063]

[0064] Among them, f Gi is the revenue of generator i; C i is the fuel cost function of generator i; h is the clearing price of G generators at time h; P h Gi is the electricity cleared by generator i at time h; k h in is the bid coefficient of generator i at time h, and n is 1, 2 or 3; is the electricity reported by generator i in the nth period at time h. i , b i are the coefficients of the quadratic and linear terms of the cost of generator i.

[0065] For step S2: construct Online Network and TargetNetwork, initialize two network parameters, the network parameter of Online Network is recorded as θ, and the network parameter of TargetNetwork is recorded as θ'. Initialization state s=(0, 0, λ 0 , P 0 Gi), set the system load and node electricity price of the previous period, the system load and node electricity price of the current period to the initial values, where λ 0 is the clearing price of G generators at time 0, P 0 Gi is the power cleared by generator i at time 0.

[0066] For step S3: input the current state s, calculate the Q value Q corresponding to each action a at time n n (s,a|θ) For step S4: Calculate the dynamic probability ε according to the following formula

[0067]

[0068] T k =δ k *T0

[0069] Q(s,a r ) is the action a selected according to the uniform random strategy r The corresponding Q value; Q(s,a g ) is the maximum Q value in the current environment; T k is the temperature corresponding to the kth simulated annealing process; T0 is the initial temperature; δ is the cooling coefficient.

[0070] Determine whether ε is greater than or equal to the random number random. If so, select the action with the largest Q value. Otherwise, select the action randomly, that is, select the three-stage quotation coefficient k. h in

[0071] For step S5: Figure 2 As shown, submit the action to ISO, and calculate the current reward r based on action a.

[0072] Select the three-stage quotation coefficient k according to action a h in , the profit f calculated by the formula Gi To represent rewards.

[0073]

[0074]

[0075] For step S6: h is updated to the next moment, and ISO calculates the current node electricity price λ according to the clearing model h and the clearing power P h Gi

[0076] s'=(λ h-1 , P h-1 Gi ,λh , P h Gi )

[0077] For step S7: store (s, a, r, s') into the experience pool, and repeat steps 3-7 for a preset M times

[0078] For step S8: randomly extract some samples from the experience pool to train the DDQN network, calculate the loss function, and perform gradient descent, and update the network parameters θ′ of TargetNetwork every N times

[0079] L(θ)=([r n +γmaxQ n (s n+1 ,a′|θ′)]-Q n (s,a|θ) 2

[0080]

[0081] Among them, L(θ) represents the loss function, r n represents the reward at the nth moment, γ represents the discount factor, θ represents the network parameter of OnlineNetwork, θ' represents the network parameter of TargetNetwork, Q n (s,a|θ) is the Q value corresponding to action a at time n, Q n (s n+1 ,a'|θ') is at time n+1, when the state is s n+1 Next, select the Q value of the TargetNetwork network corresponding to action a'.

[0082] For step S9: repeat steps 3-8 to reach the upper limit of the number of training times. After reaching the upper limit, the power supplier's quotation result is obtained, as shown in the following Table 2, where the quotation result refers to the quotation coefficient, which is the second column in Table 2.

[0083] Table 2. Quotation results of power producers without considering collusion

[0084]

[0085] The results of quoting according to marginal cost are also given in brackets for comparison. As can be seen from the table, compared with the results of quoting completely according to marginal cost, the quotations of the four power generators have improved to a certain extent, driving the node electricity price of the entire network to increase by 8.3%. From the perspective of revenue, after using the algorithm of the present invention to make a quotation, the revenue of each power generator has also increased significantly, which proves the effectiveness of the responsible bidding strategy algorithm in the two-stage bidding algorithm proposed by the present invention.

[0086] Those skilled in the art should understand that the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present invention is limited to these examples; under the concept of the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A bidding decision method for power producers based on improved MA-DDQN, characterized in that: The method comprises the following steps: S1. Model multiple power generators as multiple independent agents and construct a multi-agent DDQN model. The multi-agent DDQN model includes agents, environments, states, and actions. S2. Build the Online Network and Target Network in the multi-agent DDQN model and initialize the network state; S3, input the current state s, and calculate the Q value corresponding to each action; S4. Calculate the dynamic probability ε and determine whether it is greater than or equal to the random number random. If so, select action a with the largest Q value, otherwise randomly select action a; S5. Submit the action to the independent system operator and calculate the current reward r based on action a; S6. Execute action a, update state s, obtain the next state s', and store (s, a, r, s') in the experience pool; S7, repeat steps S3-S6 for a preset M times; S8. Randomly extract samples from the experience pool to train the DDQN network, calculate the loss function, and perform gradient descent, updating the TargetNetwork every N times; S9, repeat steps S3-S8 until the upper limit of the training times is reached, and obtain the final selected action, that is, obtain the power generator quotation result.

2. The power supplier bidding decision method based on improved MA-DDQN according to claim 1 is characterized in that: In the multi-agent DDQN model; The environment is the entire power market, and ISO is the feedback provider of the environment. ISO clears the market based on the bidding curves submitted by all power generators, and provides feedback to all power generators on the total system load level, the power generation volume that the power generator won the bid for, and the node electricity price. The state is that when calculating the bid coefficient of the power generator in a certain period, based on the historical data of the previous bidding, the system load and node electricity price of the previous period, the system load and node electricity price of the current period, a total of four known quantities; Action is the three-stage bidding coefficient k selected by generator i in time period h h in , where n is 1 or 2 or 3; The reward is the profit f calculated by clearing the generator Gi ; Among them, f Gi is the revenue of generator i, C i is the fuel cost function of generator i, λ h is the clearing price of G generators at time h, P h Gi is the electricity cleared by generator i at time h, is the electricity reported by generator i in the nth period at time h, where a i 、b i are the coefficients of the quadratic and linear terms of the cost of generator i.

3. The power generator bidding decision method based on improved MA-DDQN according to claim 1 is characterized in that: In step S2, initializing the network state includes: Initialize two network parameters, the network parameter of Online Network is recorded as θ, and the network parameter of TargetNetwork is recorded as θ'. Initialize the state s = (0, 0, λ 0 , P 0 Gi ), set the system load and node electricity price of the previous period, the system load and node electricity price of the current period to the initial values, where λ 0 is the clearing price of G generators at time 0, P 0 Gi is the power cleared by generator i at time 0.

4. The power generator bidding decision method based on improved MA-DDQN according to claim 1 is characterized in that: In step S3, the current state s is input, and the Q value corresponding to each action is calculated including: Input the current state s and calculate the Q value Q corresponding to each action a at time n n (s,a|θ).

5. The power generator bidding decision method based on improved MA-DDQN according to claim 1 is characterized in that: In step S4, the dynamic probability ε is calculated according to the following formula: T k =δ k *T0 Q(s,a r ) is the action a selected according to the uniform random strategy r The corresponding Q value, Q(s,a g ) is the maximum Q value in the current environment, T k is the temperature corresponding to the kth simulated annealing process, T0 is the initial temperature, and δ is the cooling coefficient.

6. The power supplier bidding decision method based on improved MA-DDQN according to claim 1 is characterized in that: In step S6, action a is executed to update state s, and the state s' at the next moment is obtained, which includes: updating h to the next moment, and calculating the current node electricity price λ according to the clearing model of ISO h and the clearing power P h Gi s'=(λ h-1 ,P h-1 Gi ,l h ,P h Gi )。 7. The power generator bidding decision method based on improved MA-DDQN according to claim 1 is characterized in that: In step S8, the formula for calculating the loss function is: L(θ)=([r n +γmaxQ n (s n+1 ,a′∣θ′)]-Q n (s,a∣θ)) 2 ; Among them, L(θ) represents the loss function, r n represents the reward at the nth moment, γ represents the discount factor, θ represents the network parameter of the Online Network, θ' represents the network parameter of the TargetNetwork, Q n (s,a|θ) is the Q value corresponding to action a at time n, Q n (s n+1 ,a'|θ') is at time n+1, when the state is s n+1 Next, select the Q value of the TargetNetwork network corresponding to action a'.

Images