Electric power spot market participant decision-making method and device based on deep reinforcement learning

Abstract

The invention discloses an electric power spot market participant decision-making method and device based on deep reinforcement learning, and the method comprises the steps: building a new energy output prediction model considering the uncertainty, and determining a system operation constraint corresponding to a contract decomposition variable; constructing a risk measurement model and an income calculation model according to the contract decomposition variable, and determining a risk-income adaptive coupling coefficient; constructing a risk-income self-adaptive decomposition optimization model according to the risk measurement model, the income calculation model and the risk-income self-adaptive coupling coefficient; and carrying out optimization solution on the risk-income adaptive decomposition optimization model through a whale swarm algorithm improved based on chaotic disturbance to obtain an optimal solution of a contract decomposition variable, and then carrying out electric power spot market participant decision-making based on deep reinforcement learning according to the optimal solution. The method improves the accuracy, flexibility and real-time performance of decision making of the participants in the electric power spot market based on deep reinforcement learning, also improves the stability of the new energy electric power market, and can be applied to the technical field of electric power markets.

Description

Technical Field

[0001] This invention relates to the field of electricity market technology, and in particular to a decision-making method and apparatus for electricity spot market participants based on deep reinforcement learning. Background Technology

[0002] With the advancement of market-oriented reforms in the electricity spot market, the decision-making behavior of market participants (such as power generators and electricity retailers) directly impacts their returns and market stability. While research on optimizing decision-making in the electricity spot market has made some progress, the following shortcomings still exist: Insufficient dynamic adaptability: Existing decision-making models are mostly based on static optimization or rule-based reasoning (such as fixed pricing strategies and historical data fitting), failing to fully consider real-time dynamic changes in market prices and supply-demand relationships. For example, traditional methods use intraday static pricing, which cannot respond to real-time electricity price fluctuations, resulting in revenue loss.

[0003] Lack of characterization of game theory relationships: The electricity spot market involves dynamic games (such as bidding competition and strategy interaction) among multiple participants (power generators, retailers, and users). Existing research often assumes "perfect competition" in the market, neglecting the strategic interactions among participants, leading to a disconnect between decision-making models and actual market operation. For example, independently optimizing a single party's bidding strategy without considering the impact of competitors' bid adjustments on its own revenue results in strategy failure.

[0004] Limitations of uncertainty handling: Uncertain factors such as market prices, load demand, and power generation costs significantly affect decision-making effectiveness. Although traditional methods (such as Monte Carlo simulation and robust optimization) can describe uncertainty, they suffer from high computational complexity and poor real-time performance, and cannot meet the high-frequency decision-making needs of the spot market with "daily settlement and monthly clearing".

[0005] Weak generalization ability of algorithms: When faced with diverse market scenarios (such as peak and off-peak periods, extreme weather), existing reinforcement learning methods (such as basic DQN) have the defects of slow convergence and easy overfitting. The decision accuracy drops significantly in new scenarios, which limits the practical application value.

[0006] In summary, existing decision-making schemes for electricity spot market participants lack accuracy and real-time performance, which affects the stability of the electricity market. Summary of the Invention

[0007] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0008] Therefore, one objective of this invention is to provide a decision-making method for electricity spot market participants based on deep reinforcement learning, which improves the accuracy and real-time performance of electricity spot market participants' decisions and also enhances the stability of the electricity market.

[0009] Another objective of this invention is to provide a decision-making device for electricity spot market participants based on deep reinforcement learning.

[0010] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of the present invention include: On one hand, embodiments of the present invention provide a decision-making method for electricity spot market participants based on deep reinforcement learning, including the following steps: Spatial modeling of market state and participant actions is performed, and a state transition function is established based on market clearing rules to obtain a participant decision-making environment model; Based on the participant decision-making environment model, a participant decision-making optimization model based on deep reinforcement learning is constructed. The relative value of actions is quantified based on the advantage function, and the objective function is updated based on the advantage function to determine the strategy. Determine the basic revenue item, the game-theoretic synergy item, and the risk penalty item, and construct a reward function that integrates market game theory and uncertainty based on the basic revenue item, the game-theoretic synergy item, and the risk penalty item; The participant decision optimization model is optimized based on the adaptive exploration rate and real-time decision-making mechanism to obtain the optimal decision strategy, and electricity trading is carried out according to the optimal decision strategy.

[0011] Furthermore, in one embodiment of the present invention, the participant decision-making environment model includes a market state space model, a participant action space model, and the state transition function, wherein the market state space model is:

[0012] in, Let represent the market state vector at time t. This represents the spot market clearing price at time t. This represents the total system load demand at time t. This represents the total reported power generation of other participants at time t. This represents the participant's own marginal cost of generating electricity. Represents a historical price series. Indicates weather conditions; The participant's action space model is as follows:

[0013]

[0014]

[0015] in, This represents the decision action at time t. This indicates the price declared by the power supplier. and These represent the lower limit and upper limit of the market price, respectively. This indicates the electricity volume declared by the power generator. and These represent the minimum and maximum generating capacity, respectively. The state transition function is:

[0016] in, This represents the market state vector at time t+1. This indicates the decision-making actions of other participants. This indicates a random perturbation. This represents the state transition function.

[0017] Furthermore, in one embodiment of the present invention, the participant decision optimization model includes a policy network and a value network. The policy network is used to input the market state vector and output an action probability distribution, and the value network is used to input the market state vector and output a state value. The advantage function is:

[0018] in, Represents the market state vector Next action The relative value, Represents the market state vector Next action The value of the action, Represents the market state vector The corresponding state value; The objective function for updating the strategy is:

[0019] in, This indicates the policy update objective function. This represents the new strategy parameters to be optimized. This represents the old strategy parameters before the update. Represents the market state vector New strategy selection action The probability, Represents the market state vector Choose action from old strategy The probability, Indicates will Limited to Inside, =0.2 is the range for clip. This represents the expectation operator.

[0020] Furthermore, in one embodiment of the present invention, the basic benefit item is:

[0021] in, This represents the basic revenue term at time t. Let represent the clearing price at time t. This represents the electricity cost at time t. This represents the actual trading volume at time t; The game-theoretic collaborative term is:

[0022] in, Denotes the collaborative terms in the game at time t. Indicates the game weights, This indicates the bids from other participants. Represents the sensitivity coefficient. Represents the natural exponential function; The risk penalty items are as follows:

[0023] in, This represents the risk penalty term at time t. This represents the risk aversion coefficient. This represents the price variance over a period of time before and after time t. The reward function is:

[0024] in, This represents the action reward at time t.

[0025] Furthermore, in one embodiment of the present invention, the optimization of the participant decision optimization model based on the adaptive exploration rate and real-time decision-making mechanism specifically includes: Initialize the policy function and value function of the participant decision optimization model; The current policy function is executed in the decision-making environment, and a random action or the optimal action is selected as the current action based on an adaptive exploration rate mechanism. Calculate the action reward corresponding to the current action; Calculate the advantage function based on the current value function and the action reward; The policy function is updated based on the advantage function and the policy update objective function, and the value function is updated by minimizing the mean squared error using gradient descent.

[0026] Furthermore, in one embodiment of the present invention, a random action or an optimal action is selected as the current action according to the following formula:

[0027] in, The exploration rate at time t is represented by... The probability of selecting a random action as the current action is used. The probability of selecting the optimal action as the current action is given, where t represents the number of iterations. =0.9, =0.1, This represents the attenuation coefficient.

[0028] Furthermore, in one embodiment of the present invention, when the rate of change of market state is greater than a preset threshold, the strategy function is re-optimized.

[0029] On the other hand, embodiments of the present invention provide a decision-making device for electricity spot market participants based on deep reinforcement learning, comprising: The decision environment modeling module is used to spatially model the market state and participant actions, and establish a state transition function based on the market clearing rules to obtain the participant decision environment model. The decision optimization modeling module is used to construct a participant decision optimization model based on deep reinforcement learning based on the participant decision environment model, quantify the relative value of actions based on the advantage function, and determine the strategy to update the objective function based on the advantage function. The reward function determination module is used to determine the basic revenue item, the game-theoretic cooperation item, and the risk penalty item, and to construct a reward function that integrates market game theory and uncertainty based on the basic revenue item, the game-theoretic cooperation item, and the risk penalty item. The decision optimization module is used to optimize the participant's decision optimization model based on the adaptive exploration rate and real-time decision mechanism to obtain the optimal decision strategy, and to conduct power trading according to the optimal decision strategy.

[0030] On the other hand, embodiments of the present invention provide an electronic device, including: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the above-described deep reinforcement learning-based decision-making method for electricity spot market participants.

[0031] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the aforementioned decision-making method for electricity spot market participants based on deep reinforcement learning.

[0032] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the aforementioned decision-making method for electricity spot market participants based on deep reinforcement learning.

[0033] The advantages and beneficial effects of the present invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention: This invention provides a spatial model of market conditions and participant actions, and establishes a state transition function based on market clearing rules to obtain a participant decision-making environment model. Based on this model, a deep reinforcement learning-based participant decision optimization model is constructed. The relative value of actions is quantified using an advantage function, and a strategy update objective function is determined accordingly. Basic payoff terms, game-theoretic cooperation terms, and risk penalty terms are identified. A reward function integrating market game theory and uncertainty is constructed based on these terms. The participant decision optimization model is then optimized using an adaptive exploration rate and a real-time decision-making mechanism to obtain the optimal decision strategy. Electricity trading is then conducted based on this optimal strategy. This invention, by constructing a deep reinforcement learning decision model that integrates dynamic market environment, multi-party game theory, and risk constraints, combined with a training mechanism based on adaptive exploration-utilization balance, enables the generation and real-time updating of optimal decision strategies for participants in complex market scenarios. This improves the accuracy and real-time performance of decisions made by participants in the electricity spot market and enhances the stability of the electricity market. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating the steps of a decision-making method for electricity spot market participants based on deep reinforcement learning, as provided in an embodiment of the present invention; Figure 2 A structural block diagram of a decision-making device for electricity spot market participants based on deep reinforcement learning, provided in an embodiment of the present invention; Figure 3This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0038] The decision-making method for electricity spot market participants based on deep reinforcement learning provided in this invention can be applied to terminals, servers, or software running on either terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the decision-making method for electricity spot market participants based on deep reinforcement learning, but is not limited to the above forms.

[0039] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0040] This invention aims to address the technical bottlenecks in the dynamic adaptability, game theory characterization, and uncertainty response of participants in the electricity spot market, thereby improving the strategic adaptability, decision-making timeliness, and economic benefits of market participants (power generators, electricity retailers, and electricity users) in a high-frequency volatile market environment, and enhancing their robustness in dealing with price fluctuations, supply and demand imbalances, and information asymmetry.

[0041] Reference Figure 1 This invention provides a decision-making method for electricity spot market participants based on deep reinforcement learning, specifically including the following steps: S101. Spatial modeling of market state and participant actions, and establishment of state transition function based on market clearing rules to obtain participant decision-making environment model; S102. Construct a participant decision optimization model based on deep reinforcement learning based on the participant decision environment model, quantify the relative value of actions based on the advantage function, and determine the strategy and update the objective function based on the advantage function. S103. Determine the basic revenue item, the game-theoretic synergy item, and the risk penalty item, and construct a reward function that integrates market game theory and uncertainty based on the basic revenue item, the game-theoretic synergy item, and the risk penalty item; S104. Optimize the participant decision optimization model based on the adaptive exploration rate and real-time decision-making mechanism to obtain the optimal decision strategy, and conduct power trading according to the optimal decision strategy.

[0042] This invention constructs a deep reinforcement learning decision model that integrates dynamic market environment, multi-party game and risk constraints, and combines it with an adaptive exploration-utilization balance-based training mechanism to generate and update the optimal decision strategies of participants in complex market scenarios in real time. This improves the accuracy and real-time performance of decision-making by participants in the electricity spot market and also enhances the stability of the electricity market.

[0043] As an optional implementation, the participant decision-making environment model includes a market state-space model, a participant action-space model, and a state transition function. The market state-space model is as follows:

[0044] in, Let represent the market state vector at time t. This represents the spot market clearing price at time t. This represents the total system load demand at time t. This represents the total reported power generation of other participants at time t. This represents the participant's own marginal cost of generating electricity. Represents a historical price series. Indicates weather conditions; The participant action space model is as follows:

[0045]

[0046]

[0047] in, This represents the decision action at time t. This indicates the price declared by the power supplier. and These represent the lower limit and upper limit of the market price, respectively. This indicates the electricity volume declared by the power generator. and These represent the minimum and maximum generating capacity, respectively. The state transition function is:

[0048] in, This represents the market state vector at time t+1. This indicates the decision-making actions of other participants. This indicates a random perturbation. This represents the state transition function.

[0049] Specifically, step S101 is the foundation of the decision-making model, used to describe the decision-making scenarios, interaction rules and core constraints of market participants, providing an environmental framework for deep reinforcement learning.

[0050] 1. Market State Space Modeling Define the market state perceived by participants, including dynamic variables and static parameters, as shown in the following formula:

[0051] in, Let be the market state vector at time t. Let t be the spot market clearing price. Let t be the total system load demand. Let t be the total reported power generation of other participants. For the participants' own marginal cost of generating electricity, This is a historical price series (prices from the previous 24 hours, reflecting the trend). Weather conditions (such as temperature and wind speed) affect the output of renewable energy.

[0052] Traditional models only include current price and load; this model adds historical trends. With the weather This enhances the ability to perceive market dynamics and reduces decision-making delays.

[0053] 2. Participant Action Space Modeling Define the participants' decision-making actions, which must comply with market trading rules (such as price limits and trading volume constraints). Taking power generators as an example, the action space is:

[0054]

[0055]

[0056] in, Let t be the decision action at time t. For power generators to submit price declarations, they must meet the constraints of the market price range; For power generators to declare electricity volume, they must meet their own power output capacity constraints. This represents the maximum power generation capacity.

[0057] The action space of electricity retailers can be defined as the purchase price and the amount of electricity, while the action space of users is the load adjustment amount, both of which need to be matched with the characteristics of their market roles.

[0058] Traditional models have a single action dimension (such as only considering the price), while this model links price and electricity volume in decision-making, which is more in line with the actual market declaration rules.

[0059] 3. Market Clearing and State Transition Model To characterize the impact of actions on the market state, a state transition function is established based on market clearing rules:

[0060] in, The decision-making actions of other participants (reflecting the game relationship); For random disturbances (such as load forecasting errors, sudden shutdowns); The state transition function is the core function for generating the market clearing price. If the generator declares the price (Clearing price), then its trading volume is Otherwise, it is 0, that is:

[0061] Among them, clearing price Determined by the balance of supply and demand:

[0062] Traditional state transitions ignore the actions of other participants. With random disturbances This model incorporates game theory factors and uncertainty, making it more closely reflect real market dynamics.

[0063] Understandably, section 1 defines the environmental information (state) perceived by the participant, section 2 clarifies the operations (actions) that the participant can perform, and section 3 describes the evolution of the environment (state transition) after the "state-action" interaction. Together, these three constitute the "environment-subject" interaction framework of deep reinforcement learning, providing a foundation for subsequent decision-making models.

[0064] As an optional implementation, the participant decision optimization model includes a policy network and a value network. The policy network takes the market state vector as input and outputs the action probability distribution, while the value network takes the market state vector as input and outputs the state value. The advantage function is:

[0065] in, Represents the market state vector Next action The relative value, Represents the market state vector Next action The value of the action, Represents the market state vector The corresponding state value; The policy update objective function is:

[0066] in, This indicates the policy update objective function. This represents the new strategy parameters to be optimized. This represents the old strategy parameters before the update. Represents the market state vector New strategy selection action The probability, Represents the market state vector Choose action from old strategy The probability, Indicates will Limited to Inside, =0.2 is the range for clip. This represents the expectation operator.

[0067] Specifically, step S102 constructs a deep reinforcement learning framework with a policy network at its core based on the decision environment model, thereby realizing the mapping from state to optimal action.

[0068] 1. Network structure design An improved PPO (Proximal Policy Optimization) algorithm is adopted, which includes a policy network and a value network: 1) Policy Network Input status Output action probability distribution (parameters) It employs a 3-layer fully connected network (with ReLU activation function), and the output layer uses Softmax to ensure that the sum of action probabilities is 1.

[0069] 2) Value Network Input status Output state value (parameter) ), used to evaluate the long-term benefits of the current state.

[0070] Traditional reinforcement learning (such as DQN) is only applicable to discrete action spaces. This model uses the PPO algorithm to support continuous actions (such as continuous values ​​of price quotes and electricity consumption), which is more in line with the needs of market decision-making.

[0071] 2. Advantage Function and Policy Update Define an advantage function to quantify the relative value of actions and guide policy optimization:

[0072] in, Value of an action (i.e., the value of taking an action) (cumulative rewards) The advantage value is the state value (i.e., the baseline payoff for the current state). A positive result indicates that the action is better than the benchmark. A negative value indicates that the result is inferior to the benchmark.

[0073] The policy update objective function uses a clip mechanism to avoid policy mutations.

[0074] in, =0.2 represents the clip range (control strategy update magnitude). This is the expectation operator (averaging over the sample).

[0075] Traditional policy gradient algorithms are prone to training instability due to excessively large parameter update magnitudes. This model uses a clip mechanism to limit policy changes and improve convergence stability.

[0076] It is understandable that 1 designs the network structure of the decision model (the policy network generates actions, and the value network evaluates the state), while 2 quantifies the value of actions based on the advantage function and realizes the stable update of the policy through the clip mechanism. Together, they form a closed loop of "model structure-optimization method" to ensure that the decision model can learn the optimal policy from the interaction with the environment.

[0077] As an optional implementation method, the basic benefit item is:

[0078] in, This represents the basic revenue term at time t. Let represent the clearing price at time t. This represents the electricity cost at time t. This represents the actual trading volume at time t; The collaborative terms in the game are:

[0079] in, Denotes the collaborative terms in the game at time t. Indicates the game weights, This indicates the bids from other participants. Represents the sensitivity coefficient. Represents the natural exponential function; The risk penalty items are:

[0080] in, This represents the risk penalty term at time t. This represents the risk aversion coefficient. This represents the price variance over a period of time before and after time t. The reward function is:

[0081] in, This represents the action reward at time t.

[0082] Specifically, the reward function in step S103 needs to balance short-term gains, long-term strategy effectiveness, and risk aversion, reflecting multi-objective optimization and providing "feedback signals" for reinforcement learning.

[0083] 1. Basic Revenue Items Quantify the direct economic benefits of current decisions:

[0084] in, Profit per unit of electricity The actual transaction volume is represented by the product, which represents the instantaneous profit.

[0085] 2. Game-theoretic collaborative items Considering strategic interactions with other participants, encourage the formation of stable and favorable game equilibria.

[0086] in, =0.1 represents the game weight (adjusting the importance of game factors). Offer quotes to other participants. =0.5 is the sensitivity coefficient (to control the impact of price differences on rewards). It is an exponential function (ensuring non-negative rewards). If the difference between one's own bid and that of the competitor is small (to avoid vicious competition), this term is positive, guiding strategic coordination.

[0087] 3. Risk Penalty Items Restrain high-risk decisions (such as aggressive quoting during periods of high price volatility):

[0088] in, =0.05 is the risk aversion coefficient (adjusting the intensity of risk penalty). The price variance (reflecting the degree of price volatility) is the price variance over a period of time (e.g., 3 hours) before and after time t. The negative sign indicates a penalty (reduced reward) for high volatility.

[0089] 4. Total Reward Function Combining the above three factors results in the final reward:

[0090] Traditional reward functions only consider a single benefit term. This model adds a game-theoretic cooperation term (to guide healthy competition) and a risk penalty term (to suppress aggressive decision-making under high volatility), achieving a multi-objective balance of "reward-game-risk".

[0091] Understandably, step 1 quantifies the direct economic benefits of decision-making (the basic objective), step 2 introduces multi-participant game factors (to improve the adaptability of the strategy in a competitive environment), step 3 adds risk penalties (to enhance the robustness of the strategy), and step 4 integrates the three into a total reward, providing a comprehensive feedback signal for reinforcement learning and ensuring that the strategy learned by the model pursues both gains and game-theoretic cooperation and risk control.

[0092] As an optional implementation, the participant decision optimization model is further optimized based on the adaptive exploration rate and real-time decision-making mechanism, specifically including: Initialize the policy function and value function of the participant decision optimization model; The current policy function is executed in the decision-making environment, and a random action or the optimal action is selected as the current action based on an adaptive exploration rate mechanism. Calculate the action reward corresponding to the current action; Calculate the advantage function based on the current value function and action reward; The objective function is updated based on the advantage function and the policy, and the policy function is updated by minimizing the mean squared error using gradient descent to update the value function.

[0093] As an optional implementation, a random action or the optimal action is selected as the current action according to the following formula:

[0094] in, The exploration rate at time t is represented by... The probability of selecting a random action as the current action is used. The probability of selecting the optimal action as the current action is given, where t represents the number of iterations. =0.9, =0.1, This represents the attenuation coefficient.

[0095] As an optional implementation, when the rate of change in market state exceeds a preset threshold, the strategy function is re-optimized.

[0096] Specifically, step S104 designs a dynamic training strategy and a real-time decision-making process to improve the model's generalization ability and response speed, ensuring that decisions can adapt to market changes.

[0097] 1. Adaptive exploration rate adjustment use -greedy Strategy balances exploration (trying new actions) and exploitation (choosing known superior actions). Dynamic decay with training iterations:

[0098] Where t is the number of iterations. =0.9 (Initially explore unknown actions). =0.1 (In the later stages, more known superior actions are utilized). =0.01 is the decay coefficient (controlling the rate of decrease in exploration rate). It is an exponential function (achieving smooth decay).

[0099] 2. Experience Replay and Generalization Training Build an experience pool storage The sample (state-action-reward-next state) is sampled using a priority sampling method (important samples have higher weights), and new scenario data (such as extreme prices, sudden changes in supply and demand) is periodically injected to avoid overfitting. The Adam optimizer is used during training, with a learning rate of [missing information]. =0.0003, batch size 32 (number of samples per training session).

[0100] 3. Real-time decision triggering mechanism When the rate of change in market conditions exceeds a threshold (e.g., price fluctuations exceeding 10% within 15 minutes), a strategy re-optimization is triggered: If Then update the policy network parameters, where, =0.1 is the fluctuation threshold (10%). The price is the price at the previous moment, and the parameters are updated when the conditions are met (to ensure that the strategy adapts to the new market conditions).

[0101] Traditional reinforcement learning uses a fixed exploration rate and static training data, and the decision-making strategy is updated slowly. This model solves the problems of weak generalization and slow response of traditional methods by using an adaptive exploration rate (to improve learning efficiency), generalization training (to enhance the ability to adapt to new scenarios), and a real-time triggering mechanism (to quickly respond to market changes).

[0102] Understandably, step 1 balances "exploration-exploitation" through adaptive exploration rate to ensure the model learns efficiently (neither missing potential optimal strategies nor blindly trying ineffective actions); step 2 enhances the model's adaptability to diverse market scenarios (avoiding overfitting) through experience replay and generalization training; and step 3 ensures the strategy can quickly respond to market changes (such as drastic price fluctuations) through a real-time triggering mechanism. These three elements together form a guarantee system of "training efficiency-generalization ability-real-time performance," enabling the decision-making model to be effectively applied in the real market.

[0103] The method steps of the embodiments of the present invention have been described above. It can be understood that the embodiments of the present invention, by constructing a deep reinforcement learning decision model that integrates dynamic market environment, multi-party game and risk constraints, and combining it with a training mechanism based on adaptive exploration-utilization balance, realize the generation and real-time updating of optimal decision strategies for participants in complex market scenarios, thereby improving the accuracy and real-time performance of decision-making by participants in the electricity spot market and also improving the stability of the electricity market.

[0104] Compared with the prior art, the embodiments of the present invention also have the following advantages: 1) Dynamic decision innovation: By using deep reinforcement learning to capture changes in market status in real time, the strategy is dynamically updated with S_t, breaking through the limitations of traditional static decision-making and improving response speed.

[0105] 2) Innovation in game perception: The reward function incorporates multi-participant strategy interaction terms, quantifies the impact of game relationships on decision-making, and makes the returns of strategies more secure in a competitive environment.

[0106] 3) Robustness Enhancement Innovation: By integrating risk penalty and uncertainty perturbation training, the model's return stability (variance) under price fluctuation scenarios is reduced, and its risk resistance is significantly improved.

[0107] 4) Innovation in generalization ability: The adaptive exploration and experience playback mechanism ensures that the model maintains a high level of decision-making accuracy in new market scenarios (such as policy adjustments and extreme weather), solving the problem of weak generalization in traditional methods.

[0108] Reference Figure 2 This invention provides a decision-making device for electricity spot market participants based on deep reinforcement learning, comprising: The decision environment modeling module is used to spatially model the market state and participant actions, and establish a state transition function based on the market clearing rules to obtain the participant decision environment model. The decision optimization modeling module is used to construct a participant decision optimization model based on deep reinforcement learning based on the participant decision environment model, quantify the relative value of actions based on the advantage function, and determine the strategy and update the objective function based on the advantage function. The reward function determination module is used to determine the basic revenue item, the game-theoretic synergy item, and the risk penalty item, and to construct a reward function that integrates market game theory and uncertainty based on the basic revenue item, the game-theoretic synergy item, and the risk penalty item. The decision optimization module is used to optimize the participant's decision optimization model based on the adaptive exploration rate and real-time decision mechanism to obtain the optimal decision strategy, and to conduct power trading based on the optimal decision strategy.

[0109] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0110] Reference Figure 3 This invention provides an electronic device, comprising: At least one processor; At least one memory for storing at least one program; When the above-mentioned at least one program is executed by the above-mentioned at least one processor, the above-mentioned at least one processor implements the above-mentioned decision-making method for electricity spot market participants based on deep reinforcement learning.

[0111] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0112] This invention also provides a computer-readable storage medium storing a processor-executable computer program that, when executed by a processor, implements the aforementioned deep reinforcement learning-based decision-making method for electricity spot market participants.

[0113] This invention provides a computer-readable storage medium that can execute a decision-making method for electricity spot market participants based on deep reinforcement learning, as provided in the method embodiments of this invention. It can execute any combination of the implementation steps of the method embodiments and has the corresponding functions and beneficial effects of the method.

[0114] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned decision-making method for electricity spot market participants based on deep reinforcement learning.

[0115] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0116] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0117] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0118] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0119] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the aforementioned blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0120] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the aforementioned functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0121] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0123] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the aforementioned program can be printed, because the aforementioned program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0124] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0125] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0126] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0127] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for power spot market participant decision-making based on deep reinforcement learning, characterized in that, The method comprises the following steps: spatially modeling market state and participant action, and establishing a state transition function based on market clearing rules to obtain a participant decision environment model; constructing a participant decision optimization model based on deep reinforcement learning according to the participant decision environment model, quantifying the relative value of action based on an advantage function, and determining a policy update target function according to the advantage function; determining a basic income item, a game synergy item and a risk penalty item, and constructing a reward function integrating market game and uncertainty according to the basic income item, the game synergy item and the risk penalty item; optimizing the participant decision optimization model based on an adaptive exploration rate and a real-time decision mechanism to obtain an optimal decision strategy, and performing power trading according to the optimal decision strategy.

2. The method of claim 1, wherein, The participant decision environment model comprises a market state space model, a participant action space model and the state transition function, the market state space model being: wherein, denotes the market state vector at time t, denotes the spot market clearing price at time t, denotes the total system load demand at time t, denotes the total declared generation by other participants at time t, denotes the marginal generation cost of the participant itself, denotes the historical price sequence, denotes the weather state; The participant action space model being: wherein, denotes a decision action at time t, denotes a price declared by a power generator, and denote a lower limit of a market price and an upper limit of a market price, respectively, denotes a power amount declared by a power generator, and denote a minimum power generation capacity and a maximum power generation capacity, respectively; The state transition function being: wherein, denotes the market state vector at time t+1, denotes the decision actions of other participants, denotes a random disturbance, denotes a state transition function.

3. The method of claim 2, wherein, The participant decision optimization model comprising a policy network and a value network, the policy network being used for inputting a market state vector and outputting an action probability distribution, and the value network being used for inputting a market state vector and outputting a state value, the advantage function being: wherein, market state vector the next action the relative value, market state vector the next action the action value, market state vector the corresponding state value; The policy update target function being: where, denotes the policy update target function, denotes the new policy parameters to be optimized, denotes the old policy parameters before the update, denotes the market state vector the probability of the new policy selecting action , denotes the market state vector the probability of the old policy selecting action , denotes clipping to , =0.2 the clip range, denotes the expected operator.

4. The method of claim 3, wherein, The basic income item being: wherein, represents the base revenue item at time t, represents the clearing price at time t, represents the power cost at time t, represents the actual trading volume at time t; The game synergy item being: wherein, denotes the game synergy term at time t, denotes the game weight, denotes the other participant's offer, denotes the sensitivity coefficient, denotes the natural exponential function; The risk penalty item being: wherein, represents the risk penalty term at time t, represents the risk aversion coefficient, represents the price variance over a period of time before and after time t; The reward function being: wherein, denotes the action reward at time t.

5. The method of claim 4, wherein, The optimization of the participant decision optimization model based on the adaptive exploration rate and the real-time decision mechanism specifically comprising: initializing a policy function and a value function of the participant decision optimization model; executing a current policy function in a decision environment, and selecting a random action or an optimal action as a current action based on an adaptive exploration rate mechanism; calculating an action reward corresponding to the current action; calculating an advantage function according to a current value function and the action reward; updating the policy function according to the advantage function and the policy update target function, and updating the value function by minimizing mean square error by using a gradient descent method.

6. The method of claim 5, wherein, The random action or the optimal action is selected as the current action according to the following formula: wherein, represents the exploration rate at time t, with selects a random action as the current action with probability selects the optimal action as the current action with probability = 0.9, = 0.1, represents the decay coefficient.

7. The method of claim 5, wherein, When the market state change rate is greater than a preset threshold, re-optimization of the policy function is triggered.

8. A deep reinforcement learning based power spot market participant decision apparatus, characterized by, The method comprises: a decision environment modeling module, configured to spatially model market state and participant action, and establish a state transition function based on market clearing rules to obtain a participant decision environment model; a decision optimization modeling module, configured to construct a participant decision optimization model based on deep reinforcement learning according to the participant decision environment model, quantify the relative value of action based on an advantage function, and determine a policy update target function according to the advantage function; a reward function determination module, configured to determine a basic income item, a game synergy item and a risk penalty item, and construct a reward function integrating market game and uncertainty according to the basic income item, the game synergy item and the risk penalty item; a decision optimization module, configured to optimize the participant decision optimization model based on an adaptive exploration rate and a real-time decision mechanism to obtain an optimal decision strategy, and perform power trading according to the optimal decision strategy.

9. An electronic device, comprising: The method comprises: at least one processor; at least one memory for storing at least one program; when the at least one program is executed by the at least one processor, the at least one processor is caused to implement the method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method according to any one of claims 1 to 7. The computer program, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images