Reinforcement learning-based multi-agent decision-making method and system for power market

Abstract

A power market multi-agent decision-making method and system based on reinforcement learning, the method comprising collecting market operation boundary conditions to form a standardized environment state vector; constructing an agent differentiated state space according to the standardized environment state vector; constructing a unified action space, each action type corresponding to a bid reference adjustment coefficient, and different types of market agents adopting different bid reference calculation methods for the same action; each market agent selects an action according to a strategy and generates a multi-section step bidding curve; market clearing calculation is performed according to all the bidding curves to obtain the winning electricity quantity and the node marginal price; a spread settlement mechanism is used to calculate the income of each market agent as a reward signal; a Q-learning algorithm is used to update the Q matrix; and multiple rounds of iteration are repeated until the Q matrix converges, and a power market multi-agent decision-making scheme is obtained. The present application solves the problems of strategy solidification, high architecture coupling degree and lack of strategy evolution capability in the multi-agent modeling of the simulation system.

Description

Technical Field

[0001] This invention belongs to the field of power system simulation technology, specifically relating to a multi-stakeholder decision-making method and system for the power market based on reinforcement learning. Background Technology

[0002] With the deepening of the construction of new power systems, the installed capacity of new energy sources such as wind power and photovoltaics continues to increase, and new market players such as energy storage, load aggregators, and virtual power plants are accelerating their entry into the market. The types of entities to be modeled in simulation systems are becoming increasingly diverse and their numbers are growing rapidly. Currently, power market simulation systems mainly employ rule-driven fixed strategy generation modules in their modeling methods. These modules generate quantity and price declaration data for each entity according to preset cost parameters and fixed bidding logic. Alternatively, they use matching and inference methods based on typical behavior databases, mapping and combining strategies through a pre-built behavioral pattern library. Or, they employ single-entity optimization methods based on mathematical programming models, determining the optimal declaration scheme by constructing linear or nonlinear programming problems.

[0003] However, the aforementioned existing technical solutions still suffer from the following technical shortcomings: Rule-driven strategy generation modules employ fixed function mapping relationships, and their model parameters cannot be dynamically adjusted based on feedback from market clearing results during simulation, resulting in a lack of variability in the strategy space during multiple rounds of simulation. Methods based on typical behavior databases rely on expert experience to construct the database, requiring expansion of the database structure and matching rules as the types of market participants increase, leading to insufficient scalability and high maintenance costs. Methods based on mathematical programming models suffer from excessive computational complexity in large-scale simulation scenarios involving multiple participants and time periods, making it difficult to meet the efficiency requirements of real-time or near-real-time simulation. Furthermore, existing simulation systems construct independent computational modules for different types of participants, resulting in inconsistent state variable definitions and interface protocols between modules, leading to high system integration complexity and low code reusability. Simultaneously, existing simulation architectures generally lack a universal strategy iteration engine, failing to simulate the gradual optimization process of participant strategies as the environment changes during multiple rounds of counterfactual simulation, thus limiting the simulation system's ability to explore complex market evolution patterns. Summary of the Invention

[0004] The purpose of this invention is to address the problems of rigid strategies, high architectural coupling, and lack of strategy evolution capabilities in existing simulation systems for multi-agent modeling. It provides a reinforcement learning-based multi-agent decision-making method and system for the power market. This method constructs a unified "state space-action space-reward function" multi-agent decision-making framework and introduces the Q-learning algorithm as a strategy iteration engine. By differentiating the state variables specific to each agent, a unified description of the multiple agents is achieved. Continuous updating of the Q-matrix enables progressive strategy evolution of the multiple agents in consecutive simulation iterations.

[0005] To achieve the above objectives, the present invention provides the following technical solution: Firstly, a multi-stakeholder decision-making method for the electricity market based on reinforcement learning is provided, including: Collect market operation boundary conditions to form a standardized environmental state vector; the standardized environmental state vector includes load forecast, renewable energy output forecast, market supply and demand ratio, and electricity price forecast. A subject-specific state space is constructed based on a standardized environmental state vector; the subject-specific state space consists of a general environmental state and a subject-specific state, which is used to define the current state of each market subject in each round; Construct a unified action space, define several discrete action types in the action space, and each action type corresponds to a pricing benchmark adjustment coefficient. Different types of market participants adopt different pricing benchmark calculation methods for the same action according to their own characteristics. In the current state, select an action from the action space according to the strategy, calculate the quotation benchmark price based on the quotation benchmark adjustment coefficient corresponding to the selected action, and generate a multi-segment ladder quotation curve based on the quotation benchmark price. Market clearing calculations are performed based on multi-segment tiered pricing curves to solve the safety-constrained unit combination problem and the economic dispatch problem, thereby obtaining the winning bid volume and nodal marginal price for each entity; Based on the winning bid volume and the marginal electricity price at each node, the revenue of each market participant is calculated using a differential settlement mechanism, and the revenue is used as a reward signal. The Q-learning algorithm is used to update the Q-matrix of each state based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, thus obtaining the updated policy; the next state is defined by the subject's differentiated state space. Repeat the above market clearing calculation, revenue calculation for each market participant, and Q matrix update process for multiple rounds of iterative deduction until the Q matrix converges, thus obtaining a multi-stakeholder decision-making scheme for the electricity market.

[0006] As a preferred embodiment, in the step of constructing the subject-differentiated state space based on the standardized environmental state vector, the general environmental state includes the market supply and demand state quantified by the supply and demand ratio, the electricity price forecast state characterized by the spot forecast electricity price, and the new energy output level state characterized by the new energy penetration rate.

[0007] As a preferred embodiment, in the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a thermal power unit, the subject-specific state includes the marginal cost level determined by coal cost and unit efficiency, the contract holding ratio determined by the ratio of the electricity allocated from medium- and long-term contracts to the predicted power generation, the start-stop state constraints determined by the start-stop state of the previous period and the remaining value of the minimum start-stop time, and the ramping capacity level determined by the ratio of the maximum ramping rate to the average ramping demand of the system.

[0008] As a preferred embodiment, in the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a new energy unit, the subject-specific state includes the predicted output confidence based on historical prediction error statistics, the contract holding ratio determined by the ratio of the decomposed electricity of medium- and long-term contracts to the predicted power generation, the curtailment rate determined by the curtailment rate of wind or solar power in the most recent period, and the sign and magnitude of the predicted output deviation.

[0009] As a preferred embodiment, in the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is an energy storage subject, the subject-specific state includes the state of charge level obtained by discretization of the state of charge, the expected level of peak-valley price difference calculated based on future electricity price forecasts, whether it participates in frequency regulation or the ancillary service market opportunity identifier, and the charge-discharge efficiency coefficient determined by the inherent parameters of the energy storage device.

[0010] As a preferred embodiment, in the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a load-side subject, the subject-specific state includes the predicted load deviation level reflecting the accuracy of electricity consumption forecasting, the contract position ratio determined by the ratio of the electricity decomposed from medium- and long-term contracts to the predicted electricity consumption, the user price elasticity level, and the interruptible load ratio level reflecting the demand response potential.

[0011] As a preferred embodiment, in the step of defining several discrete action types in the action space, the discrete action types include conservative high price, stable return, marginal cost, aggressive low price, extreme high price, and extreme low price, with each action type corresponding to a different price benchmark adjustment coefficient; the action of the energy storage entity is extended to a composite action consisting of charging and discharging actions, with the charging and discharging power satisfying complementary constraints and state of charge boundary constraints; the action adjustment direction of the load-side entity is opposite to that of the generation-side entity, with the load-side entity increasing the bidding volume during low-price periods and decreasing the bidding volume during high-price periods.

[0012] As a preferred embodiment, the step of generating a multi-segment tiered pricing curve includes: multiplying the pricing benchmark adjustment coefficient corresponding to the selected action by the pricing benchmark price as the benchmark price; dividing the total capacity of the unit into N equal segments; setting the price of each segment linearly increasing from the benchmark price; and outputting a set of pricing pairs consisting of the capacity of each segment and the corresponding price, where N is the preset number of segments.

[0013] As a preferred embodiment, in the step of selecting an action from the action space according to the strategy in the current state, each market participant selects the action with the highest current Q value with a probability of 1-ε, and randomly and uniformly selects actions for exploration with a probability of ε, where ε is a preset exploration probability; after training is completed, each market participant makes decisions entirely according to the action with the highest current Q value, and no longer conducts random exploration, i.e., ε=0.

[0014] As a preferred approach, the Q-learning algorithm is employed to update the Q-matrices based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, thereby obtaining the updated policy. The calculation expression is as follows:

[0015] In the formula, α is the learning rate and α∈(0,1], and γ is the discount factor and γ∈[0,1]; wherein, the learning rate α gradually decreases as the number of iteration rounds increases.

[0016] As a preferred embodiment, in the step of calculating the revenue of each market participant based on the winning bid volume and the marginal electricity price at each node using a differential settlement mechanism, the revenue calculation expression for the power generation side participant is as follows:

[0017] In the formula, express t Electricity allocation for medium- and long-term contracts; express t Prices of medium- and long-term contracts during the period; Indicates the main body of the power generation side t Electricity volume won in the spot market during the specified period; express t Marginal electricity price at specific time points; Indicates operating costs; Indicates start-up and shutdown costs; Indicates other income; T This indicates the total number of transaction periods included in the settlement cycle; The formula for calculating the revenue of the load-side main body is as follows:

[0018] In the formula, This indicates the cost of deviation assessment; Indicates the main body of the load side t The electricity consumption for which the winning bid was determined through market clearing during the specified period; R This represents the total revenue of the power generation entity during the settlement period.

[0019] As a preferred embodiment, in the step of generating multi-segment tiered pricing curves, the benchmark price for new energy generating units is adjusted based on the confidence level of predicted output. When the confidence level is low, the benchmark price is lowered, and when the confidence level is high, the benchmark price is raised. When the energy storage entity is charging, it submits a power purchase curve, and when it is discharging, it submits a power sales curve. The charging and discharging power is constrained by the state of charge boundary, and the charging and discharging efficiency coefficient is used to evaluate the economic benefits of charging and discharging to determine the optimal charging and discharging power.

[0020] As a preferred embodiment, the convergence criterion for the Q matrix is: the change in the Q matrix is ​​less than a preset threshold, or the number of iterations reaches a preset maximum number of iterations.

[0021] Secondly, a multi-stakeholder decision-making system for the electricity market based on reinforcement learning is provided, including: An environmental state vector generation module is used to collect market operation boundary conditions and form a standardized environmental state vector; the standardized environmental state vector includes load forecasting, new energy output forecasting, market supply and demand ratio, and electricity price forecasting. The differentiated state space construction module is used to construct the subject's differentiated state space based on the standardized environmental state vector; the subject's differentiated state space consists of a general environmental state and a subject-specific state, and is used to define the current state of each market subject in each round; The action space construction module is used to construct a unified action space, in which several discrete action types are defined. Each action type corresponds to a pricing benchmark adjustment coefficient. Different types of market participants use different pricing benchmark calculation methods for the same action according to their own characteristics. The multi-segment tiered pricing curve generation module is used to select an action from the action space according to a strategy in the current state, calculate the pricing benchmark price based on the pricing benchmark adjustment coefficient corresponding to the selected action, and generate a multi-segment tiered pricing curve based on the pricing benchmark price. The market clearing calculation module is used to perform market clearing calculations based on multi-segment tiered pricing curves, solve the safety-constrained unit combination problem and the economic dispatch problem, and obtain the winning bid volume and nodal marginal price for each entity. The market entity revenue calculation module is used to calculate the revenue of each market entity based on the winning bid volume and the marginal electricity price at each node, using a differential settlement mechanism, and to use the revenue as a reward signal. The policy update module is used to update the Q-matrix of each subject using the Q-learning algorithm, based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, to obtain the updated policy; the next state is defined by the subject's differentiated state space. The iterative deduction module is used to repeat the above-mentioned market clearing calculation, revenue calculation of each market participant, and Q matrix update process for multiple rounds of iterative deduction until the Q matrix converges, thus obtaining the decision-making scheme of multiple participants in the electricity market.

[0022] As a preferred embodiment, the general environmental states constructed by the differentiated state space construction module include the market supply and demand state quantified by the supply and demand ratio, the electricity price forecast state characterized by the spot forecast electricity price, and the new energy output level state characterized by the new energy penetration rate.

[0023] As a preferred embodiment, when the differentiated state space construction module constructs the subject-specific state, if the market subject type is a thermal power unit, the subject-specific state includes the marginal cost level determined by coal cost and unit efficiency, the contract holding ratio determined by the ratio of the electricity allocated from medium- and long-term contracts to the predicted power generation, the start-stop state constraints determined by the start-stop state of the previous period and the remaining value of the minimum start-stop time, and the ramping capacity level determined by the ratio of the maximum ramping rate to the system's average ramping demand.

[0024] As a preferred embodiment, when the differentiated state space construction module constructs the subject-specific state, if the market subject type is a new energy unit, the subject-specific state includes the predicted output confidence based on historical prediction error statistics, the contract holding ratio determined by the ratio of the electricity decomposed from medium- and long-term contracts to the predicted power generation, the curtailment rate determined by the recent wind curtailment rate or solar curtailment rate, and the sign and magnitude of the predicted output deviation.

[0025] As a preferred embodiment, when the differentiated state space construction module constructs the subject-specific state, if the market subject type is an energy storage subject, the subject-specific state includes the state of charge level obtained by discretization of the state of charge, the expected level of peak-valley price difference calculated based on future electricity price forecasts, whether it participates in frequency regulation or the ancillary service market opportunity identifier, and the charging and discharging efficiency coefficient determined by the inherent parameters of the energy storage device.

[0026] As a preferred embodiment, when the differentiated state space construction module constructs subject-specific states, if the market subject type is a load-side subject, the subject-specific states include a predicted load deviation level reflecting the accuracy of electricity consumption forecasting, a contract position ratio determined by the ratio of the electricity output from medium- and long-term contracts to the predicted electricity consumption, a user price elasticity level, and an interruptible load ratio level reflecting the demand response potential.

[0027] As a preferred embodiment, the discrete action types defined by the action space construction module include conservative high price, stable return, marginal cost, aggressive low price, extreme high price, and extreme low price, with each action type corresponding to a different pricing benchmark adjustment coefficient. When the action space construction module constructs a unified action space, the actions of the energy storage entity are expanded into composite actions consisting of charging and discharging actions, with the charging and discharging power satisfying complementary constraints and state of charge boundary constraints. The action adjustment direction of the load-side entity is opposite to that of the generation-side entity, with the load-side entity increasing the bidding volume during low-price periods and decreasing the bidding volume during high-price periods.

[0028] As a preferred embodiment, the multi-segment tiered pricing curve generation module uses the pricing benchmark adjustment coefficient corresponding to the selected action multiplied by the pricing benchmark price as the benchmark price, divides the total capacity of the unit into N equal segments, sets the price of each segment linearly increasing from the benchmark price, and outputs a set of pricing pairs consisting of the capacity of each segment and the corresponding price, where N is the preset number of segments.

[0029] As a preferred embodiment, when the multi-segment tiered pricing curve generation module selects an action from the action space according to the strategy in the current state, each market participant selects the action with the highest current Q value with a probability of 1-ε, and randomly and uniformly selects actions for exploration with a probability of ε, where ε is the preset exploration probability; after training is completed, each market participant makes decisions entirely according to the action with the highest current Q value, and no longer conducts random exploration, i.e., ε=0.

[0030] As a preferred embodiment, the policy update module employs the Q-learning algorithm. Based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, it updates the respective Q-matrices to obtain the updated policy. The calculation expression is as follows:

[0031] In the formula, α is the learning rate and α∈(0,1], and γ is the discount factor and γ∈[0,1]; wherein, the learning rate α gradually decreases as the number of iteration rounds increases.

[0032] As a preferred embodiment, when the market entity revenue calculation module calculates the revenue of each market entity based on the winning bid volume and the marginal electricity price at each node, using a differential settlement mechanism, the revenue calculation expression for the generation-side entity is as follows:

[0033] In the formula, express t Electricity allocation for medium- and long-term contracts; express t Prices of medium- and long-term contracts during the period; Indicates the main body of the power generation side tElectricity volume won in the spot market during the specified period; express t Marginal electricity price at specific time points; Indicates operating costs; Indicates start-up and shutdown costs; Indicates other income; T This indicates the total number of transaction periods included in the settlement cycle; The formula for calculating the revenue of the load-side main body is as follows:

[0034] In the formula, This indicates the cost of deviation assessment; Indicates the main body of the load side t The electricity consumption for which the winning bid was determined through market clearing during the specified period; R This represents the total revenue of the power generation entity during the settlement period.

[0035] As a preferred embodiment, when generating the multi-segment tiered pricing curve, the benchmark price of the new energy unit is adjusted based on the confidence level of the predicted output. When the confidence level is low, the benchmark price is lowered, and when the confidence level is high, the benchmark price is raised. The energy storage entity submits a power purchase curve when charging and a power sales curve when discharging. The charging and discharging power is constrained by the state of charge boundary, and the charging and discharging efficiency coefficient is used to evaluate the economic benefits of charging and discharging to determine the optimal charging and discharging power.

[0036] As a preferred embodiment, the convergence condition of the Q matrix in the iterative deduction module is: the change in the Q matrix is ​​less than a preset threshold, or the number of iterative deduction rounds reaches a preset maximum number of rounds.

[0037] Thirdly, an electronic device is provided, including a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the reinforcement learning-based multi-agent decision-making method for the electricity market.

[0038] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing at least one instruction, which, when executed by a processor, implements the reinforcement learning-based multi-agent decision-making method for the electricity market.

[0039] Compared with the prior art, the first aspect of the present invention has at least the following beneficial effects: By constructing a differentiated state space for entities, consisting of a general environmental state and entity-specific states, along with a unified action space, this invention integrates diverse market entities such as thermal power, new energy, energy storage, and load-side entities into a single decision-making framework. This overcomes the problems of inconsistent interfaces, low code reusability, and high system integration complexity caused by constructing independent computation modules for different types of entities in traditional simulation systems, significantly improving the versatility and scalability of the simulation method. Furthermore, this invention introduces the Q-learning algorithm to construct a closed-loop learning architecture encompassing state perception, strategy selection, payoff feedback, and strategy optimization. This allows each market entity to continuously optimize its Q-matrix based on market clearing results and payoff feedback during multiple rounds of simulation, enabling a shift in bidding strategies from fixed rule-driven to data-driven, and from static strategies to dynamic evolution. This realistically reflects the strategy learning behavior of rational market entities in repeated games, providing a more practical means of modeling entity behavior in power market simulation. This invention avoids the computational overhead and instability caused by complex function approximation by discretizing the continuous state space into finite integer levels to construct a finite-dimensional Q-matrix. Each state-action value in the Q-matrix directly corresponds to the strategy preference under a specific market scenario, exhibiting good interpretability and facilitating market analysts' understanding of the decision-making logic of various entities. By solving the safety-constrained unit combination problem and the economic dispatch problem through market clearing calculations, the strategy optimization results are strictly constrained by grid operation constraints, ensuring that the simulation results conform to the actual operation procedures of the power system. By calculating revenue through a price difference settlement mechanism and using it as a reward signal, the update direction of the Q-matrix is ​​consistent with the actual market settlement rules, ensuring that the strategy evolution goal matches the decision-making goals of entities in the real market environment.

[0040] Furthermore, the general environmental state of this invention includes the supply-demand ratio, the spot market forecast electricity price, and the penetration rate of new energy sources, so that different subjects have a consistent perception of the environmental state, providing a common environmental state basis for the collaborative evolution of multi-subject strategies.

[0041] Furthermore, this invention configures thermal power units with unique state variables such as marginal cost level, contract holding ratio, start-up and shutdown state constraints, and ramp-up capability level, enabling the strategic decision-making of thermal power units to reflect cost characteristics, physical constraints, and operating boundaries.

[0042] Furthermore, this invention incorporates unique state variables such as the confidence level of predicted output of new energy generating units, the proportion of contract holdings, the curtailment rate, and the sign and magnitude of the predicted output deviation. This enables the pricing strategy of new energy generating units to adaptively adjust according to the prediction accuracy, accurately characterizing the impact of output uncertainty on decision-making behavior.

[0043] Furthermore, this invention configures unique state variables for the energy storage entity, such as state of charge level, peak-valley price difference expectation level, ancillary service market opportunity identifier, and charge-discharge efficiency coefficient. This makes the energy storage charging and discharging strategy subject to the combined effects of energy state constraints, state of charge boundary constraints, and efficiency and economic constraints, thus solving the technical problem in the prior art that it is difficult to model the energy storage entity due to charging and discharging coupling constraints.

[0044] Furthermore, this invention configures unique state variables such as the load deviation level, contract holding ratio, user price elasticity level, and interruptible load ratio for the load-side main configuration, enabling the load-side power purchase strategy to reflect the uncertainty of electricity consumption and providing a state description dimension for demand-side resource participation in market simulation.

[0045] Furthermore, this invention sets discrete action types as a complete strategy spectrum ranging from conservative high price to extreme high price, with each action corresponding to a different pricing benchmark adjustment coefficient, which can characterize the behavior of market participants with different risk preferences. Energy storage actions are extended to composite actions of charging and discharging, satisfying complementary constraints and state of charge boundary constraints, so that the charging and discharging coupling characteristics and temporal coupling characteristics of the energy storage entity can be accurately expressed in the same action space. The direction of load-side actions is opposite to that of the generation side, so that the unified action space adapts to the decision semantics of the load-side entity under the reverse adjustment logic, without the need to design a separate action space for the load side, further improving the versatility of the framework.

[0046] Furthermore, the multi-segment stepped pricing curve of this invention is generated by equally dividing the capacity segments and linearly increasing the pricing, providing clear rules for the generation of pricing curves for each action type, ensuring that the pricing curves generated for the same action in different rounds and under different states have consistency and reproducibility, and providing a clear output mapping relationship for the stable convergence of the Q matrix.

[0047] Furthermore, this invention achieves a balance between exploration and exploitation by employing an ε-greedy strategy, and switches to a full exploitation mode with ε=0 after training is completed, thus realizing a two-stage working mode switching from exploratory learning to stable application.

[0048] Furthermore, this invention achieves a progressive learning model that enables rapid response in the early stages of training and stable convergence in the later stages of training by adopting the standard update formula of the Q-learning algorithm and limiting the learning rate α to gradually decrease with the increase of iteration rounds. The discount factor γ is set to 0.9~0.99 so that the subject can optimize immediate returns while taking into account the long-term impact of future returns, which matches the actual characteristics of the electricity market where bidding decisions affect returns in subsequent periods.

[0049] Furthermore, by fixing the calculation formulas for the revenue of the generation side and the load side in the differential settlement mechanism in the form of mathematical expressions, this invention ensures that the update direction of the Q matrix is ​​precisely aligned with the actual settlement rules of the electricity market, thereby ensuring the consistency between the reinforcement learning optimization objective and the actual market settlement caliber and avoiding strategy deviations caused by the distortion of reward signals.

[0050] Furthermore, the benchmark price of the new energy unit in this invention is differentiated based on the confidence level of the predicted output, enabling the behavior strategy of the new energy unit to adaptively respond to the accuracy of the output prediction, and providing an adjustment mechanism for strategy modeling under uncertainty in high-penetration new energy scenarios; by limiting the application of electricity purchase curves during energy storage charging and electricity sales curves during discharging, and by using the charging and discharging efficiency coefficient to participate in the evaluation of charging and discharging economic benefits to determine the optimal charging and discharging power, the charging and discharging strategy of energy storage is simultaneously subject to the combined effects of state of charge constraints and efficiency and economic constraints, ensuring that the energy storage strategy converges within the physically feasible region.

[0051] Furthermore, by setting the convergence criterion of the Q matrix to either a change in the Q matrix being less than a preset threshold or the number of iterations reaching a preset maximum number of iterations, this invention avoids the problem of infinite waiting or premature termination caused by uncertain convergence speed under a single threshold condition, thus enhancing the engineering practicality of the simulation method.

[0052] It is understood that the beneficial effects of the second to fourth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 Flowchart of a multi-stakeholder decision-making method for the electricity market based on reinforcement learning, according to an embodiment of the present invention; Figure 2 A schematic diagram of the structure of a multi-stakeholder decision-making system for the electricity market based on reinforcement learning, according to an embodiment of the present invention. Detailed Implementation

[0055] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0056] Please see Figure 1 This invention proposes a multi-stakeholder decision-making method for the electricity market based on reinforcement learning, which mainly includes the following steps: S1. Collect market operation boundary conditions to form a standardized environmental state vector; the standardized environmental state vector includes load forecasting, new energy output forecasting, market supply-demand ratio, and electricity price forecasting. S2. Construct a subject-specific state space based on the standardized environmental state vector; the subject-specific state space consists of a general environmental state and a subject-specific state, used to define the current state of each market subject in each round; S3. Construct a unified action space, define several discrete action types in the action space, and each action type corresponds to a pricing benchmark adjustment coefficient. Different types of market participants adopt different pricing benchmark calculation methods for the same action according to their own characteristics. S4. In the current state, select an action from the action space according to the strategy, calculate the quotation benchmark price according to the quotation benchmark adjustment coefficient corresponding to the selected action, and generate a multi-segment ladder quotation curve based on the quotation benchmark price. S5. Based on the multi-segment tiered pricing curve, perform market clearing calculations, solve the safety-constrained unit combination problem and the economic dispatch problem, and obtain the winning bid volume and nodal marginal price for each entity. S6. Based on the winning bid volume and the marginal electricity price at each node, calculate the revenue of each market participant using a differential settlement mechanism, and use the revenue as a reward signal; S7. Using the Q-learning algorithm, update the Q-matrix of each state based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action to obtain the updated policy; the next state is defined by the subject's differentiated state space. S8. Repeat the above market clearing calculation, revenue calculation of each market participant, and Q matrix update process for multiple rounds of iterative deduction until the Q matrix converges, and obtain the decision-making scheme of multiple participants in the electricity market.

[0057] In one possible implementation, the standardized environment state vector formed in step S1 is S. t =(L t W t ,R t ,Ptfore ), where L t For load forecasting, W t For forecasting the output of new energy sources, R t For the market supply and demand ratio, Pt fore The above data is for electricity price forecasting; it comes from load forecasting of the dispatching system, renewable energy power forecasting of the meteorological department, and market analysis reports of the trading center. It can be obtained through data interfaces or simulation data from the basic model and operation data management center of the power market simulation and deduction system.

[0058] Each market participant, based on the current market state (including the general environmental state and its own specific state), selects a pricing strategy using the Q matrix, generating a multi-segment stepped pricing curve. In one possible implementation, when constructing the participant-differentiated state space based on the standardized environmental state vector in step S2, the general environmental state includes the market supply and demand state quantified by the supply-demand ratio. s 1. The electricity price forecast status represented by the spot price forecast s 2. And the status of new energy output level as represented by the new energy penetration rate. s 3.

[0059] The supply-demand ratio is calculated as follows: ; The penetration rate of new energy sources is calculated as follows: ρt = Wt / Lt.

[0060] Furthermore, in step S2, when constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a thermal power unit, the subject-specific state includes the marginal cost level determined by coal cost and unit efficiency. s 4. Contract holding ratio determined by the ratio of electricity allocated from medium- and long-term contracts to predicted electricity generation. s 5. Start / stop state constraints determined by the start / stop status of the previous time period and the remaining minimum start / stop time. s 6. Climbing capacity level, determined by the ratio of maximum climbing rate to the system's average climbing requirement. s 7. The ratio of electricity generated under medium- and long-term contracts to the projected electricity generation is calculated using the following formula: Based on the calculation results, the contract position ratio will be adjusted. s A score of 5 indicates low, medium, or high. Furthermore, in step S2, when constructing the subject-specific state space based on the standardized environmental state vector, if the market subject type is a new energy unit, the subject-specific state includes the predicted output confidence level based on historical prediction error statistics. s 4. Contract holding ratio determined by the ratio of electricity allocated from medium- and long-term contracts to predicted electricity generation. s 5. Curtailment rate determined by the wind or solar curtailment rate in the most recent period. s6. The sign and magnitude of the predicted output deviation. s 7. Among them, the confidence level of predicted output will be... s The four categories are low (<60%), medium (60%-85%), and high (>85%). The formula for calculating the ratio of medium- and long-term contract power generation to predicted power generation is as follows: Based on the calculation results, the contract position ratio will be adjusted. s The five-point scale is low, medium, and high; the curtailment rate is... s A score of 6 indicates low (<5%), medium (5%-15%), or high (>15%); the predicted output deviation sign is categorized as positive deviation, negative deviation, or close to the expected deviation.

[0061] Furthermore, in step S2, when constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is an energy storage subject, the subject-specific states include the state of charge level obtained by discretizing the state of charge, the peak-valley price difference expectation level calculated based on future electricity price forecasts, whether it participates in frequency regulation or the ancillary service market opportunity identifier, and the charge-discharge efficiency coefficient determined by the inherent parameters of the energy storage device. Specifically, the state of charge level is divided into low (<30%), medium (30%-70%), and high (>70%); the peak-valley price difference expectation level is divided into small, medium, and large; and the charge-discharge efficiency coefficient can participate in state combinations.

[0062] Furthermore, in step S2, when constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a load-side subject, the subject-specific states include the predicted load deviation level reflecting the accuracy of electricity consumption forecasting, the contract position ratio determined by the ratio of the electricity output from medium- and long-term contracts to the predicted electricity consumption, the user price elasticity level, and the interruptible load ratio level reflecting the demand response potential. Specifically, the predicted load deviation level is divided into negative deviation (<-5%), normal (-5%~5%), and positive deviation (>5%); the user price elasticity level is divided into low, medium, and high. The aforementioned state variables are all discretized into a finite number of integer levels to form a finite Q matrix dimension.

[0063] In one possible implementation, the unified action space constructed in step S3 is: a ∈{ a 1, a 2,..., a k The pricing benchmark is uniform for all entities, but different entities use different methods to calculate the pricing benchmark for the same action. Discrete action types are defined in the table below:

[0064] For the load-side entity, the action is used to determine the electricity purchase price, using the benchmark electricity utility. U0Substitute marginal cost and adjust in the opposite direction (increase the number of bids during low-price periods and decrease the number of bids during high-price periods).

[0065] For energy storage, the action can be expanded into a composite action consisting of charging and discharging actions. a ch , a dis These correspond to the charging and discharging strategies, respectively. The charging and discharging power must satisfy the complementary constraint P. ch,t ·P dis,t =0, and satisfies the SOC boundary.

[0066] In one possible implementation, step S4 generates the generation-side / energy storage discharge quotation curve as follows: Given the current state s and the selected action a First, calculate the benchmark price P. base :

[0067] in, MC Marginal cost of thermal power (new energy) MC =0, energy storage discharge MC (Considering charging costs and aging costs).

[0068] Generate N-segment tiered pricing curves, with capacity segments divided equally:

[0069] The price segments increase linearly:

[0070] The final output is a set of quote pairs {( Q 1, P 1),( Q 2, P 2),...,( Q N , P N )}.

[0071] The method for generating the price curve of new energy units in step S4 is as follows: New energy units in P base Based on the prediction confidence level s conf Make corrections:

[0072] in, η (0) = 0.9 (Price reduction to ensure consumption when confidence level is low). η(1) = 1.0, η (2) = 1.1 (The price can be raised appropriately when the confidence level is high).

[0073] The method for generating energy storage charging and discharging power decisions in step S4 is as follows: Energy storage depends on state s and compound movements ( a ch , a dis Determine the charging and discharging power.

[0074] When charging, declare the electricity purchase curve (as load); when discharging, declare the electricity sales curve (as power source).

[0075] The charging and discharging power is constrained by the state of charge (SOC), and the optimal charging and discharging timing and power level are obtained by learning the Q matrix.

[0076] The method for generating the load-side electricity purchase price curve in step S4 is as follows: The load side generates a demand-side declaration curve based on predicted load and power purchase strategy. The actions are the opposite of those on the generation side: a low-price strategy increases off-peak electricity declarations, while a high-price strategy reduces peak-peak electricity declarations. The benchmark for the power purchase price is the market-expected electricity price, and the direction of adjustment is determined by the actions taken.

[0077] In one possible implementation, when step S4 describes selecting an action from the action space according to a strategy in the current state, in the current state... s Under these circumstances, each market participant selects the current [opportunity] with a probability of 1-ε. Q The action with the highest value is calculated using the following expression:

[0078] The exploration is carried out by randomly and uniformly selecting actions with a probability ε, where ε is the preset exploration probability. After training, each market participant makes decisions based entirely on the action with the highest current Q value, and no longer conducts random exploration, i.e., ε=0.

[0079] In one possible implementation, the electricity market generally adopts a price difference settlement mechanism of "medium- and long-term contracts + spot price deviation". The revenue of power generation entities consists of three parts: revenue from medium- and long-term contracts, revenue from spot price deviation, and other expenses (subsidies, ancillary services, capacity compensation, etc.), minus operating costs. This invention uses the price difference settlement method to calculate the entity's revenue, serving as a reward signal for updating the Q matrix. The revenue value reflects the differences in outcomes of different decision-making behaviors under specific market conditions, providing the Q matrix with "state and action" evaluation scalars. This allows the agent to establish a "state-behavior-outcome" mapping relationship during the iteration process, realizing the modeling of multi-entity decision-making patterns. Specifically, in step S6, when calculating the revenue of each market entity using the price difference settlement mechanism, the revenue calculation expression for power generation entities (thermal power, new energy, and energy storage discharge) is as follows:

[0080] In the formula, express t Electricity allocation for medium- and long-term contracts; express t Prices of medium- and long-term contracts during the period; Indicates the main body of the power generation side t Electricity volume won in the spot market during the specified period; express t Marginal electricity price at specific time points; This indicates operating costs, including fuel costs, maintenance costs, etc. Indicates start-up and shutdown costs; This refers to other income, including subsidies, ancillary service revenue, capacity compensation, etc. T This indicates the total number of transaction periods included in the settlement cycle; The formula for calculating the revenue of the load-side main body is as follows:

[0081] In the formula, This indicates the cost of deviation assessment; Indicates the main body of the load side t The electricity consumption for which the winning bid was determined through market clearing during the specified period; R This represents the total revenue of the power generation entity during the settlement period.

[0082] In one possible implementation, the calculation expression for updating the Q matrix using the Q-learning algorithm is as follows:

[0083] In the formula, α is the learning rate and α∈(0,1]; the larger the value of α, the faster the update but the more likely it is to cause oscillations; the smaller the value of α, the smoother the update but the slower the convergence. In the continuous decision-making problem of the electricity market, α is usually taken as 0.1~0.3, and gradually decreased as the number of training rounds increases, in order to balance the rapid learning in the early stage of exploration with the stable convergence in the later stage. γ is the discount factor and γ∈[0,1], which measures the importance of future rewards relative to immediate rewards. The closer γ is to 1, the more the subject values ​​long-term returns; the closer γ is to 0, the more short-sighted the subject is. Since the current price will affect the subsequent market state and returns, γ is generally taken as 0.9~0.99.

[0084] Repeat the above steps until the change in the Q matrix is ​​less than a preset threshold or the preset number of iterations is reached. After training, a converged Q matrix is ​​obtained. In practical applications, the pre-trained Q matrix is ​​directly loaded, and the subject no longer performs random exploration, but follows the established procedure. Select an action (i.e., ε=0) to quickly make a decision on the optimal pricing strategy.

[0085] In summary, the decision simulation method of this invention first constructs a market environment perception layer, collecting boundary conditions such as load forecasting, renewable energy output, supply-demand ratio, and electricity price forecasting to form a standardized environmental state vector. Secondly, it constructs a differentiated state space, setting general environmental states (supply-demand state, electricity price forecast, renewable energy penetration rate) and specific states (marginal cost and contract holdings of thermal power, confidence level and curtailment rate of renewable energy forecasts, SOC and peak-valley price difference of energy storage, load forecasting deviation and elasticity, etc.) for different types of entities such as thermal power, renewable energy, energy storage, and load side, and discretizing the continuous state into finite integer levels. Finally, it designs a unified action space, using a pricing benchmark... The system adjusts the coefficients to generate differentiated pricing strategies such as conservative high price, stable return, marginal cost, and aggressive low price. Energy storage adopts a combined charging and discharging action, with the load-side action direction opposite to the generation-side action. The entities select actions according to an ε-greedy strategy in the current state, calculate the benchmark price for bidding, and generate multi-segment tiered bidding curves. These are submitted to the market clearing module to solve the safety constraint unit combination and economic dispatch problem, obtaining the winning bid volume and nodal marginal electricity price. Then, the revenue of each entity is calculated according to the price difference settlement mechanism. Finally, the revenue is used as a reward signal, and the Q-learning algorithm is used to update the Q matrix. After multiple rounds of iterative deduction until the Q matrix converges, the decision-making behavior of multiple entities gradually evolves autonomously.

[0086] This invention overcomes the shortcomings of traditional methods, such as independent modeling of different entities, rigid strategies, and difficulty in adaptation. It can comprehensively reflect the differentiated bidding behavior of various entities, including thermal power, new energy, energy storage, and load side, under different market environments. It supports flexible configuration of multi-dimensional states such as supply and demand status, price expectations, contract positions, and their own operating characteristics, meeting the research needs of power market simulation for the heterogeneity of entities and the evolution of strategies. It covers key influencing factors such as marginal cost, prediction confidence, state of charge, and load elasticity, adapting to the market pattern of multiple entities coexisting in the new power system. All state spaces and action spaces adopt clear discretization rules and mathematical definitions, ensuring the reproducibility and interpretability of strategy learning.

[0087] Please see Figure 2 Another embodiment of the present invention proposes a multi-agent decision-making system for the electricity market based on reinforcement learning, comprising: The environmental state vector generation module 201 is used to collect market operation boundary conditions and form a standardized environmental state vector; the standardized environmental state vector includes load forecast, new energy output forecast, market supply and demand ratio and electricity price forecast. The differentiated state space construction module 202 is used to construct the subject differentiated state space based on the standardized environmental state vector; the subject differentiated state space consists of a general environmental state and a subject-specific state, and is used to define the current state of each market subject in each round; Action space construction module 203 is used to construct a unified action space, in which several discrete action types are defined. Each action type corresponds to a pricing benchmark adjustment coefficient. Different types of market participants use different pricing benchmark calculation methods for the same action according to their own characteristics. The multi-segment tiered pricing curve generation module 204 is used to select an action from the action space according to a strategy in the current state, calculate the pricing benchmark price according to the pricing benchmark adjustment coefficient corresponding to the selected action, and generate a multi-segment tiered pricing curve based on the pricing benchmark price. The market clearing calculation module 205 is used to perform market clearing calculations based on multi-segment tiered pricing curves, solve the safety-constrained unit combination problem and the economic dispatch problem, and obtain the winning bid volume and nodal marginal price of each entity. The market entity revenue calculation module 206 is used to calculate the revenue of each market entity based on the winning bid volume and the marginal electricity price at each node, using a differential settlement mechanism, and to use the revenue as a reward signal. The policy update module 207 is used to update the Q matrix of each subject using the Q-learning algorithm based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, so as to obtain the updated policy; the next state is defined by the subject's differentiated state space. The iterative deduction module 208 is used to repeat the above-mentioned market clearing calculation, revenue calculation of each market participant, and Q matrix update process for multiple rounds of iterative deduction until the Q matrix converges, thus obtaining the decision scheme of multiple participants in the power market.

[0088] In one possible implementation, the general environmental states constructed by the differentiated state space construction module 202 include the market supply and demand state quantified by the supply and demand ratio, the electricity price forecast state characterized by the spot forecast electricity price, and the new energy output level state characterized by the new energy penetration rate.

[0089] In one possible implementation, when the differentiated state space construction module 202 constructs the subject-specific state, if the market subject type is a thermal power unit, the subject-specific state includes the marginal cost level determined by coal cost and unit efficiency, the contract holding ratio determined by the ratio of the electricity generated from medium- and long-term contracts to the predicted power generation, the start-stop state constraint determined by the start-stop state of the previous period and the remaining value of the minimum start-stop time, and the ramping capacity level determined by the ratio of the maximum ramping rate to the average ramping demand of the system.

[0090] In one possible implementation, when the differentiated state space construction module 202 constructs the subject-specific state, if the market subject type is a new energy unit, the subject-specific state includes the predicted output confidence based on historical prediction error statistics, the contract holding ratio determined by the ratio of the electricity decomposed by medium and long-term contracts to the predicted power generation, the curtailment rate determined by the curtailment rate of wind or solar power in the most recent period, and the sign and magnitude of the predicted output deviation.

[0091] In one possible implementation, when the differentiated state space construction module 202 constructs the subject-specific state, when the market subject type is an energy storage subject, the subject-specific state includes the state of charge level obtained by discretization of the state of charge, the peak-valley price difference expected level calculated based on future electricity price forecasts, whether it participates in the frequency regulation or standby ancillary service market opportunity identifier, and the charge and discharge efficiency coefficient determined by the inherent parameters of the energy storage device.

[0092] In one possible implementation, when the differentiated state space construction module 202 constructs subject-specific states, if the market subject type is a load-side subject, the subject-specific states include a predicted load deviation level reflecting the accuracy of electricity consumption forecasting, a contract position ratio determined by the ratio of the electricity output from medium- and long-term contracts to the predicted electricity consumption, a user price elasticity level, and an interruptible load ratio level reflecting the demand response potential.

[0093] In one possible implementation, the discrete action types defined by the action space construction module 203 include conservative high price, stable return, marginal cost, aggressive low price, extreme high price, and extreme low price, with each action type corresponding to a different pricing benchmark adjustment coefficient. When the action space construction module constructs a unified action space, the action of the energy storage entity is expanded into a composite action consisting of charging and discharging actions, with the charging and discharging power satisfying complementary constraints and state of charge boundary constraints. The action adjustment direction of the load-side entity is opposite to that of the generation-side entity, with the load-side entity increasing the bidding volume during low-price periods and decreasing the bidding volume during high-price periods.

[0094] In one possible implementation, the multi-segment tiered pricing curve generation module 204 uses the pricing benchmark adjustment coefficient corresponding to the selected action multiplied by the pricing benchmark price as the benchmark price, divides the total capacity of the unit into N equal segments, sets the price of each segment linearly increasing from the benchmark price, and outputs a set of pricing pairs consisting of the capacity of each segment and the corresponding price, where N is the preset number of segments.

[0095] In one possible implementation, when the multi-segment tiered pricing curve generation module 204 selects an action from the action space according to the strategy in the current state, each market participant selects the action with the highest current Q value with a probability of 1-ε, and randomly and uniformly selects actions for exploration with a probability of ε, where ε is a preset exploration probability; after training is completed, each market participant makes decisions entirely according to the action with the highest current Q value, and no longer conducts random exploration, i.e., ε=0.

[0096] In one possible implementation, the policy update module 207 employs the Q-learning algorithm to update the respective Q matrices based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action. When obtaining the updated policy, the calculation expression is as follows:

[0097] In the formula, α is the learning rate and α∈(0,1], and γ is the discount factor and γ∈[0,1]; wherein, the learning rate α gradually decreases as the number of iteration rounds increases.

[0098] In one possible implementation, when the market entity revenue calculation module 206 calculates the revenue of each market entity based on the winning bid volume and the nodal marginal price, using a differential settlement mechanism, the revenue calculation expression for the generation-side entity is as follows:

[0099] In the formula, express t Electricity allocation for medium- and long-term contracts; express t Prices of medium- and long-term contracts during the period; Indicates the main body of the power generation sidet Electricity volume won in the spot market during the specified period; express t Marginal electricity price at specific time points; Indicates operating costs; Indicates start-up and shutdown costs; Indicates other income; T This indicates the total number of transaction periods included in the settlement cycle; The formula for calculating the revenue of the load-side main body is as follows:

[0100] In the formula, This indicates the cost of deviation assessment; Indicates the main body of the load side t The electricity consumption for which the winning bid was determined through market clearing during the specified period; R This represents the total revenue of the power generation entity during the settlement period.

[0101] In one possible implementation, when generating the multi-segment tiered pricing curve, the multi-segment tiered pricing curve generation module 204 adjusts the benchmark price of the new energy unit based on the predicted output confidence level. When the confidence level is low, the benchmark price is lowered, and when the confidence level is high, the benchmark price is raised. When the energy storage entity is charging, it submits a power purchase curve, and when it is discharging, it submits a power sales curve. The charging and discharging power is constrained by the state of charge boundary, and the charging and discharging efficiency coefficient is used to evaluate the economic benefits of charging and discharging to determine the optimal charging and discharging power.

[0102] In one possible implementation, the convergence condition of the Q matrix by the iterative deduction module 208 is: the change in the Q matrix is ​​less than a preset threshold, or the number of iterative deduction rounds reaches a preset maximum number of rounds.

[0103] Another embodiment of the present invention also proposes an electronic device including a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the reinforcement learning-based multi-agent decision-making method for the electricity market.

[0104] Another embodiment of the present invention also proposes a computer-readable storage medium storing at least one instruction that, when executed by a processor, implements the reinforcement learning-based multi-agent decision-making method for the electricity market.

[0105] The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals. For ease of explanation, the above content only shows the parts related to the embodiments of the present invention; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This computer-readable storage medium is non-transitory and can be stored in storage devices formed by various electronic devices, enabling the execution process described in the method of the embodiments of the present invention.

[0106] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0108] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-stakeholder decision-making method for the electricity market based on reinforcement learning, characterized in that, include: Collect market operation boundary conditions to form a standardized environmental state vector; the standardized environmental state vector includes load forecast, renewable energy output forecast, market supply and demand ratio, and electricity price forecast. A subject-specific state space is constructed based on a standardized environmental state vector; the subject-specific state space consists of a general environmental state and a subject-specific state, which is used to define the current state of each market subject in each round; Construct a unified action space, define several discrete action types in the action space, and each action type corresponds to a pricing benchmark adjustment coefficient. Different types of market participants adopt different pricing benchmark calculation methods for the same action according to their own characteristics. In the current state, select an action from the action space according to the strategy, calculate the quotation benchmark price based on the quotation benchmark adjustment coefficient corresponding to the selected action, and generate a multi-segment ladder quotation curve based on the quotation benchmark price. Market clearing calculations are performed based on multi-segment tiered pricing curves to solve the safety-constrained unit combination problem and the economic dispatch problem, thereby obtaining the winning bid volume and nodal marginal price for each entity; Based on the winning bid volume and the marginal electricity price at each node, the revenue of each market participant is calculated using a differential settlement mechanism, and the revenue is used as a reward signal. The Q-learning algorithm is used to update the Q matrix of each state based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, so as to obtain the updated policy. The next state is defined by the subject-differentiated state space; Repeat the above market clearing calculation, revenue calculation for each market participant, and Q matrix update process for multiple rounds of iterative deduction until the Q matrix converges, thus obtaining a multi-stakeholder decision-making scheme for the electricity market.

2. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, In the step of constructing the subject-differentiated state space based on the standardized environmental state vector, the general environmental state includes the market supply and demand state quantified by the supply and demand ratio, the electricity price forecast state represented by the spot forecast electricity price, and the new energy output level state represented by the new energy penetration rate.

3. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 2, characterized in that, In the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a thermal power unit, the subject-specific state includes the marginal cost level determined by coal cost and unit efficiency, the contract holding ratio determined by the ratio of the electricity allocated from medium- and long-term contracts to the predicted power generation, the start-stop state constraints determined by the start-stop state of the previous period and the remaining value of the minimum start-stop time, and the ramping capacity level determined by the ratio of the maximum ramping rate to the average ramping demand of the system.

4. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 2, characterized in that, In the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a new energy unit, the subject-specific state includes the predicted output confidence based on historical prediction error statistics, the contract holding ratio determined by the ratio of the decomposed electricity of medium and long-term contracts to the predicted power generation, the curtailment rate determined by the curtailment rate of wind or solar power in the most recent period, and the sign and magnitude of the predicted output deviation.

5. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 2, characterized in that, In the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is an energy storage subject, the subject-specific state includes the state of charge level obtained by discretization of the state of charge, the expected level of peak-valley price difference calculated based on future electricity price forecasts, whether it participates in the frequency regulation or standby ancillary service market opportunity identifier, and the charge and discharge efficiency coefficient determined by the inherent parameters of the energy storage device.

6. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 2, characterized in that, In the step of constructing the subject-specific state space based on the standardized environmental state vector, when the market subject type is a load-side subject, the subject-specific state includes the predicted load deviation level reflecting the accuracy of electricity consumption forecasting, the contract position ratio determined by the ratio of the electricity decomposed from medium- and long-term contracts to the predicted electricity consumption, the user price elasticity level, and the interruptible load ratio level reflecting the demand response potential.

7. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, In the step of defining several discrete action types in the action space, the discrete action types include conservative high price, stable return, marginal cost, aggressive low price, extreme high price, and extreme low price, and each action type corresponds to a different quotation benchmark adjustment coefficient; the action of the energy storage subject is extended to a composite action consisting of charging action and discharging action, and the charging and discharging power satisfies complementary constraints and the state of charge boundary constraints; the action adjustment direction of the load-side subject is opposite to that of the generation-side subject, and the load-side subject increases the bidding volume during low-price periods and decreases the bidding volume during high-price periods.

8. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, The steps for generating a multi-segment tiered pricing curve include: multiplying the pricing benchmark adjustment coefficient corresponding to the selected action by the pricing benchmark price as the benchmark price; dividing the total capacity of the unit into N equal segments; setting the price of each segment linearly increasing from the benchmark price; and outputting a set of pricing pairs consisting of the capacity of each segment and the corresponding price, where N is the preset number of segments.

9. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, In the step of selecting an action from the action space according to the strategy in the current state, each market participant selects the action with the highest current Q value with a probability of 1-ε, and randomly and evenly selects actions for exploration with a probability of ε, where ε is the preset exploration probability. After training is completed, each market participant makes decisions solely based on the action with the highest current Q value, and no longer engages in random exploration, i.e., ε=0.

10. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, The Q-learning algorithm is used to update the Q-matrix of each state based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, thus obtaining the updated policy. The calculation expression is as follows: In the formula, α is the learning rate and α∈(0,1], and γ is the discount factor and γ∈[0,1]; wherein, the learning rate α gradually decreases as the number of iteration rounds increases.

11. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, In the step of calculating the revenue of each market participant based on the winning bid volume and the marginal electricity price at each node using the differential settlement mechanism, the revenue calculation expression for the power generation side is as follows: In the formula, express t Electricity allocation for medium- and long-term contracts; express t Prices of medium- and long-term contracts during the period; Indicates the main body of the power generation side t Electricity volume won in the spot market during the specified period; express t Marginal electricity price at specific time points; Indicates operating costs; Indicates start-up and shutdown costs; Indicates other income; T This indicates the total number of transaction periods included in the settlement cycle; The formula for calculating the revenue of the load-side main body is as follows: In the formula, This indicates the cost of deviation assessment; Indicates the main body of the load side t The electricity consumption for which the winning bid was determined through market clearing during the specified period; R This represents the total revenue of the power generation entity during the settlement period.

12. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, In the step of generating multi-segment tiered pricing curves, the benchmark price for new energy generating units is adjusted based on the confidence level of predicted output. When the confidence level is low, the benchmark price is lowered, and when the confidence level is high, the benchmark price is raised. When the energy storage entity is charging, it submits a power purchase curve, and when it is discharging, it submits a power sales curve. The charging and discharging power is constrained by the state of charge boundary, and the charging and discharging efficiency coefficient is used to evaluate the economic benefits of charging and discharging to determine the optimal charging and discharging power.

13. The multi-stakeholder decision-making method for the electricity market based on reinforcement learning according to claim 1, characterized in that, The convergence criterion for the Q matrix is: the change in the Q matrix is ​​less than a preset threshold, or the number of iterations reaches a preset maximum number of iterations.

14. A multi-stakeholder decision-making system for the electricity market based on reinforcement learning, characterized in that, include: An environmental state vector generation module is used to collect market operation boundary conditions and form a standardized environmental state vector; the standardized environmental state vector includes load forecasting, new energy output forecasting, market supply and demand ratio, and electricity price forecasting. The differentiated state space construction module is used to construct the subject's differentiated state space based on the standardized environmental state vector; the subject's differentiated state space consists of a general environmental state and a subject-specific state, and is used to define the current state of each market subject in each round; The action space construction module is used to construct a unified action space, in which several discrete action types are defined. Each action type corresponds to a pricing benchmark adjustment coefficient. Different types of market participants use different pricing benchmark calculation methods for the same action according to their own characteristics. The multi-segment tiered pricing curve generation module is used to select an action from the action space according to a strategy in the current state, calculate the pricing benchmark price based on the pricing benchmark adjustment coefficient corresponding to the selected action, and generate a multi-segment tiered pricing curve based on the pricing benchmark price. The market clearing calculation module is used to perform market clearing calculations based on multi-segment tiered pricing curves, solve the safety-constrained unit combination problem and the economic dispatch problem, and obtain the winning bid volume and nodal marginal price for each entity. The market entity revenue calculation module is used to calculate the revenue of each market entity based on the winning bid volume and the marginal electricity price at each node, using a differential settlement mechanism, and to use the revenue as a reward signal. The policy update module is used to update the Q-matrix of each state according to the current state, the selected action, the reward signal, and the next state obtained after executing the current action, so as to obtain the updated policy. The next state is defined by the subject-differentiated state space; The iterative deduction module is used to repeat the above-mentioned market clearing calculation, revenue calculation of each market participant, and Q matrix update process for multiple rounds of iterative deduction until the Q matrix converges, thus obtaining the decision-making scheme of multiple participants in the electricity market.

15. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, The general environmental states constructed by the differentiated state space construction module include the market supply and demand state quantified by the supply and demand ratio, the electricity price forecast state represented by the spot forecast electricity price, and the new energy output level state represented by the new energy penetration rate.

16. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 15, characterized in that, When the differentiated state space construction module constructs the subject-specific state, if the market subject type is a thermal power unit, the subject-specific state includes the marginal cost level determined by coal cost and unit efficiency, the contract holding ratio determined by the ratio of the electricity allocated from medium- and long-term contracts to the predicted power generation, the start-stop state constraint determined by the start-stop state of the previous period and the remaining value of the minimum start-stop time, and the ramping capacity level determined by the ratio of the maximum ramping rate to the average ramping demand of the system.

17. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 15, characterized in that, When the differentiated state space construction module constructs the subject-specific state, if the market subject type is a new energy unit, the subject-specific state includes the predicted output confidence based on historical prediction error statistics, the contract holding ratio determined by the ratio of the electricity decomposed by medium and long-term contracts to the predicted power generation, the curtailment rate determined by the curtailment rate of wind or solar power in the most recent period, and the sign and magnitude of the predicted output deviation.

18. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 15, characterized in that, When the differentiated state space construction module constructs the subject-specific state, if the market subject type is an energy storage subject, the subject-specific state includes the state of charge level obtained by discretization of the state of charge, the expected level of peak-valley price difference calculated based on future electricity price forecasts, whether it participates in frequency regulation or the ancillary service market opportunity identifier, and the charging and discharging efficiency coefficient determined by the inherent parameters of the energy storage device.

19. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 15, characterized in that, When the differentiated state space construction module constructs the subject-specific state, if the market subject type is a load-side subject, the subject-specific state includes the predicted load deviation level reflecting the accuracy of electricity consumption forecasting, the contract position ratio determined by the ratio of the electricity decomposed from medium- and long-term contracts to the predicted electricity consumption, the user price elasticity level, and the interruptible load ratio level reflecting the demand response potential.

20. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, The discrete action types defined by the action space construction module include conservative high price, stable return, marginal cost, aggressive low price, extreme high price, and extreme low price. Each action type corresponds to a different pricing benchmark adjustment coefficient. When the action space construction module constructs a unified action space, the action of the energy storage entity is expanded into a composite action consisting of charging and discharging actions. The charging and discharging power satisfies complementary constraints and the state of charge boundary constraints. The action adjustment direction of the load-side entity is opposite to that of the generation-side entity. The load-side entity increases the bidding volume during low-price periods and decreases the bidding volume during high-price periods.

21. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, The multi-segment tiered pricing curve generation module uses the pricing benchmark adjustment coefficient corresponding to the selected action multiplied by the pricing benchmark price as the benchmark price, divides the total capacity of the unit into N equal segments, sets the price of each segment linearly increasing from the benchmark price, and outputs a set of pricing pairs consisting of the capacity of each segment and the corresponding price, where N is the preset number of segments.

22. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, When the multi-segment tiered pricing curve generation module selects an action from the action space according to the strategy in the current state, each market participant selects the action with the highest current Q value with a probability of 1-ε, and randomly and evenly selects actions for exploration with a probability of ε, where ε is the preset exploration probability. After training is completed, each market participant makes decisions solely based on the action with the highest current Q value, and no longer engages in random exploration, i.e., ε=0.

23. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, The policy update module employs the Q-learning algorithm. Based on the current state, the selected action, the reward signal, and the next state obtained after executing the current action, it updates the respective Q matrices to obtain the updated policy. The calculation expression is as follows: In the formula, α is the learning rate and α∈(0,1], and γ is the discount factor and γ∈[0,1]; wherein, the learning rate α gradually decreases as the number of iteration rounds increases.

24. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, When the market entity revenue calculation module calculates the revenue of each market entity based on the winning bid volume and the marginal electricity price at each node, using the differential settlement mechanism, the revenue calculation expression for the power generation side entity is as follows: In the formula, express t Electricity allocation for medium- and long-term contracts; express t Prices of medium- and long-term contracts during the period; Indicates the main body of the power generation side t Electricity volume won in the spot market during the specified period; express t Marginal electricity price at specific time points; Indicates operating costs; Indicates start-up and shutdown costs; Indicates other income; T This indicates the total number of transaction periods included in the settlement cycle; The formula for calculating the revenue of the load-side main body is as follows: In the formula, This indicates the cost of deviation assessment; Indicates the main body of the load side t The electricity consumption for which the winning bid was determined through market clearing during the specified period; R This represents the total revenue of the power generation entity during the settlement period.

25. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, When generating the multi-segment tiered pricing curve, the pricing benchmark price of the new energy unit is adjusted based on the confidence level of the predicted output. When the confidence level is low, the pricing benchmark price is lowered, and when the confidence level is high, the pricing benchmark price is raised. When the energy storage entity is charging, it submits a power purchase curve, and when it is discharging, it submits a power sales curve. The charging and discharging power is constrained by the state of charge boundary, and the charging and discharging efficiency coefficient is used to participate in the economic benefit assessment of charging and discharging to determine the optimal charging and discharging power.

26. The multi-stakeholder decision-making system for the electricity market based on reinforcement learning according to claim 14, characterized in that, The convergence criteria for the Q matrix in the iterative deduction module are: the change in the Q matrix is ​​less than a preset threshold, or the number of iterative deduction rounds reaches a preset maximum number of rounds.

27. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the reinforcement learning-based multi-agent decision-making method for the electricity market as described in any one of claims 1 to 13.

28. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the reinforcement learning-based multi-agent decision-making method for the electricity market as described in any one of claims 1 to 13.

Images