A multi-agent simulation system for electricity market bidding decision-making

Abstract

This invention discloses a multi-agent simulation system for bidding decision-making in the electricity market, comprising an environment simulation module and a multi-agent module. The environment simulation module includes a market clearing unit and a market settlement unit, used to execute the market clearing algorithm and perform financial settlement, generating reward signals and market environment states. The multi-agent module contains multiple agents, each consisting of an agent policy network and an agent action decoder. The multi-agent module receives reward signals and market environment state information from the environment simulation module, and performs autonomous policy learning and bidding decision-making. The agent action decoder includes a raw action receiving and parsing unit, a normalization and scaling unit, a constraint projection and structure integration unit, and a compliant bid output unit. This invention significantly improves the convergence speed and policy economy of multi-agent reinforcement learning in electricity market simulation.

Description

Technical Field

[0001] This invention relates to the technical field of power market simulation and multi-agent reinforcement learning, and in particular to a multi-agent simulation system for power market bidding decision-making. Background Technology

[0002] With the deepening construction of new power systems primarily based on new energy sources, the structure, participants, and trading mechanisms of the electricity market are becoming increasingly complex. The large-scale grid connection of intermittent power sources such as wind and solar power, along with the emergence of new market participants such as independent energy storage, virtual power plants, and flexible loads, has resulted in highly heterogeneous, dynamic, and interactive behavioral strategies among market participants. To gain a deeper understanding of the dynamic evolution of the market, evaluate the effectiveness of new market mechanism designs, predict participant strategies and behaviors, and mitigate market risks, simulation methods based on Multi-Agent Systems (MAS) have become an indispensable core tool in electricity market research.

[0003] In a multi-agent simulation framework, traditional market participants such as power generators, electricity retailers, and large users are modeled as agents with autonomous perception, decision-making, and learning capabilities. Each agent typically employs reinforcement learning (RL) algorithms to learn optimal bidding or trading strategies by continuously interacting with the environment (i.e., the simulated market) to maximize its long-term gains (such as profits). This approach effectively simulates the dynamic process of participants making independent decisions and engaging in game-playing based on local information in the real market, thus providing insights into dynamic equilibrium and strategic interactions that are difficult to obtain with traditional optimization models.

[0004] However, introducing high-fidelity electricity market rules into large-scale multi-agent reinforcement learning simulations faces two major technical challenges:

[0005] First, there's the "curse of dimensionality" in the action space. To accurately reflect the operation of the electricity market, agents typically need to submit high-dimensional segmented bidding curves, i.e., supply or demand curves composed of multiple price-quantity pairs arranged in a monotonically non-decreasing order. For an agent that needs to submit B segments of bids, the dimension of its original action space is at least B (if only bidding is done) or 2B (if bidding includes quantity), and the value of B is usually large (e.g., 10 segments or more). When the system contains dozens or even hundreds of agents, the dimension of the joint action space will expand exponentially, resulting in extremely low training efficiency, slow convergence, or even failure to converge for reinforcement learning algorithms, severely limiting the scale and practicality of the simulation system.

[0006] Second, the "hard constraints" of market rules must be met. Electricity market bidding is subject to strict rule constraints, the most crucial of which are monotonicity constraints and price boundary constraints. Standard reinforcement learning agents output unconstrained continuous or discrete values, failing to guarantee that their output action vectors naturally satisfy these complex engineering and economic constraints. Traditional solutions often employ "post-processing" methods, such as forcing monotonicity through projection algorithms (e.g., isotonic regression PAVA) after the agent outputs an action, and satisfying boundary constraints through pruning operations. These methods have significant drawbacks: 1) Post-processing operations are often non-differentiable, severing the gradient backflow path from the final market revenue (reward) to the policy network parameters, preventing the agent from directly learning how to generate "compliant and optimal" actions through gradient signals, resulting in unstable training and suboptimal policies; 2) They increase additional computational overhead.

[0007] In summary, existing technologies struggle to achieve both high efficiency and feasibility in large-scale multi-agent reinforcement learning simulations while maintaining simulation realism. Therefore, there is an urgent need for an innovative system architecture that can "embed" market rules into the decision-making process of agents, fundamentally achieving intelligent dimensionality reduction of the action space and automatic constraint fulfillment, thereby supporting large-scale, high-efficiency, and high-fidelity multi-agent simulations of the power market. This invention presents a systematic solution to address this pressing need. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a multi-agent simulation system for bidding decision-making in the electricity market. This system introduces a differentiable agent action decoder based on Bézier curves to automatically decode the low-dimensional, unconstrained parameters output by the agent policy network into a high-dimensional segmented bidding curve that fully conforms to market rules. This greatly reduces the learning difficulty of the agents and improves the training efficiency and stability of the entire simulation system while ensuring the realism of the simulation.

[0009] To achieve the above objectives, the technical solution provided by this invention is: a multi-agent simulation system for electricity market bidding decision-making, comprising:

[0010] An environment simulation module is used to simulate the core trading and settlement processes of the electricity market. The environment simulation module includes a market clearing unit and a market settlement unit. The market clearing unit is used to receive bidding information submitted by market participants, execute a preset market clearing algorithm, and determine the market clearing result, including the winning bid volume and the clearing price. The market settlement unit is used to perform financial settlement based on the market clearing result determined by the market clearing unit, generate reward signals to drive the intelligent agent's bidding strategy update, and generate market environment status information including the market clearing result and financial settlement information.

[0011] A multi-agent module receives reward signals and market environment status information from the environment simulation module, performs autonomous strategy learning and bidding decisions, and generates bidding information to be submitted to the market clearing unit. The multi-agent module contains multiple independent agents, each representing a market participant. Each agent includes an agent policy network and an agent action decoder. The agent policy network learns and updates its internal agent bidding strategy based on the reward signals received from the environment simulation module, calculates bidding decisions based on the received market environment status information, and outputs a low-dimensional control point parameter vector representing the agent's bidding strategy. The agent action decoder receives the low-dimensional control point parameter vector and converts it into a high-dimensional piecewise bidding curve that fully conforms to the electricity market bidding specifications through a data decoding process based on the mathematical properties of Bézier curves, and submits it to the market clearing unit.

[0012] Furthermore, the intelligent agent action decoder includes:

[0013] The original action receiving and parsing unit is used to receive the low-dimensional control point parameter vector output by the agent policy network, and use the softplus function to perform a differentiable transformation on the low-dimensional control point parameter vector to obtain a positive value sequence in which all components are positive.

[0014] The normalization and scaling unit is used to map the positive value sequence into a set of effective control point sequences for defining the Bézier curve. The effective control point sequence achieves price boundary constraints by fixing the first and last control points, and constructs intermediate control points by accumulating the normalized positive value sequence to achieve the monotonically increasing condition.

[0015] The constraint projection and structure integration unit is used to receive the effective control point sequence and generate a Bézier curve through Bernstein polynomial. Based on the mathematical properties of the Bézier curve, the Bézier curve can be determined as the continuous bidding curve function of the agent under the strict monotonically increasing condition and price boundary constraint.

[0016] The compliant pricing output unit is used to uniformly discretely sample the continuous bidding curve function, with the number of sampling points equal to the number of bidding segments stipulated by the market. The function value corresponding to each sampling point is used as the price of the corresponding bidding segment, thereby generating the final high-dimensional segmented bidding curve and returning it to the environment simulation module.

[0017] Furthermore, the market clearing unit executes a pre-set market clearing algorithm, which is a safety-constrained unit combination algorithm and a safety-constrained economic dispatch algorithm. The safety-constrained unit combination algorithm and the safety-constrained economic dispatch algorithm are based on a mixed-integer linear programming model. The electricity market clearing is achieved by calling a commercial solver to solve the mixed-integer linear programming model and determining the market clearing result, which includes the winning bid amount and clearing price of each agent.

[0018] Furthermore, the specific process of financial settlement by the market settlement unit includes: calculating the net settlement income of each agent based on the market clearing result, which serves as a reward signal to drive its strategy update; and generating market environment status information to characterize the market operation based on the market clearing result and financial settlement information.

[0019] Furthermore, the agent policy network is a policy model constructed based on a deep reinforcement learning algorithm. It updates the internal agent bidding policy parameters according to the reward signal and calculates based on the market environment state information to output a low-dimensional control point parameter vector representing the agent bidding policy.

[0020] Furthermore, the original action receiving and parsing unit performs the following operations:

[0021] a. Receive the low-dimensional control point parameter vector output by the intelligent agent policy network. The dimension of the low-dimensional control point parameter vector is n-1, which is less than the number of segments B of the high-dimensional segmented bidding curve that conforms to the power market bidding specifications, thereby realizing the dimensionality reduction of the action space.

[0022] b. The specific mathematical process of transforming the low-dimensional control point parameter vector using the softplus function to obtain the positive value sequence includes:

[0023] Let the low-dimensional control point parameter vector output by the agent policy network be... ,in Represents an n-1 dimensional real vector. express The j-th component is subjected to a differentiable transformation of the low-dimensional control point parameter vector using the softplus function to obtain the positive value sequence. The specific mathematical expression is:

[0024] ;

[0025] In the formula, For the softplus function, The transformed positive value sequence The j-th component, the differentiable transformation can ensure that each component obtained is always greater than zero and is a smooth, continuously differentiable sequence of positive values ​​over the entire domain.

[0026] Furthermore, the normalization and scaling unit performs the following operations:

[0027] a. The specific process of satisfying boundary constraints by fixing the first and last control points includes:

[0028] a1. Definition Let be a sequence of effective control points for an nth-order Bézier curve, where the th... One effective control point The horizontal axis represents a fixed normalized electricity value, and the vertical axis represents the quoted price.

[0029] a2. Define the market-regulated price range as follows: ,in As the lower limit of the price, This is the upper limit of the price;

[0030] a3. Fix the first control point of the Bézier curve The vertical axis represents the lower limit of the price, with fixed control points at the tail of the Bézier curve. The vertical axis represents the upper limit of the price, and its mathematical expression is:

[0031] ;

[0032] The x-coordinates of the first and last control points are fixed at 0 and 1; by fixing the first and last control points, it is ensured that the starting point and ending point of the generated Bézier curve fall precisely within the price range boundary.

[0033] b. The specific process of constructing intermediate control points from the cumulative normalized positive value sequence includes:

[0034] b1. Calculate the sum S of the positive value sequence. The mathematical expression is:

[0035] ;

[0036] b2. Construct intermediate control points by accumulating the normalized sequence of positive values. The mathematical expression is:

[0037] ;

[0038] In the formula, Let be the ordinate of the i-th valid control point;

[0039] The original action receiving and parsing unit maps the positive value sequence into a set of effective control point sequences that strictly satisfy the monotonically increasing condition and price boundary constraints.

[0040] Furthermore, the specific process by which the constraint projection and structure integration unit generates a continuous bid curve function based on Bézier curves according to the Bernstein polynomial formula is as follows:

[0041] Based on effective control point sequence Based on Bernstein's polynomial formula, an n-order Bézier curve is generated and determined as a continuous bidding curve function for the agent. The mathematical expression is:

[0042] ;

[0043] In the formula, t is a curve parameter, and its value range [0,1] corresponds to the complete path of the curve from the starting point to the ending point; The Bernstein basis polynomial has the following mathematical expression:

[0044] ;

[0045] In the formula, The coefficients are binomial coefficients;

[0046] The continuous bidding curve function passes through the first valid control point at t=0, i.e. ; It passes through the last valid control point at t=1, that is The above process ensures that the entire curve lies inside the convex hull formed by the sequence of effective control points, thereby ensuring that the curve satisfies the boundary constraints.

[0047] Furthermore, the specific process by which the compliant bid output unit generates a high-dimensional piecewise bid curve that conforms to the electricity market bidding specifications through uniform discrete sampling is as follows:

[0048] The curve parameter t is uniformly sampled within the interval [0,1], and the sampling points are... Defined as:

[0049] ;

[0050] In the formula, the price corresponding to the k-th quote is... Take the function value of the continuous bidding curve function at the sampling point. ,Right now:

[0051] ;

[0052] Finally, a high-dimensional piecewise bidding curve is output. By combining the bidding intervals corresponding to the high-dimensional discrete bidding vectors, a high-dimensional piecewise bidding curve that conforms to the market monotonicity constraint and boundary constraint is formed.

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] 1. This invention solves the curse of dimensionality problem by achieving efficient dimensionality reduction of the action space through Bézier curves. Its core lies in enabling the agent policy network to learn a low-dimensional control point parameter vector instead of directly learning a high-dimensional bidding curve. The agent action decoder then maps this low-dimensional vector to a high-dimensional compliance bidding curve. This significantly reduces the dimensionality of the action space that the agent policy network needs to explore, allowing reinforcement learning algorithms to converge quickly and stably, and significantly improving the computational efficiency and engineering feasibility of multi-agent simulation systems.

[0055] 2. This invention utilizes the inherent mathematical properties of Bézier curves to achieve the endogeneity satisfaction of market rule constraints. The agent action decoder ensures price boundary constraints by fixing the first and last control points, and ensures monotonicity constraints by constructing a strictly monotonically increasing sequence of effective control points. The entire generation process naturally and deterministically guarantees the compliance of the output curve, completely avoiding the non-differentiable post-processing steps in traditional methods, and fundamentally eliminating the gradient breakage problem caused by external constraint processing.

[0056] 3. The entire action decoding process provided by this invention possesses end-to-end differentiability, providing a solid foundation for efficient policy optimization. All stages, from original parameter transformation and normalization mapping to curve generation and sampling, consist of differentiable mathematical operations. This ensures smooth backpropagation of gradients from market settlement rewards to agent policy network parameters, enabling the agent to directly learn how to adjust its low-dimensional policy representation to maximize long-term gains. This achieves true end-to-end optimization, significantly improving training stability and efficiency.

[0057] 4. The bidding curve generated by this invention is smooth and reasonable, and improves the interpretability of the strategy. The continuous bidding curve generated based on Bézier curves has smooth mathematical properties, avoiding the abrupt changes in traditional segmented bidding, and more closely reflects the rational bidding behavior of participants in the real market. At the same time, the low-dimensional control point parameter vector provides an intuitive and concise intermediate representation for analyzing and understanding the agent's strategic intent, enhancing the transparency and interpretability of the model's decision-making process.

[0058] 5. The system architecture constructed in this invention is clear and flexible, possessing good versatility and scalability. Each module in the system has a clearly defined responsibility and low coupling. The complexity of the Bézier curve and the number of bidding segments required by the market are independent of each other, allowing for flexible adjustment of simulation accuracy and efficiency according to actual needs. This architecture can easily accommodate different market clearing algorithms, diverse reinforcement learning algorithms, and various power grid topology models, making it suitable for various market bidding scenarios, including those on the generation and user sides, demonstrating broad applicability.

[0059] In summary, this invention, through innovative system design, deeply integrates Bézier curves with multi-agent reinforcement learning to construct an efficient, compliant, and scalable simulation platform for electricity market bidding decisions. It provides strong technical support for market mechanism design, market participant behavior analysis, and market risk assessment under the new power system, and is worthy of promotion. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the overall architecture of the system of the present invention.

[0061] Figure 2 This is a schematic diagram illustrating the working principle and process of an intelligent agent action decoder.

[0062] Figure 3 This is a comparison of the training convergence curves of different methods in the embodiments. In the figure, the horizontal axis represents the training period (Episodes), and the vertical axis represents the reward value obtained by the agent (Reward, on the order of 10). 5 The legend in the lower right corner distinguishes the three methods by line color: the dark solid line corresponds to the system of this invention (BC), the medium gray solid line corresponds to the post-correction method (PHC), and the dark gray solid line corresponds to the architecture constraint method (AC). The vertical dashed line at x=1000 (Start learning) and the "Warm-up" label in the lower left corner together define the experimental phase, i.e., the first 1000 cycles are the warm-up period, followed by the formal learning phase. Furthermore, the light-shaded area around the curves reflects the range of reward fluctuations (variance) during the training process for each method, while the solid lines depict the trajectory of the average reward, visually demonstrating the significant advantages of the system of this invention in terms of convergence speed and stability.

[0063] Figure 4 This is a comparison chart of the bidding curves and marginal costs generated by different methods in the embodiments. In the chart, the horizontal axis represents the generator unit's capacity (in MW), and the vertical axis represents the price (in $ / MWh). The legend on the left side details the meaning of each curve: the legend uses line type and dot color to indicate the meaning of each curve: the dark solid line with dark dots corresponds to the present invention system (BC), the medium gray solid line with medium gray dots corresponds to the post-correction method (PHC), and the light gray solid line with light gray dots corresponds to the architecture constraint method (AC). In addition, the chart includes multiple dashed lines as benchmark references, where the upward-curving dark dashed line represents the unit's true marginal cost, the lower light dashed line represents the average cost, and the vertical dashed line marks the winning power position under different methods (such as BC Power), intuitively showing the degree of fit between the bidding strategy of each method and the true cost. Detailed Implementation

[0064] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0065] like Figure 1 and Figure 2 As shown, this embodiment discloses a multi-agent simulation system for electricity market bidding decision-making. It is a software simulation system preferably developed using the Python programming language and mainstream deep learning frameworks such as PyTorch and TensorFlow, and can be deployed and run on Linux or Windows server clusters equipped with GPU accelerators. It includes the following functional modules:

[0066] An environment simulation module is used to simulate the core trading and settlement processes of the electricity market. The environment simulation module includes a market clearing unit and a market settlement unit. The market clearing unit is used to receive bidding information submitted by market participants, execute a preset market clearing algorithm, and determine the market clearing result, including the winning bid volume and the clearing price. The market settlement unit is used to perform financial settlement based on the market clearing result determined by the market clearing unit, generate reward signals to drive the intelligent agent's bidding strategy update, and generate market environment status information including the market clearing result and financial settlement information.

[0067] A multi-agent module receives reward signals and market environment status information from the environment simulation module, performs autonomous strategy learning and bidding decisions, and generates bidding information to be submitted to the market clearing unit. The multi-agent module contains multiple independent agents, each representing a market participant. Each agent includes an agent policy network and an agent action decoder. The agent policy network learns and updates its internal agent bidding strategy based on the reward signals received from the environment simulation module, calculates bidding decisions based on the received market environment status information, and outputs a low-dimensional control point parameter vector representing the agent's bidding strategy. The agent action decoder receives the low-dimensional control point parameter vector and converts it into a high-dimensional piecewise bidding curve that fully conforms to the electricity market bidding specifications through a data decoding process based on the mathematical properties of Bézier curves, and submits it to the market clearing unit.

[0068] Specifically, the intelligent agent action decoder includes:

[0069] The original action receiving and parsing unit is used to receive the low-dimensional control point parameter vector output by the agent policy network, and use the softplus function to perform a differentiable transformation on the low-dimensional control point parameter vector to obtain a positive value sequence in which all components are positive.

[0070] The normalization and scaling unit is used to map the positive value sequence into a set of effective control point sequences for defining the Bézier curve. The effective control point sequence achieves price boundary constraints by fixing the first and last control points, and constructs intermediate control points by accumulating the normalized positive value sequence to achieve the monotonically increasing condition.

[0071] The constraint projection and structure integration unit is used to receive the effective control point sequence and generate a Bézier curve through Bernstein polynomial. Based on the mathematical properties of the Bézier curve, the Bézier curve can be determined as the continuous bidding curve function of the agent under the strict monotonically increasing condition and price boundary constraint.

[0072] The compliant pricing output unit is used to uniformly discretely sample the continuous bidding curve function, with the number of sampling points equal to the number of bidding segments stipulated by the market. The function value corresponding to each sampling point is used as the price of the corresponding bidding segment, thereby generating the final high-dimensional segmented bidding curve and returning it to the environment simulation module.

[0073] Specifically, the market clearing unit executes a preset market clearing algorithm, which is a safety-constrained unit combination algorithm and a safety-constrained economic dispatch algorithm. The safety-constrained unit combination algorithm and the safety-constrained economic dispatch algorithm are based on a mixed-integer linear programming model. The electricity market clearing is achieved by calling a commercial solver to solve the mixed-integer linear programming model and determining the market clearing result, which includes the winning bid amount and clearing price of each agent.

[0074] Specifically, the financial settlement process of the market settlement unit includes: calculating the net settlement income of each agent based on the market clearing result, which serves as a reward signal to drive its strategy update; and generating market environment status information to characterize the market operation based on the market clearing result and financial settlement information.

[0075] Specifically, the agent policy network is a policy model built based on a deep reinforcement learning algorithm. It updates the internal agent bidding policy parameters according to the reward signal and calculates based on the market environment state information to output a low-dimensional control point parameter vector representing the agent bidding policy.

[0076] Specifically, the original action receiving and parsing unit performs the following operations:

[0077] a. Receive the low-dimensional control point parameter vector output by the intelligent agent policy network. The dimension of the low-dimensional control point parameter vector is n-1, which is less than the number of segments B of the high-dimensional segmented bidding curve that conforms to the power market bidding specifications, thereby realizing the dimensionality reduction of the action space.

[0078] b. The specific mathematical process of transforming the low-dimensional control point parameter vector using the softplus function to obtain the positive value sequence includes:

[0079] Let the low-dimensional control point parameter vector output by the agent policy network be... ,in Represents an n-1 dimensional real vector. express The j-th component is subjected to a differentiable transformation of the low-dimensional control point parameter vector using the softplus function to obtain the positive value sequence. The specific mathematical expression is:

[0080] ;

[0081] In the formula, For the softplus function, The transformed positive value sequence The j-th component, the differentiable transformation can ensure that each component obtained is always greater than zero and is a smooth, continuously differentiable sequence of positive values ​​over the entire domain.

[0082] Specifically, the normalization and scaling unit performs the following operations:

[0083] a. The specific process of satisfying boundary constraints by fixing the first and last control points includes:

[0084] a1. Definition Let be a sequence of effective control points for an nth-order Bézier curve, where the th... One effective control point The horizontal axis represents a fixed normalized electricity value, and the vertical axis represents the quoted price.

[0085] a2. Define the market-regulated price range as follows: ,in As the lower limit of the price, This is the upper limit of the price;

[0086] a3. Fix the first control point of the Bézier curve The vertical axis represents the lower limit of the price, with fixed control points at the tail of the Bézier curve. The vertical axis represents the upper limit of the price, and its mathematical expression is:

[0087] ;

[0088] The x-coordinates of the first and last control points are fixed at 0 and 1; by fixing the first and last control points, it is ensured that the starting point and ending point of the generated Bézier curve fall precisely within the price range boundary.

[0089] b. The specific process of constructing intermediate control points from the cumulative normalized positive value sequence includes:

[0090] b1. Calculate the sum S of the positive value sequence. The mathematical expression is:

[0091] ;

[0092] b2. Construct intermediate control points by accumulating the normalized sequence of positive values. The mathematical expression is:

[0093] ;

[0094] In the formula, Let be the ordinate of the i-th valid control point.

[0095] The original action receiving and parsing unit maps the positive value sequence into a set of effective control point sequences that strictly satisfy the monotonically increasing condition and price boundary constraints.

[0096] Specifically, the process by which the constraint projection and structural integration unit generates a continuous bid curve function based on Bézier curves according to the Bernstein polynomial formula is as follows:

[0097] Based on effective control point sequence Based on Bernstein's polynomial formula, an n-order Bézier curve is generated and determined as a continuous bidding curve function for the agent. The mathematical expression is:

[0098] ;

[0099] In the formula, t is a curve parameter, and its value range [0,1] corresponds to the complete path of the curve from the starting point to the ending point; The Bernstein basis polynomial has the following mathematical expression:

[0100] ;

[0101] In the formula, is the binomial coefficient.

[0102] The continuous bidding curve function passes through the first valid control point at t=0, i.e. ; It passes through the last valid control point at t=1, that is The above process ensures that the entire curve lies inside the convex hull formed by the sequence of effective control points, thereby ensuring that the curve satisfies the boundary constraints.

[0103] Specifically, the process by which the compliant bid output unit generates a high-dimensional piecewise bid curve that conforms to the electricity market bidding specifications through uniform discrete sampling is as follows:

[0104] The curve parameter t is uniformly sampled within the interval [0,1], and the sampling points are... Defined as:

[0105] ;

[0106] In the formula, the price corresponding to the k-th quote is... Take the function value of the continuous bidding curve function at the sampling point. ,Right now:

[0107] ;

[0108] Finally, a high-dimensional piecewise bidding curve is output. By combining the bidding intervals corresponding to the high-dimensional discrete bidding vectors, a high-dimensional piecewise bidding curve that conforms to the market monotonicity constraint and boundary constraint is formed.

[0109] Below we combine Figure 3 and Figure 4 This embodiment verifies the effectiveness, superiority, and potential of the multi-agent simulation system described above in solving practical technical problems. The experimental design follows strict control principles to ensure the fairness of the comparison and the reliability of the conclusions.

[0110] To construct a high-fidelity simulation test environment, the IEEE 24-bus reliability testing system, widely recognized in the field of power system engineering and research, was adopted as the physical foundation. The system's complete network parameters, including bus data, generator technical parameters, transmission line impedance and capacity limits, were all obtained from the publicly available and standard PGLib-OPF benchmark library, ensuring the objectivity and reproducibility of the experimental basis. To simulate day-ahead spot market rules, each agent was required to submit a monotonically non-decreasing quote curve containing 10 segments (B=10) for each trading session, with the quote range limited to [10, 100] USD / MWh. Load data used processed real historical sequences, scaled according to system size, with each training cycle simulating a complete trading day (24 hours) of market operation.

[0111] To evaluate the effectiveness of the system of this invention, two representative comparative benchmark methods were set up, and the core differences are shown in the table below:

[0112]

[0113] This experiment consisted of 5000 training cycles, with the first 1000 cycles serving as a warm-up period. A comparison of the training convergence performance of the three methods is provided below. Figure 3As shown, the system of this invention exhibits a fast and stable learning process. Its average reward curve rapidly climbs to a high level and stabilizes in the early stages of training, ultimately converging to a reward value of approximately 1.6 × 10⁻⁶. 5 In contrast, while the post-correction method eventually converges, the process is slow, and its stable reward value of 0.7 × 10^5 is significantly lower than that of the method in this invention, less than half. The architecture constraint method, on the other hand, fails to learn effectively, with its reward curve fluctuating wildly around zero. This result directly proves that the system in this invention fundamentally solves the exploration and learning difficulties brought about by high-dimensional action spaces through action space dimensionality reduction and intrinsic constraint satisfaction mechanisms, resulting in a qualitative improvement in training efficiency.

[0114] Figure 4 This paper compares the bid curves generated by three different methods for the same generating unit with its actual marginal cost curve during a typical market period. The graph shows that the bid curve generated by the system of this invention is a smooth, continuous, and strictly monotonic curve, with its overall shape closely matching the actual marginal cost curve of the generating unit. Specifically, when the unit's output is low, the bid closely follows the marginal cost, demonstrating strong competitiveness to secure the bid; as output increases, the bid rises smoothly and gradually, accurately reflecting the increase in generation costs, thus ensuring a reasonable profit at high output levels. This indicates that the system of this invention guides the agent to learn a complex, economical, and adaptive high-quality bidding strategy. In contrast, the curves generated by the ex-post correction method and the architecture constraint method either exhibit unreasonable step-like abrupt changes or deviate significantly from the cost curve, showing the coarseness and suboptimal nature of the strategy. This comparison strongly confirms that the system of this invention not only accelerates training but also ensures that the agent converges to a high-quality strategy that highly conforms to economic principles.

[0115] In summary, the rigorous experimental verification based on the standard testing system fully demonstrates that the system of this invention significantly outperforms traditional methods in terms of convergence speed and strategy quality. This system effectively solves the core challenge of generating high-dimensional compliance actions in large-scale electricity market simulation, providing an efficient, reliable, and advanced simulation analysis tool for market mechanism design, agent behavior analysis, and risk assessment, and is worthy of widespread application.

[0116] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A multi-agent simulation system for electricity market bidding decision-making, characterized in that, include: An environment simulation module is used to simulate the core trading and settlement processes of the electricity market. The environment simulation module includes a market clearing unit and a market settlement unit. The market clearing unit is used to receive bidding information submitted by market participants, execute a preset market clearing algorithm, and determine the market clearing result, including the winning bid volume and the clearing price. The market settlement unit is used to perform financial settlement based on the market clearing result determined by the market clearing unit, generate a reward signal to drive the intelligent agent's bidding strategy update, and generate market environment status information including the market clearing result and financial settlement information. A multi-agent module receives reward signals and market environment status information from the environment simulation module, performs autonomous strategy learning and bidding decisions, and generates bidding information to be submitted to the market clearing unit. The multi-agent module comprises multiple independent agents, each representing a market participant. Each agent includes an agent policy network and an agent action decoder. The agent policy network learns and updates its internal agent bidding strategy based on the reward signals received from the environment simulation module, calculates bidding decisions based on the received market environment status information, and outputs a low-dimensional control point parameter vector representing the agent's bidding strategy. The agent action decoder receives the low-dimensional control point parameter vector and, through a data decoding process based on the mathematical properties of Bézier curves, converts it into a high-dimensional segmented bidding curve that fully conforms to the electricity market bidding specifications, and submits it to the market clearing unit. The agent action decoder includes: The original action receiving and parsing unit is used to receive the low-dimensional control point parameter vector output by the agent policy network, and use the softplus function to perform a differentiable transformation on the low-dimensional control point parameter vector to obtain a positive value sequence in which all components are positive. The normalization and scaling unit is used to map the positive value sequence into a set of effective control point sequences for defining the Bézier curve. The effective control point sequence achieves price boundary constraints by fixing the first and last control points, and constructs intermediate control points by accumulating the normalized positive value sequence to achieve the monotonically increasing condition. The constraint projection and structure integration unit is used to receive the effective control point sequence and generate a Bézier curve through Bernstein polynomial. Based on the mathematical properties of the Bézier curve, the Bézier curve can be determined as the continuous bidding curve function of the agent under the strict monotonically increasing condition and price boundary constraint. The compliant pricing output unit is used to uniformly discretely sample the continuous bidding curve function, with the number of sampling points equal to the number of bidding segments stipulated by the market. The function value corresponding to each sampling point is used as the price of the corresponding bidding segment, thereby generating the final high-dimensional segmented bidding curve and returning it to the environment simulation module. The original action receiving and parsing unit performs the following operations: a. Receive the low-dimensional control point parameter vector output by the intelligent agent policy network. The dimension of the low-dimensional control point parameter vector is n-1, which is less than the number of segments B of the high-dimensional segmented bidding curve that conforms to the power market bidding specifications, thereby realizing the dimensionality reduction of the action space. b. The specific mathematical process of transforming the low-dimensional control point parameter vector using the softplus function to obtain the positive value sequence includes: Let the low-dimensional control point parameter vector output by the agent policy network be... ,in Represents an n-1 dimensional real vector. express The j-th component is subjected to a differentiable transformation of the low-dimensional control point parameter vector using the softplus function to obtain the positive value sequence. The specific mathematical expression is: ； wherein is a softplus function, is the jth component of the transformed positive sequence The differentiable transformation is able to ensure that each component obtained is always greater than zero, and is a smooth continuous differentiable positive sequence over the entire domain.

2. The multi-agent simulation system for electricity market bidding decision according to claim 1, characterized in that, The market clearing unit executes a pre-set market clearing algorithm consisting of a safety-constrained unit combination algorithm and a safety-constrained economic dispatch algorithm. The safety-constrained unit combination algorithm and the safety-constrained economic dispatch algorithm are based on a mixed-integer linear programming model. The electricity market clearing is achieved by calling a commercial solver to solve the mixed-integer linear programming model and determining the market clearing result, which includes the winning bid amount and clearing price of each agent.

3. The multi-agent simulation system for electricity market bidding decision according to claim 2, characterized in that, The specific process of financial settlement by the market settlement unit includes: calculating the net settlement income of each agent based on the market clearing result, which serves as a reward signal to drive its strategy update; and generating market environment status information to characterize the market operation based on the market clearing result and financial settlement information.

4. The multi-agent simulation system for electricity market bidding decision according to claim 3, characterized in that, The agent policy network is a policy model built based on a deep reinforcement learning algorithm. It updates the internal agent bidding strategy parameters according to the reward signal and calculates based on the market environment state information to output a low-dimensional control point parameter vector representing the agent bidding strategy.

5. The multi-agent simulation system for electricity market bidding decision according to claim 4, wherein, The normalization and scaling unit performs the following operations: a. The specific process of satisfying boundary constraints by fixing the first and last control points includes: a1. Definition is a sequence of effective control points of an n-th order Bezier curve, wherein the horizontal coordinate of the i-th effective control point is a fixed normalized electric quantity value, and the vertical coordinate is a bid. ​ a2. Define the market-regulated price range as follows: ,in As the lower limit of the price, This is the upper limit of the price; a3. Fix the first control point of the Bézier curve The vertical axis represents the lower limit of the price, with fixed control points at the tail of the Bézier curve. The vertical axis represents the upper limit of the price, and its mathematical expression is: ； The x-coordinates of the first and last control points are fixed at 0 and 1; by fixing the first and last control points, it is ensured that the starting point and ending point of the generated Bézier curve fall precisely within the price range boundary. b. The specific process of constructing intermediate control points from the cumulative normalized positive value sequence includes: b1. Calculate the sum S of the positive value sequence. The mathematical expression is: ； b2. Construct intermediate control points by accumulating the normalized sequence of positive values. The mathematical expression is: ； In the formula, Let be the ordinate of the i-th valid control point; The original action receiving and parsing unit maps the positive value sequence into a set of effective control point sequences that strictly satisfy the monotonically increasing condition and price boundary constraints.

6. A multi-agent simulation system for electricity market bidding decision-making according to claim 5, characterized in that, The specific process by which the constraint projection and structural integration unit generates a continuous bid curve function based on Bézier curves according to the Bernstein polynomial formula is as follows: Based on effective control point sequence Based on Bernstein's polynomial formula, an n-order Bézier curve is generated and determined as a continuous bidding curve function for the agent. The mathematical expression is: ； In the formula, t is a curve parameter, and its value range [0,1] corresponds to the complete path of the curve from the starting point to the ending point; The Bernstein basis polynomial has the following mathematical expression: ； In the formula, The coefficients are binomial coefficients; The continuous bidding curve function passes through the first valid control point at t=0, i.e. ; It passes through the last valid control point at t=1, that is The above process ensures that the entire curve lies inside the convex hull formed by the sequence of effective control points, thereby ensuring that the curve satisfies the boundary constraints.

7. A multi-agent simulation system for electricity market bidding decision-making according to claim 6, characterized in that, The specific process by which the compliant bid output unit generates a high-dimensional piecewise bid curve that conforms to the electricity market bidding specifications through uniform discrete sampling is as follows: The curve parameter t is uniformly sampled within the interval [0,1], and the sampling points are... Defined as: ； In the formula, the price corresponding to the k-th quote is... Take the function value of the continuous bidding curve function at the sampling point. ,Right now: ； Finally, a high-dimensional piecewise bidding curve is output. By combining the bidding intervals corresponding to the high-dimensional discrete bidding vectors, a high-dimensional piecewise bidding curve that conforms to the market monotonicity constraint and boundary constraint is formed.

Images