Novel power market behavior strategy determination method for multiple users in power system

By establishing a two-layer decision optimization model for the wholesale and retail markets in the new power system, and using deep reinforcement learning algorithms to optimize the market behavior strategy on the user side, the problem of insufficient research on the market behavior strategy on the user side is solved and the transaction in the power market is improved Efficiency and new energy consumption efficiency have promoted the optimal allocation of power resources and the reduction of carbon emissions.

CN118944048AActive Publication Date: 2024-11-12NORTH CHINA ELECTRIC POWER UNIV

Patent Information

Application Number
CN202410910678.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-12
Estimated Expiration
2044-07-08

AI Technical Summary

Technical Problem

In the new power system, the research on the market behavior strategy of the user side has not been fully reflected. The existing technology mainly focuses on the power generation side, and fails to effectively utilize the quotation strategy of the user side, resulting in low trading efficiency and new energy consumption efficiency of the power market. There are shortcomings in the market operation mechanism.

Method used

A method for determining the power market behavior strategy of multiple users under a new power system is proposed. By establishing a two-layer decision optimization model for the power wholesale market and retail market, and using deep reinforcement learning algorithms, the transaction strategies of each entity, including power generation and users The declaration strategy of the application to achieve coordinated optimization of market behavior.

Benefits of technology

It has improved the trading efficiency of the power market and the consumption efficiency of new energy, promoted the optimized allocation of power resources, reduced carbon emissions, enhanced the guiding role of price signals, and improved the stability and sustainable development capabilities of market construction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118944048A_ABST
    Figure CN118944048A_ABST
Patent Text Reader

Abstract

The invention relates to the field of electric power systems, and particularly discloses a novel electric power market behavior strategy determination method for multiple users under an electric power system, an electric power market comprises an electric power wholesale market and an electric power retail market, and the main body of the retail market comprises retail users on a demand side and aggregation users on a power supply side. The main body of the wholesale market comprises wholesale users on the demand side and generators on the power supply side, and the wholesale users comprise aggregated users on the power supply side of the retail market. The method comprises the following steps: by taking maximum social welfare as a lower-layer target and maximum income as an upper-layer target, constructing a double-layer decision optimization model of power generators and wholesale users in an electric power wholesale market and a double-layer decision optimization model of aggregation users in an electric power retail market; and then, establishing a corresponding Markov decision process, and solving each model based on the Markov decision process by using a DQN algorithm to obtain a behavior decision of each subject. According to the invention, multiple participants and multi-level markets of the power market are fully considered, and the construction of a novel power system can be promoted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of power systems, and in particular to a method for determining power market behavior strategies of multiple users in a new power system. Background Art

[0002] With the proposal of carbon peak and carbon neutrality goals, building a new power system with new energy as the main body has become the future development form of energy and electricity. The access of a high proportion of new energy has brought new challenges to the power system, requiring the market to play a more important role in resource allocation to promote the safe and stable operation and sustainable development of the system. Among them, under the background of the continuous deepening of the market-oriented reform of the power industry, the user's collaborative market behavior has increasingly profoundly affected the consumption of new energy and the efficiency and stability of the new power system, and the user side has gradually become the focus of market construction.

[0003] However, the construction of my country's unified electricity market is still in its infancy, and the market operation mechanism still has many shortcomings, especially on the user side. Summary of the invention

[0004] To this end, the present invention provides a method for determining a power market behavior strategy for multiple users in a new power system, in an effort to solve or at least alleviate the above problems.

[0005] According to one aspect of the present invention, a method for determining power market behavior strategies of multiple users under a new power system is provided. The power market includes a wholesale power market and a retail power market. The subjects of the retail power market include retail users on the demand side and aggregated users on the power supply side. The subjects of the wholesale power market include wholesale users on the demand side and power generators on the power supply side, and wholesale users include aggregated users on the power supply side of the retail power market. The method includes: establishing a wholesale power market clearing model and a retail power market clearing model with the goal of maximizing social welfare, and for each subject in the wholesale power market, constructing its wholesale power market transaction strategy model with the goal of maximizing its profit, and for each aggregated user, constructing its retail power market transaction strategy model with the goal of maximizing its profit; for each subject in the wholesale power market, using its wholesale power market transaction strategy model as the upper model and the wholesale power market clearing model as the lower model, constructing its two-layer decision optimization model in the wholesale power market, and for each aggregated user, constructing its For aggregated users, a two-layer decision-making optimization model in the electricity retail market is constructed, with its electricity retail market trading strategy model as the upper model and the electricity retail market clearing model as the lower model. For any subject, the quotation coefficient of each trading period is used as the action of each trading period, the income of the unit declared electricity volume of each trading period is used as the reward of each trading period, and the status of each trading period is determined based on the declared electricity price, declared electricity volume, successful bid electricity volume and the clearing price of the electricity wholesale market. The Markov decision process of each wholesale user and power generator in the electricity wholesale market and the Markov decision process of each aggregated user in the electricity retail market are established. Based on the established Markov decision process, the deep reinforcement learning algorithm is used to solve the two-layer decision optimization models, and the declaration strategy of each wholesale user and power generator in the electricity wholesale market and the declaration strategy of each aggregated user in the electricity retail market are obtained. The declaration strategy includes the declared electricity price and declared electricity volume of each trading period.

[0006] According to another aspect of the present invention, there is provided a computing device, comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be suitable for execution by the at least one processor, and the program instructions include instructions for executing a method for determining a power market behavior strategy for multiple users under a new power system according to the present invention.

[0007] According to another aspect of the present invention, a readable storage medium storing program instructions is provided. When the program instructions are read and executed by a computing device, the computing device executes the method for determining the electricity market behavior strategy for multiple users in a new power system according to the present invention.

[0008] In summary, the present invention fully considers the multiple participants and multi-level markets in the electricity market, takes the user side as the research object, focuses on the collaborative market behavior of users, establishes a two-layer decision-making optimization model for multiple users, and uses a deep reinforcement learning algorithm to solve it, accurately and efficiently obtaining the market behavior decisions of each subject. In this way, carbon emissions can be effectively reduced and new energy can be absorbed through the electricity market. Moreover, for the construction of the electricity market, the role of price signals can be more fully utilized to guide users to actively increase production and efficiency, green electricity use, and optimize electricity use methods, thereby promoting the optimal allocation of power resources and providing guidance for the construction of new power systems and power market mechanisms. In addition, the two-layer decision-making optimization model of the present invention takes into account factors such as carbon prices and demand response, so that the market behavior decisions of each subject obtained can be more accurate and more representative. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles disclosed herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The above and other purposes, features and advantages of the present disclosure will become more apparent by reading the following detailed description in conjunction with the accompanying drawings. Throughout the present disclosure, the same reference numerals generally refer to the same parts or elements.

[0010] Figure 1 A structural block diagram of a computing device 100 according to an embodiment of the present invention is shown;

[0011] Figure 2 A flowchart of a method 200 for determining a power market behavior strategy for multiple users in a new power system according to an embodiment of the present invention is shown;

[0012] Figure 3 A schematic diagram showing a power market structure according to an embodiment of the present invention is shown;

[0013] Figure 4 A schematic diagram showing a centralized bidding market clearing according to an embodiment of the present invention is shown;

[0014] Figure 5 A schematic diagram showing a decision-making behavior process of a subject under a DQN algorithm according to an embodiment of the present invention;

[0015] Figure 6 A schematic diagram showing a new power system market-oriented operation process according to an embodiment of the present invention;

[0016] Figure 7 A schematic diagram showing a change in a user neural network training error according to an embodiment of the present invention is shown;

[0017] Figure 8 A schematic diagram of quotation iteration of some users according to an embodiment of the present invention is shown;

[0018] Fig. 9 A schematic diagram showing average bids of supply and demand entities and average market clearing prices according to an embodiment of the present invention is shown;

[0019] Fig.10 A schematic diagram showing the market electricity supply-demand ratio and wholesale market transaction electricity price for each month according to an embodiment of the present invention;

[0020] Fig.11 A schematic diagram showing retail market pricing for each month according to an embodiment of the present invention is shown;

[0021] Fig.12 A schematic diagram showing the output of various types of units according to an embodiment of the present invention is shown;

[0022] Fig.13 A schematic diagram showing the impact of renewable energy power generation on market clearing prices according to an embodiment of the present invention;

[0023] Fig.14 The figure shows the proportion of renewable energy power generation to power generation and actual power purchase, as well as a schematic diagram of the total renewable energy power generation and total power purchase according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0025] The new power system uses new energy sources such as wind power and photovoltaic power generation as the main power source, gradually replacing thermal power generation, and using thermal power, energy storage, and elastic loads with changed functional roles as flexible adjustment resources to ensure the safe and stable operation of the system. Among them, the user side is a potential driver of carbon emissions in the new power system. When power users participate in transactions, the impact of resources on the environment on the demand side of the power market gradually becomes prominent. Without the active participation of power users, it is difficult to effectively reduce carbon emissions and absorb new energy through the power market. Users participating in the power market can reduce their own electricity costs. For the construction of the power market, it can give full play to the role of price signals, guide users to actively increase production and efficiency, use green electricity, and optimize electricity usage methods to promote the optimal allocation of power resources.

[0026] However, existing research on market bidding mostly focuses on the power generation side, and most of them target a single market player, without reflecting the user-side bidding strategy. In addition, existing research on power user market strategies mostly only considers a single market process, and rarely involves research on the entire power market transaction process and the connection between markets at all levels. To this end, the present invention provides a method for determining power market behavior strategies for multiple users under a new power system. The method fully considers the multiple participants, multi-level markets, and transaction models in the power market, and focuses on the strategic bidding behavior of users participating in the market.

[0027] The method for determining the electricity market behavior strategy of multiple users in the novel power system of the present invention can be executed in a computing device. Figure 1 A block diagram of the physical components (i.e., hardware) of a computing device 100 is shown. In a basic configuration, the computing device 100 includes at least one processing unit 102 and a system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 can be implemented as a processor. The system memory 104 includes, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, the system memory 104 includes an operating system 105 and a program module 106, and the program module 106 includes a behavior strategy determination module 120, which is configured to execute the power market behavior strategy determination method 200 for multiple users under the new power system of the present invention.

[0028] According to one aspect, operating system 105 is suitable for controlling the operation of computing device 100, for example. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other application programs, and are not limited to any particular application or system. Figure 1 This basic configuration is illustrated in FIG. 1 by those components within dashed line 108. According to one aspect, computing device 100 has additional features or functionality. For example, according to one aspect, computing device 100 includes additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 1 1 is illustrated by removable storage 109 and non-removable storage 110.

[0029] As stated above, according to one aspect, program modules are stored in the system memory 104. According to one aspect, the program modules may include one or more application programs, and the present invention does not limit the type of application programs, for example, the application programs may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slide show applications, drawing or computer-aided applications, web browser applications, etc.

[0030] According to one aspect, the examples may be practiced on a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. Figure 1 Each or many components shown in can be integrated into a system on a chip (SOC) on a single integrated circuit to practice examples. According to one aspect, such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via SOC, the functions described in this article can be operated via a dedicated logic integrated with other components of the computing device 100 on a single integrated circuit (chip). Embodiments of the present invention can also be practiced using other technologies capable of performing logical operations (such as AND, OR, and NOT), including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present invention can be practiced in a general-purpose computer or in any other circuit or system.

[0031] According to one aspect, the computing device 100 may also have one or more input devices 112, such as a keyboard, a mouse, a pen, a voice input device, a touch input device, etc. Output devices 114 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples and other devices may also be used. The computing device 100 may include one or more communication connections 116 that allow communication with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver and / or transceiver circuits; Universal Serial Bus (USB), parallel and / or serial ports.

[0032] The term computer-readable medium as used herein includes computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, or program modules). System memory 104, removable storage 109, and non-removable storage 110 are all examples of computer storage media (i.e., memory storage). Computer storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other products that can be used to store information and can be accessed by computer device 100. According to one aspect, any such computer storage medium can be a part of computing device 100. Computer storage media do not include carrier waves or other propagated data signals.

[0033] According to one aspect, communication media is implemented by computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transport mechanism), and includes any information delivery media. According to one aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0034] Figure 2 A flowchart of a method 200 for determining a power market behavior strategy for multiple users in a new power system according to an embodiment of the present invention is shown. The method 200 is suitable for determining a power market behavior strategy for multiple users in a new power system according to an embodiment of the present invention. Figure 1 Executed in the computing device 100 shown.

[0035] It should be explained here that the power market of the present invention includes the power wholesale market and the power retail market, and the power market entities include the power wholesale market entities and the power retail market entities. Among them, the entities of the power retail market include retail users on the demand side (i.e. users who purchase electricity from the power retail market) and aggregated users on the power supply side, and the entities of the power wholesale market include wholesale users on the demand side (i.e. users who purchase electricity from the power wholesale market) and power generators on the power supply side, and the wholesale users on the demand side of the power wholesale market include aggregated users who sell electricity to retail users in the power retail market. That is, this embodiment is composed of two-level markets, the power wholesale market and the power retail market, and the two-level markets are connected through aggregated users.

[0036] Further, aggregated users may include power sales companies, microgrids, and distributed parks, retail users may include small and medium-sized power users, and wholesale users may also include large power users. Specifically, large power users may be power users whose power supply voltage is greater than or equal to the first predetermined value, while small and medium-sized power users are users whose power supply voltage is less than the first predetermined value. Among them, the first predetermined value may be 10kV, of course, this is only an example. In addition, in some embodiments, large power users and small and medium-sized power users may also be divided according to annual power consumption, such as users whose annual power consumption is greater than or equal to the second predetermined value as large power users, and users whose annual power consumption is less than the second predetermined value as small and medium-sized power users, which is not limited by the present invention. In addition, the aggregated users are explained here, which are different from ordinary power users. The main bodies such as power sales companies and grid-connected microgrids usually contain multiple users, and may also contain multiple equipment resources accordingly. In this embodiment, they are uniformly modeled as aggregated users. Further, the market participation model of power grid enterprises such as microgrids is unified with that of power sales companies and modeled as aggregated users.

[0037] It can be seen that the electricity users of the present invention include large, medium and small ordinary electricity users, as well as aggregated users such as power sales companies and microgrids. Among them, it should be noted that users can only participate in one of wholesale electricity transactions or retail transactions in a single transaction cycle. Therefore, the operation structure of the market throughout the year is designed as follows in this embodiment: large electricity users only participate in electricity wholesale market transactions and freely declare electricity prices and electricity quantities; small and medium-sized electricity users only participate in electricity retail market transactions and only report quantities but not quotes; when aggregated users participate in the electricity wholesale market, their reporting method is consistent with that of large electricity users. In the retail market, they sell electricity to retail users as electricity sellers with unilateral pricing, and distributed power sources are sold by aggregated users. In addition, wholesale electricity transactions in the electricity wholesale market include medium- and long-term transactions (monthly markets) and spot transactions (day-ahead markets, real-time markets). Among them, the connection between medium- and long-term transactions and spot transactions can adopt a decentralized market model, mainly based on the physical execution of medium- and long-term contracts, with part of the electricity in the spot market (deviation electricity) participating, and the power generation and consumption parties determine the power generation and consumption curves themselves in the day-ahead stage, and adjust the deviation electricity through day-ahead and real-time balancing transactions. Among them, the electricity market structure of the present invention can be found in Figure 3 .

[0038] After clearly understanding the power market structure of the present invention, the power market behavior strategy determination method 200 for multiple users in the new power system of the present invention is described below. Figure 2 As shown, the method 200 begins at 210 .

[0039] In 210, a wholesale electricity market clearing model and a retail electricity market clearing model are established with the goal of maximizing social welfare, and for each entity in the wholesale electricity market, a wholesale electricity market trading strategy model is constructed with the goal of maximizing its profit, and for each aggregated user, a retail electricity market trading strategy model is constructed with the goal of maximizing its profit.

[0040] Let me explain it first. The present invention adopts a centralized bidding model in wholesale electricity transactions (monthly medium- and long-term transactions and spot transactions), and the clearing method adopts quotations from both supply and demand parties, and the queuing method is used for unified clearing. The main steps include: power generators and users declare electricity and electricity prices within the specified time, and the management subject accumulates the declared electricity in the order of low to high electricity sales prices and high to low electricity purchase prices to form a ladder curve of supply and demand. The price difference is large and the transaction is given priority to form a transaction pair. When the price difference is negative, the transaction cannot be completed. Among all the price difference pairs that have been traded, the arithmetic mean of the price difference declared by the power buyer and the power seller of the last transaction is the unified clearing price, and finally a unified settlement is made according to this price. If the electricity declared by both the supply and demand parties is the same, a transaction pair is formed directly; if the declared electricity is not completely matched, the party with a larger declared electricity will continue to look for the next transaction object to form a new transaction pair until all the declared purchase or sales electricity is zero, or the user's quotation is lower than that of the power generator. Regarding power generators, an explanation is given here. Each power generator is mainly composed of thermal power generating sets and new energy power generating sets. The new energy power generating sets mainly include photovoltaic power generating sets and wind power generating sets, which are not limited in the present invention.

[0041] In the past, under the mode of reporting quantity but not quoting, users entered the market as price takers. In comparison, under the mode of reporting quantity and quoting quantity in this embodiment, users have greater autonomy, so that the market can play a full role. In addition, when both power generators and users quote quantity, the market management entity can clear the market with the goal of maximizing social welfare and calculate the unified clearing price of the market. The schematic diagram of the step-shaped curve formed by the quotations submitted by various power generators and power sellers can be seen in Figure 4 .

[0042] Based on this, according to an embodiment of the present invention, a wholesale power market clearing model is constructed to characterize the clearing process of the centralized bidding of the bids on both sides of power generation and consumption in a single centralized bidding market transaction. Among them, the maximum social welfare in the wholesale power market can be specifically the maximum difference in the declared electricity prices of wholesale users and power generators (that is, the maximum difference in the declared electricity prices on both sides of supply and demand), so the wholesale power market clearing model can be specifically the following formula.

[0043]

[0044] In the formula, y represents the difference in electricity prices reported by wholesale users and power generators, which can also be expressed as y(f i ,d j,e k ), B Uj , B Ok , B Gi They represent the electricity prices declared by large power users Uj, aggregate users Ok and power generators Gi in any trading period of the wholesale power market, N U 、N O 、N G They respectively represent the number of large power users, aggregated users and power generators in the wholesale power market.

[0045] As mentioned above, wholesale electricity transactions in the wholesale electricity market include medium- and long-term transactions (monthly market) and spot transactions (day-ahead market, real-time market). Therefore, in the wholesale electricity market, this embodiment includes three time scales: monthly (month), day-ahead (day), and real-time (hour). Therefore, this embodiment includes three time scales of wholesale electricity market clearing models. It should be noted that the wholesale electricity market clearing model of any time scale can be expressed as the above formula, only the meaning of the trading period is different. For example, for the month, the trading period refers to the month (any trading period is any month), for the day-ahead, the trading period refers to the day (any trading period is any day), and for the real-time, the trading period refers to the hour (any trading period is any hour). Among them, the various models in the wholesale electricity market involved below are the same, and will not be repeated later. In addition, for the retail electricity market, this embodiment only includes one time scale of the day-ahead, so the trading period in the various models in the retail electricity market involved below refers to the day (any trading period is any day).

[0046] Next, the electricity prices declared by power generators, large power users, and aggregated users in any trading period in the wholesale power market are specifically described (this embodiment adopts a single-segment quotation mode).

[0047] Power generators: In this embodiment, capacity retention is not considered. Power generators declare electricity according to actual output, that is, when power generators have multiple units, they quote based on their weighted average marginal cost. In addition, in the actual power generation process, the marginal cost of renewable energy generation is much lower than that of thermal power, so the benchmark price in this embodiment does not consider renewable energy. Based on this, the electricity price declared by power generators in any trading period of the wholesale power market can be specifically expressed as follows.

[0048] B Gi (f i )=f i C Gi

[0049]

[0050] In the formula, B Gi (f i) represents the electricity price declared by the power generator Gi in any trading period of the electricity wholesale market (i.e., the actual electricity bidding price), C Gi represents the benchmark price of electricity bidding of power generator Gi, f i represents the power bidding coefficient (i.e., quotation coefficient) of the power generator Gi in any trading period in the wholesale power market, f min ≤f i ≤f max , when f i =1, the power generator quotes based on marginal cost, f max 、f min They represent the upper and lower limits of the power supplier's bid coefficient, C gx (p gx ) represents the marginal cost of the thermal power generating unit gx, p gx Indicates the upper limit of the output of the thermal power generating unit gx, m gx 、n gx , l gx are the quadratic coefficient, linear coefficient and constant term reflecting the characteristics of the thermal power generating unit gx. Among them, the value range of x is 1 to X, and X is the total number of thermal power generating units in the power generation company Gi. In addition, B Gi (f i ) and the above B Gi same.

[0051] Large power users: Large power users mainly purchase electricity from the wholesale power market, first conduct monthly transactions, and then declare in the day-ahead market and real-time market based on the deviation between the monthly contract decomposed electricity and the actual demand electricity, so as to achieve real-time electricity balance. When participating in power market transactions, users declare the demand electricity and the desired price. Since the transaction electricity of users is greatly affected by the quotation and actual load demand, only the declared electricity price of large power users is modeled. Assuming that the declared electricity is based on the actual demand electricity and the desired price is a fixed value, the quotation of large power users can be expressed as follows:

[0052] B Uj (d j ) = d j C Uj

[0053] In the formula, B Uj (d j ) represents the electricity price declared by large power user Uj in any trading period of the wholesale power market, C Uj represents the benchmark price of large power users Uj in the wholesale power market, d j It represents the bid coefficient of large power users in any trading period of the wholesale power market, where d min ≤d j ≤dmax , when d j =1, the user quotes according to the benchmark price, d max d min They represent the upper and lower limits of the bid coefficients of large power users, among which, B Uj (d j ) and the above B Uj same.

[0054] Aggregate users: Aggregate users first participate in electricity wholesale market transactions in the order of monthly transactions, day-ahead transactions, and real-time transactions. They then retail the purchased electricity to small and medium-sized electricity users in the spot retail market. When they declare electricity, their actual demand and demand response electricity can be derived by summarizing the electricity of the agents, and they can be regarded as ordinary electricity users when trading in the wholesale market. The specific quotation model can be seen in the following formula.

[0055] B Ok (e k )=e k C Ok

[0056] In the formula, B Ok (e k ) represents the electricity price declared by the aggregated user Ok in any trading period in the wholesale electricity market, C Ok represents the benchmark price of aggregated user Ok in the wholesale electricity market, e k represents the bid coefficient of aggregated user Ok in any trading period of the wholesale electricity market, where e min ≤e k ≤e max , when e k = 1, the aggregated user quotes according to the benchmark price, e max 、e min They represent the upper and lower limits of the bid coefficient of the aggregated user Ok in the wholesale electricity market. Ok (e k ) and the above B Ok same.

[0057] In addition, the electricity price (i.e., retail pricing) declared by aggregated users in any trading period of the electricity retail market is further explained here. Among them, the quotations of aggregated users in the wholesale market will affect the clearing electricity price of the wholesale market, and when formulating the retail electricity price, it is necessary to refer to the electricity price of the wholesale contract and transmit the market signal of the wholesale market to the retail market. Therefore, the retail pricing strategy of the aggregated user will be affected by its quotation strategy in the wholesale market, and because the purchased electricity volume of the aggregated user is mainly monthly medium- and long-term transactions, in some embodiments, the retail pricing of the aggregated user can be characterized based on the monthly market clearing price of the aggregated user, wherein the declared price of the aggregated user in any trading period of the electricity retail market can be specifically expressed as follows.

[0058] b Ok (h k )=h k λ'

[0059] Where b Ok (h k ) represents the declared electricity price of the aggregated user Ok in any trading period in the electricity retail market, λ' represents the electricity wholesale market clearing price of the trading period associated with the trading period in the electricity retail market. Specifically, as mentioned above, the time scale of the electricity retail market is the day before, so λ' represents the electricity wholesale market clearing price of the previous month of the month to which the trading period (i.e., any of the aforementioned trading periods) belongs, and h k represents the pricing coefficient (i.e., quotation coefficient) of the aggregated user Ok in any trading period in the electricity retail market, where h min ≤h k ≤h ma x,h ma x, h min They represent the upper and lower limits of the pricing coefficient of aggregated user Ok (i.e., the upper and lower limits of the bid coefficient of aggregated user Ok in the retail market). In addition, it has been pointed out above that aggregated users purchase electricity in the wholesale electricity market and sell electricity in the retail electricity market. Therefore, the number of aggregated users in the retail electricity market is the same as that in the wholesale electricity market.

[0060] The above is the objective function of the electricity wholesale market clearing model. Among them, the electricity wholesale market clearing model also includes constraints, as follows.

[0061]

[0062] Where, X Gi , X Uj , X Ok They represent the winning bid quantity of generator Gi, large power user Uj, and aggregated user Ok in any trading period of the wholesale power market, respectively. Gi , H Uj , HOk They represent the declared power of generator Gi, large power user Uj, and aggregate user Ok in any trading period of the wholesale power market, Q Uj , Q Ok They represent the actual power demand of large power users Uj and aggregated users Ok in any trading period of the wholesale power market, respectively. min 、p max They represent the lower and upper limits of the quotations set by the wholesale electricity market for the participants in the wholesale electricity market, respectively. min ,q max They respectively represent the lower and upper limits of the electricity volume reported by the electricity wholesale market for the participants.

[0063] So far, the electricity wholesale market clearing model consisting of the objective function and the constraint conditions has been obtained. The electricity retail market clearing model is described below. The electricity retail market also adopts a competitive market model, that is, multiple aggregated users participate in the market to bid, adopt a market pricing mechanism, aggregated users determine the retail price according to the wholesale market price, and retail users choose retailers according to their own needs and preferences. In order to reflect the pricing mechanism of the electricity retail trading market, a listing transaction model is adopted. That is: the aggregated user (power sales side) unilaterally quotes and reports the quantity, and is listed on the power retail platform. The retail user reports the quantity without quoting, and is sorted in order from low to high according to the retail pricing to form a retail contract. The contract price is the retail pricing of the aggregated user, so as to reflect the market behavior of the retail user who gives priority to the power sales company with lower pricing with the goal of minimizing the cost of electricity. This process can be described as the retail market management subject clearing with the goal of minimizing the cost of electricity purchase. Based on this, according to an embodiment of the present invention, a wholesale electricity market clearing model is constructed to characterize the single clearing process, and the maximum social welfare in the power retail market is the minimum electricity cost of the retail user. In this way, the power retail market clearing model can be specifically as follows.

[0064]

[0065] In the formula, y' represents the electricity cost of retail users, which can also be expressed as y'(h k ), b Ok represents the electricity price declared by the aggregated user Ok in any trading period of the electricity retail market (same as b above) Ok (h k ) is the same), N O Represents the number of aggregated users in the electricity retail market.

[0066] Furthermore, the electricity retail market clearing model also includes constraints, as follows.

[0067]

[0068] Where X' Ok N′ represents the winning bid quantity of aggregated user Ok in a certain trading period (i.e. any trading period) in the electricity retail market. u represents the number of retail users in the electricity retail market, and further, the number of small and medium-sized electricity users in the electricity retail market. uj' , Q u ' j' They represent the actual power demand and demand response power of retail user uj' in any trading period of the electricity retail market, and p' min 、p' max They respectively represent the lower and upper limits of quotations imposed on participants in the electricity retail market.

[0069] Next, the market trading strategy models are explained one by one.

[0070] Wholesale electricity market trading strategy model for power generators: In this embodiment, the model takes into account the carbon cost of power generators, as follows.

[0071]

[0072] Where M Gi represents the revenue of the generator Gi in any trading period of the wholesale electricity market, λ represents the clearing price in any trading period of the wholesale electricity market, C Gi represents the benchmark price of electricity bidding of power generator Gi, X Gi represents the amount of electricity that the generator Gi bids for in any trading period in the wholesale electricity market, p CO2 represents the carbon price, B CO2 represents the carbon quota coefficient per unit of electricity generation, It represents the weighted average carbon emission coefficient of all units in the power generator Gi.

[0073] Furthermore, the wholesale electricity market trading strategy model of the power generator also includes constraints corresponding to the above objective function, as follows.

[0074] f min C Gi ≤B Gi ≤f max C Gi

[0075] Electricity wholesale market trading strategy model for large power users: Since most users participating in the electricity wholesale market are large industrial users, their electricity utility maximization goal is quantified as output maximization goal, minus the cost to get the revenue maximization goal, and the upper limit of the amount of electricity allowed to respond to demand is set as a fixed proportion of the maximum electricity load. Based on this, the electricity wholesale market trading strategy model for large power users can be expressed as the following formula.

[0076] max M Uj =(I Uj -λ)X Uj +δQ' Uj

[0077] Among them, M Uj represents the revenue of large power users Uj in any trading period in the wholesale power market, I Uj represents the electricity benefit (kilowatt-hour output) of large-scale power user Uj, that is, the benefit that large-scale power user Uj can obtain by purchasing a unit of electricity, λ represents the clearing price in any trading period of the wholesale power market, and X Uj represents the amount of electricity that a large power user Uj bids for in any trading period in the wholesale power market, δ represents the compensation coefficient when participating in demand response and receiving compensation, Q' Uj It represents the demand response amount of large power user Uj in any trading period in the wholesale power market.

[0078] Furthermore, the electricity wholesale market transaction strategy model for large electricity users also includes constraints corresponding to the above objective function, as follows.

[0079]

[0080] Where η represents the upper limit of the proportion of the electricity demand response reported to the maximum electricity load.

[0081] Electricity wholesale market trading strategy model for aggregated users: The bidding strategy of aggregated users in the electricity wholesale market is the same as that of large electricity users, as follows.

[0082] max M Ok =(I Ok -λ)X Ok +δQ' Ok

[0083]

[0084] Where M Ok represents the revenue of aggregated user Ok in any trading period in the wholesale electricity market, I Ok represents the electricity efficiency of aggregated user Ok, λ represents the clearing price of any trading period in the wholesale electricity market, and X Ok represents the winning bid quantity of aggregated user Ok in any trading period of the wholesale electricity market, δ represents the compensation coefficient when participating in demand response and receiving compensation, Q' Ok It represents the demand response power of aggregated user Ok in any trading period in the wholesale electricity market, and η represents the upper limit of the proportion of the demand response power reported to the maximum power load.

[0085] The electricity retail market trading strategy model of aggregated users: Aggregated users with higher bids in the electricity wholesale market have relatively stable transaction volumes, but may increase the clearing price of the wholesale market, resulting in a decrease in the price difference between the wholesale market and the retail market, thereby affecting revenue. When the wholesale market bid is low, the amount of electricity purchased may decrease, thereby affecting revenue, but the price difference is large, which gives aggregated users more room to bid in the retail market. Therefore, aggregated users need to weigh the purchase price, sales price, purchase volume and transaction volume to determine the strategy quotation coefficient. In addition, aggregated users will only participate in retail transactions after the monthly and day-ahead electricity wholesale market transactions are completed. Specifically, the electricity retail market trading strategy model of aggregated users is as follows.

[0086] max M Ok '=(b Ok -λ')X' Ok

[0087] Where M Ok ' represents the revenue of aggregated user Ok in any trading period in the electricity retail market, X' Ok represents the winning bid quantity of aggregated user Ok in any trading period in the electricity retail market, b Ok represents the declared electricity price of the aggregated user Ok in any trading period in the electricity retail market, λ' represents the clearing price of the trading period in the electricity wholesale market associated with the trading period in the electricity retail market, and further, λ' represents the clearing price of the electricity wholesale market in the previous month of the month to which the trading period in the electricity retail market belongs.

[0088] Furthermore, the electricity retail market transaction strategy model for aggregated users also includes constraints corresponding to the above objective function, as shown in the following formula.

[0089] h min λ'≤b Ok ≤h max λ'

[0090] So far, the wholesale power market clearing model, the retail power market clearing model, the retail power market trading strategy model of each aggregated user, and the wholesale power market trading strategy model of each power generator, large power user, and aggregated user are obtained. Then, enter 220, for each subject in the wholesale power market, use its wholesale power market trading strategy model as the upper model and the wholesale power market clearing model as the lower model to construct its two-layer decision-making optimization model in the wholesale power market, and for each aggregated user, use its retail power market trading strategy model as the upper model and the retail power market clearing model as the lower model to construct its two-layer decision-making optimization model in the retail power market, and obtain the two-layer decision-making optimization model of each power generator in the wholesale power market, the two-layer decision-making optimization model of each large power user in the wholesale power market, the two-layer decision-making optimization model of each aggregated user in the wholesale power market, and the two-layer decision-making optimization model of each aggregated user in the retail power market.

[0091] Then, enter 230, for any subject, by taking the quotation coefficient of each trading period as the action of each trading period, taking the income of the unit declared electricity volume of each trading period as the reward of each trading period, and determining the status of each trading period based on the declared electricity price, declared electricity volume, successful bid electricity volume, and clearing price of the wholesale electricity market of each trading period, a Markov decision process of each wholesale user and power generator in the wholesale electricity market, as well as a Markov decision process of each aggregated user in the retail electricity market are established.

[0092] Here, we first explain the Markov decision process, which is a modeling of the behavior process on a discrete time series. The model mainly includes five types of elements: agent, environment, state, action, and reward. The agent is the subject, the environment is the system in which the agent is located (in this invention, it is a new power system including the subject object and the environment object), the state generally refers to the state of the entire system, the action is the action taken by the subject, and the reward is the feedback given to the agent by the environment.

[0093] Specifically, at each discrete time step t (monthly, day-ahead, and real-time in this invention), the subject obtains environmental observations (market information and equipment information in this invention) and determines its own state s t ∈S (state set), execute action a according to the behavior strategy t ∈A(action set), then P=P ( s'|s,a ) The state transition probability migrates to the new state s t+1 ∈S, get instant reward r t = R(s,a) and update the policy parameters and environment information to form, for example, s 0 、a 0 、r0 、s 1 、a 1 、r 1 、s 2 、a 2 、r 2 ,...The Markov chain of this Markov decision process can be defined as (S,A,P,R,γ), where γ is the discount factor.

[0094] According to the above analysis, the behavior process of the subject can be divided into three stages: state observation, strategy selection and state update. At the same time, the behavior of the subject in the new power system has obvious hierarchy and diversity. There are three types of actions: actions taken immediately after obtaining environmental observation values, actions taken independently after strategy selection, and actions selected according to strategies after multi-agent collaboration and game. Therefore, the behavior level of the intelligent agent can be divided into stress layer, autonomous layer and collaborative layer.

[0095] Next, the Markov decision process established in this embodiment is described one by one.

[0096] The Markov decision process of generators in the wholesale electricity market includes:

[0097]

[0098] a it =[f it ]

[0099]

[0100] In the formula, s it represents the status of the generator Gi in the tth trading period of the wholesale electricity market, represents the electricity price reported by generator Gi in the tth trading period of the electricity wholesale market, represents the maximum value of the electricity prices declared by all generators in the tth trading period of the wholesale electricity market, λ t represents the clearing price of the wholesale electricity market in the tth trading period, They represent the winning bid and declared quantity of power of generator Gi in the tth trading period of the wholesale power market, respectively. it represents the action of generator Gi in the tth trading period of the wholesale electricity market, f jt represents the bid coefficient of the generator Gi in the tth trading period of the wholesale electricity market, r it represents the reward of generator Gi in the tth trading period of the wholesale electricity market, represents the revenue of generator Gi in the tth trading period of the wholesale electricity market, P i (s i(t+1) |sit ,a it ) represents the state transition probability of the generator Gi in the tth trading period of the wholesale electricity market. Further speaking, it is the probability that the generator Gi performs action a in the tth trading period of the wholesale electricity market. it By state it Transfer to state s i(t+1) In addition, the transaction pairs are explained here. Here, it refers to whether the power generator Gi forms a transaction pair with large power users, and whether the power generator Gi forms a transaction pair with aggregated users.

[0101] The Markov decision process of large electricity users in the wholesale electricity market includes:

[0102]

[0103] a jt =[d jt ]

[0104]

[0105]

[0106] Among them, s jt represents the state of large power user Uj in the tth trading period of the wholesale power market, represents the electricity price declared by large power user Uj in the tth trading period of the wholesale power market, represents the minimum value of the electricity price declared by all large power users in the tth trading period of the wholesale power market, λ t represents the clearing price of the wholesale electricity market in the tth trading period, They represent the bid power, actual demand power, and demand response power of large power user Uj in the tth trading period of the wholesale power market, respectively. jt represents the action of large power user Uj in the tth trading period of the wholesale power market, d jt It represents the quotation coefficient of large power user Uj in the tth trading period of the wholesale power market. For the convenience of research, points can be evenly selected in its value set, and the quotation coefficient can be expressed as a discrete value, r jt I represents the reward of large power user Uj in the tth trading period of the wholesale power market. The monthly transaction volume is much larger than the day-ahead and real-time spot market transaction volume. Therefore, in order to train the agent in the whole process of monthly transaction, day-ahead transaction and real-time transaction, the reward can be expressed as the reward per unit of declarable power. Specifically, the income is divided by the actual power demand of the user in the tth trading period (or the tth transaction). Ujrepresents the electricity efficiency of large power users Uj, δ represents the compensation coefficient when participating in demand response and obtaining compensation, P j (s j(t+1) |s jt ,a jt ) represents the state transition probability of large power user Uj in the tth trading period of the wholesale power market. Similarly, whether a trading pair is formed here refers to whether large power user Uj and power generator form a trading pair.

[0107] The Markov decision process of aggregated users in the wholesale electricity market includes:

[0108]

[0109] a kt =[e kt ]

[0110]

[0111] In the formula, s kt represents the state of aggregated user Ok in the tth trading period of the wholesale electricity market, represents the electricity price declared by the aggregated user Ok in the tth trading period in the wholesale electricity market, represents the minimum value of the electricity price declared by all aggregated users in the tth trading period of the wholesale electricity market, λ t represents the clearing price of the wholesale electricity market in the tth trading period, They represent the bid power, actual demand power, and demand response power of aggregated user Ok in the tth trading period of the wholesale electricity market, respectively. kt represents the action of aggregated user Ok in the tth trading period of the wholesale electricity market, e kt represents the quotation coefficient of aggregated user Ok in the tth trading period of the wholesale electricity market, r kt represents the reward of aggregated user Ok in the tth trading period of the wholesale electricity market, I Ok represents the electricity efficiency of aggregated user Ok, δ represents the compensation coefficient when participating in demand response and obtaining compensation, P k (s k(t+1) |s kt ,a kt ) represents the state transition probability of the aggregated user Ok in the tth trading period of the electricity wholesale market. Similarly, whether a transaction is formed here refers to whether the aggregated user Ok and the power generator form a trading pair.

[0112] The Markov decision process of aggregated users in the electricity retail market includes:

[0113]

[0114] a kt '=[h kt ]

[0115]

[0116] Among them, s kt ' represents the state of aggregated user Ok in the tth trading period of the electricity retail market, represents the electricity price declared by the aggregated user Ok in the tth trading period of the electricity retail market, represents the maximum value of the electricity price declared by all aggregated users in the tth trading period of the electricity retail market, λ t ' represents the clearing price of the trading period in the wholesale electricity market that is associated with the tth trading period in the retail electricity market (to put it further, the clearing price of the wholesale electricity market in the previous month of the month to which the tth trading period in the retail electricity market belongs), Represents the aggregated user O k The winning bid volume in the tth trading period in the electricity retail market, represents the aggregated user O in the electricity wholesale market for the trading period associated with the tth trading period in the electricity retail market k The winning bid electricity can be decomposed into the electricity volume of the tth trading period in the electricity retail market (further speaking, the aggregated user O k The winning electricity volume in the target trading period in the wholesale electricity market can be decomposed into the electricity volume in the tth trading period in the retail electricity market. The target trading period is the month before the month to which the tth trading period in the retail electricity market belongs, or the month before the month to which the tth trading day in the retail electricity market belongs). kt ' represents the action of aggregated user Ok in the tth trading period of the electricity retail market, h kt represents the quotation coefficient of aggregated user Ok in the tth trading period of the electricity retail market, r kt ' represents the reward of aggregated user Ok in the tth trading period of the electricity retail market, P k '(s k(t+1) |s kt ,a kt ) represents the state transition probability of the aggregate user Ok in the tth trading period of the electricity retail market. Similarly, whether a transaction is formed here refers to whether the aggregate user Ok and the retail user form a trading pair.

[0117] At this point, the establishment of the Markov decision process is completed. Then, entering 240, based on the established Markov decision process, using the deep reinforcement learning algorithm, each two-layer decision optimization model is solved to obtain the declaration strategy of each wholesale user and power generator in the wholesale power market and the declaration strategy of each aggregated user in the retail power market.

[0118] Among them, in order to use the deep reinforcement learning algorithm (Deep Q-Network, DQN) to solve the two-layer decision optimization model of each power generator in the electricity wholesale market, the two-layer decision optimization model of each large power user in the electricity wholesale market, the two-layer decision optimization model of each aggregated user in the electricity wholesale market, and the two-layer decision optimization model of each aggregated user in the electricity retail market, this embodiment designs the decision-making behavior process of the subject under the deep reinforcement learning algorithm, as follows.

[0119] The first step is to initialize the network parameters, exploration factor, target network update interval and the current state of the subject of the evaluation network. Furthermore, in some embodiments, the update step size, discount factor, memory bank capacity and experience playback batch of the evaluation network need to be initialized.

[0120] In the second step, a greedy strategy (ε-greedy strategy) is used to determine the declared electricity price of the subject in the current trading period and update the exploration factor. Among them, the ε-greedy strategy is used to determine the declared electricity price of the subject in the current trading period, which specifically includes: generating a random number β∈[0,1], if the random number β is less than the exploration factor, a random quotation is selected within the quotation range, otherwise the optimal quotation is selected according to the evaluation network. In addition, the exploration factor can be updated by the following formula.

[0121]

[0122] Among them, ε t represents the exploration factor of the t-th step (i.e., the exploration factor of the t-th trading period); ε start represents the initial exploration factor; ε end It represents the exploration factor at the end of the simulation, specifically, it can be a preset value; T is the total number of iterations (number of steps), that is, the total number of trading periods, specifically, it can be 8759.

[0123] The third step is to execute the action of declaring the electricity price determined by the declaration, obtain the status and reward of the subject in the next trading period, and save the current status, action, reward, and status of the next trading period as a sample to the memory bank. When saving the sample, first determine whether the current number of samples in the memory bank has reached the memory bank capacity. If not, store it directly. If it has reached, delete the earliest sample stored in the memory bank before storing it.

[0124] The fourth step is to update the network parameters of the evaluation network based on the first prediction value and the second prediction value. The first prediction value is the prediction value of the evaluation network for the current state and the current action, and the second prediction value is the maximum value of the prediction values ​​of the target network for all actions in the subject state and action set in any sample selected from the memory bank. A batch of experience playback samples can be selected from the memory bank. Further, the network parameters of the evaluation network can be updated by the following formula.

[0125]

[0126] L(w)=E s,a,r,s' [(r+γmax A Q(s',a';w')-Q(s,a;w)) 2 ]

[0127] Among them, w' and w represent the updated network parameters and the current network parameters respectively, α represents the update step size, L(w) represents the loss function, and E s,a,r,s' [] represents the expected value of the experience replay batch samples (i.e., the expected value of the experience replay batch samples), r represents the reward for executing the action of the declared electricity price determined by the declaration, γ represents the discount factor, Q(s,a;w) represents the predicted value (i.e., the first predicted value) when the input of the evaluation network is the current state s and the current action a, max A Q(s',a';w') represents the maximum value (i.e., the second prediction value) of all predicted values ​​obtained by inputting the state s' (the state is a state in a sample selected from the memory bank) and each action in the action set A in the target network.

[0128] Step 5: Check whether the current number of steps reaches an integer multiple of the target network update interval. If not, no processing is performed. If it does, the network parameters of the evaluation network are copied to the target network, and it is determined whether the current number of steps reaches the preset value (regardless of whether the current number of steps reaches an integer multiple of the target network update interval, a judgment is made as to whether the preset value is reached).

[0129] Step 6. If the preset value is not reached, the current state will be updated to the state of the next trading period, the current step number will be increased by one, and the steps of determining the declared electricity price, updating the exploration factor, executing the action, saving the sample, updating the network parameters, copying the network parameters, and judging whether the step number has reached the preset value (i.e. the second to fifth steps mentioned above) will be continued until the current step number reaches the preset value and ends.

[0130] For the above-constructed two-layer decision optimization model, based on the established Markov decision process, through the above process, the declared electricity price and declared electricity quantity of each large power user, aggregated user, and power generator in the wholesale power market each month, each day before, and each hour, as well as the declared electricity price and declared electricity quantity of each aggregated user in the retail power market each day before can be obtained. Specifically, in some embodiments, the transaction period corresponding to each step can be set to one hour, and then for monthly transactions, it is carried out in the last hour of the last day of each month (running the monthly two-layer decision optimization model), and for daily transactions, it is carried out in the last hour of each day (running the daily two-layer decision optimization model).

[0131] In order to better understand the decision-making behavior process of the designed subject under the DQN algorithm, the following Figure 5 For the model constructed by the present invention, the subject state s, action a and reward r in the process can refer to the Markov decision process established above, and the specific steps of the algorithm process are as follows:

[0132] (1) Generate evaluation network parameters w. If the current step size step = 0, copy the evaluation network parameters to the target network, generate exploration factor ε, update step size α, discount factor γ, experience replay batch k, memory bank capacity Z and network update interval N. If the current step size step = 0, initialize the subject's current state s.

[0133] (2) Adopting the ε-greedy strategy for action selection, that is, the subject selects the quotation. The ε-greedy strategy is applied to the reinforcement learning algorithm to determine the probabilities of two action modes: randomly exploring the environment and using the strategy set to make decisions. The specific process is: first define an exploration rate ε∈[0,1]. Each time in the quotation selection link, generate a random number β∈[0,1]. If the random number is less than ε, select a random quotation within the quotation range; otherwise, choose the optimal quotation according to the neural network. The purpose of using the ε-greedy strategy is to enable the agent to fully explore the environment at the beginning of the iteration, so as to accumulate as much experience as possible in the memory bank. As the experience increases, the exploration rate is gradually reduced, and the possibility of using the learned knowledge is increased. In this way, the needs of exploration and utilization can be balanced, and the learning and convergence process of the strategy can be continuously optimized. Therefore, ε is reduced linearly in each training round. In the initial stage, the agent will be more inclined to explore, and then as the learning progresses, the proportion of using known information will gradually increase, that is:

[0134]

[0135] Among them, ε t represents the exploration factor at step t, ε start represents the initial exploration factor, ε endrepresents the exploration factor at the end of the simulation, and T is the total number of iterations.

[0136] (3) Execute action a, that is, declare the electricity quantity and price. Participating in market bidding belongs to the collaborative layer behavior, because the result of executing action a needs to be reflected in the market clearing result obtained through multi-agent collaboration. According to the market clearing model, the clearing result is obtained, and then the next state s' and reward r are obtained. Save the information [s, a, r, s'] to the memory bank. If the memory bank data reaches the capacity Z, clear the earliest data put into the memory bank. It should be noted that the present embodiments all belong to the collaborative layer behavior.

[0137] (4) Gradient descent process. The core of the DQN algorithm process is to train the neural network parameters w to approximate the value function. This paper adopts the gradient descent method to minimize the loss function, that is, the deviation between the target value and the evaluation network output value, to continuously train and adjust the network parameters. For state s, the predicted value of the evaluation network output is Q(s,a;w), the reward for executing action a is r, the state transition is s', the target network output is Q(s',a;w'), k samples are selected from the memory bank, and if there are less than k samples, all the information in the memory bank is taken out. The loss function can be expressed as:

[0138] L(w)=E s,a,r,s' [(r+γmax a Q(s',a;w')-Q(s,a;w)) 2 ]

[0139] Next, find the gradient of w and finally update w:

[0140]

[0141] (5) Update environmental information, step size step+1, when the step size reaches the network update interval N, copy the evaluation network parameters to the target network. In some embodiments, the updated environmental information may specifically update the power of the subject, or more specifically, the power in the contract signed by the subject. During the operation of the new power system, the power generation and consumption parties record the interaction results in the power contract as a medium for mutual implementation of the behavior. At the same time, the interaction process between the subject and the environment is also recorded and executed in the form of generating and updating the power contract. In addition, in some embodiments, updating environmental information also includes updating the capacity of the unit and load.

[0142] (6) Repeat the above steps (2) to (5) until step = 8759, the one-year simulation cycle ends, and the training is completed.

[0143] Furthermore, the present invention also designs a simulation process for the market-oriented operation of a new power system, such as Figure 6. Among them, the cycle of a single simulation is one year, and the simulation step length is refined to the hourly level. The simulation process is advanced step by step according to the time scale, covering medium- and long-term transactions, spot transactions, retail / distributed transactions, as well as ancillary services and demand response, simulating the whole process of market players participating in the market-oriented transactions of the new power system. According to the time scale, medium- and long-term transactions (monthly transactions), spot transactions, retail / distributed transactions, and finally ancillary services and demand response and other real-time balancing behaviors of electric energy are carried out. In each level of the trading market, the transaction process can be refined into market players participating in information declaration, market clearing, contract execution and settlement, and real-time electricity quantity determination. Among them, information declaration is the autonomous layer behavior of the subject, and market clearing is the collaborative layer behavior, which is carried out according to the subject quotation and quantity reporting model and market clearing model established above. Contract execution and settlement, and real-time electricity quantity determination are the stress layer behaviors of the subject. In each transaction link, multiple subjects identify external information and call behavior functions to participate in the simulation process. The behavior process is as above. Figure 5 In addition, since the simulation cycle is one year, annual transactions are not considered. The estimated monthly output, daily output and real-time output curves of units and loads are derived from empirical data. The specific overall development process of the power market is as follows:

[0144] (1) First, input the initial data and conduct monthly centralized bidding transactions. The power generation companies and users declare 80% of the monthly estimated power generation and consumption. The quotations are made according to the model established above and the decision is made through the deep reinforcement learning algorithm. The management entity aggregates the information and makes unified clearing. The transaction results are recorded in the contract object. After the transaction is completed, the environmental information and the neural network parameters of the subject are updated. A total of 12 monthly transactions are conducted throughout the year, and the step length is the last hour of the last day of each month.

[0145] (2) A decentralized market model is adopted to connect medium- and long-term transactions with spot transactions, with the physical execution of medium- and long-term contracts as the main method and the participation of spot market deviation electricity. That is, in the day-ahead stage, the power generation and consumption parties decompose the monthly contract electricity, expected power generation and expected demand electricity into days, where the contract electricity is decomposed according to the proportion of demand electricity on each day of the month. After determining the power generation and demand electricity, a day-ahead centralized bidding transaction is conducted. The declared electricity is the daily deviation electricity that is not met after the contract is decomposed. The quotation is obtained through a deep reinforcement learning algorithm. After the transaction is completed, the contract information and neural network parameters are also updated. A total of 365 day-ahead transactions are conducted throughout the year, and the step length is carried out in the last hour of each day.

[0146] (3) After the day-ahead wholesale market transaction, the day-ahead retail market transaction is carried out. The aggregated user sells wholesale electricity to the retail user. If there is a distributed power source within the aggregated user, its electricity is sold by the aggregated user agent. The price signal of the wholesale market is transmitted to the retail market by market pricing. The retail user is the power seller and the price is set unilaterally. The retail contract is formed in the order of its price from low to high. After the transaction is completed, the retail contract information and the retail user neural network parameters are updated.

[0147] (4) Further refine the transaction scale and conduct real-time spot transactions. First, the expected power generation and expected demand are still broken down into hours. Then, the daily power consumption of the day-ahead contract and the monthly contract is broken down into hours. The real-time expected power generation and consumption of both parties are determined and declared based on this power. The quotation is also obtained through the deep reinforcement learning algorithm. After the transaction is completed, the contract information and neural network parameters are updated. A total of 8,760 real-time transactions were conducted throughout the year, one for each step.

[0148] (5) Considering the real-time output fluctuations of the units and loads, determine the actual output and actual load at a certain point in time, and take the smaller value of the two as the actual executable power of the contract, that is, the actual power purchased by the user. Compare the contract purchase power with the actual executable power, adjust the contract power, and compare the executable power of all contracts at a certain point in time with the demand power at that point in time. If there is an imbalance between supply and demand, the thermal power generator and the user shall perform auxiliary services and demand response behaviors, and the user shall perform real-time balancing behaviors first to maintain the real-time balance of supply and demand. The maximum power that can be provided by auxiliary services and demand response is obtained by the ramp rate of the units and loads, and at the same time, it is necessary to ensure that the output of the equipment is within the upper and lower limits of the allowable output.

[0149] In addition, the present invention also takes a regional power system as an example, applies the above model and algorithm, and gives the following examples.

[0150] 1. Parameter and solution settings

[0151] This example test is based on NVIDIA RTX 3060Intel(R)Core(TM)i7-12650h CPU. Python is mainly used in object-oriented programming to complete the code compilation of deep reinforcement learning DQN algorithm, market bidding model, subject strategy model and Markov decision process. The neural network of DQN is generated using pytorch. In addition, the database background of the simulation system is built based on SQLServer database, and the visualization interface of the case is built by connecting AnyLogic software through pypeline plug-in. In order to prove the superiority and effectiveness of the DQN algorithm, the Q-learning algorithm is used to solve the same example, and finally compare the training effect.

[0152] (1) Simulation experiment setting: The purpose includes two aspects. First, to compare algorithms to prove the effectiveness and superiority of the DQN deep reinforcement learning algorithm for user quotation strategy selection. Second, to conduct a macro-micro analysis to analyze the changes in the user's decision-making process under the DQN algorithm training and its impact on market operation results. Among them, in order to compare algorithms, Experiment 1 uses the DQN deep reinforcement learning algorithm for simulation as the experimental group, and Experiment 2 uses the Q-learning algorithm for simulation as the control group. The main parameter settings of the model algorithm are shown in Table 1 below.

[0153] Table 1

[0154]

[0155] Among them, the action set, that is, the bid coefficient set of power generators and users, is defined as follows: the bid lower limit of ordinary users is set to 430 yuan / MW·h, the corresponding minimum bid coefficient is 0.86, the bid upper limit is 627.2 yuan / MW·h, the corresponding maximum bid coefficient is 1.2544, and 28 points are evenly taken in the range of the upper and lower limits of the bid coefficient to form a discrete action interval with 30 bid coefficients, corresponding to 30 bid values. The lower limit of the wholesale power market bid of aggregated users is set to 410 yuan / MW·h, the upper limit is 607.2 yuan / MW·h, the upper and lower limits of the bid coefficient are [0.854,1.265], and the upper and lower limits of the pricing coefficient in the power retail market are [1.01,1.3]; the bid lower limit of power generators is set to (C Gi -60) yuan / MW·h, the upper limit is (C Gi +144) yuan / MW·h, and its action space is also defined as a set of 30 discrete quotation coefficients. For the Q-learning algorithm, since it cannot analyze high-dimensional states, its state set is simplified to the quotation of the subject in the previous training step.

[0156] (2) Experimental parameter setting: The installed capacity ratio of renewable energy in the power system is 65%, the power generation ratio of renewable energy is 40%, the total power generation is 58700000MWh, and the overall supply and demand in the system is balanced. The simulation experiment simulates the operation of the system in one year, with a single-step operation time of 1 hour and a total duration of 8760 hours. The experimental parameter inputs of the two groups of experiments are the same, and the main experimental data are shown in Table 2 below.

[0157] Table 2

[0158]

[0159] In the experiment, the power generators include 5 thermal power generation entities, 2 wind power generation entities, 2 photovoltaic power generation entities and 3 distributed photovoltaic power generation entities; the demand entities include 15 small and medium-sized power users and 7 large power users; the aggregated user entities include 2 power sales companies and one distributed park. Among them, the thermal power generating units include 330MW, 600MW, 660MW and 1000MW coal-fired units. The relevant marginal cost technical parameters are obtained by fitting the actual data, and the marginal cost of the units is finally calculated. The annual utilization hours, declarable electricity and annual electricity decomposition curves of each unit are obtained based on empirical data. The specific information of the power generators and users is shown in Tables 3 and 4 below.

[0160] Table 3

[0161]

[0162] Table 4

[0163]

[0164]

[0165] 2. Analysis of user market collaborative bidding strategy

[0166] (1) Analysis of algorithm effectiveness: The user's market collaborative behavior refers to a series of behaviors in which the user actively participates in the electricity market in collaboration with other entities under market signals, including self-declaration of electricity volume and electricity price, participation in wholesale and retail transactions, and demand response behavior. In the model constructed in this embodiment, the main research is on the adaptive bidding behavior of users and power generators in the market bidding process, as well as the deep reinforcement learning process under environmental changes and market information feedback. The simulation simulates the transaction situation for one year, with an hour as the simulation step length. After each hour of transaction is completed, a training iteration is performed, for a total of 8760 iterations. For each intelligent agent, since it may not participate in market transactions of all time scales at the same time, the number of iterations and training is about 8000.

[0167] ① Convergence analysis: For the two types of intelligent agents, users and generators, their dual Q networks and memory banks are set up respectively, such as Figure 7 The figure shows the training error value of the user DQN neural network under 8760 steps. It can be seen that as the iteration proceeds, the training error of the double-layer Q neural network gradually decreases and eventually tends to be stable, and the network parameters are basically stable as the iteration proceeds.

[0168] Through the user's quotation, we can have a more intuitive analysis of the algorithm's convergence. In order to prove the superiority of the DQN algorithm, the user's quotation convergence under the Q-learning algorithm and the DQN algorithm is compared. Among them, in the electricity wholesale market transaction, the quotation convergence of some users is as follows: Figure 8 As shown (Q-learning on the left and DQN on the right).

[0169] From the distribution of user quotations, it can be seen that the Q-learning algorithm is unstable in convergence and is prone to fall into local optimal solutions. As the number of states increases, the algorithm cannot fully traverse the state actions, making it difficult to achieve good results. Under the DQN algorithm, users also undergo a random exploration process, but because it adopts the form of a neural network, it does not need to traverse all states and action spaces. With the existing experience of the memory bank, the quotation strategy can be trained to converge. From the overall trend of user quotations, it can be seen that as the iteration proceeds, the user's quotation gradually converges under the training of the DQN neural network: when the number of market transactions is insufficient, the user's quotation selection is relatively random, and it is displayed in a scattered manner on the image. As the transaction proceeds, the user's neural network training is basically completed. With the goal of maximizing benefits, the user's quotation gradually moves towards the strategy that may obtain the maximum benefit, and the quotation gradually concentrates. In general, the convergence speed and effect of the DQN deep reinforcement learning algorithm are significantly better than those of the Q-learning algorithm, and the quotation when it finally reaches convergence is smaller than the latter, which is conducive to users obtaining lower wholesale market electricity purchase prices to obtain greater benefits.

[0170] ② Comparison of algorithm effectiveness: In order to eliminate the impact of chance on the experimental results, 50 experiments were conducted for each of Experiments 1 and 2, and a one-way analysis of variance was performed on the relevant power market operation indicators obtained from each experiment to verify the significant impact of the algorithm application on the experimental results. The settings are shown in Table 5 below.

[0171] Table 5

[0172]

[0173] For the sample data obtained by repeated simulation of the two experiments (50 times each), a one-way ANOVA was performed on them, with the following assumptions: H0: μ1 = μ2 (the algorithm selection has no significant effect on the average clearing price of the market); H1: μ1 ≠ μ2 (the algorithm selection has a significant effect on the average clearing price of the market). SPSS software was used to perform a one-way ANOVA on the algorithm selection factor, with the average clearing price of the market as the comparison indicator. The normality test results and the variance homogeneity test results are shown in Tables 6 and 7 below, respectively.

[0174] Table 6

[0175]

[0176] Table 6 shows the descriptive statistics and normality test results of the average market clearing price, including the median, mean, etc., which are used to test the normality of the data. Due to the small sample size, the results of the Shapiro-Wilk test are used, and the significant P value is 0.294, which is not significant at the level, and the original hypothesis (data conforms to the normal distribution) cannot be rejected, so the data conforms to the normal distribution.

[0177] Table 7

[0178]

[0179] The results of the variance homogeneity test in Table 7 show that for the clearing price, the significant P value is 0.535, which is not significant at the level, and the null hypothesis cannot be rejected (null hypothesis: the variance homogeneity is satisfied), so the data satisfies the variance homogeneity. The final variance analysis results are shown in Table 8 below.

[0180] Table 8

[0181]

[0182] The mean values ​​of the clearing prices of Scheme 1 and Scheme 2 are 429.269 and 437.392 respectively; due to the homogeneity of variance, a single sample variance test (significance level 5%) is used, and the variance analysis result P≤0.05, so the statistical results are significant, and the null hypothesis H0: μ1=μ2 (the algorithm selection has no significant effect on the average clearing price of the market) is rejected, indicating that there are significant differences in the average clearing prices of the market obtained by different schemes, that is, different user market behavior decision-making methods have a significant impact on the market clearing results. Among them, μ1=429.26, which is significantly lower than μ2=437.39, indicates that under the DQN algorithm, the training effect of the user market quotation strategy is better, the market clearing efficiency is high, and the average price is low, which is conducive to improving the user's benefits. At the same time, the standard deviation of the average market clearing price of Scheme 1 is lower than that of Scheme 2, indicating that the algorithm is less random and the experimental results are more stable.

[0183] (2) Analysis of user bidding strategy and market clearing price: Under the condition of two-sided bidding, users participate in the electricity market and respond to price signals. They need to make a trade-off between the transaction volume and the clearing price. When the bid is high, the required volume can be purchased stably, but the market clearing price may be increased, thereby increasing the cost of electricity. When the bid is low, the market clearing price may be reduced, but it cannot fully meet the electricity demand for production and life. For power generators, user participation in bidding will inevitably affect their overall bidding strategy, and then affect the market clearing results under the joint action of both supply and demand. The above analysis has analyzed the market transaction bids of some users and power generators throughout the year. Next, we will analyze their overall trends and the market clearing prices under two-sided bidding, and finally analyze their final profit results under the main bidding strategy.

[0184] ① Analysis of wholesale power market quotation strategy: In wholesale power transactions, the simulation results of the average monthly market quotations of each generator and user and the average monthly market clearing prices of each level are as follows: Fig. 9 First, the user-side quotations are analyzed. It can be seen that although the initial benchmark values ​​of the quotations are relatively close, as the training progresses, the quotations of users have diverged. The average quotations of some users have decreased and stabilized, while the average quotations of some users have gradually increased. This is mainly due to the different kilowatt-hour benefits of each user. 4 , U 6 , U 7 The main body, due to its higher per-kilowatt-hour efficiency, has a larger quotation space and a larger profit margin. The total benefit is mainly related to the winning bid volume. In order to maximize the benefit, users tend to give up part of the benefits to the power generation side, declare a higher price to guarantee the purchase volume to ensure output, and reduce the demand response volume. 2 , U 1 , U 3 Due to the low efficiency of electricity, the main body will have to make a trade-off between the winning bid volume and the electricity cost, and gradually reduce the bid price to approach the market clearing price to reduce the electricity cost. Similarly, the bid price on the power generation side is mainly affected by the marginal cost. The units with higher marginal costs tend to bid higher prices, and they are more likely to become marginal units. Units with cost advantages choose lower bid prices in order to stabilize the transaction volume. 1 , G 4 From the perspective of wind power units, their bids have gradually decreased and stabilized. This is mainly due to the continuous decrease in the lowest bids on the user side. In order to ensure the winning bids, power generators have lowered their bids accordingly, while wind power sellers G 6 , G 7 and photovoltaic power sales electricity supplier G 8 , G 9 They are all declared at lower prices. This is not only because their benchmark declared prices are lower, but also related to the volatility and instability of their output. In order to obtain stable electricity sales as much as possible, renewable energy power generation entities have adopted lower declared prices.

[0185] From the overall trend of the quotation curve and the clearing price curve, it can be observed that in the process of continuous participation in the market, power generators and users in the market have trained and iterated their own quotation strategies based on market information and transaction results. Under the influence of the quotations on both the supply and demand sides, the market clearing price has gradually stabilized and has a slightly decreasing trend. Its fluctuations are mainly related to the supply and demand ratio of electricity in that month. In the end, the strategic quotation behaviors of users and power generators in pursuit of maximizing their own interests gradually converged, and both parties found the quotation strategy with the highest expected return. The final converged quotations of all users are shown in Table 9 below.

[0186] Table 9

[0187]

[0188] It can be seen that the bid is closely related to the electricity revenue, which is consistent with the analysis results of the above average bid. Users with greater electricity benefits have greater bidding space, and their final bids that reach convergence are higher. Therefore, in general, users with high electricity benefits tend to make high bids, which have less impact on the unified market clearing price, while users with low electricity benefits are likely to form the last pair of trading pairs to reach a transaction. The same is true for the power generation side. The units with greater marginal costs are more likely to become marginal units, affecting the market clearing price.

[0189] ② Analysis of wholesale electricity market clearing prices: Market transaction prices directly affect the main body of electricity purchase costs, such as Fig.10 The figure shows the market electricity supply-demand ratio and wholesale market transaction electricity price in each month. As can be seen from the figure, under the influence of the bidding behavior on both the supply and demand sides, the market electricity price has not been continuously increasing or decreasing. Its overall trend is basically determined by the supply-demand ratio. In the months with high supply-demand ratios, such as November and December, the market supply is sufficient and the overall electricity price is low. In the months with low supply-demand ratios, such as August and September, the overall electricity price in the market is high, reflecting the general commodity characteristics of electricity prices being affected by the supply and demand relationship in the free bidding market on both sides of supply and demand. In addition, from the training effect of the model algorithm, with the increase in the number of iterations, the market electricity price gradually tends to be stable. Comparing the months with close supply-demand ratios, such as April and June, March and July, May and November, it can be seen that when the market supply-demand ratio is similar, the later the month, the smaller the fluctuation of the market price, and the lower the average market clearing price, reflecting that as the training progresses, the market clearing price has a downward trend and gradually stabilizes, and the user's electricity cost has been reduced. The total average market clearing price λ in the experiment C =419.17 yuan / MW·h. The following Table 10 shows the details of the average market clearing price.

[0190] Table 10

[0191]

[0192] ③ Analysis of electricity retail market quotation strategy and market price: In the retail market, the listing transaction method is adopted, and the aggregated users on the power supply side set the price unilaterally. Retail users choose to place orders independently. The market clearing price is as follows: Fig.11 As shown. It can be seen that since the market pricing is determined unilaterally by the power supply side, with the convergence of the aggregated user quotation strategy, the market retail pricing also converges significantly. The monthly market clearing price is used as the benchmark value for the retail market declaration of the month in the model. It can be seen that the electricity price trend of the final market clearing result is relatively consistent with the overall trend of the wholesale market, reflecting the transmission of electricity wholesale to retail market prices. Among them, the market pricing and the quotation coefficients of aggregated users in each month are shown in Table 11 below.

[0193] Table 11

[0194]

[0195]

[0196] It can be seen that in the case of one-sided pricing transactions, when retail users basically do not implement market behavior, the pricing coefficient of the aggregated users on the power supply side continues to increase in order to maximize benefits, and the final converged quotation coefficient is 1.4, reaching the upper limit, reflecting that its pricing strategy has been optimized under the reinforcement learning algorithm. Compared with the experimental results of the two-sided centralized bidding in the wholesale market, it can be seen that the market behavior on the demand side can well restrict the supply side quotation, which plays an important role in stabilizing the market clearing electricity price and reflecting the supply and demand relationship of electricity.

[0197] 3. Analysis of green synergy of user market behavior

[0198] User green behavior refers to the user's purchase of renewable energy power generation, and user collaborative behavior refers to a series of behaviors in which users actively cooperate with other entities to participate in the power market under market signals, including self-declaration of electricity volume and electricity price, participation in wholesale and retail electricity transactions, and demand response behavior. Before analyzing the user's electricity purchase volume, the overall power generation of the system is analyzed first. The estimated power generation curves of various resources are obtained by fitting empirical data, and the final actual output results are obtained by market trading conditions, such as Fig.12 As shown. It can be seen that the thermal power unit has the smallest output fluctuation, operates smoothly and reliably, and is the most stable power supply unit in the new power system. The output fluctuation of photovoltaic power generation is affected by the day and night sunshine conditions, and the output curve of wind power is basically irregular. Both can only generate electricity under specific conditions, and the output is 0 when there is no wind or light. Therefore, with the large-scale integration of renewable energy into the power grid, the safe and stable operation of the new power system is greatly challenged, and users are required to actively respond to price signals to purchase new energy power generation, and actively provide auxiliary services and demand response.

[0199] On this basis, the consumption of new energy power generation by users in the wholesale electricity market is analyzed. In particular, this embodiment sets the lowest marginal cost for renewable energy according to actual conditions, and takes carbon prices into account when calculating the revenue of power generators. Therefore, under the training of the reinforcement learning algorithm, new energy power generators quote lower prices in the market transactions throughout the year, and the final converged average price is 283.9 yuan / MW·h, which is significantly lower than the 343.9 yuan / MW·h of thermal power generators. In the case of fixed user demand, in trading periods with high new energy power generation, the market clearing price is generally low. Under the guidance of price signals, users adjust their own quotation strategies to increase the overall purchase of electricity. The process of new energy power generation affecting the market clearing price is shown in the figure below. Fig.13 shown.

[0200] Since the electricity demand of users fluctuates every month, the proportion of the electricity purchased by users from new energy sources to the total electricity purchased and the proportion of the expected electricity generated by new energy sources to the total expected electricity generated are analyzed, such as Fig.14 As shown on the left. It can be seen that the user's purchase of new energy power generation gradually approaches the actual power generation trend of renewable energy as the number of training times progresses, and the deviation gradually decreases in the final stage of training. Under the training of the DQN algorithm, the user's consumption of new energy power generation is basically consistent with the output trend of new energy, indicating that under the training of the deep reinforcement learning algorithm, it is possible to avoid the instability of clearing electricity caused by the randomness of the supply and demand quotations as much as possible, and improve the consumption effect of new energy power generation. Under the Q-learning algorithm, the non-convergence of the supply and demand quotation strategies leads to large fluctuations in the transaction volume.

[0201] In addition, the total purchase amount of renewable energy power generation in each month is also shown, such as Fig.14 As shown on the right. It can be seen that under the DQN algorithm, the user's purchase of new energy power generation not only conforms to the unit's expected output in terms of trend, but also has a higher total purchase amount of new energy power generation and a higher ratio of the total purchase amount to the actual power generation of new energy. Calculate the utilization rate of new energy θ R , in is the actual grid-connected power of the renewable energy generator Gri′ after considering the power transaction in trading period t as well as the ancillary services and demand response process; is the real-time maximum power generation of the new energy power generator Gri′ during the trading period t, N Gr is the number of new energy generators. In the DQN case, θ R=100%, while this value is reduced to 98.79% in the Q-learning case, indicating that the deviation of renewable energy exceeds the limits of thermal power and the auxiliary services and demand response provided by the user side, resulting in wind and solar power abandonment. The experiment shows that the user's adaptive learning behavior has a profound impact on the power market and the new power system, reflecting the green collaborative market behavior of users actively responding to price signals to purchase renewable energy power generation, and fully proving the important role of user side participation in the power market.

[0202] In addition, an embodiment of the present invention further includes: A8, a method as described in any one of A3-A7, wherein the electricity wholesale market transaction strategy model for aggregated users includes:

[0203] max M Ok =(I Ok -λ)X Ok +δQ' Ok

[0204] Among them, M Ok represents the revenue of aggregated user Ok in any trading period in the wholesale electricity market, I Ok represents the electricity efficiency of aggregated user Ok, λ represents the clearing price of any trading period in the wholesale electricity market, and X Ok represents the winning bid quantity of aggregated user Ok in any trading period of the wholesale electricity market, δ represents the compensation coefficient when participating in demand response and receiving compensation, Q' Ok It represents the demand response quantity of aggregated user Ok in any trading period in the wholesale electricity market.

[0205] A9. A method as described in any one of A3-A8, wherein the electricity retail market transaction strategy model for aggregated users includes:

[0206] max M Ok '=(b Ok -λ')X' Ok

[0207] Among them, M Ok ' represents the revenue of aggregated user Ok in any trading period in the electricity retail market, X' Ok represents the winning bid quantity of aggregated user Ok in any trading period in the electricity retail market, b Ok represents the electricity price declared by the aggregated user Ok in any trading period in the electricity retail market, and λ' represents the clearing price in the electricity wholesale market for the trading period associated with the trading period in the electricity retail market.

[0208] A11. A method as described in any one of A3-A10, wherein the Markov decision process of a large power user in the wholesale power market comprises:

[0209]

[0210]

[0211] Among them, s jt represents the state of large power user Uj in the tth trading period of the wholesale power market, represents the electricity price declared by large power user Uj in the tth trading period of the wholesale power market, represents the minimum value of the electricity price declared by all large power users in the tth trading period of the wholesale power market, λ t represents the clearing price of the wholesale electricity market in the tth trading period, They represent the bid power, actual demand power, and demand response power of large power user Uj in the tth trading period of the wholesale power market, respectively. jt represents the action of large power user Uj in the tth trading period of the wholesale power market, d jt represents the quotation coefficient of large power user Uj in the tth trading period of the wholesale power market, r jt represents the reward of large power user Uj in the tth trading period of the wholesale power market, I Uj It represents the electricity efficiency of large power user Uj, and δ represents the compensation coefficient when participating in demand response and obtaining compensation.

[0212] A12. A method as described in any one of A3-A11, wherein the Markov decision process of aggregating users in the wholesale electricity market comprises:

[0213]

[0214]

[0215]

[0216] Among them, s kt represents the state of aggregated user Ok in the tth trading period of the wholesale electricity market, represents the electricity price declared by the aggregated user Ok in the tth trading period in the wholesale electricity market, represents the minimum value of the electricity price declared by all aggregated users in the tth trading period of the wholesale electricity market, λ t represents the clearing price of the wholesale electricity market in the tth trading period, They represent the bid power, actual demand power, and demand response power of aggregated user Ok in the tth trading period of the wholesale electricity market, respectively. kt represents the action of aggregated user Ok in the tth trading period of the wholesale electricity market, e ktrepresents the quotation coefficient of aggregated user Ok in the tth trading period of the wholesale electricity market, r kt represents the reward of aggregated user Ok in the tth trading period of the wholesale electricity market, I Ok represents the electricity efficiency of aggregated user Ok, and δ represents the compensation coefficient when participating in demand response and obtaining compensation.

[0217] A13. A method as described in any one of A3-A12, wherein the Markov decision process of aggregating users in the electricity retail market comprises:

[0218]

[0219] a kt '=[h kt ]

[0220]

[0221] Among them, s kt ' represents the state of aggregated user Ok in the tth trading period of the electricity retail market, represents the electricity price declared by the aggregated user Ok in the tth trading period of the electricity retail market, represents the maximum value of the electricity price declared by all aggregated users in the tth trading period of the electricity retail market, λ t ' represents the clearing price of the trading period in the wholesale electricity market associated with the tth trading period in the retail electricity market, Represents the aggregated user O k The winning bid volume in the tth trading period in the electricity retail market, represents the aggregated user O in the electricity wholesale market for the trading period associated with the tth trading period in the electricity retail market k The winning bid electricity can be decomposed into the electricity volume in the tth trading period of the electricity retail market, a kt ' represents the action of aggregated user Ok in the tth trading period of the electricity retail market, h kt represents the quotation coefficient of aggregated user Ok in the tth trading period of the electricity retail market, r kt ' represents the reward of aggregated user Ok in the tth trading period of the electricity retail market.

[0222] A16. A readable storage medium storing program instructions, when the program instructions are read and executed by a computing device, the computing device executes the method as described in any one of A1-A14.

[0223] In the specification provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be put into practice without these specific details. In some examples, known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification. It should be understood that in order to simplify the present disclosure and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features clearly recorded in each claim. In addition, unless otherwise specified, the use of ordinal numbers "first", "second", "third" and the like to describe common objects only represents different instances involving similar objects, and is not intended to imply that the objects described in this way must have a given order in time, space, sorting or in any other way.

[0224] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments may be envisioned within the scope of the invention thus described. In addition, it should be noted that the language used in this specification is selected primarily for readability and teaching purposes, rather than for explaining or defining the subject matter of the present invention. Therefore, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is illustrative, not restrictive, with respect to the scope of the present invention, which is defined by the appended claims.

Claims

1. A method for determining a power market behavior strategy for multiple users under a new power system, wherein the power market includes a power wholesale market and a power retail market, wherein the main bodies of the power retail market include retail users on the demand side and aggregate users on the power supply side, and the main bodies of the power wholesale market include wholesale users on the demand side and power generators on the power supply side, and wherein the wholesale users include aggregate users on the power supply side of the power retail market. include: The wholesale power market clearing model and the retail power market clearing model are established with the goal of maximizing social welfare. For each entity in the wholesale power market, the wholesale power market trading strategy model is constructed with the goal of maximizing its profit. For each aggregated user, the retail power market trading strategy model is constructed with the goal of maximizing its profit. For each subject in the wholesale electricity market, the wholesale electricity market transaction strategy model is used as the upper model, and the wholesale electricity market clearing model is used as the lower model to construct a two-layer decision-making optimization model in the wholesale electricity market. For each aggregated user, the retail electricity market transaction strategy model is used as the upper model, and the retail electricity market clearing model is used as the lower model to construct a two-layer decision-making optimization model in the retail electricity market. For any subject, by taking the bid coefficient of each trading period as the action of each trading period, the income of the unit declared electricity volume of each trading period as the reward of each trading period, and determining the status of each trading period based on the declared electricity price, declared electricity volume, winning bid electricity volume, and clearing price of the wholesale electricity market, the Markov decision process of each wholesale user and power generator in the wholesale electricity market and the Markov decision process of each aggregated user in the retail electricity market are established; Based on the established Markov decision process, the deep reinforcement learning algorithm is used to solve the two-layer decision optimization models to obtain the declaration strategies of each wholesale user and power generator in the wholesale electricity market and the declaration strategies of each aggregated user in the retail electricity market. The declaration strategies include the declared electricity price and declared electricity volume in each trading period.

2. The method according to claim 1, in, The trading periods include monthly, day-ahead, hourly in the wholesale electricity market and day-ahead in the retail electricity market. Accordingly, the declaration strategy includes the declared electricity price and declared electricity quantity of each wholesale user and power generator in the wholesale electricity market each month, day-ahead, and hourly, as well as the declared electricity price and declared electricity quantity of each aggregated user in the retail electricity market each day-ahead.

3. The method according to claim 1 or 2, in, The wholesale users also include large electricity users, the retail users include small and medium-sized electricity users, the aggregate users include power sales companies, microgrids, and distributed parks, the large electricity users are electricity users whose supply voltage is greater than or equal to a first predetermined value, and the small and medium-sized electricity users are the remaining electricity users except the aggregate users and large electricity users.

4. The method according to claim 3, in, The social welfare in the wholesale electricity market is maximized when the difference between the declared electricity prices of wholesale users and power generators is the largest, and the wholesale electricity market clearing model includes: Where y represents the difference in electricity prices declared by wholesale users and power generators, B Uj , B Ok , B Gi They represent the electricity prices declared by large power users Uj, aggregate users Ok and power generators Gi in any trading period of the wholesale power market, N U 、N O 、N G They respectively represent the number of large power users, aggregated users and power generators in the wholesale power market.

5. The method according to claim 3 or 4, in, The maximum social welfare in the electricity retail market is when the electricity cost of retail users is minimized, and the electricity retail market clearing model includes: Among them, y' represents the electricity cost of retail users, b Ok N represents the electricity price declared by the aggregated user Ok in any trading period of the electricity retail market, O Represents the number of aggregated users in the wholesale electricity market.

6. The method according to any one of claims 3 to 5, in, The wholesale electricity market trading strategy model for power generators includes: Among them, M Gi represents the revenue of the generator Gi in any trading period of the wholesale electricity market, λ represents the clearing price in any trading period of the wholesale electricity market, C Gi represents the benchmark price of electricity bidding of power generator Gi, X Gi It represents the winning bid quantity of the generator Gi in any trading period of the wholesale electricity market. represents the carbon price, represents the carbon quota coefficient per unit of electricity generation, It represents the weighted average carbon emission coefficient of all units in the power generator Gi.

7. The method according to any one of claims 3 to 6, in, The electricity wholesale market trading strategy model for large electricity users includes: maxM Uj =(I Uj -λ)X Uj +δQ' Uj Among them, M Uj represents the revenue of large power users Uj in any trading period in the wholesale power market, I Uj represents the electricity efficiency of large power users Uj, λ represents the clearing price of any trading period in the wholesale power market, and X Uj represents the amount of electricity that a large power user Uj bids for in any trading period in the wholesale power market, δ represents the compensation coefficient when participating in demand response and receiving compensation, Q' Uj It represents the demand response amount of large power user Uj in any trading period in the wholesale power market.

8. The method according to any one of claims 3 to 7, in, The Markov decision process of generators in the wholesale electricity market includes: a it =[f it ] Among them, s it represents the status of the generator Gi in the tth trading period of the wholesale electricity market, represents the electricity price reported by generator Gi in the tth trading period of the electricity wholesale market, represents the maximum value of the electricity prices declared by all generators in the tth trading period of the wholesale electricity market, λ t represents the clearing price of the wholesale electricity market in the tth trading period, They represent the winning bid and declared quantity of power of generator Gi in the tth trading period of the wholesale power market, respectively. it represents the action of generator Gi in the tth trading period of the wholesale electricity market, f jt represents the bid coefficient of the generator Gi in the tth trading period of the wholesale electricity market, r it represents the reward of generator Gi in the tth trading period of the wholesale electricity market, It represents the revenue of generator Gi in the tth trading period in the wholesale electricity market.

9. The method according to any one of claims 1 to 8, in, The declaration strategies of each subject are obtained through the following decision-making behaviors constructed under the deep reinforcement learning algorithm: Initialize the network parameters, exploration factor, target network update interval and the current state of the subject of the evaluation network; A greedy strategy is used to determine the subject's declared electricity price in the current trading period and to update the exploration factor; Execute the action of declaring the electricity price determined by the declaration, obtain the state and reward of the subject in the next trading period, and save the current state, action, reward, and state of the next trading period as a sample to the memory bank; Based on the first prediction value and the second prediction value, the network parameters of the evaluation network are updated, the first prediction value is the prediction value of the evaluation network for the current state and the current action, and the second prediction value is the maximum value of the prediction values ​​of the target network for all actions in the subject state and action set in any sample selected from the memory bank; If the current number of steps reaches an integer multiple of the target network update interval, the network parameters of the evaluation network are copied to the target network, and it is determined whether the current number of steps reaches a preset value; If the preset value is not reached, the current state will be updated to the state of the next trading period, the current step number will be increased by one, and the steps of determining the declared electricity price, updating the exploration factor, executing the action, saving the sample, updating the network parameters, copying the network parameters, and judging whether the step number has reached the preset value will be continued until the current step number reaches the preset value and ends.

10. A computing device, include: at least one processor; as well as A memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, and the program instructions include instructions for executing the method according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Power purchasing and selling joint strategy optimization method based on multi-task deep reinforcement learning

    CN116029415A

  • Electricity market balancing method, system and equipment and storage medium

    CN116885714A

  • Renewable energy consumption method based on game model

    CN117726418A

Cited By

  • Electric power market split-field bidding system considering unit heterogeneous cost characteristics

    CN119624597A

  • Power transaction method, device and system based on Markov optimization

    CN119850247A

  • Electric power spot strategy processing method based on intelligent agent and computer equipment

    CN120069597A

  • Novel power market behavior strategy determination method for multiple users in power system

    CN120373889A