Agent model training method for power market

Abstract

This application provides a method for training an intelligent agent model for the electricity market, applicable to the field of artificial intelligence. The method includes: the intelligent agent model is a reinforcement learning model built on an actor-commentator network framework; the actions output by the intelligent agent model are input into an electricity market environment model to obtain the market environment state; wherein the actions are the declared electricity price and declared power, and the market environment state is the node price and winning bid power at the electricity market clearing point; the actor network of the reinforcement learning model is trained using a first loss function; wherein the function value of the first loss function is determined based on the values ​​of the main loss term and the regularization term; the main loss term represents the negative value of the Q-value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price; the Q-value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment state and the action.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method for training an intelligent agent model for the electricity market. Background Technology

[0002] With the deepening of power market reform, virtual power plants, as an important form of distributed energy aggregation, have been widely used in the research of bidding and trading in the power market. Multi-Agent Reinforcement Learning (MARL) has gradually become an important means of solving multi-player game problems in the power market. Because this method can simulate the independent decision-making processes of multiple market participants (such as thermal power units, wind power, photovoltaics, energy storage, and power users) and continuously learn the optimal strategy in a game environment, it has received widespread attention in power market bidding modeling.

[0003] In related technologies, in traditional multi-agent reinforcement learning, the training of actor networks is usually based on policy gradients. Its core objective is to continuously improve the expected return of the current policy in the environment by maximizing the state-action value function given by the critic network. Therefore, during the training process, actor networks mainly focus on how to obtain higher Q values, but lack direct constraints on whether the policy after iterative convergence conforms to the real market operation rules. This can easily lead to the agent excessively pursuing local optimal returns while ignoring the consistency between bidding behavior and real market prices, resulting in problems such as bidding deviating from reality, price signal distortion, and even oscillations in the training process. Summary of the Invention

[0004] In view of this, this application provides a method, apparatus, electronic device, and readable storage medium for training intelligent agent models for the electricity market.

[0005] Firstly, this application provides a method for training an agent model for the electricity market. The agent model is a reinforcement learning model built based on an actor-critic network framework. The method includes: The actions output by the intelligent agent model are input into the electricity market environment model to obtain the market environment state; among them, the actions are the bid electricity price and bid power, and the market environment state is the node electricity price and winning bid power at the electricity market clearing point; The actor network of the reinforcement learning model is trained using the first loss function; the function value of the first loss function is determined based on the values ​​of the main loss term and the regularization term; the main loss term represents the negative value of the Q value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price; the Q value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment state and the action.

[0006] In the embodiments of this application, the reinforcement learning model includes at least two critic networks, which include an online network and a target network; the action-corresponding Q-value is the minimum Q-value generated by all online networks.

[0007] According to embodiments of this application, the method further includes: The commentator network is trained using a second loss function; the value of the second loss function is determined based on the Q-value generated by the online network and the target value generated by the target network; the target value is determined based on the minimum Q-value generated by all target networks and the reward value of the corresponding action; the reward value is determined based on the node electricity price, the winning power, and the corresponding operating cost obtained.

[0008] According to an embodiment of this application, the first loss function is composed of a linear summation of a main loss term and a regularization term. The value of the regularization term is determined based on the difference between the actual market electricity price and the declared electricity price, as well as the regularization term coefficient. The regularization term coefficient can be adaptively adjusted according to changes in the value of the main loss term.

[0009] According to an embodiment of this application, the regularization coefficient can be adaptively adjusted according to the numerical change of the main loss term, including: the adaptive adjustment of the numerical change direction of the regularization coefficient is negatively correlated with the numerical change direction of the main loss term.

[0010] According to an embodiment of this application, the electricity market environment model includes: an objective function and constraints; wherein, the objective function represents minimizing the total electricity purchase cost, and the constraints are used to constrain the safe dispatch constraints that the objective function must satisfy. The actions output by the intelligent agent model are input into the electricity market environment model to obtain the market environment state, including: Based on the actions and constraints, the objective function is solved using a mathematical programming solver to obtain the market environment state.

[0011] According to embodiments of this application, the reward value is determined based on the nodal price, the awarded power, and the corresponding operating costs, including: The difference between the declared total price and the operating cost is determined as the reward value; whereby the declared total price is the product of the node electricity price and the winning bid power.

[0012] Secondly, this application provides a training device for an intelligent agent model in the electricity market, comprising: The module for generating market environment states is used in the agent model, which is a reinforcement learning model built on an actor-commentator network framework. The actions output by the agent model are input into the electricity market environment model to obtain the market environment state. The actions are the bid price and bid power, and the market environment state is the node price and winning bid power at the electricity market clearing point. The training module is used to train the actor network of the reinforcement learning model using the first loss function. The function value of the first loss function is determined based on the values ​​of the main loss term and the regularization term. The main loss term represents the negative value of the Q value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price. The Q value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment and the action.

[0013] Thirdly, this application provides an electronic device, comprising: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to implement the methods described above.

[0014] Fourthly, this application provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the above-described method.

[0015] This application provides a method for training an agent model for the electricity market, including a device, an electronic device, and a readable storage medium. Utilizing an electricity market environment model, the market environment state can be continuously updated. Based on the market environment state and the actions output by the agent model, a commentator network in a reinforcement learning model is used to determine the Q-value corresponding to each action. Furthermore, a main loss term represents the negative value of the Q-value corresponding to the action, and a regularization term represents the difference between the actual market electricity price and the declared electricity price. Since the function value of the first loss function is determined based on the values ​​of the main loss term and the regularization term, training a reinforcement learning model based on an actor-commentator network framework using the first loss function can simultaneously achieve high Q-values ​​for the agent model's output actions and consistency between the actions and the actual market electricity price. This effectively reduces problems such as declared electricity price deviation and policy distortion caused by the actions output by the agent model, achieving higher quality training of the reinforcement learning model. Attached Figure Description

[0016] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a method for training an intelligent agent model for an electricity market according to an embodiment of this application is shown schematically. Figure 2 This illustration schematically shows a model structure diagram of an agent model training method for an electricity market according to an embodiment of this application; Figure 3 A schematic diagram of an agent model training apparatus for an electricity market according to an embodiment of this application is shown. Figure 4 The illustration shows a schematic diagram of an electronic device for training an agent model for an electricity market according to an embodiment of this application. Detailed Implementation

[0017] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0019] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0020] In the embodiments of this application, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0021] In the embodiments of this application, the user's authorization or consent was obtained before obtaining or collecting the user's personal information.

[0022] With the deepening of power market reform, virtual power plants, as an important form of distributed energy aggregation, have been widely applied in the research of bidding and trading in the power market. In recent years, Multi-Agent Reinforcement Learning (MARL) has gradually become an important means to solve multi-player game problems in the power market. Because this method can simulate the independent decision-making processes of multiple market participants (such as thermal power units, wind power, photovoltaics, energy storage, and power users) and continuously learn the optimal strategy in a game environment, it has received widespread attention in power market bidding modeling. The earliest power market bidding method constructed a two-layer bidding model, with the upper layer being the operating entity layer and the lower layer being the power trading center layer. The two-layer bidding model was transformed into a single-layer model using KKT conditions. Then, linearization methods such as duality theory and the Big M method were used to transform the single-layer nonlinear programming model into a single-layer mixed linear integer programming model. Finally, CPLEX or GUROBI commercial solvers were used to solve the problem. This two-layer model incorporates conditional value at risk (VaR) and robust optimization methods to enhance its resilience. It also considers solving a two-layer model for combined markets such as the electricity-carbon-green energy market, ancillary services market, and day-ahead real-time market, analyzing the bidding performance of each entity in each market. However, this two-layer model solution requires that the bidding proposals of all entities except the research entity be known, and linearizing the two-layer model incurs nonlinear losses. Each formula needs linearization, increasing the workload of preliminary work. These drawbacks make this method lack universality and unsuitable for practical application.

[0023] Reinforcement learning is a machine learning method that uses an agent to learn the optimal strategy through trial and error by interacting with the environment. At each step, the agent takes an action, and the environment provides rewards and new states. The agent adjusts its decisions based on cumulative rewards, gradually learning long-term goals. Unlike supervised learning, reinforcement learning does not require labeled samples but relies on reward signals to drive learning. It is commonly used in self-learning, game theory, and market optimization scenarios. The process of an agent learning through interaction with the environment closely matches market bidding. Multiple agents correspond to multiple market participants, the reinforcement learning environment corresponds to market clearing, and actions, rewards, and states correspond to bidding volume, revenue, and clearing results, respectively. Therefore, reinforcement learning has gradually begun to be applied in electricity market bidding. In the early stages of reinforcement learning development, market bidding using reinforcement learning was introduced as single-agent reinforcement learning. The actions and states of each bidding entity were different dimensions of that agent, essentially representing a non-competitive market. With the development of artificial intelligence, multi-agent reinforcement learning has developed rapidly and begun to be applied in electricity market bidding. Each bidding entity is abstracted as an agent, and each agent possesses independent decision-making and perception capabilities. The initial Multi Agent Deep Q-Network (MADQN) algorithm used discrete actions and continuous states. Discrete actions are not conducive to convergence in multi-agent learning. Even Multi Agent Deep Q-Network, which addresses the overestimation problem of DQN, still suffers from poor convergence. One issue is the discretized action space; another is the simultaneous interaction of multiple agents with the environment. For a single agent, the environment is unstable; even if an agent takes the same actions multiple times, the rewards will be inconsistent. The later Multi Agent Deep Deterministic Policy Gradient (MADDPG) algorithm solved the problem of unstable environments in multi-agent reinforcement learning. Applied to market bidding, it achieved better convergence and bidding results. Its improved version, Multi Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3), solved the overestimation problem and accelerated convergence. However, the convergence results of the above linearization solution method and multi-agent reinforcement learning solution method do not take into account the consistency with the real market electricity price. Their convergence results often differ significantly from the real market electricity price.

[0024] Figure 1 The flowchart illustrates a method for training an agent model for an electricity market according to an embodiment of this application.

[0025] According to embodiments of the present invention, such as Figure 1As shown, a training method for an agent model for the electricity market is presented. The agent model is a reinforcement learning model built based on an actor-critic network framework. The method includes: Step S101: Input the actions output by the intelligent agent model into the electricity market environment model to obtain the market environment state; The actions involved are submitting the electricity price and the declared power capacity, while the market environment status refers to the electricity price and the winning bid power at the power market clearing point. For example, the action is represented as , It is expressed as follows:

[0026] in, Indicates the declared electricity price. This indicates the winning bid power.

[0027] Market environment status is represented as , It is expressed as follows:

[0028] in, and These represent the winning bid power for wind power and solar power, respectively. This indicates the winning bid capacity of the energy storage; Indicates the state of charge of the energy storage; Indicates the controllable load level. Indicates the first t Time period b The node electricity price.

[0029] Step S102: Train the actor network of the reinforcement learning model using the first loss function; The value of the first loss function is determined based on the values ​​of the main loss term and the regularization term. The main loss term represents the negative value of the Q value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price. The Q value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment and the action.

[0030] This invention discloses a method for training an agent model for the electricity market. Utilizing an electricity market environment model, the method continuously updates the market environment state. Based on the market environment state and the actions output by the agent model, a commentator network within a reinforcement learning model is used to determine the Q-value corresponding to each action. Furthermore, a main loss term represents the negative value of the Q-value corresponding to the action, and a regularization term represents the difference between the actual market electricity price and the declared electricity price. Since the value of the first loss function is determined based on the values ​​of the main loss term and the regularization term, training a reinforcement learning model based on an actor-commentator network framework using the first loss function can simultaneously achieve high Q-values ​​for the agent model's output actions and consistency between the actions and the actual market electricity price. This effectively reduces problems such as declared electricity price deviation and policy distortion caused by the agent model's output actions, achieving higher-quality training of the reinforcement learning model.

[0031] According to embodiments of the present invention, such as Figure 2 As shown, the reinforcement learning model includes at least two critic networks, which consist of an online network and a target network; the action-corresponding Q-value is the minimum Q-value generated by all online networks.

[0032] For example, such as Figure 2 As shown, the reinforcement learning model can be the MATD3 model.

[0033] According to an embodiment of the present invention, a commentator network is trained using a second loss function; wherein the function value of the second loss function is determined based on the Q-value generated by the online network and the target value generated by the target network; the target value is determined based on the minimum Q-value generated by all target networks and the reward value of the corresponding action; the reward value is determined based on the node electricity price, the winning bid power, and the corresponding operating cost obtained.

[0034] For example, the reward value is represented as , It is expressed as follows:

[0035] in, Indicates the first t Time period b The nodal electricity price at a node; Indicates the winning bid power; This indicates the operating cost corresponding to the awarded power.

[0036] For example, the target value is represented by y, and the formula for calculating the target value y is as follows:

[0037] in, r It's a reward value. γIt is a discount factor; This represents the minimum Q-value generated by the two target networks.

[0038] According to an embodiment of the present invention, the first loss function is composed of a linear summation of a main loss term and a regularization term. The value of the regularization term is determined based on the difference between the actual market electricity price and the declared electricity price, as well as the regularization term coefficient. The regularization term coefficient can be adaptively adjusted according to the changes in the value of the main loss term.

[0039] For example, the first loss function is expressed as , It is represented as follows;

[0040] First loss function middle: Indicates the main loss term; Represents a regular term; It is the coefficient of the regularization term. Represents the true market electricity price. Indicates the declared electricity price; According to an embodiment of the present invention, the regularization coefficient can be adaptively adjusted according to the numerical change of the main loss term, including: the adaptive adjustment of the numerical change direction of the regularization coefficient is negatively correlated with the numerical change direction of the main loss term.

[0041] Regularization coefficient The self-regulation method is represented as follows:

[0042] in, These are gradient coefficients. yes The maximum value; For example, in the early stages of training, the value of the main loss term typically fluctuates significantly, indicating that the agent is still in the policy exploration and reward search phase. At this time, imposing overly strong regularization constraints may inhibit the model's necessary exploration of high-reward regions, hindering the formation of an effective bidding strategy. Therefore, the regularization coefficient should be adjusted in the early stages of training. The first value is taken; however, as training progresses, the value of the main loss term gradually decreases and stabilizes in the later stages, indicating that the agent's strategy has begun to shift from disordered exploration to convergence optimization. At this point, enhancing the guiding role of real market electricity prices in strategy updates helps to further reduce price deviations and improve the consistency between the final strategy and real market patterns. Therefore, the regularization coefficient is increased in the later stages of training. The second value is taken, and the second value is greater than the first value. Therefore, by using adaptive coefficients instead of fixed parameters, the relative importance of the main loss term and the regularization term in different training stages can be matched more accurately, thereby enhancing the robustness and applicability of the regularization mechanism.

[0043] According to an embodiment of the present invention, the electricity market environment model includes: an objective function and constraints; wherein, the objective function represents minimizing the total electricity purchase cost, and the constraints are used to constrain the safe dispatch constraints satisfied by the objective function; The actions output by the intelligent agent model are input into the electricity market environment model to obtain the market environment state, including: Based on the actions and constraints, the objective function is solved using a mathematical programming solver to obtain the market environment state.

[0044] For example, the objective function can be solved using CPLEX.

[0045] According to an embodiment of the present invention, the reward value is determined based on the node electricity price, the winning bid power, and the corresponding operating costs, including: The difference between the declared total price and the operating cost is determined as the reward value; whereby the declared total price is the product of the node electricity price and the winning bid power.

[0046] For example, the internal components of a virtual power plant can include various distributed energy sources (such as wind power, photovoltaics, energy storage, and controllable loads).

[0047] For example, the objective function is expressed as follows:

[0048] in, It is the first i The declared power output of each thermal power unit It is the first i Quotation for a thermal power unit; It is the first j The declared power of each wind turbine unit It is the first j Price quote for individual wind turbine units; It is the first k Each photovoltaic power output declared, It is the first k A photovoltaic quote; It is the first m Each energy storage application power, It is the first m Individual energy storage quotes; It is the first u Each controllable load reporting power It is the first u A controllable load quote; T It refers to the total bidding period within a day;I This represents the total number of thermal power units; J This refers to the total number of wind turbine units; K This refers to the total number of photovoltaic (PV) displays; M It represents the total energy storage capacity; U It is the total number of controllable loads.

[0049] For example, the constraints include the following: The global power balance constraint is as follows: in, B This represents the total number of nodes. It is the first b Nodes t Load power at any given time.

[0050] The power constraint for winning bids for thermal power units is as follows:

[0051] in, This indicates the winning bid power of the thermal power unit.

[0052] The ramp-up constraint for thermal power units is as follows:

[0053] in, This refers to thermal power units breaking the lower limit of their ramp-climbing power. It is the upper limit of climbing power.

[0054] Wind power constraints are as follows:

[0055] in, Indicates the first j The winning bid power for each wind power project.

[0056] The photovoltaic power constraint is as follows:

[0057] in, Indicates the first k The winning bid power for each photovoltaic unit.

[0058] The energy storage power constraint is as follows:

[0059] in, Indicates the first m The winning bid power for energy storage.

[0060] The constraints of the energy storage SOC are as follows:

[0061] in, Indicates the first m The charging power of the energy storage unit; Indicates charging power; Indicates the first m The discharge power of the energy storage unit; Indicates discharge power; Indicates the first m The maximum capacity of each energy storage unit. and This indicates the minimum and maximum states of charge of the stored energy.

[0062] soc The constraint that ensures consistent initial and final power is as follows:

[0063] The node power balance constraint is as follows:

[0064] in, Representing nodes respectively i and nodes j The side road; K Represents the set of branch paths; Representative node Active power generation at time t At the node The active load power at time t; It is a node i To the node j exist t Net injection power at any given time.

[0065] The line power constraint is as follows:

[0066] in, It is a node i To the node j exist t Net injection power at any given time It is a node i phase angle, It is a node j phase angle, It is a node i and nodes j Branch reactor, It is a node i and nodes j The maximum transmission power of the branch between them.

[0067] Figure 3The diagram illustrates a structural block diagram of an agent model training device for an electricity market according to an embodiment of this application.

[0068] like Figure 3 As shown, the intelligent agent model training device for the electricity market includes a market environment state generation module 301 and a training module 302. The market environment state generation module 301 is used to generate the market environment state by inputting the actions output by the agent model into the power market environment model. The agent model is a reinforcement learning model built on the actor-critic network framework. The actions involved are submitting the electricity price and the declared power capacity, while the market environment status refers to the electricity price and the winning bid power at the power market clearing point. Training module 302 is used to train the actor network of the reinforcement learning model using the first loss function; The value of the first loss function is determined based on the values ​​of the main loss term and the regularization term. The main loss term represents the negative value of the Q value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price. The Q value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment and the action.

[0069] Figure 4 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of this application, is illustrated schematically. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0070] like Figure 4 As shown, an electronic device 400 according to an embodiment of this application includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0071] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 402 and / or RAM 403. It should be noted that programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0072] According to embodiments of this application, the electronic device 400 may further include an input / output (I / O) interface 404, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the input / output (I / O) interface 404: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output (I / O) interface 404 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0073] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0074] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0075] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0076] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 402 and / or RAM 403 described above and / or one or more memories other than ROM 402 and RAM 403.

[0077] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the computer system to implement the methods provided in the embodiments of this application.

[0078] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0079] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0080] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this application embodiment. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0081] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0083] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0084] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for training an agent model for the electricity market, characterized in that, The agent model is a reinforcement learning model built on an actor-critic network framework, and the method includes: The actions output by the intelligent agent model are input into the electricity market environment model to obtain the market environment state; wherein, the actions are the bid electricity price and bid power, and the market environment state is the node electricity price and winning bid power at the electricity market clearing point; The actor network of the reinforcement learning model is trained using a first loss function; wherein the function value of the first loss function is determined based on the value of the main loss term and the value of the regularization term; the main loss term represents the negative value of the Q-value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price; the Q-value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment state and the action.

2. The method according to claim 1, characterized in that: The reinforcement learning model includes at least two critic networks, which include an online network and a target network; the action-corresponding Q-value is the minimum Q-value generated by all the online networks.

3. The method according to claim 2, characterized in that, The method further includes: The critic network is trained using a second loss function; wherein the value of the second loss function is determined based on the Q-value generated by the online network and the target value generated by the target network; the target value is determined based on the minimum Q-value generated by all the target networks and the corresponding reward value of the action; the reward value is determined based on the node electricity price, the winning power, and the corresponding operating cost obtained.

4. The method according to claim 1, characterized in that, The first loss function is composed of the linear summation of the main loss term and the regularization term. The value of the regularization term is determined based on the difference between the actual market electricity price and the declared electricity price, as well as the regularization term coefficient. The regularization term coefficient can be adaptively adjusted according to the changes in the value of the main loss term.

5. The method according to claim 4, characterized in that, The regularization coefficient can be adaptively adjusted according to the numerical change of the main loss term, including: the adaptive adjustment of the numerical change direction of the regularization coefficient is negatively correlated with the numerical change direction of the main loss term.

6. The method according to claim 1, characterized in that, The electricity market environment model includes: an objective function and constraints; wherein, the objective function represents minimizing the total electricity purchase cost, and the constraints are used to constrain the safe dispatch constraints satisfied by the objective function; The step of inputting the actions output by the intelligent agent model into the electricity market environment model to obtain the market environment state includes: Based on the actions and constraints, the objective function is solved using a mathematical programming solver to obtain the market environment state.

7. The method according to claim 1, characterized in that, The step of determining the reward value based on the node electricity price, the winning bid power, and the corresponding operating costs includes: The difference between the declared total price and the operating cost is determined as the reward value; wherein the declared total price is the product of the node electricity price and the winning bid power.

8. A training device for an intelligent agent model in an electricity market, comprising: A market environment state generation module is used to input the actions output by the agent model into the electricity market environment model to obtain the market environment state. The actions are the bid price and bid power, and the market environment state is the node price and winning bid power at the electricity market clearing stage. The training module is used to train the actor network of the reinforcement learning model using a first loss function; wherein the function value of the first loss function is determined based on the value of the main loss term and the value of the regularization term; the main loss term represents the negative value of the Q-value corresponding to the action, and the regularization term represents the difference between the actual market electricity price and the declared electricity price; the Q-value corresponding to the action is determined by the commentator network in the reinforcement learning model based on the market environment state and the action.

9. An electronic device, comprising: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 7.

Images