Power demand immediate response reinforcement learning control method, system, device and medium

Abstract

The application discloses a power demand instant response reinforcement learning control method, system, device and medium, which are corresponding solutions, in which: in order to make the reinforcement learning intelligent agent consider both the operation cost and the response accuracy in real-time control, on the basis of a reinforcement learning algorithm (for example, a Q-value-based off-policy reinforcement learning algorithm), an expert model for predicting the power of a balancing node is additionally trained, the power of the balancing node is predicted according to the action output by a policy network through the expert model, an instant response deviation auxiliary loss is constructed according to the predicted power, and then the instant response deviation is explicitly optimized when the policy network is updated, so that the optimization direction of the response error is more finely guided. Through the above solution, the stability and interpretability of the intelligent agent training can be improved, and then the instant response accuracy is improved while the economy is ensured.

Description

Technical Field

[0001] This invention relates to the fields of power artificial intelligence and power system automation technology, and in particular to a reinforcement learning control method, system, device and medium for real-time response to power demand. Background Technology

[0002] In recent years, deep reinforcement learning (DRL) has been widely applied to economic operation of distribution networks and optimization of multi-energy systems due to its end-to-end policy learning, strong function approximation capabilities, and adaptability to complex constraints. By modeling the economic dispatch problem of distribution networks as a Markov decision process (MDP) and using power flow computation as environmental dynamics, agents can directly learn policies from the interaction data of "state-action-reward-next state," and quickly make control decisions based on real-time states during the deployment phase, thereby significantly reducing real-time optimization overhead.

[0003] Existing power system optimization research based on DRL (Dynamic Power Response) primarily focuses on operating costs as the optimization objective. Typical reward designs are usually derived from a linear combination of economic cost items such as fuel costs for thermal power units, electricity purchase and sale costs, renewable energy curtailment costs, peak shaving costs for virtual power plants, and grid loss costs. While these methods can achieve high control performance in purely economic operation scenarios, in practical engineering, increasingly more upstream and downstream power grids are signing "instant response" agreements. These agreements connect the downstream and upstream grids through a balancing node. When the upstream grid issues refined power demand over several consecutive time steps, the downstream grid must not only reduce overall operating costs but also ensure that the actual power at the balancing node tracks this demand as accurately as possible at each moment. If instant response accuracy is characterized solely by adding an "instant response deviation penalty" to the economic cost, the following problems arise: the economic operating cost and response deviation are mixed into a single scalar reward, making it difficult for reinforcement learning algorithms to distinguish the contribution directions of "reducing operating costs" and "improving response accuracy" in the gradient space when updating policies. This leads to fluctuating learning directions and unstable training.

[0004] Therefore, it is necessary to propose a reinforcement learning control method that can enhance the gradient signal of "instant response" while ensuring low economic operating costs, so as to achieve comprehensive optimization of distribution network operating costs and response indicators. Summary of the Invention

[0005] The purpose of this invention is to provide a reinforcement learning control method, system, device, and medium for real-time response to power demand, which can improve the accuracy of real-time response while ensuring low-cost and economical operation.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A reinforcement learning control method for real-time response to power demand includes:

[0008] Construct an intelligent agent that includes a policy network and an evaluation network;

[0009] The distribution network operation process under the immediate response scenario is modeled as a Markov decision process, and samples during the interaction with the agent are collected and stored in the replay buffer. Each sample includes a state, an action, and a reward. The state is the current state of the distribution network under the immediate response scenario, the action is the control action for the distribution network decided by the agent, and the reward is determined by the economic operating cost of the distribution network after the action is executed and the immediate response deviation penalty. The immediate response deviation penalty is determined based on the power obtained by power flow calculation based on the state and action and the power demand issued by the upper-level grid.

[0010] The agent is trained by sampling a batch of samples from the replay buffer: In the first stage, the evaluation network is trained using the samples through temporal difference loss, and the expert model is trained through supervised learning loss; In the second stage, the policy network is trained, the action output by the policy network is evaluated using the evaluation network in the first stage, the policy loss is constructed, the prediction power of the action output by the policy network is calculated using the expert model in the first stage, the instant response bias auxiliary loss is constructed, and the policy network is trained by combining the policy loss and the instant response bias auxiliary loss; The above two stages are executed alternately.

[0011] The trained agent is deployed in a distribution network under a real-time response scenario, and the action decision is made by a policy network.

[0012] A power demand real-time response reinforcement learning control system for implementing the aforementioned method includes:

[0013] The agent building unit is used to build agents that include policy networks and evaluation networks;

[0014] The Markov Decision Process Modeling and Sample Collection Unit is used to model the distribution network operation process under the immediate response scenario as a Markov Decision Process and collect samples during the interaction with the agent and store them in the replay buffer. Each sample includes a state, an action, and a reward. The state is the current state of the distribution network under the immediate response scenario, the action is the control action for the distribution network decided by the agent, and the reward is determined by the economic operating cost of the distribution network after the action is executed and the immediate response deviation penalty. The immediate response deviation penalty is determined based on the power obtained by power flow calculation from the state and action and the power demand issued by the upper-level grid.

[0015] The agent and expert model training unit is used to sample a batch of samples from the replay buffer to train the agent: In the first stage, the sampled samples are used to train the evaluation network using temporal difference loss and the expert model using supervised learning loss; In the second stage, the policy network is trained by evaluating the actions output by the policy network using the evaluation network from the first stage to construct the policy loss, and simultaneously calculating the prediction power of the actions output by the policy network using the expert model from the first stage to construct the immediate response bias auxiliary loss; Finally, the policy network is trained by combining the policy loss and the immediate response bias auxiliary loss; The above two stages are executed alternately.

[0016] The control unit is used to deploy trained agents in the distribution network under real-time response scenarios and make action decisions through the policy network therein.

[0017] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0020] As can be seen from the technical solution provided by the present invention, the present invention additionally trains an expert model to predict the power of the balancing node based on the reinforcement learning algorithm (e.g., a Q-value-based off-policy reinforcement learning algorithm). The expert model predicts the power of the balancing node based on the actions output by the policy network, and thereby constructs an immediate response bias auxiliary loss. This immediately responds bias is explicitly optimized when the policy network is updated, thus providing more precise guidance on the optimization direction to reduce response error. Through this approach, the stability and interpretability of agent training can be improved, thereby enhancing immediate response accuracy while ensuring cost-effectiveness. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic diagram of a power demand real-time response reinforcement learning control method provided in an embodiment of the present invention.

[0023] Figure 2This is a schematic diagram of a training framework for predicting auxiliary tasks using expert models, provided in an embodiment of the present invention.

[0024] Figure 3 A complete training flowchart is provided for embodiments of the present invention.

[0025] Figure 4 This is a schematic diagram illustrating an example of a demand curve provided in an embodiment of the present invention.

[0026] Figure 5 The graph shows the average step-by-step requirement response error comparison across 20 tasks, as provided in this embodiment of the invention.

[0027] Figure 6 This is a visual diagram of the response curve provided in an embodiment of the present invention.

[0028] Figure 7 This is a schematic diagram of a power demand real-time response reinforcement learning control system provided in an embodiment of the present invention.

[0029] Figure 8 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0031] First, the following explanations are provided for the terms that may be used in this article:

[0032] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0033] The following is a detailed description of a power demand real-time response reinforcement learning control method, system, device, and medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, conventional conditions in the art or conditions recommended by the manufacturer shall apply. Where the manufacturers of the instruments used in the embodiments of the present invention are not specified, they are all conventional products that can be purchased commercially.

[0034] Example 1

[0035] This invention provides a reinforcement learning control method for real-time response to power demand, such as... Figure 1 As shown, it mainly includes:

[0036] Step 1: Construct an agent that includes a policy network and an evaluation network.

[0037] In this embodiment of the invention, the reinforcement learning algorithm used by the agent can be a policy actor-critic algorithm based on the Q-value function, where the actor is the policy network and the critic is the evaluation network.

[0038] Step 2: MDP modeling and sample collection.

[0039] In this embodiment of the invention, the distribution network operation process under the immediate response scenario is modeled as a Markov decision process. Simultaneously, an agent is designed to interact with the environment, and samples from the interaction process are stored in a replay buffer. Data is then sampled from the replay buffer to learn and optimize strategies. Each sample includes a state, an action, and a reward. The state is the current state of the distribution network under the immediate response scenario; the action is the control action for the distribution network decided by the agent; and the reward is determined by the economic operating cost of the distribution network after the action is executed and the immediate response deviation penalty. The immediate response deviation penalty is determined based on the power of the balancing node obtained through power flow calculation using the state and action, and the power demand issued by the upper-level grid.

[0040] In this embodiment of the invention, the state of the distribution network under the real-time response scenario includes: (1) active and reactive power of the load, active power of photovoltaic and wind power, active and reactive power of thermal power units, active power regulation of virtual power plants, status of energy storage devices, and power of balancing nodes; (2) topology coding: used to distinguish different distribution network topologies; (3) electricity price information: current and future multi-step electricity price information; (4) power demand information: current and future multi-step power demand.

[0041] In this embodiment of the invention, the control actions for the power distribution network include: (1) energy storage control: the charging and discharging power of the energy storage device; (2) generator control: the power generation power of the thermal power unit; (3) renewable energy output regulation: the power of renewable energy curtailment; (4) virtual power plant control: the active power regulation of the virtual power plant.

[0042] In this embodiment of the invention, the calculation method for the instantaneous response deviation penalty is expressed as follows:

[0043] ;

[0044] in, This is the penalty coefficient; In the state and actions The power of the slack node is obtained through power flow calculation. The power demand issued by the upper-level power grid; To provide an immediate response to deviation penalties, t represents time. It is the absolute value symbol.

[0045] In this embodiment of the invention, the reward is calculated as follows: the cost of purchasing and selling electricity at time t is defined as... , remove Other economic operating costs are ; combination , and immediate response deviation penalty Calculate the total cost Among them, the total cost within the set demand period is yes and immediate response deviation penalty The sum; the total cost during the designated non-demand periods. yes Electricity purchase and sales costs The sum; take the total cost The opposite number as a reward ,Right now: .

[0046] Step 3: Train the intelligent agent.

[0047] In this embodiment of the invention, in addition to the policy network and evaluation network of the basic algorithm, an expert model is introduced to assist in the training of the policy network. A batch of samples is sampled from the replay buffer to train the agent: In the first stage, the sampled samples are used to train the evaluation network using temporal difference loss, and the expert model is trained using supervised learning loss; In the second stage, the policy network is trained, and the actions output by the policy network are evaluated using the evaluation network from the first stage to construct the policy loss. At the same time, the predicted power of the actions output by the policy network is calculated using the expert model from the first stage to construct the immediate response bias auxiliary loss; Finally, the policy network is trained by combining the policy loss and the immediate response bias auxiliary loss; The above two stages are executed alternately.

[0048] The power of the equilibrium node predicted by the expert model is denoted as... ;in, For state, For the action, t represents time. For expert models, These are the parameters of the expert model.

[0049] The training loss of the expert model is constructed by combining the power of the balancing nodes in the samples:

[0050] ;

[0051] in, The training loss of the expert model, For mathematical expectation, For replay buffer; For state With action The power (power at the slack node) is obtained by power flow calculation.

[0052] In this embodiment of the invention, the expert model can adopt a Transformer architecture based on graph attention mechanism or a graph neural network to better capture the topology and power flow characteristics of the distribution network; considering that the method of training the expert model based on the above training loss can be implemented with reference to conventional technology, it will not be described in detail.

[0053] Based on the above introduction, the training process of the policy network can be described as follows:

[0054] (1) For the sampled time t, the state of the sampled time is... Input is fed into the policy network, and the policy network outputs actions. .

[0055] (2) Evaluate the network's response to actions Conduct an assessment and construct a strategy loss. , The policy represented by the output of the policy network. The parameters represent the policy network.

[0056] (3) The expert model is based on the state With action Calculate and predict the power of the slack node And in combination with power requirements Construct an immediate response bias auxiliary loss:

[0057] ;

[0058] in, To provide immediate response to deviations and auxiliary losses, For mathematical expectation, This is a replay buffer.

[0059] (4) Construct the total loss by combining the strategy loss and the immediate response bias auxiliary loss. :

[0060] ;

[0061] in, These are the weighting coefficients.

[0062] (5) Train the policy network using the total loss. This process can be implemented with reference to conventional techniques, so it will not be described in detail.

[0063] In this embodiment of the invention, the two phases of the training process are executed alternately until a set stopping condition is reached (e.g., the set maximum number of training steps is reached).

[0064] Step 4: Deploy and apply the intelligent agent.

[0065] The trained agent is deployed in a distribution network under an instant response scenario, and action decisions are made through the policy network therein. That is, the policy network makes corresponding decisions based on the real-time state.

[0066] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0067] I. Overall Overview of the Plan

[0068] This invention provides a reinforcement learning control method for real-time response to power demand. This method improves the accuracy of the "real-time response" between upstream and downstream power grids while meeting the economic constraints of power system operation. It can be applied to collaborative optimization decision-making and real-time operation control of power generation, grid, load, and energy storage systems. Here, "power generation," "grid," "load," and "energy storage" refer to the generation side, grid topology, load, and energy storage system, respectively.

[0069] The above-mentioned solution provided by the embodiments of the present invention proposes to construct a power expert model of the balancing node based on power flow data supervision training on the basis of the Q-value-based off-policy reinforcement learning algorithm. Since the power of the balancing node as the supervision signal comes from a stable physical mapping, the expert model has a fast convergence speed and high prediction accuracy. At the same time, under the reinforcement learning algorithm framework, a prediction auxiliary task based on the expert model is introduced. By constructing the deviation between the power prediction of the balancing node and the demand of the upper-level power grid as an auxiliary loss term, the instantaneous response deviation is explicitly optimized during policy network update. This provides a high-quality, interpretable instantaneous response error evaluation signal for the policy network update process, thereby providing more refined guidance on the optimization direction of reducing response error and improving instantaneous response accuracy while ensuring economy.

[0070] II. Detailed introduction of the plan.

[0071] The following sections will introduce MDP modeling, expert models, policy network training schemes, and the complete training process.

[0072] 1. MDP modeling.

[0073] (1) Economic operation model and immediate response target of distribution network.

[0074] The lower-level power grid environment is constructed based on the distribution network, taking into account factors such as thermal power units, electricity purchase and sale prices, renewable energy sources (wind power, photovoltaic), energy storage devices, virtual power plants, and balance nodes connected to the upper-level power grid. Denotes the discrete time step, denoted as For time t, excluding the purchase and sale of electricity Other economic operating costs, including thermal power unit costs Cost of renewable energy curtailment Peak shaving costs of virtual power plants and network loss costs Etc., that is:

[0075] .

[0076] Based on this, the upper-level power grid is introduced during the demand response period. Internally issued power demand curve ,use Indicates the state and actions Below, the power at the slack node obtained through power flow calculation is used to define the instantaneous response deviation penalty as:

[0077] ;

[0078] in, This represents the penalty coefficient for demand response deviation. During the demand period, the environment does not consider electricity purchase and sale costs; the total cost is... yes and demand response deviation penalty The sum; during non-demand periods, there is no environmental response bias penalty, and the total cost is... yes Electricity purchase and sales costs The sum is:

[0079] .

[0080] For example, a day can be divided into 96 time steps, i.e., t=0,...,95; the required time period can be set as multiple consecutive time periods within 41-64, and the rest can be non-required time periods.

[0081] Therefore, for a given strategy Overall optimization goal It can be written as:

[0082] .

[0083] In a reinforcement learning environment, the reward for each step is the inverse of the total cost mentioned above, i.e.:

[0084] .

[0085] (2) State space modeling.

[0086] This invention divides the agent's observation state vector into the following four parts:

[0087] Power grid operation status characteristics: active and reactive power of load, active power of photovoltaic and wind power, active and reactive power of thermal power units, active power regulation of virtual power plants, status of energy storage devices, and power of balance nodes.

[0088] Topology coding: used to fully utilize topology information and distinguish different distribution network topologies;

[0089] Electricity price information: Current electricity prices and prices at multiple future time points;

[0090] Power demand information: Power demand at present and at multiple future time steps.

[0091] (3) Action space modeling.

[0092] The action vector is in continuous form and is composed of the following concatenation of sub-vectors:

[0093] Energy storage control: such as the charging and discharging power of multiple energy storage devices;

[0094] Conventional generator control: output regulation of thermal power units;

[0095] Renewable energy output regulation: regulation of the curtailment of renewable energy sources (solar power, wind power, etc.);

[0096] Virtual power plant control: Active power regulation of virtual power plants.

[0097] At each step, the environment constructs power flow equations by combining the current state with the actions output by the agent, solving for quantities such as node voltage, branch power, and slack node power, and outputting the state at the next moment. and rewards Therefore, the real-time response problem of the distribution network is uniformly modeled as a Markov decision process in a continuous action space.

[0098] 2. Expert model.

[0099] For reinforcement learning algorithms, it is only necessary to incorporate the immediate rewards of the reinforcement learning environment. Taking the negative of the total cost mentioned above allows for direct policy optimization around the combined objective of "operating cost + immediate response accuracy". However, this purely scalar reward-based approach makes it difficult for the gradient signal acquired by the reinforcement learning policy network to distinguish the contributions of economic operating cost and immediate response penalty, resulting in an ambiguous and unstable training objective. Therefore, this invention further introduces a balanced node power expert model and a prediction auxiliary task to explicitly guide the policy.

[0100] (1) Definition of expert model.

[0101] Each step of the environment process involves solving the power flow equations based on the state. With action The power of the slack node was calculated. This quantity strictly satisfies the power flow equations, has a clear physical meaning, and is numerically accurate. This invention trains a power expert model for a slack node using supervised learning. , This represents the predicted power of the balancing node. For expert models The parameters. The training process minimizes the predicted values. Actual power of the balancing node in power flow calculation Mean square error between:

[0102] .

[0103] because Completely determined by the power flow equations, the expert model is essentially fitting a stable physical mapping, thus it has a fast convergence speed and high prediction accuracy.

[0104] (2) Data acquisition and network structure.

[0105] The data required to train the expert model comes from samples accumulated during the interaction between reinforcement learning and the environment. Each sample contains states. ,action ,award and the power of the slack node obtained from power flow calculation Because this physical mapping is relatively stable and has low noise, the expert model can use a relatively simple fully connected neural network to directly regress the power of the balancing nodes after concatenating the state and action. Alternatively, a Transformer architecture based on graph attention or a graph neural network can be introduced as needed to enhance the ability to represent topological information.

[0106] 3. Strategy Network Training Scheme.

[0107] When training using only the comprehensive reward, the gradient signals obtained by the policy network during updates are mixed. To clarify the gradient components corresponding to the immediate response objective, this invention introduces a prediction auxiliary task based on an expert model during the policy network update process, as follows:

[0108] (1) Sample a batch of states from the replay buffer.

[0109] (2) State Input the policy network and sample the corresponding actions. :

[0110] ;

[0111] in, Indicates the state The lower policy network outputs a probability distribution of actions, with the symbol... To show obedience, Indicates action Follows a probability distribution, that is, actions From probability distribution Randomly selected from the list.

[0112] (3) and Input the expert model to obtain the predicted power of the equilibrium node. ,in, It is an expert model.

[0113] (4) Compare the predicted power with the power demand of the upstream power grid at time t. By comparison, construct an immediate response bias auxiliary loss:

[0114] .

[0115] (5) Evaluate the network's response to actions Conduct an assessment and construct a strategy loss. ;in, To evaluate the network based on and The specific calculation method for the calculated Q value can be referred to conventional techniques, and will not be elaborated upon in this invention.

[0116] (6) Combining the above two losses, the total loss of the policy network is:

[0117] ;

[0118] in, These are weighting coefficients used to balance the relative importance of policy loss and immediate response bias auxiliary loss. Throughout training: the evaluation network is updated only using the reward returned by the environment, which includes both economic cost and immediate response bias, ensuring that the value function comprehensively considers both economy and response accuracy; the policy network, based on the value estimate provided by the evaluation network, additionally receives the response error signal from the expert model, explicitly guiding the policy to reduce the deviation between the power of the balancing node and the demand value, such as... Figure 2 As shown, the training framework for predicting auxiliary tasks using expert models is illustrated. Solid arrows represent forward propagation, dashed arrows represent backward propagation, and ⊕ represents gradient addition.

[0119] 4. Complete training program.

[0120] like Figure 3 The image shows an example of a complete training process.

[0121] (1) Initialize the parameters of the policy network (Actor), evaluation network (Critic), expert model (Predictor) network, and training hyperparameters in reinforcement learning, and set the training time step k=0.

[0122] (2) Samples (including states, actions, and rewards) are collected using a random exploration method and placed in the replay buffer.

[0123] (3) Determine whether the minimum number of samples required for training has been reached. If so, proceed to step 4; otherwise, return to step 2.

[0124] (4) The policy network interacts with the environment to collect samples (including states, actions, and rewards) and place them in the replay buffer.

[0125] The above steps (2) and (4) belong to the traditional step logic in reinforcement learning, that is, first explore the interaction randomly to ensure diversity, and then use the policy network interaction.

[0126] (5) Sample data from the replay buffer, update the evaluation network, and update the expert model.

[0127] (6) Sample data from the replay buffer and update the policy network by guiding the evaluation network and expert model.

[0128] (7) Let k←k+1, where ← represents assignment.

[0129] (8) Determine whether the training time step k has reached the maximum number of training steps. If yes, the process ends; otherwise, return to step (5).

[0130] As previously mentioned, the reinforcement learning algorithm in this embodiment of the invention can employ a policy actor-commentator algorithm based on the Q-value function, such as: Deep Deterministic Policy Gradient (DDPG), Double Delay Deep Deterministic Policy Gradient (TD3), and Soft Actor-Commentator (SAC). Similar results can be achieved by simply training an expert model to predict the power of the balancing node, and then introducing the expert model to predict the auxiliary loss during its policy network update phase.

[0131] III. Example Explanation.

[0132] To illustrate the effectiveness of the above-mentioned solution of the present invention, the Soft Actor-Commentator (SAC) algorithm will be used as an example.

[0133] 1. Distribution network model under real-time response.

[0134] The IEEE 123 node distribution network (a standard example of a power grid with 123 nodes) is used as the underlying power grid topology. An economic operation model is established, incorporating controllable resources such as thermal power units, wind farms, photovoltaic power plants, energy storage, and virtual power plants. Four operating scenarios (with different proportions of renewable energy output, load levels, and electricity prices) and five topology variants are constructed, resulting in 20 tasks. Task 1 consists of scenario 1 and topology 1, Task 2 consists of scenario 2 and topology 1, and so on. The runtime for each station is set. =96 means that the day is divided into 96 discrete time steps, each corresponding to 15 minutes. Demand response period Time steps 41–64 are set, during which the upper-level power grid issues refined power demands. Each time the environment is reset, one demand curve is randomly selected from 342 training demand patterns to construct diverse demand scenarios. An example of a demand curve is shown below. Figure 4 As shown, "mod" represents the demand pattern, and the number following it indicates the sequence number of the demand pattern. The horizontal axis represents the time period from 41 to 64, and each demand pattern corresponds to a specific demand curve. The status, actions, and rewards are the same as in the example above.

[0135] 2. Training and assessment setup.

[0136] The SAC algorithm using an expert model (the scheme of this invention) is compared with the SAC algorithm without an expert model (the baseline SAC algorithm). The expert model, the policy network in the SAC algorithm, and the evaluation network all employ three-layer fully connected networks. Each algorithm is trained using mixed sampling on 20 tasks. Each experiment consists of 1 million training steps, with evaluation performed every 10,000 steps under all 15 evaluation requirement modes (not seen in the training requirement modes). Five random seeds are used in each experiment to reduce the interference from randomness. The average response error per step and the normalized average economic operating cost of each algorithm on the 20 tasks are statistically analyzed.

[0137] 3. Results Analysis.

[0138] (1) such as Figure 5 As shown, the graph compares the average step-by-step demand response error across 20 tasks, demonstrating the performance comparison between the SAC algorithm using an expert model (the proposed solution, sac+pred) and the SAC algorithm without an expert model (the baseline SAC algorithm, sac). Figure 5 The lower the midline curve, the smaller the response error per step, demonstrating the significant reduction in response error achieved by the method of this invention. Specifically, the average demand response error per step across 20 tasks decreased from 0.036 at the baseline SAC to 0.015 at the method of this invention, a reduction of approximately 58%.

[0139] (2) For example Figure 6 As shown, a visual comparison of the response curves of the SAC algorithm using an expert model (the proposed solution, sac+pred) and the SAC algorithm without an expert model (the baseline SAC algorithm, SAC) is presented. The closer the response curve is to the demand curve, the higher the response accuracy. From Figure 6 As can be seen, the response curve of the present invention almost overlaps with the demand curve, demonstrating the high-precision response of the method of the present invention compared with the baseline algorithm.

[0140] (3) In terms of economic operation, the average running cost per round of the SAC algorithm using the expert model (the proposed solution, sac+pred) is 1.117 after normalization across 20 tasks, while the average running cost per round of the SAC algorithm without the expert model (the baseline SAC algorithm, sac) is 1.123. The average running cost per round of the proposed solution is slightly better than the baseline algorithm, demonstrating that the proposed solution does not sacrifice economic performance and may even have a lower economic operating cost.

[0141] The above results verify the comprehensive advantages of the method of the present invention in terms of both real-time response accuracy and economical operation.

[0142] Based on the above examples, it is evident that the present invention has the following beneficial effects:

[0143] (1) Significantly improved instant response accuracy: The comprehensive evaluation results on 20 tasks show that, compared with the SAC algorithm, the average demand response error per step of the method of the present invention is reduced from about 0.036 to about 0.015, a reduction of about 58%, which can respond to power demand more accurately.

[0144] (2) The economic operating cost is basically unaffected: The comprehensive evaluation results on 20 tasks show that the method of the present invention performs slightly better than the baseline SAC algorithm in terms of operating cost per game, indicating that economic performance is not sacrificed while significantly reducing response error.

[0145] (3) Improved training stability and interpretability: By explicitly using the physical quantity of the balance node power as the output of the expert model and constructing the prediction auxiliary loss, this invention provides a clear gradient direction for "reducing response error" for reinforcement learning actors, making the training process more stable and improving the interpretability of the policy for the immediate response target.

[0146] (4) Enhanced applicability to multiple scenarios and topologies: By conducting mixed training on 20 tasks consisting of 4 scenarios, 5 topologies and 342 demand patterns, the strategy can maintain high real-time response accuracy under complex and ever-changing operating conditions, and has good engineering practical value.

[0147] (5) Simple to implement and easy to integrate into engineering: This invention is an extension of the existing economic operation and power flow simulation environment of distribution network. The expert model training and reinforcement learning algorithm framework are both based on mature deep learning technology. During the training phase, only the expert model supervision training needs to be added and the expert model is used to assist the policy network update, which is easy to integrate into the existing scheduling and control platform.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0149] Example 2

[0150] This invention also provides a power demand real-time response reinforcement learning control system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 7 As shown, the system mainly includes:

[0151] The agent building unit is used to build agents that include policy networks and evaluation networks;

[0152] The Markov Decision Process Modeling and Sample Collection Unit is used to model the distribution network operation process under the immediate response scenario as a Markov Decision Process and collect samples during the interaction with the agent and store them in the replay buffer. Each sample includes a state, an action, and a reward. The state is the current state of the distribution network under the immediate response scenario, the action is the control action for the distribution network decided by the agent, and the reward is determined by the economic operating cost of the distribution network after the action is executed and the immediate response deviation penalty. The immediate response deviation penalty is determined based on the power obtained by power flow calculation from the state and action and the power demand issued by the upper-level grid.

[0153] The agent and expert model training unit is used to sample a batch of samples from the replay buffer to train the agent: In the first stage, the sampled samples are used to train the evaluation network using temporal difference loss and the expert model using supervised learning loss; In the second stage, the policy network is trained by evaluating the actions output by the policy network using the evaluation network from the first stage to construct the policy loss, and simultaneously calculating the prediction power of the actions output by the policy network using the expert model from the first stage to construct the immediate response bias auxiliary loss; Finally, the policy network is trained by combining the policy loss and the immediate response bias auxiliary loss; The above two stages are executed alternately.

[0154] The control unit is used to deploy trained agents in the distribution network under real-time response scenarios and make action decisions through the policy network therein.

[0155] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0156] Example 3

[0157] The present invention also provides a processing device, such as Figure 8 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0158] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0159] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0160] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0161] The output device can be a display terminal;

[0162] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0163] Example 4

[0164] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0165] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0166] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A reinforcement learning control method for real-time response to power demand, characterized in that, include: Construct an intelligent agent that includes a policy network and an evaluation network; The operation process of the distribution network under the real-time response scenario is modeled as a Markov decision process, and samples from the interaction process with the intelligent agent are collected and stored in the replay buffer. Each sample includes a state, an action, and a reward. The state is the state of the distribution network under the current real-time response scenario. The action is the control action for the distribution network decided by the agent. The reward is determined by the economic operating cost of the distribution network after the action is executed and the real-time response deviation penalty. The real-time response deviation penalty is determined based on the power obtained by power flow calculation from the state and action and the power demand issued by the upper-level grid. A batch of samples is sampled from the replay buffer to train the agent: In the first stage, the evaluation network is trained using the samples through temporal difference loss, and the expert model is trained through supervised learning loss. In the second stage, the policy network is trained. The evaluation network from the first stage is used to evaluate the actions output by the policy network, and the policy loss is constructed. The expert model from the first stage is used to calculate the prediction power of the actions output by the policy network, and the immediate response bias auxiliary loss is constructed. The policy loss and the immediate response bias auxiliary loss are combined to train the policy network. The above two stages are executed alternately. The trained agent is deployed in a distribution network under a real-time response scenario, and the action decision is made by a policy network.

2. The power demand real-time response reinforcement learning control method according to claim 1, characterized in that, The state of the power distribution network includes: Power grid operation status characteristics: active and reactive power of load, active power of photovoltaic and wind power, active and reactive power of generating units, active power regulation of virtual power plants, status of energy storage devices, and power of balancing nodes. Topology coding: used to distinguish different distribution network topologies; Electricity price information: Current and future multi-step electricity price information; Power demand information: Current and future multi-step power demand.

3. The power demand real-time response reinforcement learning control method according to claim 1, characterized in that, The control actions for the power distribution network include: Energy storage control: the charging and discharging power of energy storage devices; Generator control: Power generation capacity of thermal power units; Renewable energy output regulation: Renewable energy curtailment; Virtual power plant control: Active power of the virtual power plant.

4. The power demand real-time response reinforcement learning control method according to claim 1, characterized in that, The calculation method for the immediate response deviation penalty is expressed as follows: ； in, This is the penalty coefficient; In the state and actions The power of the slack node is obtained through power flow calculation. The power demand issued by the upper-level power grid; To provide an immediate response to deviation penalties, t represents time. It is the absolute value symbol.

5. A reinforcement learning control method for real-time response to power demand according to claim 1 or 4, characterized in that, The reward is calculated in the following manner: The cost of purchasing and selling electricity at time t is defined as follows: , remove Other economic operating costs are ; Combination , and immediate response deviation penalty Calculate the total cost Among them, the total cost within the set demand period is yes and immediate response deviation penalty The sum; the total cost during the designated non-demand periods. yes Electricity purchase and sales costs sum; Take the total cost The opposite number as a reward ,Right now: .

6. The power demand real-time response reinforcement learning control method according to claim 1, characterized in that, The method of training the expert model using supervised learning loss includes: Each sample also includes: the power calculated by power flow under the given state and action, i.e., the power of the slack node; the power of the slack node predicted by the expert model is defined as... ;in, For state, For the action, t represents time. For expert models, These are the parameters of the expert model; The training loss of the expert model is constructed by combining the power of the balancing nodes in the samples: ； in, The training loss of the expert model, For mathematical expectation, For replay buffer; For state With action The power of the balancing node is obtained from the power flow calculation.

7. A reinforcement learning control method for real-time response to power demand according to claim 1 or 6, characterized in that, The first-stage evaluation network is used to evaluate the actions output by the policy network, constructing a policy loss. The first-stage expert model is used to calculate the predicted power of the actions output by the policy network, constructing an instantaneous response bias auxiliary loss. The policy network is trained by combining the policy loss and the instantaneous response bias auxiliary loss, including: For the sampled time t, the state within it Input is fed into the policy network, and the policy network outputs actions. ; Evaluation network for action Conduct an assessment and construct a strategy loss. , The policy representing the policy network, The parameters representing the policy network; Expert Model According to the status With action Calculate predicted power In conjunction with the power demand issued by the upper-level power grid Construct an immediate response bias auxiliary loss: ； in, To provide immediate response to deviations and auxiliary losses, For mathematical expectation, For replay buffer; The total loss is constructed by combining the policy loss and the immediate response bias auxiliary loss. : ； in, These are the weighting coefficients; The policy network is trained using the total loss.

8. A power demand real-time response reinforcement learning control system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The agent building unit is used to build agents that include policy networks and evaluation networks; The Markov Decision Process Modeling and Sample Collection Unit is used to model the distribution network operation process under the immediate response scenario as a Markov Decision Process and collect samples during the interaction with the agent and store them in the replay buffer. Each sample includes a state, an action, and a reward. The state is the current state of the distribution network under the immediate response scenario, the action is the control action for the distribution network decided by the agent, and the reward is determined by the economic operating cost of the distribution network after the action is executed and the immediate response deviation penalty. The immediate response deviation penalty is determined based on the power obtained by power flow calculation from the state and action and the power demand issued by the upper-level grid. The agent and expert model training unit is used to sample a batch of samples from the replay buffer to train the agent: In the first stage, the sampled samples are used to train the evaluation network using temporal difference loss and the expert model using supervised learning loss; In the second stage, the policy network is trained by evaluating the actions output by the policy network using the evaluation network from the first stage to construct the policy loss, and simultaneously calculating the prediction power of the actions output by the policy network using the expert model from the first stage to construct the immediate response bias auxiliary loss; Finally, the policy network is trained by combining the policy loss and the immediate response bias auxiliary loss; The above two stages are executed alternately. The control unit is used to deploy trained agents in the distribution network under real-time response scenarios and make action decisions through the policy network therein.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images