A training method and apparatus for a power dispatching agent, and a power dispatching method based on a power dispatching agent.

By constructing a scheduling optimization model and rigid constraints, designing a comprehensive reward function, and utilizing a dual experience replay pool and preset weight sampling, the problem of blind exploration in the early stage of agent training was solved, thereby improving the training efficiency and strategy effectiveness of the power scheduling agent.

CN123117246APending Publication Date: 2026-11-17POWER DISPATCHING CONTROL CENT OF GUANGDONG POWER GRID CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611187673.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-06
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing intelligent agents blindly explore in the early stages of training, resulting in low training efficiency and a tendency to generate illegal actions that violate power system safety constraints, making it difficult to optimize the reward function.

Method used

A scheduling optimization model and rigid constraints are constructed, a comprehensive reward function is designed, a dual experience replay pool is established, and a pre-set weighted mixed sampling method is used to screen feasible actions that meet the constraints. The network parameters of the power scheduling agent are updated through iterative training.

Benefits of technology

It improves the training efficiency of the agent, reduces invalid exploration, ensures that the agent follows the power grid constraints during training, reduces the probability of illegal samples, and improves the effectiveness of the scheduling strategy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117246A_ABST
    Figure CN123117246A_ABST
Patent Text Reader

Abstract

This invention discloses a training method and apparatus for a power dispatching agent, as well as a power dispatching method based on the power dispatching agent, relating to the field of power dispatching technology. The training method includes: constructing a dispatching optimization model and rigid constraints with the objective of minimizing the total generation cost within a look-ahead time window; constructing a comprehensive reward function based on the model and constraints, and constructing a power dispatching agent; during the training process, storing all dispatching actions output by the power dispatching agent in an original experience replay pool, and selecting feasible actions that satisfy the rigid constraints and storing them in a feasible action experience replay pool; sampling from the dual experience replay pools according to preset sampling weights to form training batch data, updating the agent network parameters, until the network parameters converge or the maximum number of iterations is reached, thus obtaining a trained power dispatching agent. By implementing this invention, the problem of low training efficiency caused by blind exploration in the early stages of existing agents is solved, thereby improving the training efficiency of the agent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of power dispatching technology, and in particular to a training method and apparatus for a power dispatching intelligent agent, as well as a power dispatching method based on a power dispatching intelligent agent. Background Technology

[0002] As the global energy transition progresses, new power systems based on renewable energy are facing unprecedented technological challenges. The integration of a high proportion of renewable energy (such as wind and solar power) brings significant output uncertainty, while the requirements for power system security (such as the N-1 safety criterion) are becoming increasingly stringent.

[0003] In recent years, researchers have attempted to use deep reinforcement learning algorithms (Deep Deterministic Policy Gradient, or DDPG) to learn optimal scheduling strategies through continuous interaction between agents and the environment in order to cope with the uncertainties brought about by new energy sources.

[0004] However, conventional deep reinforcement learning algorithms are blind in the initial exploration phase of training, which easily generates a large number of invalid actions and illegal actions that violate power system safety constraints (such as line over-limit and slope over-limit), making it difficult to optimize the reward function and resulting in low efficiency in the agent training process. Summary of the Invention

[0005] This invention provides a training method, apparatus, and power dispatching method based on a power dispatching agent, which can solve the problem of low training efficiency caused by blind exploration in the early stage of existing agents and improve the training efficiency of agents.

[0006] An embodiment of the present invention provides a training method for a power dispatching agent, comprising: Obtain power grid operating status parameters; Based on the power grid operating status parameters, a scheduling optimization model is constructed with the objective function of minimizing the total power generation cost of all generating units at all time steps within the forward time window, and rigid constraints are also constructed. Based on the scheduling optimization model and the rigid constraints, a comprehensive reward function is constructed, and an action space and a state space are also constructed. Based on the state space, action space, and comprehensive reward function, a power dispatching intelligent agent is constructed; The power dispatching agent is iteratively trained. During each training process, all dispatching actions output by the power dispatching agent are stored in the original experience replay pool, and feasible actions that satisfy all rigid constraints of the dispatching optimization model are stored in the feasible action experience replay pool. Samples are extracted from the feasible action experience replay pool and the original experience replay pool according to preset sampling weights to form training batch data. The network parameters of the power dispatching agent are updated according to the training batch data until the network parameters converge or the number of training rounds reaches the preset maximum number of iterations, thus obtaining the trained power dispatching agent.

[0007] Furthermore, the rigid constraints include: unit output limit constraints, line power flow limit constraints, unit ramping limit constraints, and real-time power balance constraints.

[0008] Furthermore, based on the scheduling optimization model and the rigid constraints, a comprehensive reward function is constructed, including: The negative value of the total generation cost in the scheduling optimization model is used as the reward benchmark component; Based on the unit output limit constraints, construct a penalty item for exceeding the unit output limit; Based on the line flow restriction constraints, construct a line flow violation penalty item; Based on the unit ramping limit constraints, construct a unit ramping over-limit penalty item; The comprehensive reward function is obtained by superimposing the reward benchmark component with the unit output over-limit penalty, the line power flow over-limit penalty, and the unit ramp over-limit penalty.

[0009] Furthermore, the action space and state space are constructed, including: Obtain the node load forecast data for all time steps within the forward time window and the corresponding action vector for the previous forward time window; The state space is constructed based on the node load forecast data of all time steps within the look-ahead time window and the corresponding action vector of the previous look-ahead time window. Obtain the normalized output of balancing units in the power system; The motion space is constructed based on the normalized output of the balancing unit.

[0010] Furthermore, the training batch data includes: several training samples; each training sample corresponds to the current state, current action, immediate reward, next state, and round termination flag for a single scheduling time step; the network parameters include: current Critic network parameters, current Actor network parameters, target Critic network parameters, and target Actor network parameters. The step of updating the network parameters of the power dispatching agent based on the training batch data includes: The current state and current action in the training batch data are input into the current Critic network to obtain the value evaluation output of the current Critic network for the corresponding action in each state. The next state from the training batch data is input into the target Actor network to generate the action corresponding to the next state. The next state and its corresponding action are then input into the target Critic network to obtain the target value output. The target Q value is determined based on the immediate reward, the round end indicator, and the target value output; Based on the value assessment output and the target Q value, determine the mean squared error loss of the current Critic network; Based on the mean squared error loss, the current Critic network parameters are updated using the backpropagation gradient algorithm. The current state of the training samples is input into a pre-defined scheduler supervision network to obtain expert scheduling actions; The current state in the training batch data is input into the current Actor network to obtain the policy action corresponding to each current state; Each current state and its corresponding policy action are input into the current Critic network to obtain the value assessment of the policy action by the current Critic network, and the expected action value loss of the current Actor network is determined based on the value assessment. The expert supervision loss is calculated based on the mean square error between the strategy action and the expert scheduling action. The comprehensive loss is obtained by weighting and integrating the expected action value loss and the expert supervision loss. Based on the comprehensive loss, the current Actor network parameters are updated using gradient descent. Based on the preset soft update smoothing coefficient, the parameters of the target Critic network and the target Actor network are softly updated.

[0011] Furthermore, the preset dispatcher monitoring network is determined in the following way: Acquire a number of historical scheduling data; the historical scheduling data includes: the historical operating status of the power grid and the corresponding historical scheduling actions; Based on the current network structure of the Actor network, construct the scheduler supervision network to be trained; Several historical scheduling data are input into the dispatcher supervision network to be trained, so that the dispatcher supervision network takes the historical operating state of the power grid as input and the predicted scheduling action as output for iterative training. In each training process, a loss function is calculated based on the predicted scheduling action and the historical scheduling action, and the network parameters of the dispatcher supervision network are adjusted according to the loss function until the loss function converges, thus obtaining the preset dispatcher supervision network.

[0012] Furthermore, after obtaining the trained power dispatching agent, it also includes: In the event of a detected line fault in the power system, a target domain environment with an updated topology is constructed. Based on the network topology of the trained power dispatch agent, a fault state transfer agent is constructed, and the network parameters of the trained power dispatch agent are used as the initial network parameters of the fault state transfer agent. The fault-state transition agent is placed in the target domain environment after the topology is updated, and experience samples of fault condition state transition are collected. Based on the experience samples of fault state transition, the parameters of the Actor network and Critic network of the fault state transition agent are fine-tuned by backpropagation to obtain the fault scheduling agent.

[0013] Based on the above method embodiments, the present invention provides corresponding device embodiments, including: a power grid static parameter acquisition module, a scheduling optimization model construction module, a reinforcement learning framework construction module, a power dispatching agent construction module, and a power dispatching agent training module; The power grid static parameter acquisition module is used to acquire power grid operating status parameters; The scheduling optimization model construction module is used to construct a scheduling optimization model based on the power grid operating status parameters, with the objective function being to minimize the total power generation cost of all generating units at all time steps within the forward time window, and to construct rigid constraints. The reinforcement learning framework construction module is used to construct a comprehensive reward function and an action space and a state space based on the scheduling optimization model and the rigid constraints. The power dispatching agent construction module is used to construct a power dispatching agent based on the state space, action space, and comprehensive reward function. The power dispatching agent training module is used to iteratively train the power dispatching agent. During each training process, all dispatching actions output by the power dispatching agent are stored in the original experience replay pool, and feasible actions that satisfy all rigid constraints of the dispatching optimization model are stored in the feasible action experience replay pool. Samples are extracted from the feasible action experience replay pool and the original experience replay pool according to preset sampling weights to form training batch data. The network parameters of the power dispatching agent are updated according to the training batch data until the network parameters converge or the number of training rounds reaches the preset maximum number of iterations, thus obtaining the trained power dispatching agent.

[0014] An embodiment of the present invention provides a power dispatching method based on a power dispatching intelligent agent, comprising: Obtain power grid status data for the current forward-looking time window; The power grid status data of the current forward time window is input into the power dispatching agent to obtain the normalized action response of each unbalanced unit; wherein, the power dispatching agent is obtained by the training method of the power dispatching agent according to any embodiment of the present invention; The generator units of the power system are dispatched based on the normalized action response of each unbalanced unit.

[0015] Furthermore, the power grid status data includes: the predicted load of each node; The method of scheduling generator units in the power system based on the normalized action response of each unbalanced unit includes: The normalized action response of each unbalanced unit is denormalized to obtain the corresponding active power output. Summing up the active power output of all unbalanced units yields the total active power output of the unbalanced units. The total system load is calculated based on the predicted load of each node; Subtracting the total output of the unbalanced units from the total system load yields the active power output of the balanced units. Based on the active power output of the balanced units and its upper and lower limits, determine the dynamic available capacity margin of each unbalanced unit. Based on the active power output of each unbalanced unit, determine the consumption increment rate of the corresponding unbalanced unit; The total over-limit power of the balancing unit is determined based on the active power output of the balancing unit. Based on the consumption increment rate and dynamic available capacity margin of each unbalanced unit, the total over-limit power of the balanced units is allocated to each unbalanced unit, and the physical scheduling instructions of each unit are obtained. The generator units of the power system are dispatched according to the physical dispatch instructions of each unit.

[0016] Compared with the prior art, the beneficial effects of this embodiment are as follows: This invention acquires power grid operating state parameters. Based on these parameters, and using the objective function of minimizing the total generation cost of all generating units across all time steps within a forward time window, a scheduling optimization model is constructed, along with rigid constraints. A comprehensive reward function is then built based on the scheduling optimization model and rigid constraints. This embeds the power grid optimization objective and rigid constraints into reinforcement learning, guiding the agent to optimize strategies in a direction that satisfies the constraints and reduces generation costs, reducing meaningless random exploration, and constructing action and state spaces. A power dispatching agent is then constructed based on the state space, action space, and comprehensive reward function. Iterative training is performed on the power dispatching agent. During each training process, all dispatching actions output by the power dispatching agent are stored in the original... The experience replay pool filters out feasible actions that satisfy all rigid constraints of the scheduling optimization model and stores them in the feasible action experience replay pool, thus separately aggregating feasible scheduling experiences and distinguishing them from a large number of invalid exploration samples that violate constraints. Samples are drawn from the feasible action experience replay pool and the original experience replay pool according to preset sampling weights to form training batch data. This enables the agent to learn effective scheduling experiences that satisfy the rigid constraints of the power grid, greatly reducing the probability of the agent randomly learning invalid and illegal samples and avoiding the agent blindly trying and making mistakes in the inactive action range. Based on the training batch data, the network parameters of the power scheduling agent are updated until the network parameters converge or the number of training rounds reaches the preset maximum number of iterations, resulting in a trained power scheduling agent.

[0017] In summary, this invention constructs a dual experience replay pool and employs pre-defined weighted mixed sampling, enabling feasible scheduling experience to continuously influence agent parameter iteration. This reduces meaningless trial and error by the agent in inactive action zones, thereby solving the problem of low training efficiency caused by blind exploration in the early stages of existing agent training and improving the training efficiency of the agent. Attached Figure Description

[0018] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a training method for a power dispatching intelligent agent according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a training device for a power dispatching intelligent agent according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a power dispatching method based on a power dispatching intelligent agent, provided in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0022] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0025] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0026] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0027] like Figure 1 As shown, in order to solve the problem of low training efficiency caused by blind exploration in the early stage of existing intelligent agents, an embodiment of the present invention provides a training method for a power dispatching intelligent agent, which includes at least the following steps: Step S101: Obtain power grid operating status parameters; For step S101, the above-mentioned power grid operation state parameters of the present invention include: the total time step included in the look-ahead time window. Total number of generators in the power system Total number of nodes in the power system Total number of branches in the power system ,dynamo The quadratic coefficient of the power generation cost function ,dynamo The first coefficient of the power generation cost function ,dynamo The constant term coefficient of the power generation cost function Traditional generators The lower bound of active power output Traditional generators Upper limit of active power output Renewable energy generators In the forward-looking time window Inner The real-time active power output limit at each time step. ,node Forecast node load values ​​for the corresponding time period Branch roads For including generators DC power transfer distribution factor at nodes Branch roads For nodes DC power transfer distribution factor Branch roads Active power flow limit upper limit ,dynamo Downhill climbing limit ,dynamo The limit of climbing uphill ; By collecting power grid operating state parameters, multidimensional observation information characterizing the power grid operating conditions is used as the observation input corresponding to the state space in the reinforcement learning framework.

[0028] Step S102: Based on the power grid operating status parameters, with the objective function of minimizing the total power generation cost of all generating units at all time steps within the forward time window, construct a scheduling optimization model and establish rigid constraints; For step S102, in order to achieve forward-looking and refined control of the future power grid operation status, this embodiment adopts a rolling optimization time window mechanism. The total span of each look-ahead window is set to 4 hours, and the time resolution is set to 15 minutes. Let the starting time of the current look-ahead window be... The window contains A time step (from 0 to ...) ), model over time It rolls forward as it increases.

[0029] It should be noted that, considering the main focus is on the distribution of active power in the transmission network, this invention uses a DC power flow model for modeling and solving, in order to balance computational efficiency and accuracy.

[0030] To ensure optimal power generation costs in power system dispatch, a dispatch optimization model is constructed with the objective function of minimizing the total power generation cost of all generating units at all time steps within the look-ahead time window.

[0031] Specifically, the scheduling optimization model is as follows: ; In the formula, Forward-looking time window The total power generation cost of the system within the system, For generator In the forward-looking time window The first The active power output (in MW) at each time step is the core decision variable of the scheduling optimization model. This represents the total number of generators in the power system. This represents the total time steps contained within the lookahead time window. For generator The power generation cost function.

[0032] In this embodiment, the power generation cost function of a traditional unit is assumed to be a quadratic function: ; In the formula, , , Generators The quadratic coefficient, linear coefficient, and constant coefficient of the power generation cost function.

[0033] It should be noted that, in order to promote the consumption of renewable energy (new energy), this invention specifically sets the cost function coefficient of renewable energy generators to zero, classifying them as zero marginal cost generators.

[0034] In addition, to ensure the absolute safety and feasibility of the scheduling scheme at the physical level, rigid constraints are constructed for the scheduling optimization model to define the boundary of safe operation of the power grid.

[0035] In a preferred embodiment, the rigid constraints include: unit output limit constraints, line power flow limit constraints, unit ramping limit constraints, and real-time power balance constraints.

[0036] In one embodiment of the present invention, the generator output limit includes both traditional generator output limit and new energy generator output limit. The aforementioned traditional generator unit output limit constraint is used to restrict the active power output of traditional generator units to within a physical upper and lower limit, specifically: ; In the formula, and Traditional generators The lower and upper bounds of active power output; The aforementioned output limit constraint for new energy units is used to restrict the upper limit of active power output of renewable energy (new energy) generator units, ensuring that the dispatched output does not exceed the real-time available generating power of new energy units in the current period. This adapts to the random and fluctuating operating characteristics of new energy output. Specifically: ; In the formula, Generators for renewable energy In the forward-looking time window Inner The real-time active power output limit for each time step is predicted based on the system operating environment.

[0037] The power flow limit constraint is used to ensure that the power flow meets the line capacity, prevent power overload in transmission branches, and ensure the safe and stable operation of power grid transmission equipment. Specifically: ; In the formula, branch road In the forward-looking time window Inner A time step of line current. branch road The upper limit of active power flow; Specifically, the DC power transfer distribution factor (dc PTDF) is used for fast linearization calculation of line power flow to adapt to efficient solutions for large-scale power grids. The formula for calculating line power flow is: ; In the formula, The total number of nodes in the power system. For nodes The node load forecast value for the corresponding time period, For the road For including generators The DC power transfer distribution factor of a node is used to represent its contribution to the power supply characteristics. branch road For nodes The DC power transfer distribution factor is used to represent the contribution of all nodes to the load characteristic power flow.

[0038] The unit ramping limit constraint is used to constrain the adjustment range of unit output between adjacent time steps, ensuring a smooth transition of unit output. Specifically: ; In the formula, and Generators The downhill and uphill limits.

[0039] It should be noted that, since renewable energy units respond extremely quickly, the output of renewable energy units is not subject to the above-mentioned unit ramping limit constraints in this dispatch optimization model; only conventional generator units are subject to the ramping regulation limit requirements.

[0040] The real-time power balance constraint is used to ensure that the total power generation of the system must match the total load demand within any time step, thus maintaining the active power supply and demand balance of the power system. Specifically: ; By constructing the aforementioned unit output limit constraints, line power flow limit constraints, unit ramping limit constraints, and real-time power balance constraints, operating boundaries are set from multiple dimensions, including units, lines, timing regulation, and overall network power balance. This effectively avoids various equipment safety operation risks and ensures the physical feasibility of the dispatching scheme.

[0041] Step S103: Based on the scheduling optimization model and the rigid constraints, construct a comprehensive reward function, and construct the action space and state space; In a preferred embodiment, a comprehensive reward function is constructed based on the scheduling optimization model and the rigid constraints, including: The negative value of the total generation cost in the scheduling optimization model is used as the reward benchmark component; Based on the unit output limit constraints, construct a penalty item for exceeding the unit output limit; Based on the line flow restriction constraints, construct a line flow violation penalty item; Based on the unit ramping limit constraints, construct a unit ramping over-limit penalty item; The comprehensive reward function is obtained by superimposing the reward benchmark component with the unit output over-limit penalty, the line power flow over-limit penalty, and the unit ramp over-limit penalty.

[0042] For step S103, in order to balance cost optimization and multidimensional physical security during the scheduling process, the constructed comprehensive reward function includes a cost-benefit term and a security constraint violation penalty term. Since the goal of reinforcement learning is to maximize the cumulative reward, the negative value of the total generation cost in the scheduling optimization model is used as the reward baseline component, and unit output violation penalty term, line power flow violation penalty term, and unit ramping violation penalty term are added to characterize the degree of constraint violation.

[0043] Indicative, forward-looking time window The comprehensive reward function is as follows: ; In the formula, Forward-looking time window The comprehensive reward function, Penalties for exceeding the unit's power output limit This is a penalty item for exceeding the line power flow limit. This is a penalty for exceeding the ramp limit during the unit's hill climb.

[0044] Among them, the penalty for exceeding the unit output limit Specifically: ; In the formula, To penalize the weighting factor, a large negative value is used.

[0045] By defining the upper and lower limits of active power output based on unit output limits, a penalty item for exceeding the unit output limit is constructed. This transforms violations of unit output limits into quantifiable negative reward losses, thereby significantly reducing the overall reward value corresponding to scheduling actions that involve unit output limits. This guides the power dispatching agent to proactively avoid scheduling schemes that exceed the physical output limits of units during iterative training.

[0046] Line flow over-limit penalty Specifically: ; In the formula, The weighting factor for penalizing trends that exceed limits. This represents the total number of branches in the power system.

[0047] By defining the upper limit of active power flow limit for branch paths through line power flow constraint, a penalty item for exceeding the line power flow limit is constructed. This transforms violations of line power flow exceeding the transmission capacity limit into quantifiable negative reward losses, thereby significantly reducing the comprehensive reward value corresponding to scheduling actions with line power flow exceeding the limit. This guides the power dispatching agent to proactively avoid scheduling schemes that cause power overload of transmission branches during iterative training.

[0048] Unit over-limit climbing penalty Specifically: ; In the formula, The weighting factor for exceeding the climbing limit penalty. The change in the unit's active power output between adjacent time periods is calculated using the following logic: ; By defining the allowable adjustment range of unit output between adjacent time steps based on the unit ramp-up limit constraints, a unit ramp-up over-limit penalty term is constructed. This transforms violations where the unit's output increases or decreases beyond the ramp rate limit into quantifiable negative reward losses. This significantly reduces the overall reward value of dispatch actions that involve units exceeding ramp rate limits, guiding the power dispatching agent to proactively avoid dispatch schemes that cause sudden increases or decreases in output and damage to unit equipment during iterative training.

[0049] In a preferred embodiment, constructing the action space and state space includes: Obtain the node load forecast data for all time steps within the forward time window and the corresponding action vector for the previous forward time window; The state space is constructed based on the node load forecast data of all time steps within the look-ahead time window and the corresponding action vector of the previous look-ahead time window. Obtain the normalized output of balancing units in the power system; The motion space is constructed based on the normalized output of the balancing unit.

[0050] In one embodiment of the present invention, the state space is used to provide the agent with comprehensive observation information of the current environment to support its long-term benefit assessment and next action decision. In order to take into account the temporal correlation in multi-time-period look-ahead scheduling, the present invention designs the state space to include node load prediction data for all time steps within the look-ahead time window, as well as the corresponding action output of the previous look-ahead time window.

[0051] Specifically, forward-looking time windows The corresponding state space matrix Represented as: ; in, This represents the action vector of the previous look-ahead time window. Indicates the forward time window Inner The normalized node load vector at each time step, whose dimension is equal to the total number of nodes in the power system. The details are as follows: ; In the formula, For nodes The normalized load value aims to eliminate the adverse effects of differences in load magnitude between different nodes on the backpropagation of neural network gradients.

[0052] To avoid generating a large number of illegal scheduling actions that violate real-time power balance constraints during the random search process of the reinforcement learning agent, this invention introduces the concept of a balancing node from the optimal power flow model when constructing the action space, selecting the generator unit with the largest content volume in the power system as the balancing unit (for lack of generality, its index is set to...). The unit's output is then automatically solved using a real-time power balance equation, only considering the remaining power... The active power output of the generator sets is used as a variable for the autonomous decision-making of the intelligent agent, thereby expanding the degrees of freedom in the action space from... Down to This effectively reduces the learning difficulty and exploration cost of intelligent agents.

[0053] Specifically, forward-looking time windows Corresponding motion space Represented as: ; in, Indicates the first The normalized active power output vector at each time step has a dimension of . Specifically: ; In the formula, For generator Normalized values ​​of active and passive output, mapped to The interval is calculated using the following formula: ; Before obtaining After the Taiwanese generator set outputs power, the balancing generator set (the first one) The output of the unit is calculated using the following formula: ; By reducing the dimensionality of the action space and linking the output of the balancing unit with the above-mentioned calculation, the dimensionality of the agent's decision variables is reduced, and the learning convergence efficiency of the agent is improved.

[0054] Step S104: Based on the state space, action space, and comprehensive reward function, construct a power dispatching intelligent agent; For step S104, the state space carries the full-dimensional operation condition observation information of the power grid, the action space after dimensionality reduction and optimization limits the range of unit dispatching instructions that the agent can output, and the comprehensive reward function that integrates multiple constraint penalty terms quantifies the quality and degree of violation of the dispatching action. The power grid dispatching scenario is transformed into a standard reinforcement learning interaction framework to build a power dispatching agent.

[0055] Step S105: Iteratively train the power dispatching agent; during each training process, store all dispatching actions output by the power dispatching agent into the original experience replay pool, and select feasible actions that satisfy all rigid constraints of the dispatching optimization model and store them into the feasible action experience replay pool; extract samples from the feasible action experience replay pool and the original experience replay pool according to the preset sampling weights to form training batch data; update the network parameters of the power dispatching agent according to the training batch data until the network parameters converge or the number of training rounds reaches the preset maximum iteration training round, and obtain the trained power dispatching agent.

[0056] For step S105, the power dispatching agent observes the power grid status based on the current network parameters and outputs dispatching actions. It stores all the complete experience samples of the status, actions, rewards, and next status generated in this interaction into the original experience replay pool. At the same time, it verifies the dispatching actions according to the unit output limit constraints, line power flow limit constraints, unit ramping limit constraints, and real-time power balance constraints. If the dispatching action meets all constraints, it stores the experience sample corresponding to the dispatching action into the feasible action experience replay pool, realizing the compliant experience pool storage.

[0057] The sampling weights of the feasible action experience replay pool and the original experience replay pool are pre-configured. According to the weight ratio, the corresponding number of training samples are extracted from the feasible action experience replay pool and the original experience replay pool to form training batch data, thereby increasing the proportion of compliant and feasible actions in the training samples.

[0058] The network parameters of the power dispatching agent are iteratively updated using the constructed training batch data, combined with the comprehensive reward function to calculate the policy loss and value loss, and then updated in a single iteration using the backpropagation algorithm.

[0059] In a preferred embodiment, the training batch data includes: a plurality of training samples; each training sample corresponds to the current state, current action, immediate reward, next state, and round termination flag of a single scheduling time step; the network parameters include: current Critic network parameters, current Actor network parameters, target Critic network parameters, and target Actor network parameters; The step of updating the network parameters of the power dispatching agent based on the training batch data includes: The current state and current action in the training batch data are input into the current Critic network to obtain the value evaluation output of the current Critic network for the corresponding action in each state. The next state from the training batch data is input into the target Actor network to generate the action corresponding to the next state. The next state and its corresponding action are then input into the target Critic network to obtain the target value output. The target Q value is determined based on the immediate reward, the round end indicator, and the target value output; Based on the value assessment output and the target Q value, determine the mean squared error loss of the current Critic network; Based on the mean squared error loss, the current Critic network parameters are updated using the backpropagation gradient algorithm. The current state of the training samples is input into a pre-defined scheduler supervision network to obtain expert scheduling actions; The current state in the training batch data is input into the current Actor network to obtain the policy action corresponding to each current state; Each current state and its corresponding policy action are input into the current Critic network to obtain the value assessment of the policy action by the current Critic network, and the expected action value loss of the current Actor network is determined based on the value assessment. The expert supervision loss is calculated based on the mean square error between the strategy action and the expert scheduling action. The comprehensive loss is obtained by weighting and integrating the expected action value loss and the expert supervision loss. Based on the comprehensive loss, the current Actor network parameters are updated using gradient descent. Based on the preset soft update smoothing coefficient, the parameters of the target Critic network and the target Actor network are softly updated.

[0060] In one embodiment of the present invention, the deep reinforcement learning framework used is the deep deterministic policy gradient algorithm. The network architecture of this algorithm includes the current Critic network, the current Actor network, the target Actor network, and the target Critic network, and the network parameters of the current Critic network, the current Actor network, the target Actor network, and the target Critic network are iteratively updated.

[0061] Specifically, the current parameter update process of the Critic network is as follows: When updating the network parameters of the power dispatching agent using training batch data consisting of several training samples, the first batch of training batch data is used... The current state within each training sample and current action Input the current Critic network to obtain the current state of the Critic network. Next action Value assessment output ,in, These are the current Critic network parameters; The first The next state of each training sample Input is fed into the target Actor network to generate the action corresponding to the next state. and the next state and their corresponding actions The input is fed into the target Critic network to obtain the target value output. ,in, Target Critic network parameters; When the interactive simulation reaches the last time step of the current lookahead window, it signifies the complete end of this round of lookahead scheduling simulation cycle. There is no next scheduling period within the window, and the round ends. for The current step's comprehensive instant reward will be directly applied. The target Q value is used; if the current time step does not belong to the end of the lookahead window, there are still subsequent scheduling periods to be deduced within the window, and there is a next state and subsequent scheduling benefits, then the round ends. for The current instant reward With discount factor Weighted target value of the next state The target Q value is obtained by adding the results. The specific formula for calculating the target Q value is as follows: ; In the formula, For the target Q value, As a discount factor, For the comprehensive reward function, For the target Actor network in the next state The action selected below.

[0062] Using the temporal difference error, the mean squared error loss of the current Critic network is calculated using the following formula: ; In the formula, This represents the mean squared error loss of the current Critic network. This represents the number of training samples in the training batch data.

[0063] By minimizing the mean square error loss The backpropagation gradient algorithm is used to update the current Critic network parameters. .

[0064] Specifically, the parameter update process of the current Actor network is as follows: The first The current state of each training sample The input is fed into a pre-trained supervisory network for dispatchers, which then outputs expert dispatching actions based on the dispatching experience it has learned. ; will the first The current state of each training sample Inputting the data into the current Actor network yields the corresponding policy action. ; According to the current Actor network in the... The expert supervision loss is calculated by taking the mean square error between the policy action output in the current state of each training sample and the expert scheduling action. ; In the formula, For expert-monitored loss, this is used to quantify the policy actions of the current Actor network based on real-time output of state input. Expert scheduling actions output by the dispatcher overseeing the network. The mean square error between them.

[0065] The current state and corresponding policy action are input into the value evaluation function of the current Critic network. The value evaluations for all training samples are summed and averaged to calculate the expected action value loss of the current Actor network. The specific formula is as follows: ; in, The expected action value loss of the current Actor network. For the current Actor network in state The output strategy action, These are the current Actor network parameters. These are the current Critic network parameters; Loss of value of expected action Loss monitoring by experts Weighted fusion is performed to obtain the comprehensive loss, as shown in the following formula: ; In the formula, The overall loss of the current Actor network, To control the proportion of expert supervision loss weight.

[0066] To minimize overall loss To achieve the goal, the gradient descent algorithm is used to update the current Actor network parameters. By integrating loss into gradient backpropagation updates, while pursuing optimal scheduling costs, it avoids a large number of illegal exploration behaviors based on manual scheduling experience, significantly reducing the proportion of illegal actions in the early stage of training and stabilizing the strategy for updating gradients.

[0067] Specifically, the parameter update process for the target Actor network and the target Critic network is as follows: To maintain the stability of agent parameter iteration during the forward rolling regulation of the large power grid and prevent training divergence caused by multi-constraint, multi-time-period coupling, the target Actor network parameters are adjusted after each training round. and target Critic network parameters Periodically introducing soft update slip from the current network, the specific recursive formula of which satisfies: ; ; In the formula, Let be the soft update smoothing coefficient of the target Actor network. Let be the soft update smoothing coefficient of the target Critic network, where and All are positive constants much less than 1.

[0068] Repeat the above process of weighted sampling with dual experience replay pools, updating the current network gradient, and soft updating the target network. When the change in network parameters is less than the preset convergence threshold, the parameters are considered to have converged. Or, when the cumulative number of iteration training rounds reaches the preset maximum number of iteration training rounds, the Actor network is retained as the final trained power dispatching agent.

[0069] In a preferred embodiment, the preset dispatcher monitoring network is determined in the following way: Acquire a number of historical scheduling data; the historical scheduling data includes: the historical operating status of the power grid and the corresponding historical scheduling actions; Based on the current network structure of the Actor network, construct the scheduler supervision network to be trained; Several historical scheduling data are input into the dispatcher supervision network to be trained, so that the dispatcher supervision network takes the historical operating state of the power grid as input and the predicted scheduling action as output for iterative training. In each training process, a loss function is calculated based on the predicted scheduling action and the historical scheduling action, and the network parameters of the dispatcher supervision network are adjusted according to the loss function until the loss function converges, thus obtaining the preset dispatcher supervision network.

[0070] In one embodiment of the present invention, the dispatcher-supervised network has the same network structure as the current actor network. It completes supervised training using historical scheduling data, aiming to mimic the actual operating habits of a dispatcher and act as an expert control strategy. During reinforcement learning, this network is used to supervise the output of the reinforcement learning control network, thereby effectively accelerating the training of the agent and improving the reliability of neural network training.

[0071] Preferably, based on the size of the input state data of the Actor network, a single-interval training mode or a multi-interval training mode is adaptively selected to balance the accuracy and computational efficiency of the algorithm.

[0072] Specifically, the multi-period training mode is suitable for small and medium-sized power systems. It merges the predicted node loads of all time periods within the complete forward-looking time window and the output results of the corresponding units in the previous scheduling window into a unified overall state vector, which is then fed into the neural network to complete the forward calculation at once. In this mode, the network input size is the product of the state feature dimension of a single time period and the time step of the forward-looking window, and the output size is the product of the action dimension of a single time period and the time step of the forward-looking window. It can completely preserve the temporal coupling and correlation information of each time period within the window, ensuring the scheduling accuracy in small and medium-sized power grid scenarios.

[0073] The single-time training mode is suitable for large-scale systems (such as the SG126-node system). It splits the complete temporal input state into slices according to a single scheduling time step and feeds them into the network for computation in batches of small samples. In this mode, the network input size is simplified to the single-step state feature dimension, and the output size is synchronously reduced to the single-step action dimension, which greatly reduces the complexity of the network structure, reduces the amount of computation per forward pass, and significantly shortens the model convergence time.

[0074] In a preferred embodiment, after obtaining the trained power dispatching agent, the method further includes: In the event of a detected line fault in the power system, a target domain environment with an updated topology is constructed. Based on the network topology of the trained power dispatch agent, a fault state transfer agent is constructed, and the network parameters of the trained power dispatch agent are used as the initial network parameters of the fault state transfer agent. The fault-state transition agent is placed in the target domain environment after the topology is updated, and experience samples of fault condition state transition are collected. Based on the experience samples of fault state transition, the parameters of the Actor network and Critic network of the fault state transition agent are fine-tuned by backpropagation to obtain the fault scheduling agent.

[0075] In one embodiment of the present invention, after obtaining the power dispatching agent trained under normal operating conditions, in order to solve the technical problems of the power grid topology change and the scarcity of historical fault operation data in the N-1 fault scenario, the large computing power consumption, long convergence period, inability to meet the real-time fault dispatching requirements, and the easy occurrence of power flow exceeding the limit by directly reusing the normal operating condition agent, a fault dispatching agent migration process is added for the N-1 line disconnection fault scenario of the power grid.

[0076] Specifically, when a line disconnection fault is detected in the power system, the physical topology of the power grid undergoes a step change, the underlying DC power transfer distribution factor matrix changes synchronously, and the system automatically updates the fault state distribution factor matrix. and Available branch set and line power flow limit Complete the fault topology update and build the target domain environment adapted to the fault conditions.

[0077] Considering the scarcity of historical operational data in the fault target domain and the time-consuming and computationally expensive nature of training from scratch, transfer learning is introduced. A novel fault-state transfer agent is built by referencing the complete network topology of a pre-trained power dispatch agent under normal operating conditions, directly replicating and reusing the Actor network parameters of the power dispatch agent. With Critic network parameters , as the initial parameters for the fault-transferring agent network (Actor network) and the Critic network, i.e., setting , This allows the general scheduling prior knowledge learned by the agent under normal operating conditions, such as power balance and unit output constraints, to be imported into the fault scenario model, avoiding the meaningless cold start exploration caused by training from scratch; among them, These are the initial parameters for the fault-transferring Actor network. These are the initial parameters for the fault-state transition agent Critic network.

[0078] After parameter initialization, the fault-state transition agent is deployed to the fault target domain environment with updated topology for short-cycle interactive exploration. The agent outputs scheduling actions based on the new state space corresponding to the fault topology, and the environment responds according to the fault state power transition matrix. and Recalculate the power flow over-limit penalty and return to the comprehensive reward for fault conditions. The system synchronously collects and stores experience samples of fault state transitions, including fault status, fault scheduling actions, immediate fault rewards, and the next fault state. Based on the collected interaction experience samples carrying fault topology features, it reuses the gradient update logic from the power dispatching agent training, calculates the corresponding loss function, and executes the backpropagation algorithm to update the Actor network parameters of the fault state transition agent. Critic network parameters Online fine-tuning is performed to adapt to the shrinking feasible domain of safe actions under fault conditions, ultimately resulting in a fault scheduling agent that can adapt to N-1 fault scenarios and has fault-safe scheduling capabilities.

[0079] This invention employs a model-level migration adaptive fine-tuning mechanism to reuse the network structure and parameters of the scheduling agent under normal operating conditions. It can adapt to N-1 fault topologies by using only a small number of fault samples for online fine-tuning. This reduces the computation time for responding to topology changes from tens of hours (offline retraining scale) to minutes / seconds, significantly shortening the model adaptation time after power grid topology changes and enabling rapid self-healing and highly reliable adaptive scheduling under large-scale power grid faults.

[0080] like Figure 2 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided; One embodiment of the present invention provides a training device for a power dispatching agent, comprising: a power grid static parameter acquisition module 1, a dispatching optimization model construction module 2, a reinforcement learning framework construction module 3, a power dispatching agent construction module 4, and a power dispatching agent training module 5; The power grid static parameter acquisition module 1 is used to acquire power grid operating status parameters; The scheduling optimization model construction module 2 is used to construct a scheduling optimization model based on the power grid operating status parameters, with the objective function being to minimize the total power generation cost of all generator units at all time steps within the forward time window, and to construct rigid constraints. The reinforcement learning framework construction module 3 is used to construct a comprehensive reward function and an action space and a state space based on the scheduling optimization model and the rigid constraints. The power dispatching intelligent agent construction module 4 is used to construct a power dispatching intelligent agent based on the state space, action space, and comprehensive reward function; The power dispatching agent training module 5 is used to iteratively train the power dispatching agent. In each training process, all dispatching actions output by the power dispatching agent are stored in the original experience replay pool, and feasible actions that meet all rigid constraints of the dispatching optimization model are stored in the feasible action experience replay pool. Samples are extracted from the feasible action experience replay pool and the original experience replay pool according to preset sampling weights to form training batch data. The network parameters of the power dispatching agent are updated according to the training batch data until the network parameters converge or the number of training rounds reaches the preset maximum iteration training round, and the trained power dispatching agent is obtained.

[0081] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the power dispatching method based on a power dispatching agent provided by any of the above-described method embodiments of the present invention.

[0082] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0083] like Figure 3 As shown, an embodiment of the present invention provides a training method for a power dispatching agent, the method comprising at least the following steps: Step S201: Obtain the power grid status data for the current look-ahead time window; In a preferred embodiment, the power grid status data includes: the predicted load of each node; For step S201, during the online scheduling phase, the power grid state data of the current look-ahead time window is collected, and the power grid state data is consistent with the state space features used in the offline training phase of the power dispatching agent.

[0084] Step S202: Input the power grid state data of the current look-ahead time window into the power dispatching agent to obtain the normalized action response of each unbalanced unit; wherein, the power dispatching agent is obtained by the training method of the power dispatching agent according to any embodiment of the present invention; For step S202, the power grid state data of the current look-ahead time window is input into the power dispatching agent obtained by the training method of any of the above embodiments of the present invention. Based on the load-dispatch mapping relationship and power grid security constraint rules obtained by offline training, forward inference calculation is performed, and the normalized action response of each unbalanced unit is output. The range of values ​​is .

[0085] Step S203: Dispatch the generator sets of the power system according to the normalized action response of each unbalanced unit.

[0086] In a preferred embodiment, the generator sets of the power system are dispatched based on the normalized action response of each unbalanced unit, including: The normalized action response of each unbalanced unit is denormalized to obtain the corresponding active power output. Summing up the active power output of all unbalanced units yields the total active power output of the unbalanced units. The total system load is calculated based on the predicted load of each node; Subtracting the total output of the unbalanced units from the total system load yields the active power output of the balanced units. Based on the active power output of the balanced units and its upper and lower limits, determine the dynamic available capacity margin of each unbalanced unit. Based on the active power output of each unbalanced unit, determine the consumption increment rate of the corresponding unbalanced unit; The total over-limit power of the balancing unit is determined based on the active power output of the balancing unit. Based on the consumption increment rate and dynamic available capacity margin of each unbalanced unit, the total over-limit power of the balanced units is allocated to each unbalanced unit, and the physical scheduling instructions of each unit are obtained. The generator units of the power system are dispatched according to the physical dispatch instructions of each unit.

[0087] For step S203, in order to convert it into a physical power command executable by the grid, the following formula is used for each unbalanced unit (previous) Normalized action response of a generator By performing inverse normalization, the corresponding active power output is obtained: ; In the formula, Unbalanced unit The contribution after reverse normalization.

[0088] After obtaining the active power output of all unbalanced generating units, in order to meet the real-time power balance constraints of the power system, the active power output of all unbalanced generating units is summed to calculate the total output of the unbalanced generating units. Simultaneously, the predicted loads of each node are summed to obtain the total system load. The active power output of the balanced units is obtained by subtracting the total output of the unbalanced units from the total system load, as shown in the following formula: ; In the formula, To balance the active power output of the generator unit.

[0089] Since the output of the balancing unit is calculated without considering the physical capacity constraints of the unit's own output limits, the calculated result may exceed the unit's permissible safe operating range. Therefore, this embodiment first calculates the total over-limit power of the balancing unit using the following formula: ; In the formula, To balance the total over-limit power of the generating units, To balance the safety lower limit of the unit, To balance the safety limits of the generator units.

[0090] like This indicates that the calculated output of the balancing unit exceeds the limit, and the final physical output of the balancing unit needs to be forcibly reduced. Cut off to its safety boundary; when the output exceeds the upper limit, cause When the effort exceeds the lower limit, it causes The excess power difference caused by the cut-off will be distributed to the remaining unbalanced units to restore the active power balance of the entire network.

[0091] Before performing over-limit power transfer, assess the remaining unbalanced units. Dynamic available capacity margin during this period This margin is limited by both the unit's maximum output and the ramp rate.

[0092] Specifically, if the output of the balancing unit exceeds its active power output limit, the other units need to increase their power output to share the shortfall. At this time, the dynamic available capacity margin of each unit is... Calculate using the piecewise function as follows: ; If the output of the balancing unit exceeds its lower limit of active power output, the other units need to reduce their power output to absorb the excess output. At this time, the dynamic available capacity margin of each unit is... Calculate using the piecewise function as follows: ; By calculating the dynamic available capacity margin of each unit, the maximum power adjustment range that each unit can safely withstand is quantified, thus avoiding the secondary over-limit of unbalanced units after power allocation.

[0093] To ensure optimal power generation costs while correcting over-limit power consumption, this invention employs the principle of equal consumption incremental rate to perform secondary allocation of over-limit power. First, the output points of each unbalanced unit are calculated. The differential value of the power generation cost function (i.e., the incremental rate of consumption). Subsequently, based on the magnitude of the consumption increment rate and the dynamic available capacity margin calculated in the above steps, The over-limit power of the balanced units is allocated to each unbalanced unit proportionally, and the final physical dispatch instructions issued to each unit are calculated. .

[0094] Specifically, if the output of the balancing unit exceeds its active power output limit, the other units need to increase their power output, and the final physical dispatch command will be issued. The calculation formula is: ; If the initial output of the balancing unit exceeds its lower limit of active power output, the remaining units need to reduce their power output, and the final physical dispatch command will be issued. The calculation formula is: ; If a power deficit still exists in the entire system after a single redistribution calculation, the capacity assessment and redistribution steps described above will be iteratively executed based on the updated physical output until... Ultimately, the system will transmit the physical scheduling instructions for each unit. The data is sent to each generator unit in real time for power dispatching.

[0095] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A method for training an electric power dispatching agent, characterized in that, include: Obtain power grid operating status parameters; Based on the power grid operating status parameters, a scheduling optimization model is constructed with the objective function of minimizing the total power generation cost of all generating units at all time steps within the forward time window, and rigid constraints are also constructed. Based on the scheduling optimization model and the rigid constraints, a comprehensive reward function is constructed, and an action space and a state space are also constructed. Based on the state space, action space, and comprehensive reward function, a power dispatching intelligent agent is constructed; The power dispatching agent is iteratively trained. During each training process, all scheduling actions output by the power dispatching agent are stored in the original experience replay pool, and feasible actions that satisfy all rigid constraints of the scheduling optimization model are stored in the feasible action experience replay pool. Samples are drawn from the feasible action experience replay pool and the original experience replay pool according to the preset sampling weights to form training batch data. The network parameters of the power dispatching agent are updated according to the training batch data until the network parameters converge or the number of training rounds reaches the preset maximum number of iterations, and the trained power dispatching agent is obtained. 2.The method of Claim 1, wherein, The rigid constraints include: unit output limit constraints, line power flow limit constraints, unit ramping limit constraints, and real-time power balance constraints.

3. The training method for a power dispatching intelligent agent according to claim 2, characterized in that, Based on the scheduling optimization model and the rigid constraints, a comprehensive reward function is constructed, including: The negative value of the total generation cost in the scheduling optimization model is used as the reward benchmark component; Based on the unit output limit constraints, construct a penalty item for exceeding the unit output limit; Based on the line power flow constraints, construct a line power flow violation penalty item; Based on the unit ramping limit constraints, construct a unit ramping over-limit penalty item; The comprehensive reward function is obtained by superimposing the reward benchmark component with the unit output over-limit penalty, the line power flow over-limit penalty, and the unit ramp over-limit penalty.

4. The training method for a power dispatching intelligent agent according to claim 1, characterized in that, Construct the action space and state space, including: Obtain the node load forecast data for all time steps within the forward time window and the corresponding action vector for the previous forward time window; The state space is constructed based on the node load forecast data of all time steps within the look-ahead time window and the corresponding action vector of the previous look-ahead time window. Obtain the normalized output of balancing units in the power system; The motion space is constructed based on the normalized output of the balancing unit.

5. The training method for a power dispatching intelligent agent according to claim 1, characterized in that, The training batch data includes: several training samples; each training sample corresponds to the current state, current action, immediate reward, next state, and round termination flag of a single scheduling time step; the network parameters include: current Critic network parameters, current Actor network parameters, target Critic network parameters, and target Actor network parameters. The step of updating the network parameters of the power dispatching agent based on the training batch data includes: The current state and current action in the training batch data are input into the current Critic network to obtain the value evaluation output of the current Critic network for the corresponding action in each state. The next state from the training batch data is input into the target Actor network to generate the action corresponding to the next state. The next state and its corresponding action are then input into the target Critic network to obtain the target value output. The target Q value is determined based on the immediate reward, the round end indicator, and the target value output; Based on the value assessment output and the target Q value, determine the mean squared error loss of the current Critic network; Based on the mean squared error loss, the current Critic network parameters are updated using the backpropagation gradient algorithm. The current state of the training samples is input into a pre-defined scheduler supervision network to obtain expert scheduling actions; The current state in the training batch data is input into the current Actor network to obtain the policy action corresponding to each current state; Each current state and its corresponding policy action are input into the current Critic network to obtain the value assessment of the policy action by the current Critic network, and the expected action value loss of the current Actor network is determined based on the value assessment. The expert supervision loss is calculated based on the mean square error between the strategy action and the expert scheduling action. The comprehensive loss is obtained by weighting and integrating the expected action value loss and the expert supervision loss. Based on the comprehensive loss, the current Actor network parameters are updated using gradient descent. Based on the preset soft update smoothing coefficient, the parameters of the target Critic network and the target Actor network are softly updated.

6. The training method for a power dispatching intelligent agent according to claim 5, characterized in that, The preset dispatcher monitoring network is determined in the following way: Acquire a number of historical scheduling data; the historical scheduling data includes: the historical operating status of the power grid and the corresponding historical scheduling actions; Based on the current network structure of the Actor network, construct the scheduler supervision network to be trained; Several historical scheduling data are input into the dispatcher supervision network to be trained, so that the dispatcher supervision network takes the historical operating state of the power grid as input and the predicted scheduling action as output for iterative training. In each training process, a loss function is calculated based on the predicted scheduling action and the historical scheduling action, and the network parameters of the dispatcher supervision network are adjusted according to the loss function until the loss function converges, thus obtaining the preset dispatcher supervision network.

7. The training method for a power dispatching intelligent agent according to claim 6, characterized in that, After obtaining the trained power dispatch agent, it also includes: In the event of a detected line fault in the power system, a target domain environment with an updated topology is constructed. Based on the network topology of the trained power dispatch agent, a fault state transfer agent is constructed, and the network parameters of the trained power dispatch agent are used as the initial network parameters of the fault state transfer agent. The fault-state transition agent is placed in the target domain environment after the topology is updated, and experience samples of fault condition state transition are collected. Based on the experience samples of fault state transition, the parameters of the Actor network and Critic network of the fault state transition agent are fine-tuned by backpropagation to obtain the fault scheduling agent.

8. A training device for a power dispatching intelligent agent, characterized in that, include: The system includes a power grid static parameter acquisition module, a scheduling optimization model construction module, a reinforcement learning framework construction module, a power dispatching agent construction module, and a power dispatching agent training module. The power grid static parameter acquisition module is used to acquire power grid operating status parameters; The scheduling optimization model construction module is used to construct a scheduling optimization model based on the power grid operating status parameters, with the objective function being to minimize the total power generation cost of all generating units at all time steps within the forward time window, and to construct rigid constraints. The reinforcement learning framework construction module is used to construct a comprehensive reward function and an action space and a state space based on the scheduling optimization model and the rigid constraints. The power dispatching agent construction module is used to construct a power dispatching agent based on the state space, action space, and comprehensive reward function. The power dispatching agent training module is used to iteratively train the power dispatching agent; During each training process, all scheduling actions output by the power dispatching agent are stored in the original experience replay pool, and feasible actions that satisfy all rigid constraints of the scheduling optimization model are stored in the feasible action experience replay pool. Samples are drawn from the feasible action experience replay pool and the original experience replay pool according to the preset sampling weights to form training batch data. The network parameters of the power dispatching agent are updated according to the training batch data until the network parameters converge or the number of training rounds reaches the preset maximum number of iterations, and the trained power dispatching agent is obtained.

9. A power dispatching method based on a power dispatching intelligent agent, characterized in that, include: Obtain power grid status data for the current forward-looking time window; The power grid state data of the current forward time window is input into the power dispatching agent to obtain the normalized action response of each unbalanced unit; wherein, the power dispatching agent is obtained by the training method of the power dispatching agent according to any one of claims 1-7; The generator units of the power system are dispatched based on the normalized action response of each unbalanced unit.

10. The power dispatching method based on a power dispatching intelligent agent according to claim 9, characterized in that, The power grid status data includes: the predicted load of each node; The method of scheduling generator units in the power system based on the normalized action response of each unbalanced unit includes: The normalized action response of each unbalanced unit is denormalized to obtain the corresponding active power output. Summing up the active power output of all unbalanced units yields the total active power output of the unbalanced units. The total system load is calculated based on the predicted load of each node; Subtracting the total output of the unbalanced units from the total system load yields the active power output of the balanced units. Based on the active power output of the balanced units and its upper and lower limits, determine the dynamic available capacity margin of each unbalanced unit. Based on the active power output of each unbalanced unit, determine the consumption increment rate of the corresponding unbalanced unit; The total over-limit power of the balancing unit is determined based on the active power output of the balancing unit. Based on the consumption increment rate and dynamic available capacity margin of each unbalanced unit, the total over-limit power of the balanced units is allocated to each unbalanced unit, and the physical scheduling instructions of each unit are obtained. The generator units of the power system are dispatched according to the physical dispatch instructions of each unit.