Hybrid expert reinforcement learning power grid dispatch optimization method, system, device and medium

Abstract

This invention discloses a hybrid expert reinforcement learning method, system, device, and medium for power grid dispatch optimization. These are corresponding solutions. The solutions employ a hybrid expert model combined with a reinforcement learning framework to achieve adaptive selection and dynamic updating of strategies under different power grid topologies and operating scenarios. This not only enhances cross-scenario generalization capabilities but also supports batch experience pools and hybrid sampling training mechanisms, enabling the strategy to learn stably in heterogeneous environments, thus possessing good engineering deployability. Furthermore, from the perspective of power grid operation and dispatch optimization, it can reduce economic operating costs while achieving zero out-of-bounds constraints on line power, ensuring the safe operation of the system in multi-topology and multi-scenario environments. By introducing an externally enhanced Markov decision process and internal / external reward design, it can effectively coordinate new energy generation, energy storage charging and discharging, and load demand, improving the overall operating efficiency and economy of the power grid.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence technology and intelligent power system dispatching technology, and in particular to a hybrid expert reinforcement learning power grid dispatching optimization method, system, equipment and medium. Background Technology

[0002] With the integration of high proportions of renewable energy, the large-scale deployment of energy storage systems, and the rapid growth of electrification loads, the power grid system is transforming from a traditional operating paradigm characterized by centralized dispatch and static networks to a highly dynamic, strongly coupled, and autonomous "source-grid-load-storage" collaborative network. Within the power grid, the uncertainty of renewable energy generator power, load and electricity price fluctuations, and frequent adjustments to the network topology result in scenario non-stationarity and structural dynamism in grid operation.

[0003] Existing optimal power flow methods rely on accurate models, but they incur high computational costs and depend on global grid information, making them difficult to meet real-time scheduling requirements. While reinforcement learning does not require accurate models, traditional methods typically assume fixed topologies and a single scenario, resulting in limited generalization capabilities and difficulty adapting to topology switching and multiple operational scenarios in real power grids. Furthermore, existing methods struggle to address the degradation of energy storage devices' performance due to delayed returns, thereby affecting the overall reliability of scheduling and increasing economic operating costs. Summary of the Invention

[0004] The purpose of this invention is to provide a hybrid expert reinforcement learning-based power grid dispatch optimization method, system, device, and medium, which combines reinforcement learning with a hybrid expert model to achieve multi-topology, multi-scenario power grid optimization dispatch.

[0005] The objective of this invention is achieved through the following technical solution: A hybrid expert reinforcement learning method for power grid dispatch optimization includes: Construct a hybrid expert reinforcement learning network that includes a policy network and a value network, wherein a hybrid expert structure is introduced into both the policy network and the value network; The power grid with multiple topologies and multiple operating scenarios is modeled as a unified power grid environment, and exogenous variables are introduced in the modeling process to characterize environmental uncertainties. The power grid environment is divided into internal and external states, thus modeling the power grid dispatch optimization problem as an externally enhanced Markov decision process. In the externally enhanced Markov decision process: the internal state is the state directly affected by actions, while the external state is the state independent of actions. Actions are used to control the power of relevant equipment in the power grid environment. The reward includes internal and external rewards. The internal reward is calculated based on the cost of each device, while the external reward is calculated based on the relationship between actions and the constraints of various physical quantities in the power grid. By interacting with the power grid environment through the policy network, and based on the exogenous enhanced Markov decision process, training samples for reinforcement learning are collected and stored in the global experience pool. Training samples are sampled from the global experience pool, and the value network and policy network are trained alternately. The trained policy network is deployed in a real power grid scenario, and it makes action decisions based on the state of the real power grid scenario to achieve power grid scheduling optimization.

[0006] A hybrid expert reinforcement learning power grid dispatch optimization system, used in the aforementioned method, includes: A hybrid expert reinforcement learning network building unit is used to construct a hybrid expert reinforcement learning network that includes a policy network and a value network, wherein a hybrid expert structure is introduced into both the policy network and the value network. The power grid environment modeling unit is used to unify the modeling of power grids with multiple topologies and multiple operating scenarios into a power grid environment, and introduces external variables to characterize environmental uncertainties during the modeling process; The exogenous augmented Markov decision process modeling unit is used to divide the power grid environment state into intrinsic and extrinsic states, thereby modeling the power grid dispatch optimization problem as an exogenous augmented Markov decision process. In the exogenous augmented Markov decision process: intrinsic states are states directly affected by actions, while extrinsic states are states independent of actions. Actions are used to control the power of relevant equipment in the power grid environment. The reward includes intrinsic and extrinsic rewards. Intrinsic rewards are calculated based on the cost of each device, while extrinsic rewards are calculated based on the relationship between actions and constraints of various physical quantities in the power grid. The sample generation and storage unit is used to interact with the power grid environment through the policy network, collect training samples for reinforcement learning based on the externally enhanced Markov decision process, and store them in the global experience pool. The network training unit is used to sample training samples from the global experience pool and alternately train the value network and the policy network. The power grid dispatch optimization unit is used to deploy the trained policy network in the actual power grid scenario. It makes action decisions based on the state of the actual power grid scenario to achieve power grid dispatch optimization.

[0007] As can be seen from the technical solutions provided by the present invention above: (1) From the perspective of intelligent control and generalization, the present invention adopts a hybrid expert model combined with a reinforcement learning framework to realize the adaptive selection and dynamic update of the strategy under different power grid topologies and operating scenarios; it not only improves the cross-scenario generalization capability, but also supports batch experience pools and hybrid sampling training mechanisms, enabling the strategy to learn stably in heterogeneous environments, thus possessing good engineering deployment capability. (2) From the perspective of power grid operation and scheduling optimization, it can reduce economic operating costs, while realizing zero out-of-bounds constraints on line power, ensuring the safe operation of the system in multi-topology and multi-scenario environments; by introducing an externally enhanced Markov decision process and an internal and external reward design, the present invention can effectively coordinate new energy power generation, energy storage charging and discharging and load demand, and improve the overall operating efficiency and economy of the power grid. Attached Figure Description

[0008] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a schematic diagram of a hybrid expert reinforcement learning power grid dispatch optimization method provided in an embodiment of the present invention.

[0010] Figure 2 This is a schematic diagram of the overall framework of the hybrid expert reinforcement learning power grid dispatch optimization method provided in the embodiments of the present invention.

[0011] Figure 3 A schematic diagram of an exogenously enhanced Markov decision process provided in an embodiment of the present invention.

[0012] Figure 4 The structural design diagram of the hybrid expert reinforcement learning network provided in the embodiments of the present invention.

[0013] Figure 5 Schematic diagram of multiple topologies and scenarios in the IEEE 123 power grid environment provided for embodiments of the present invention.

[0014] Figure 6 The figure shows the experimental results of the present invention in economic operation, as provided in the embodiments of the present invention.

[0015] Figure 7 The decision power of each unit and the power line for Task 0 provided in this embodiment of the invention.

[0016] Figure 8 This is a schematic diagram of a hybrid expert reinforcement learning power grid dispatch optimization system provided in an embodiment of the present invention.

[0017] Figure 9 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0020] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0021] The following provides a detailed description of a hybrid expert reinforcement learning-based power grid dispatch optimization method, system, equipment, and medium provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, conventional conditions or conditions recommended by the manufacturer in the art shall apply. Reagents or instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all commercially available conventional products.

[0022] Example 1 This invention provides a hybrid expert reinforcement learning-based power grid scheduling optimization method, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Construct a hybrid expert reinforcement learning network.

[0023] In this embodiment of the invention, the hybrid expert reinforcement learning network mainly includes: a policy network and a value network; both the policy network and the value network introduce a hybrid expert structure; the hybrid expert structure includes a gating network and M expert subnetworks, including one shared expert subnetwork and M-1 scenario expert subnetworks.

[0024] The gated network is used to calculate the routing weight of each scene expert subnetwork based on the current state and action, and to determine the correlation between each scene expert subnetwork and the current state through weight normalization. The top K-1 scene expert subnetworks with the highest correlation, along with the shared expert subnetwork, are all used as activated experts. Each activated expert takes the current state and action as input, and the outputs of all activated experts are weighted and summed according to the normalized weights to obtain the final feature representation.

[0025] The policy network combines the current state with the final feature representation to make action decisions; the value network combines the current state, action, and final feature representation to evaluate the state-action value.

[0026] Step 2: Unified modeling of power grids with multiple topologies and multiple operating scenarios.

[0027] In this embodiment of the invention, power grids with multiple topologies and operating scenarios are uniformly modeled as a power grid environment, and exogenous variables are introduced during the modeling process to characterize environmental uncertainties. The unified power grid modeling scheme is as follows:

[0028] (1) Constructing the power grid topology ,in, For the set of power grid nodes, The number of nodes; divide the nodes into multiple regions. , For the k-th region, Number of regions; The set of branches includes branches within the region and critical lines for power exchange across regions. , For the branch between node i and node j; configure the following equipment on each node: a set of thermal power generator sets. New energy power generation equipment collection Energy storage device collection Load node set and interconnection nodes for power exchange with the external power grid; among which , , , , , , , The corresponding entities are thermal power generators, new energy generators, energy storage devices, and load nodes.

[0029] (2) Modeling the power grid operation sequence: Discretize the power grid operation process into multiple time steps. T is the maximum time step; at each time step t, the power vector corresponding to each type of device is represented as follows: These represent the power of thermal power generators, the power of new energy generators, the power of energy storage charging and discharging, the load power, and the power exchanged with the external power grid, respectively.

[0030] (3) Introduce exogenous variables to characterize environmental uncertainty, including: load change factor Power factor of new energy generators Real-time electricity price The load power and the upper limit of the power of new energy generators are modeled as follows: , ;in: Indicates the rated load power; Indicates the maximum power of the new energy generator; This involves element-wise multiplication of vectors. This is the upper limit of the power output of new energy generators.

[0031] (4) Modeling power flow calculation, introducing a power flow solver to describe the mapping relationship between power injection into the grid and the hybrid expert reinforcement learning network, expressed as: ;in Node voltage; Transmit power to each critical line; For power exchange with the external power grid; This refers to the power flow operator in the power flow solver.

[0032] (5) Model the constraints of each physical quantity in the power grid.

[0033] (5.1) Voltage and line power constraints: ; ; in, These are the upper and lower bounds of the node voltage, respectively. Power of the critical circuit. This represents the upper limit of the power of the critical circuit.

[0034] (5.2) Power constraints of power generation equipment: ; in, These represent the upper and lower limits of the power output of thermal power generators.

[0035] (5.3) Power constraints of new energy generators: ; in, This represents the maximum power output of the new energy generator.

[0036] (5.4) Power and energy constraints of energy storage devices: ; ; in, These represent the upper and lower limits of energy storage capacity. These represent the upper and lower bounds of the energy storage state.

[0037] (5.5) The branch power flow adopts an approximate linearization model, and the non-convex constraints are relaxed by second-order cones; ; ; ; in, branch road Reactance, branch road Active and reactive power, branch road Active and reactive power, Let be the square of the voltage magnitudes at nodes i and j.

[0038] (6) Modeling the economic operation objective function: ; Where t is the time step t, The total cost for time step t, , The cost of a thermal power generator is described using a quadratic cost function. ,coefficient For second-order, first-order, and constant terms respectively, Let g be the power output of the thermal power generator at time step t; The cost of electricity generation for new energy generators is expressed as , This refers to the cost coefficient for electricity reduction in new energy generators. Reduce the power output of new energy generators; The cost of line loss is expressed as , branch road The resistance, branch road The square of the branch current amplitude This is the line loss coefficient; External electricity purchase cost, expressed as .

[0039] Step 3: Modeling the exogenous enhanced Markov decision process.

[0040] In this embodiment of the invention, the power grid environment state is divided into internal state and external state, thereby modeling the power grid dispatch optimization problem as an externally enhanced Markov decision process. In the externally enhanced Markov decision process: the internal state is the state directly affected by actions, and the external state is the state independent of actions. Actions are used to control the power of relevant equipment in the power grid environment. The reward includes internal reward and external reward. The internal reward is calculated based on the cost of each equipment, and the external reward is calculated based on the relationship between actions and the constraints of various physical quantities in the power grid.

[0041] (1) Modeling the power grid environment state and actions: Power grid environment state , It is an endogenous state. ; It is a matrix composed of normalized characteristics of load power, thermal power generator power, new energy generator power, and energy storage status in all regions of the power grid. To obtain the electrical value, the external state , and For future time steps t+h and H represents the set future time step; action ,in, For power regulation of thermal power generators, For the proportion of new energy generators connected to the grid, The ratio of energy storage charge and discharge power is converted into physically controllable power through linear mapping and trimming.

[0042] (2) In this embodiment of the invention, intrinsic reward .

[0043] Exogenous rewards .

[0044] ; ; ; ; in, To constrain the external rewards for the operation of thermal power generators, To constrain the external rewards for energy storage operations, To incentivize energy storage activities through external rewards, External rewards are used to penalize excessive line power. , , and All are set coefficients; For the out-of-bounds quantity of the thermal power generator g, The out-of-bounds amount of the new energy generator s. The power of the new energy generator s, The discharge preference factor is expressed as: , Discount aggregation is represented as: , For the normalized features at future time step t+h, L, R, and ele are external variables corresponding to load, new energy generator, and electricity price, respectively, and b is the global baseline, represented as... ; It is the mean of the external variable over the full step. Power of the critical circuit. This represents the upper limit of the power of the critical circuit.

[0045] Step 4: Training sample generation and storage.

[0046] In this embodiment of the invention, the policy network interacts with the power grid environment, and based on the exogenous reinforcement Markov decision process, training samples for reinforcement learning are collected and stored in the global experience pool.

[0047] Step 5: Network training.

[0048] In this embodiment of the invention, training samples are sampled from the global experience pool, and the value network and policy network are trained alternately.

[0049] In this embodiment of the invention, the overall logic of steps 4 and 5 above is as follows: Step A1: From the number of environmental interaction steps The following steps are executed: The policy network makes action decisions based on the current grid environment state at the current time step, thereby obtaining the next grid environment state. Intrinsic and extrinsic rewards are calculated to form the total reward. This reward, combined with the grid environment state, action, and reward at the corresponding time step and the grid environment state at the next time step, forms a reinforcement learning training sample, which is stored in the global experience pool. This process is repeated until the maximum number of sampling steps is reached. Proceed to step A2; Step A2: Update the number of steps from the gradient. The process begins by randomly sampling training samples from the global experience pool and using these samples to optimize the parameters of the evaluation network. Then, the number of gradient update steps is determined. If the policy update frequency is divisible by the preset policy update frequency, then the policy network parameters are optimized; otherwise, the gradient update steps are reduced. Increment by 1, then continue to randomly sample training samples from the global experience pool. Use these training samples to optimize the parameters of the evaluation network, repeating this process until the maximum gradient update step is reached. Proceed to step A3; Step A3: Perform steps A1 to A2 once, which completes one training iteration. If the number of training iterations has not reached the maximum number of training iterations... If the network training is successful, proceed to step A1; otherwise, the network training is complete.

[0050] In this embodiment of the invention, the alternating training of the value network and the policy network includes:

[0051] The value network employs a dual-Q network structure. The training loss is constructed by minimizing the temporal difference objective and the importance loss, and parameters are optimized accordingly. Represented as: ; in, For mathematical expectation, For the global experience pool, For the state and operation of the power grid environment, The state action value output by the v-th Q-network. For time-series difference objectives; This is a significant loss.

[0052] In this embodiment of the invention, the dual-Q network uses two independent Q functions or Q networks to estimate action value in reinforcement learning. By separating "action selection" and "action evaluation", the overestimation problem of Q value is reduced, making value evaluation more stable and accurate, thereby improving training stability and final policy performance.

[0053] The policy network employs a maximum entropy policy optimization framework, combining the state-action values ​​evaluated by the value network to calculate the policy loss (Actor loss), and incorporating the importance loss to construct the training loss. Parameters are then optimized using this framework. Represented as: ; in, For adaptive temperature parameters; Given the power grid environment state At that time, the action probability distribution output by the policy network, Given the power grid environment state At that time, the value network evaluates the state action values ​​of all actions.

[0054] Step 6: Deploy the application on the network.

[0055] The trained policy network is deployed in a real power grid scenario, and it makes action decisions based on the state of the real power grid scenario to achieve power grid scheduling optimization.

[0056] The above-described solution provided in the embodiments of the present invention mainly achieves the following beneficial effects: (1) From the perspective of intelligent control and generalization, this invention adopts a hybrid expert model combined with a reinforcement learning framework to realize the adaptive selection and dynamic update of the strategy under different power grid topologies and operating scenarios. It not only improves the cross-scenario generalization capability, but also supports batch experience pools and hybrid sampling training mechanisms, enabling the strategy to learn stably in heterogeneous environments, thus having good engineering deployment.

[0057] (2) From the perspective of power grid operation and dispatch optimization, it can reduce economic operating costs and achieve zero out-of-bounds constraints on line power, ensuring the safe operation of the system in multi-topology and multi-scenario environments. By introducing external enhanced Markov decision process and internal and external reward design, this invention can effectively coordinate new energy power generation, energy storage charging and discharging and load demand, and improve the overall operating efficiency and economy of the power grid.

[0058] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0059] I. Overall Overview of the Plan

[0060] This invention proposes a unified power grid modeling method for multiple topologies and operating scenarios, enabling the representation of power grid operating states. It introduces external state and external reward mechanisms, modeling the power grid dispatch optimization problem as an externally enhanced Markov decision process, thereby explicitly characterizing non-stationary factors such as environmental dynamics. Simultaneously, a policy network based on a hybrid expert structure is constructed, achieving adaptive decision-making for different tasks through a gating mechanism. Furthermore, this invention achieves cross-task training through a hybrid sampling mechanism and introduces an importance regularization term to further enhance the model's generalization ability. Thanks to these improvements, this invention can achieve power grid dispatch optimization that balances economic operation and safety across multiple topologies and operating scenarios.

[0061] II. Detailed introduction of the plan.

[0062] 1. Power grid modeling.

[0063] A directed graph topology model (power grid topology) is constructed, including thermal power generators, renewable energy generators, energy storage devices, and loads. Node and line power constraints, energy storage state of charge (SoC) constraints, and node voltage constraints are established. To characterize the operating characteristics and physical constraints of multi-source heterogeneous devices in complex power grid scenarios, this invention proposes a unified power grid modeling method for multiple topologies and operating scenarios. Through graph structure modeling, external variable modeling, power flow equation modeling, and constraint integration, it achieves a representation of the power grid's operating state. The overall process is as follows: Figure 2 As shown, it mainly includes:

[0064] (1) Modeling the power grid topology. Define the power grid system as a graphical model. ,in: A set of power grid nodes; the nodes are divided into multiple regions. , The set of branches includes branches within the region and critical lines for power exchange across regions. Configure the following units on each node: a collection of thermal power generating units. New energy power generation equipment collection Energy storage device collection Load node set And interconnection nodes for power exchange with the external power grid, etc.; among which , , , , , , , The corresponding entities are thermal power generators, new energy generators, energy storage devices, and load nodes.

[0065] (2) Modeling the power grid operation sequence. Discretize the power grid operation process into multiple time steps. At each time step t, the power vector corresponding to each type of device is represented as follows: These represent the power of thermal power generators, the power of new energy generators, the power of energy storage charging and discharging, the load power, and the power exchanged with the external power grid, respectively.

[0066] (3) Modeling exogenous variables. Exogenous variables are introduced to characterize environmental uncertainty, including: load variation factors. Power factor of new energy generators Real-time electricity price The load power and the power of the new energy generator are modeled as follows: , ;in: Indicates the rated load power; Indicates the maximum power of the new energy generator; This involves multiplying vectors element by element.

[0067] (4) Modeling power flow calculation. A power flow solver is introduced to describe the mapping relationship between power injection into the grid and the hybrid expert reinforcement learning network. The power flow solver can be used to directly solve the mixed integer second-order cone optimization problem, expressed as: ;in, Node voltage; Transmit power to each branch; The power that the power grid purchases from external sources; This refers to the power flow operator in the power flow solver.

[0068] (5) Model the constraints of each physical quantity in the power grid.

[0069] (5.1) Voltage and line power constraints: ; ; in, These are the upper and lower bounds of the node voltage, respectively. This represents the upper limit of the power of the critical circuit.

[0070] (5.2) Power constraints of power generation equipment: ; in, These represent the upper and lower limits of the power output of thermal power generators.

[0071] (5.3) Power constraints of new energy generators: ; in, This represents the maximum power output of the new energy generator.

[0072] (5.4) Power and energy constraints of energy storage devices: ; ; in, These represent the upper and lower limits of energy storage capacity. The upper and lower bounds of the SoC in the energy storage state.

[0073] (5.5) The branch power flow adopts an approximate linearization model, and the non-convex constraints are relaxed by second-order cones; ; ; ; in, branch road Reactance, branch road Active and reactive power, branch road Active and reactive power, Let be the square of the voltage magnitudes at nodes i and j.

[0074] 2. Modeling an exogenous augmented Markov decision process.

[0075] This invention proposes an exogenous augmented Markov decision process modeling method for power grid system optimization, such as... Figure 3 As shown, by dividing the power grid state into internal and external states, modeling the actions of controllable equipment and the influence of the external environment is achieved, thereby guiding the reinforcement learning strategy to complete the optimal scheduling of the power grid while maintaining physical constraints. The main points are as follows:

[0076] (1) Modeling the power grid environment state and actions. Based on the Markov decision process, external state and external reward are introduced to form an externally enhanced Markov decision process. The system state is divided into: affected by actions... Directly affected endogenous state External states that are independent of actions and driven by the external environment. The strategy objective is to maximize cumulative discount rewards, where the intrinsic reward is represented as... External rewards are recorded as : ; in, , As a discount factor, Let be the policy, representing the probability of taking action a in state s.

[0077] Power grid environment status It consists of intrinsic and extrinsic states. The intrinsic state is... .in, Including regional load Thermal power and new energy generator power Energy storage status The matrix composed of the normalized features, where k represents the region, contains the four types of normalized features of all regions mentioned above. The calculation method is as follows: ; in, For load nodes The load power, For load nodes The load normalization characteristics; Represents the k-th region. Represents the b-th region;

[0078] Similarly, normalized features It was obtained by normalizing in the same way.

[0079] External state is It consists of the load, power of new energy generators and electricity price forecasts for the next H steps, and the cross-day information usage cycle mapping ensures continuity.

[0080] The strategy outputs a continuous action vector. ,in, For power regulation of thermal power generators, For the proportion of new energy generators connected to the grid, For the energy storage charge / discharge power ratio, the range of values ​​for the above three items is constrained within... It is converted into physically controllable power through linear mapping and pruning.

[0081] (2) Map the controllable device actions to power.

[0082] (2.1) Operation of thermal power generator Mapped to power adjustment, where Active power The slope coefficient is denoted by `clip`, and the clipping operation is denoted by `clip`. ; ; in, This indicates the target power (or uncut temporary output) of the thermal power unit after adjustment based on the action, i.e., the initial active power. Based on this, the adjustment results were calculated according to the action ratio, maximum output, and gradient coefficient.

[0083] To measure the degree of action exceeding the bounds, the bounds quantity is... definition: .

[0084] (2.2) Operation of new energy generator Mapped to the grid connection ratio, the power of the new energy generator (grid-connected power) and the power reduction of new energy generators for: ; ; ; in, To utilize The calculated grid connection ratio is related to the upper limit of the power output of new energy generators. Element-wise multiplication yields grid-connected power. .

[0085] (2.3) Operation of energy storage equipment The positive and negative signs correspond to the charging and discharging power ratios, respectively. To maximize energy storage capacity, and These represent charge and discharge efficiencies, For time step: ; in, Indicates passage The calculated actual charging and discharging power of the energy storage device.

[0086] Energy Storage Status SoC Update: .

[0087] power Trimming ensures SoC constraints: .

[0088] (2.4) The power of the critical path is obtained through power flow calculation and used by the reward function.

[0089] (3) Calculate the intrinsic reward and extrinsic reward functions.

[0090] ; .

[0091] External rewards are mainly used to constrain the legal actions of the generator units and to incentivize the energy storage units to charge and discharge reasonably. They are calculated by combining the relationship between actions and constraints, so that the generator units can learn to act reasonably.

[0092] The strategy of generating physically feasible actions through negative rewards thus constrains the external rewards for the actions of the thermal generator. Defined as: ; in, This represents the out-of-bounds quantity of the thermal power generator g.

[0093] New energy generators do not involve internal state constraints, therefore the cost of curtailment is primarily reflected through primary rewards. Similar to thermal power generators, external rewards constrain energy storage actions. Defined as: ; in, The out-of-bounds amount of the new energy generator s; .

[0094] To alleviate the problem of motion degradation, this invention introduces preference rewards based on predictive information. External variables corresponding to load, renewable energy sources, and electricity prices are defined. Discounts are aggregated into ,in For a moment The normalized characteristics at the location (calculation method is described above). Based on this, a discharge preference factor is introduced. Where b is the global baseline, ensuring that rewards remain comparable across different time periods. It is the mean of the external variable over the entire step. Therefore, the external reward that incentivizes energy storage action is defined as: ; .

[0095] This incentive system provides positive rewards to energy storage systems during periods of high electricity prices, high load, and insufficient supply of renewable energy generators, thereby guiding strategy learning to align with the economic goals of system operation.

[0096] Exceeding line power limits induces system risks; therefore, the external reward for penalizing line power exceeding limits is defined as: .

[0097] The learning objective of the strategy is to maximize cumulative reward. : .

[0098] Those skilled in the art will understand that the environmental state should use the state of the previous time step (t-1) as the decision input. However, in reinforcement learning, it is assumed that after the decision of the previous action is made, the state will transition to the state of the next time step (t), which is s. t s t Used to generate actions The power P at time step t is obtained by mapping. t In other words, in the above (1) modeling of the power grid environment state and actions, the states involved are all states that have been transferred from the state of the previous time step t-1 to the state after time step t, and the actions at time step t are determined accordingly. This allows us to map the power at time step t, and then use the power flow solver to solve for information such as node voltage and critical line transmission power. After applying this to the power grid environment, we can obtain the state at the next time step t+1 and continue making action decisions at time step t+1.

[0099] For example: at t=0, state s0 is observed. Under state s0, action a0 is decided, mapped, and solved by the power flow solver. This action is then applied to the power grid environment, and the state transitions from s0 to s1. The reward r0 is calculated, thus completing the entire process at t=0. A similar process is executed at each subsequent time step, repeating continuously.

[0100] 3. Construct a hybrid expert reinforcement learning strategy network.

[0101] This step proposes a hybrid expert reinforcement learning method for modeling power grid dispatching policy networks. This method introduces a hybrid expert structure into the policy network and value network, enabling the model to dynamically adapt to different power grid topologies and various scenario environments. Simultaneously, it calculates expert weights through a gating network and employs a Top-K mechanism to select active experts, ensuring that the policy network utilizes expert resources in a balanced manner while maintaining performance. The network structure design is as follows: Figure 4 As shown, solid arrows represent forward propagation, and dashed arrows represent backward propagation.

[0102] (1) Construct a hybrid expert module.

[0103] The hybrid expert architecture comprises M expert subnetworks, each containing one shared expert and M-1 scenario experts. (Shared expert) Always participate in feature aggregation to capture consistent global control patterns across tasks; scene expert This is used to model the differences in local features under different topologies or scenarios. Through this structure, the model can simultaneously retain global control experience and adapt to changes in local scenarios.

[0104] The gating network G is based on the current input state and actions The routing weights for each expert are calculated. The network evaluates the adaptability of each input sample across different experts and normalizes the weights using the Softmax function. ,in, These represent the weights of the linear layers in the gated network. Next, the TopK activation mechanism selects the K-1 most relevant scene experts, which, together with the shared experts, form the activation expert set. : .

[0105] The outputs of the activation experts are summed by normalized weights to obtain the final feature representation. : ; in, Let q be the normalized weight of the activated expert. , They are respectively the qth activated expert, ... and the qth activated expert. The weights are output by the Softmax function of each activation expert; This is the output of the q-th activated expert.

[0106] The final feature representation obtained through the above formula In subsequent steps, the input policy network and value network will be used for action generation and state value assessment, thereby achieving adaptive control of complex power grid environments.

[0107] The above introduction uses the hybrid expert reinforcement learning network in the evaluation network as an example. For the policy network, the hybrid expert reinforcement learning network depends on the input state. The above process yields the final feature representation, which will not be elaborated here.

[0108] In this embodiment of the invention, each expert is an independent sub-neural network (such as a multilayer perceptron), and different experts learn different feature patterns. The activation indicates that the expert is selected and participates in the forward propagation calculation under the current sample.

[0109] (2) Design the Actor-Critic network.

[0110] The policy network (Actor) uses a hybrid expert structure to output continuous control actions. The policy is modeled using a Gaussian distribution, and the mean of the output actions is used. and variance Actions are sampled and tanh mapping is used to ensure that the actions are within the physically feasible range: ; ; in, This indicates that the random variable ϵ follows a Gaussian distribution with a mean of 0 and a variance of 1; MoEActor represents the processing procedure of the policy network, which includes the processing procedure of the hybrid expert reinforcement learning network mentioned above.

[0111] The Critic network evaluates the value of state-action pairs. A dual-Q network structure is employed to improve estimation stability. State-action values ​​are evaluated using a hybrid expert module. ; MoECritic refers to the evaluation network's processing procedure, which includes the processing procedure of the hybrid expert reinforcement learning network mentioned above.

[0112] The value output (state-action value) of the value network is used to construct the TD objective (temporal difference objective). To guide policy updates in the policy network: ; in, This represents the state / action value output by the Q-network for the v-th target. This represents the state and action of the next step (i.e., step t+1) during training; This is the temperature coefficient.

[0113] To avoid overusing some experts during training while others remain inactive for extended periods, this method introduces expert activation expectation and importance regularization terms to measure the balance of expert usage and incorporates them into the loss functions of the policy network and value network. This ensures effective utilization of each expert and mitigates training bias. The expert activation expectation and importance loss are defined as follows: ; ; in, Let q be the activation expectation of the activating expert; This is an indicator function; if the condition in square brackets is met, the output is 1; otherwise, the output is 0. To set coefficients, This indicates the calculation of the standard deviation of the expected value of all activations. This indicates that the mean of all expected activations is calculated.

[0114] The policy network employs a maximum entropy policy optimization framework, aiming to simultaneously improve long-term cumulative rewards and policy exploration.

[0115] .

[0116] The evaluation network is updated by minimizing the temporal difference objective error, while incorporating importance loss.

[0117] ; In the above formula, the adaptive temperature parameter It is used to balance strategy exploration and utilization, so that the strategy can be fully explored in the early stage of training and gradually converge to a stable decision in the later stage of training.

[0118]

[0119] in, The target entropy is preset; the temperature parameters are continuously optimized and adjusted using the above formula. .

[0120] 4. Complete training process.

[0121] Based on the aforementioned scheme, the following section introduces the hybrid expert reinforcement learning training method. To fully utilize multi-scene data and the hybrid expert structure, this invention uniformly stores all topology and scene samples into a global experience pool and performs batch training through a hybrid sampling mechanism. Samples from different tasks are averaged in the same training batch, improving training stability. Based on this mechanism, the specific steps of the training process are described in the algorithm steps shown in Table 1.

[0122] Table 1: Description of the Training Process

[0123] Step 4.1: Initialization steps.

[0124] In this step, we initialize the policy and network parameters: setting the policy and network parameters. Evaluate network parameters and its target network parameters Initialize the global experience pool (Replay Buffer). Training-related hyperparameters, including maximum number of training steps. Maximum number of sampling steps (maximum number of environmental interaction steps) and the maximum gradient update steps Strategy update frequency Target network update frequency wait.

[0125] From training iteration steps Begin by executing steps 4.2 to 4.3 until the maximum number of training steps is reached. .

[0126] Step 4.2: Generate and store samples.

[0127] In each training iteration step In the middle, from the number of environmental interaction steps Begin executing steps (1.1) to (1.2) below until the maximum number of sampling steps is reached. Then proceed to step 4.3.

[0128] (1.1) Sampling state and action: The policy network outputs the action. And mapped to specific unit power Then input the power flow calculation model to calculate The power grid is used to obtain the next state. .

[0129] (1.2) Calculate rewards and store experience: based on the current state ,action And the formula for calculating exogenous rewards, and the formula for calculating endogenous rewards. and external rewards And combine them into the total reward Then take the sample Store in the global experience pool This provides mixed data from multiple scenarios for subsequent training. Each environmental interaction step is considered a time step.

[0130] Step 4.3: Training the policy network and evaluating the network.

[0131] In each training iteration step In the middle, update the number of steps from the gradient. Begin executing steps (2.1) to (2.4) below until the maximum gradient update step is reached. After completing one training iteration, proceed to step 4.2.

[0132] (2.1) Batch sampling and calculation of TD targets. From the global experience pool Random sampling training experience And calculate the TD target value by evaluating the network: ; in, This is used to guide the value estimation update of the evaluation network.

[0133] (2.2) Update the evaluation network parameters. Calculate the training loss of the evaluation network, including the main loss and the importance loss: .

[0134] Update and evaluate network parameters based on gradient descent: ; In the above formula, For assignment operator, This is the gradient symbol.

[0135] (2.3) Update the policy network parameters. Determine the gradient update step number. Can it be updated by the policy frequency? If divisible, then the policy network loss is calculated using the following formula: .

[0136] Update network parameters according to the gradient descent strategy: ; Otherwise, the gradient update steps Add 1 and return to step (2.1) of step 4.3.

[0137] (2.4) Update the target network. Determine the gradient update step number. Can the target network update frequency? If divisible, the target network of the evaluation network is updated using the following formula: ; in, and These are all hyperparameters (learning rate) for gradient updates. The soft update coefficient represents the proportion of new information introduced from the current evaluation network parameters when updating the target network parameters.

[0138] Otherwise, the gradient update steps Add 1 and return to step (2.1) of step 4.3.

[0139] Through the above steps, unified training of the hybrid expert reinforcement learning model was achieved in multi-topology, multi-scenario power grid environments, ensuring that the policy network and value network maintain stability and generalization ability while making full use of diverse task samples. After training, the obtained policy network can be directly used for power grid optimization decision-making.

[0140] III. Example Introduction.

[0141] Based on the above scheme, the present invention can deploy the trained model in real-world scenarios. The following example, using an improved scenario based on the IEEE 123 standard power grid environment, will verify the practical application effect of the present invention. Specifically, as... Figure 5 As shown, five topologies (as shown in Part (a) and four operational scenarios (as shown in Part (b)) were designed, forming a total of twenty power grid dispatching tasks. The actual time series data used was granular at 15 minutes, including operational data such as load demand, photovoltaic and wind power generation, and electricity price changes.

[0142] The controllable unit configuration includes four thermal power generators, ten new energy generators, and three energy storage devices. In actual operation, the power grid is dispatched by invoking the policy network in the hybrid expert reinforcement learning network to achieve safe and economical operation under various tasks. Therefore, the preferred embodiment of the present invention is as follows.

[0143] Figure 6 The graph shows the experimental results of the present invention in economic operation. Line 1 represents the result of adding external rewards to the model, and line 2 represents the result of removing external rewards from the model. The vertical axis represents the negative value of the cost of economic operation of the power grid. The larger the value, the lower the cost of economic operation. Figure 7The diagram shows the decision power of each generator unit and the critical path power of the grid topology in Task 0. Part (a) shows the power of thermal power units: 1) it shows the real-time active power output of four thermal power generator units (Gen 0 - Gen 3) over 96 time steps (24 hours); 2) the unit power is dynamically adjusted with time steps. The method provided by this invention can adaptively allocate power output within the upper and lower bounds of the unit power based on load and electricity price fluctuations to minimize the total generation cost. Part (b) shows the power of renewable energy generators: 1) it shows the actual power injected into the grid by 10 renewable energy nodes (including photovoltaic and wind turbine units Sgen 0 - Sgen 9); 2) the model makes full use of renewable energy by reducing power cuts, and the output curve is consistent with the photovoltaic / wind power factor in the external environment. In addition, the decision power involved in Sgen2 and Sgen4 (i.e., lines b2 and b4) is exactly the same, so the two curves completely overlap. (c) is a diagram of energy storage charging and discharging power. 1) It shows the charging and discharging behavior of three energy storage systems (Storage 0-2). Positive values ​​represent charging, and negative values ​​represent discharging. 2) Under the guidance of external rewards, energy storage does not exhibit the "action degradation" phenomenon of continuously discharging to the minimum capacity and then failing. Instead, it learns the logic of charging during low demand / low electricity price and discharging during high demand / high electricity price. (d) is a diagram of energy storage state. 1) It shows the state of charge of the three energy storage systems, reflecting their energy surplus percentage (20%-80%). 2) The curve shows a significant increase in SoC during certain periods (such as after step size 48), corresponding to the charging stage; then it decreases, corresponding to the discharging stage supporting peak electricity demand. This proves that the model balances the long-term value of energy storage with real-time constraints. (e) is a diagram of external grid power purchase. 1) It shows the power purchased by the grid from the external main grid. 2) This power increases when internal thermal power and new energy sources are insufficient to cover the load or when the purchase cost is lower. Its peak value usually occurs during peak electricity consumption periods or periods when internal power generation is limited. (f) is a critical line power diagram, 1) showing the active power flow of 6 critical transmission lines (such as Line 13-18, 60-119, etc.); 2) strictly adhering to safety thresholds, due to the introduction of external rewards to punish over-limit behavior, even during periods of high operating pressure, the line power is successfully controlled within the preset upper limit, ensuring the operational safety of the power grid.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0145] Example 2 This invention also provides a hybrid expert reinforcement learning power grid dispatch optimization system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 8 As shown, the system mainly includes: A hybrid expert reinforcement learning network building unit is used to construct a hybrid expert reinforcement learning network that includes a policy network and a value network, wherein a hybrid expert structure is introduced into both the policy network and the value network. The power grid environment modeling unit is used to unify the modeling of power grids with multiple topologies and multiple operating scenarios into a power grid environment, and introduces external variables to characterize environmental uncertainties during the modeling process; The exogenous augmented Markov decision process modeling unit is used to divide the power grid environment state into intrinsic and extrinsic states, thereby modeling the power grid dispatch optimization problem as an exogenous augmented Markov decision process. In the exogenous augmented Markov decision process: intrinsic states are states directly affected by actions, while extrinsic states are states independent of actions. Actions are used to control the power of relevant equipment in the power grid environment. The reward includes intrinsic and extrinsic rewards. Intrinsic rewards are calculated based on the cost of each device, while extrinsic rewards are calculated based on the relationship between actions and constraints of various physical quantities in the power grid. The sample generation and storage unit is used to interact with the power grid environment through the policy network, collect training samples for reinforcement learning based on the externally enhanced Markov decision process, and store them in the global experience pool. The network training unit is used to sample training samples from the global experience pool and alternately train the value network and the policy network. The power grid dispatch optimization unit is used to deploy the trained policy network in the actual power grid scenario. It makes action decisions based on the state of the actual power grid scenario to achieve power grid dispatch optimization.

[0146] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0147] Example 3 The present invention also provides a processing device, such as Figure 9 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0148] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0149] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0150] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0151] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0152] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A hybrid expert reinforcement learning method for power grid dispatch optimization, characterized in that, include: Construct a hybrid expert reinforcement learning network that includes a policy network and a value network, wherein a hybrid expert structure is introduced into both the policy network and the value network; The power grid with multiple topologies and multiple operating scenarios is modeled as a unified power grid environment, and exogenous variables are introduced in the modeling process to characterize environmental uncertainties. The power grid environment is divided into internal and external states, and the power grid dispatch optimization problem is modeled as an external enhanced Markov decision process. In the externally enhanced Markov decision process: the internal state is the state directly affected by the action, while the external state is the state independent of the action. The action is used to control the power of relevant equipment in the power grid environment; the reward includes internal reward and external reward. The internal reward is calculated based on the cost of each device, while the external reward is calculated based on the relationship between the action and the constraints of various physical quantities in the power grid. By interacting with the power grid environment through the policy network, and based on the exogenous enhanced Markov decision process, training samples for reinforcement learning are collected and stored in the global experience pool. Training samples are sampled from the global experience pool, and the value network and policy network are trained alternately. The trained policy network is deployed in a real power grid scenario, and it makes action decisions based on the state of the real power grid scenario to achieve power grid scheduling optimization.

2. The hybrid expert reinforcement learning power grid dispatch optimization method according to claim 1, characterized in that, The hybrid expert structure includes a gated network and M expert subnetworks, including a shared expert subnetwork and M-1 scenario expert subnetworks; The gated network is used to calculate the routing weight of each scene expert sub-network based on the current state and action, and to determine the correlation between each scene expert sub-network and the current state through weight normalization. The top K-1 scene expert sub-networks with the highest correlation, along with the shared expert sub-network, are all used as active experts. Each activation expert takes its current state and action as input, and the outputs of all activation experts are weighted and summed according to normalized weights to obtain the final feature representation; The policy network combines the current state with the final feature representation to make action decisions; The value network combines the current state, action, and the final feature representation to evaluate the state-action value.

3. The hybrid expert reinforcement learning power grid dispatch optimization method according to claim 1, characterized in that, The process of unifying the modeling of power grids with multiple topologies and operating scenarios into a power grid environment includes: Constructing power grid topology ,in, For the set of power grid nodes, The number of nodes; divide the nodes into multiple regions. , For the k-th region, Number of regions; The set of branches includes branches within the region and critical lines for power exchange across regions. , For the branch between node i and node j; configure the following equipment on each node: a set of thermal power generator sets. New energy power generation equipment collection Energy storage device collection Load node set And communication nodes for power exchange with the external power grid; Modeling the power grid operation sequence: Discretizing the power grid operation process into multiple time steps T is the maximum time step; at each time step The power vectors corresponding to various types of equipment are represented as follows: These represent the power of thermal power generators, the power of new energy generators, the charging and discharging power of energy storage, the load power, and the power exchanged with the external power grid, respectively. Exogenous variables are introduced to characterize environmental uncertainty, including: load variation factors. Power factor of new energy generators Real-time electricity price The load power and the upper limit of the power of new energy generators are modeled as follows: , ;in: Indicates the rated load power; Indicates the maximum power of the new energy generator; This involves element-wise multiplication of vectors. This represents the upper limit of the power output of new energy generators. To model power flow calculations, a power flow solver is introduced to describe the mapping relationship between power injection into the grid and a hybrid expert reinforcement learning network, expressed as: ;in Node voltage; Transmit power to each critical line; For power exchange with the external power grid; For power flow operators in the power flow solver; Modeling the constraints of various physical quantities in the power grid; Modeling the economic objective function: ； Where t is the time step t, The total cost for time step t, , The cost of a thermal power generator is described using a quadratic cost function. ,coefficient For second-order, first-order, and constant terms respectively, Let g be the power output of the thermal power generator at time step t; The cost of electricity generation for new energy generators is expressed as , This refers to the cost coefficient for electricity reduction in new energy generators. Reduce the power output of new energy generators; The cost of line loss is expressed as , branch road The resistance, branch road The square of the branch current amplitude This is the line loss coefficient; External electricity purchase cost, expressed as .

4. The hybrid expert reinforcement learning power grid dispatch optimization method according to claim 3, characterized in that, The constraints on each physical quantity in the modeled power grid include: Voltage and line power constraints: ； ； in, These are the upper and lower bounds of the node voltage, respectively. This represents the upper limit of the critical circuit power. This represents the upper limit of the critical line power. Power constraints of power generation equipment: ； in, These represent the upper and lower limits of the power output of thermal power generators; Power constraints of new energy generators: ； in, This represents the maximum power output of the new energy generator. Energy storage device power and energy constraints: ； ； in, These represent the upper and lower limits of energy storage capacity. These are the upper and lower bounds of the energy storage state; The branch power flow adopts an approximate linear model, and second-order cone relaxation is used for non-convex constraints; ； ； ； in, branch road Reactance, branch road Active and reactive power, branch road Active and reactive power, Let be the square of the voltage magnitudes at nodes i and j.

5. The hybrid expert reinforcement learning power grid dispatch optimization method according to claim 3, characterized in that, The process of modeling the power grid dispatch optimization problem as an externally enhanced Markov decision process includes: Modeling power grid environment states and actions: Power grid environment states , It is an endogenous state. ; This represents the normalized characteristics of load power, thermal power generator power, new energy generator power, and energy storage status across all regions of the power grid. To obtain the electrical value, the external state , and For future time steps t+h and H represents the set future time step; action ,in, For power regulation of thermal power generators, For the proportion of new energy generators connected to the grid, To determine the energy storage charging and discharging power ratio, the action is converted into physically controllable power through linear mapping and trimming; Endogenous rewards external rewards ; ； ； ； ； in, To constrain the external rewards for the operation of thermal power generators, To constrain the external rewards for energy storage operations, To incentivize energy storage activities through external rewards, External rewards are used to penalize excessive line power. , , and All are set coefficients; For the out-of-bounds quantity of the thermal power generator g, The out-of-bounds amount of the new energy generator s. The power of the new energy generator s, The discharge preference factor is expressed as: , Discount aggregation is represented as: , For the normalized features at future time step t+h, L, R, and ele are external variables corresponding to load, new energy generator, and electricity price, respectively, and b is the global baseline, represented as... ; It is the mean of the external variable over the full step. Power of the critical circuit. This represents the upper limit of the power of the critical circuit.

6. The hybrid expert reinforcement learning power grid dispatch optimization method according to claim 1, characterized in that, The process involves interacting with the power grid environment through a policy network, collecting reinforcement learning training samples based on the exogenous enhanced Markov decision process, and storing them in a global experience pool. The process of sampling training samples from the global experience pool and alternately training the value network and policy network includes: Step A1: From the number of environmental interaction steps The following steps are executed: The policy network makes action decisions based on the current grid environment state at the current time step, thereby obtaining the next grid environment state. Intrinsic and extrinsic rewards are calculated to form the total reward. This reward, combined with the grid environment state, action, and reward at the corresponding time step and the grid environment state at the next time step, forms a reinforcement learning training sample, which is stored in the global experience pool. This process is repeated until the maximum number of sampling steps is reached. Proceed to step A2; Step A2: Update the number of steps from the gradient. The process begins by randomly sampling training samples from the global experience pool and using these samples to optimize the parameters of the evaluation network. Then, the number of gradient update steps is determined. If the policy update frequency is divisible by the preset policy update frequency, then the policy network parameters are optimized; otherwise, the gradient update steps are reduced. Increment by 1, then continue to randomly sample training samples from the global experience pool. Use these training samples to optimize the parameters of the evaluation network, repeating this process until the maximum gradient update step is reached. Proceed to step A3; Step A3: Perform steps A1 to A2 once, which completes one training iteration. If the number of training iterations has not reached the maximum number of training iterations... If the network training is successful, proceed to step A1; otherwise, the network training is complete.

7. A hybrid expert reinforcement learning power grid dispatch optimization method according to claim 1 or 6, characterized in that, The alternating training of the value network and the policy network includes: The value network employs a dual-Q network structure. The training loss is constructed by minimizing the temporal difference objective and the importance loss, and parameters are optimized accordingly. Represented as: ； Where t is the time step, For mathematical expectation, For the global experience pool, For the state and operation of the power grid environment, The state action value output by the v-th Q-network. For time-series difference objectives; This is a significant loss; The policy network employs a maximum entropy policy optimization framework, combining the state-action values ​​evaluated by the value network with the importance loss to construct the training loss, thereby optimizing the parameters. This results in the training loss of the policy network. Represented as: ； in, For adaptive temperature parameters; Given the power grid environment state At that time, the action probability distribution output by the policy network, Given the power grid environment state At that time, the value network evaluates the state action values ​​of all actions.

8. A hybrid expert reinforcement learning power grid dispatch optimization system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: A hybrid expert reinforcement learning network building unit is used to construct a hybrid expert reinforcement learning network that includes a policy network and a value network, wherein a hybrid expert structure is introduced into both the policy network and the value network. The power grid environment modeling unit is used to unify the modeling of power grids with multiple topologies and multiple operating scenarios into a power grid environment, and introduces external variables to characterize environmental uncertainties during the modeling process; The exogenous augmented Markov decision process modeling unit is used to divide the power grid environment state into intrinsic and extrinsic states, thereby modeling the power grid dispatch optimization problem as an exogenous augmented Markov decision process. In the exogenous augmented Markov decision process: intrinsic states are states directly affected by actions, while extrinsic states are states independent of actions. Actions are used to control the power of relevant equipment in the power grid environment. The reward includes intrinsic and extrinsic rewards. Intrinsic rewards are calculated based on the cost of each device, while extrinsic rewards are calculated based on the relationship between actions and constraints of various physical quantities in the power grid. The sample generation and storage unit is used to interact with the power grid environment through the policy network, collect training samples for reinforcement learning based on the externally enhanced Markov decision process, and store them in the global experience pool. The network training unit is used to sample training samples from the global experience pool and alternately train the value network and the policy network. The power grid dispatch optimization unit is used to deploy the trained policy network in the actual power grid scenario. It makes action decisions based on the state of the actual power grid scenario to achieve power grid dispatch optimization.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images