A method, device, system, and medium for power grid load frequency control based on large-scale multi-agent deep meta-reinforcement learning

Abstract

This invention provides a method, device, system, and medium for power grid load frequency control based on large-scale multi-agent deep meta-reinforcement learning. It constructs a cooperative load frequency control (CC-LFC) framework by mimicking a squid distributed neural network, modeling each regional controller and power distributor as agents. A large-scale counterfactual multi-agent deep meta-policy gradient (LSCMA-DMPG) algorithm is proposed, integrating centralized training with distributed execution and meta-reinforcement learning to achieve multi-task collaborative learning. Validated using a microgrid model of the four districts of Sansha Island in the Southern Power Grid, this method outperforms existing deep reinforcement learning algorithms and traditional centralized strategies in reducing frequency deviation, suppressing tie-line power fluctuations, and lowering power generation costs, significantly improving the reliability and economy of multi-regional microgrid operation.

Description

Technical Field

[0001] This invention relates to the field of power system operation and control technology, specifically a power grid load frequency control method, device, system, and medium based on large-scale multi-agent deep meta-reinforcement learning. Background Technology

[0002] In recent years, to address the dual pressures of climate change and fossil fuel depletion, power systems are accelerating their transition to a higher proportion of renewable energy. At the ends of the power grid, such as islands and remote mining areas, isolated multi-regional microgrids (MAIMs) composed of wind power, solar power, and energy storage have become the mainstream power supply model. These systems achieve inter-regional mutual support through tie lines, effectively mitigating the randomness and volatility of renewable energy output. However, they place higher demands on load frequency control (LFC): frequency deviation must be ≤ ±0.2 Hz (NERC BAL-001-1), tie line power fluctuation must be < ±3% of rated capacity, and the control cycle must be shortened to within 1 second.

[0003] Existing MAIM-LFC solutions are mainly divided into two categories:

[0004] 1. Centralized hierarchical architecture: Measurements from each region are uploaded to the central controller, where a single optimizer calculates the ACE and issues adjustment commands. CN109123456A (2019-08-20) uses an upper-layer PI + lower-layer interior-point method. In high-frequency sampling scenarios, the average iteration is >300 times, with a command delay of 5–8 seconds, resulting in a frequency deviation >0.08 Hz, which fails to meet the 0.05Hz limit requirement of GB / T 19964-2012; furthermore, a single point of failure in the central node can cause the entire system to crash.

[0005] 2. Fully Distributed Architecture: Each region makes independent decisions and exchanges marginal cost information through a consensus algorithm. While the consensus transitive Q-learning (CTQ) proposed in some literature reduces communication, it requires approximately 150-200 iterations to converge in a high-dimensional continuous action space, and the power of the connection line only returns to zero after 150 seconds, making it difficult to adapt to second-level control timing.

[0006] Furthermore, traditional deep reinforcement learning methods such as MADDPG and MATD3 still exhibit a frequency deviation >0.06 Hz after 60,000 training steps in the MAIM scenario, and their robustness to 30% wind power step disturbances decreases by >25%, failing to distinguish individual contributions and resulting in excessively large policy gradient variance.

[0007] In summary, existing technologies cannot simultaneously achieve: (1) global optimality; (2) fast convergence (iterations < 50); (3) high robustness (frequency deviation ≤ 0.03 Hz); and (4) plug-and-play within a 1 s control cycle. Therefore, a novel LFC architecture and algorithm are urgently needed to break through the binary opposition of centralized and distributed systems, complete multi-region collaborative optimization within a second-level time limit, and possess adaptive capabilities to strong random disturbances. Summary of the Invention

[0008] To address the aforementioned problems in existing technologies, this invention proposes a power grid load frequency control method, device, system, and medium based on large-scale multi-agent deep meta reinforcement learning. It provides a Cuttlefish-like Cooperative Load Frequency Control (CC-LFC) method based on a Cuttlefish-like distributed neural network structure. This method models the controllers and power distributors in each region as agents by simulating the distributed decision-making mechanism of a Cuttlefish, and employs the Large-Scale Counterfactual Multi-Agent Deep Meta Policy Gradient (LSCMA-DMPG) algorithm to achieve centralized training and distributed execution.

[0009] The present invention adopts the following technical solution:

[0010] A power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning includes the following steps:

[0011] Step (1): Construct dynamic response models of control agents and scheduling agents for each region of the multi-region microgrid. Configure control agents and scheduling agents for each region to collect the local status of each region during the current control cycle.

[0012] Step (2): Define the optimization problem for training the LSCMA-DMPG policy network, and complete the offline training of the policy network based on the centralized training and distributed execution framework:

[0013] Step (3): Collect real-time local status and calculate the area control error ACE according to the NERC BAL-001-1 standard;

[0014] Step (4): Based on the ACE obtained in step (3), update the local state spaces of the control agent and the scheduling agent, and input them into the trained LSCMA-DMPG policy network to output the optimal overall adjustment instruction. and unit participation factors ;

[0015] (5) Control each unit and energy storage system to execute the optimal overall regulation command. And when the next control cycle arrives, return to step (3) to repeat the closed-loop process of data acquisition-decision generation-execution.

[0016] Furthermore, in step (1), each region is equipped with 2n control agents and scheduling agents to replace the traditional centralized controller, and each agent only collects local status.

[0017] Furthermore, in step (1), the local state includes frequency deviation Δf and tie-line power ΔP. ij and unit output ΔP G .

[0018] Furthermore, step (2) specifically includes:

[0019] a) Definition and Role of the Optimization Problem: Define the objective function and constraints of the optimization problem. The objective function is used to minimize the system frequency deviation and the total power generation cost. Its core elements are directly transformed into the reward function of the agent to guide the agent's collaborative learning. The constraints include the ramp-up rate limit of the power generation unit and the regional power balance requirement. The constraints are structurally implemented through the normalization design of the agent's action space, and the penalty term is set in the reward function to punish the constraint violation.

[0020] b) Training architecture and process: A two-layer architecture including meta-learners and basic learners is adopted for centralized training;

[0021] The basic learner runs in multiple parallel systems, with its control and scheduling agents acting as explorers, performing actions and collecting experience samples in the environment; the learners are trained in a centralized manner, using global information to update the critic network and the actor network based on policy gradients; by introducing a counterfactual credit allocation mechanism, the individual contribution of each agent is accurately evaluated.

[0022] The meta-learner uses a deep Q-network to dynamically generate optimal exploration noise parameters for different perturbation tasks and inject them into the base learner to achieve adaptive exploration and multi-task collaborative learning.

[0023] c) Policy network acquisition: After sufficient offline training under the above architecture, when the average reward of the agent converges or reaches the preset training period, a mature LSCMA-DMPG policy network is obtained.

[0024] Furthermore, the objective function and constraints of the dynamic response model are set, wherein the objective function adopts a non-quadratic form:

[0025] (2);

[0026] In the formula, For the target in the j-th region, Δf j Let ΔP be the frequency deviation of the j-th region. Gi For the output of the i-th unit, ΔP ij For tie-line power, α i β i γ i Conversion factor;

[0027] The constraints include the ramp rate constraint and power balance constraint of the power generation unit:

[0028] (3);

[0029] Where ΔP i rate It is the ramp rate of the i-th unit, ΔP order-Σ It is the command for the i-th unit, ΔP i min It is the minimum technical output of the i-th unit, ΔP i max It is the maximum technical output of the i-th unit, ΔP Gi This is the output of the i-th unit.

[0030] Furthermore, step (4) specifically includes:

[0031] Step 4.1: Initialize algorithm parameters and agent network

[0032] Initialize the network parameters of all control agents and schedulers (Actor and Critic) using a random distribution, including Xavier initialization or He initialization.

[0033] Set the training hyperparameters, where: the critic network learning rate α critic The value ranges from 0.001 to 0.0001; the actuator network learning rate α actor The value ranges from 0.0001 to 0.00001; the discount factor γ is 0.99; the capacity of the experience replay pool is set to 1×10⁶, 5×10⁶ or any value between these values ​​for samples.

[0034] Initialize N parallel systems, where N is an integer, and load perturbation scenarios of different magnitudes are applied to each system to simulate a multi-task learning environment;

[0035] Step 4.2: Exploration Phase

[0036] In each parallel system, the explorer (i.e., the control agent and the scheduling agent) determines the current system state s based on the current system state. t The action a is output through its actuator network μ(·). t :

[0037] (4);

[0038] Here, OU Noise is the exploration noise generated by the Ornstein-Uhlenbeck process, and its variance σ is dynamically provided by the meta-learner to achieve adaptive exploration for different perturbation scenarios;

[0039] Step 4.3: Experience Collection and Storage

[0040] Each explorer will generate a state transition tuple from its interaction with the environment. Stored in the shared experience replay pool, where r t For instant rewards;

[0041] Step 4.4: Learning Phase

[0042] The learner randomly samples a batch of data from the experience pool and updates the network according to the following steps:

[0043] a) Update the Critic network:

[0044] The joint action-value function is estimated using a centralized critic network Q(·) by minimizing the mean squared error loss function L(θ). Q To update network parameters:

[0045] (5);

[0046] in, , and These are the target commentator network and the target executor network, respectively.

[0047] b) Update the Actor network:

[0048] Update the actuator network parameters for each agent using policy gradient ascent:

[0049] (6);

[0050] c) Counterfactual credit allocation:

[0051] A counterfactual benchmark module is introduced to achieve credit allocation by constructing counterfactual scenarios: the action of agent i is replaced with a default reference action, and the Q-value in the scenario is calculated. The advantage function of agent i is then approximated as: This allows for more precise guidance in updating their strategies;

[0052] Step 4.5: Meta-learner parameter injection

[0053] Obtain exploration noise parameters optimized for the current perturbation scenario from the meta-learner. And dynamically injected into the explorers of each parallel system to replace the default noise parameters;

[0054] Step 4.6: Policy Synchronization and Deployment

[0055] The learner's updated actuator network parameters are transmitted via a parameter server or shared memory mechanism. Synchronize explorers across all parallel systems for the next round of interaction and data collection;

[0056] Step 4.7: Termination Condition Determination

[0057] If the number of training rounds reaches the preset maximum value, or the rate of change of the average reward for several consecutive rounds is lower than the preset threshold, the strategy is determined to have converged, training is terminated and the final network parameters are output; otherwise, return to step (2) to continue training.

[0058] Furthermore, in step 4.7, the preset maximum value for the number of training rounds ranges from 5000 to 10000 rounds, and the preset threshold for the rate of change of the average reward per round is... .

[0059] Furthermore, a power grid load frequency control device based on large-scale multi-agent deep meta-reinforcement learning is characterized by comprising the following steps:

[0060] The dynamic response model construction module is used to construct dynamic response models of control agents and scheduling agents in each region of a multi-region microgrid. Each region is configured with control agents and scheduling agents to collect the local status of each region in the current control cycle.

[0061] The policy network training module defines the optimization problem for training the LSCMA-DMPG policy network and completes the offline training of the policy network based on a centralized training and distributed execution framework.

[0062] The area control error calculation module is used to collect real-time local status and calculate the area control error ACE according to the NERC BAL-001-1 standard.

[0063] The global optimal decision generation module updates the local state spaces of the control agent and the scheduling agent based on the obtained ACE, inputs the control agent and the scheduling agent into the pre-trained LSCMA-DMPG policy network, and outputs the optimal overall adjustment instruction. and unit participation factors ;

[0064] The optimal decision execution module is used to control each generating unit and energy storage system to execute the optimal overall regulation command. It then returns to the area control error calculation module when the next control cycle arrives, repeating the closed-loop process of data acquisition-decision generation-execution.

[0065] A power grid load frequency control system based on large-scale multi-agent deep meta-reinforcement learning includes: a computer-readable storage medium and a processor;

[0066] The computer-readable storage medium is used to store executable instructions;

[0067] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning.

[0068] A non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning.

[0069] This invention achieves the following technical effects: the system can complete global optimization decisions within a 1-second control cycle, controlling the frequency deviation within 0.03 Hz under a wind power step disturbance of 30% of rated capacity, while significantly reducing the total power generation cost. Experimental verification based on the Sansha Island four-zone microgrid model of the China Southern Power Grid shows that this method outperforms traditional centralized and fully distributed LFC strategies in terms of frequency stability, dynamic response speed, and economy. Attached Figure Description

[0070] Figure 1 This is a schematic diagram of the CC-LFC framework of the present invention;

[0071] Figure 2 This is a schematic diagram of CC-LFC's MDP modeling;

[0072] Figure 3 This is a schematic diagram of the LSCMA-DMPG training framework;

[0073] Figure 4 This is a flowchart of the meta-learner training process;

[0074] Figure 5 This is a flowchart of the distributed training process for the basic learner;

[0075] Figure 6 This is a flowchart of CC-LFC based on parallel systems;

[0076] Figure 7This is a diagram of a four-region transfer function model;

[0077] Figure 8 This is a flowchart of a power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning according to the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Please see Figure 8 The first aspect of this invention provides a grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning. This embodiment takes an islanded microgrid system comprising four interconnected regions as an example to describe the solution in detail. Each region is equipped with various distributed power sources (such as diesel generators, photovoltaic power generation, and wind power generation) and energy storage devices. The method includes the following steps:

[0080] (1) Build dynamic response models of controllers and power dividers in each region of a multi-region microgrid.

[0081] In this embodiment, the traditional controller and power distributor of each region are constructed as two autonomous agents based on the distributed neural architecture principle of an octopus: a control agent and a dispatch agent. A total of 2n=8 agents are configured (n=4).

[0082] Each agent only collects local state information for its own region, as follows: (1)

[0083] The principle of this biomimetic distributed architecture is as follows: Figure 1 (As shown in the CC-LFC framework). To achieve reinforcement learning training, a Markov Decision Process (MDP) model needs to be built for each agent. For example... Figure 2 As shown, this model clarifies the design of the state space, action space, and reward function of the controlling agent and the scheduling agent, providing a formal framework for subsequent policy learning. The specific system structure is as follows: Figure 7 (Four-region transfer function model) is shown.

[0084] (2) Set the objective function and constraints

[0085] The optimization objective in this embodiment is to minimize the overall operating cost of the entire multi-region microgrid while satisfying all system operational constraints. Objective function Set as the comprehensive operating cost of the j-th region.

[0086] Preferably, the objective function is in a non-quadratic form:

[0087] (2)

[0088] In the formula For the target in the j-th region, Δf j Let ΔP be the frequency of the j-th region. Gi For the output of the i-th unit, ΔP ij For tie-line power, α i β i γ i This is the conversion factor.

[0089] The constraints are as follows:

[0090] (3)

[0091] To efficiently train multi-agent cooperative strategies, this embodiment adopts... Figure 3 The LSCMA-DMPG training framework is shown. This framework adopts a "centralized training, distributed execution" architecture, which includes a meta-learner and a base learner. Through parallel system exploration, experience replay, and counterfactual credit assignment, it jointly optimizes the agent's policy.

[0092] This mainly includes ramp-up rate constraints and power balance constraints for each power generation unit. Among them, It is the climbing rate limit of the i-th unit. This is the total regulation power command required by the system. This constraint ensures that the power adjustment speed of each unit is within its physical limits, and that the sum of the power adjustments of all units equals the total system demand. In this embodiment, the specific system ramp rate ranges for each region are: Region A [-29.9, 29.9] kW / s, Region B [-17.8, 17.8] kW / s, Region C [-25.1, 25.1] kW / s, Region D [-22.2, 22.2] kW / s, and the total ramp rate range for the entire system is [-94.8, 94.8] kW / s.

[0093] (3) Collect real-time data and calculate the area control error (ACE).

[0094] In each control cycle (e.g., 2-4 seconds), real-time frequency deviation Δf and tie-line power deviation ΔP are collected from each area using sensors and monitoring systems deployed in each area. tie (i.e., ΔP)ij ) and the real-time output P of the unit G According to NERC standard BAL-001-1, the area control error (ACE) is calculated. This error quantifies the degree of imbalance between generation and load, and is typically correlated with frequency deviation Δf and tie-line power deviation ΔP. tie Related.

[0095] (4) Update the state and input the LSCMA-DMPG policy network to generate decisions.

[0096] The state information such as ACE calculated in step (3) is input into the offline-trained LSCMA-DMPG policy network. This network outputs two major decision variables:

[0097] 1. Total regulation power command required for this region Its expression is referenced in formula (4); as follows:

[0098] (4)

[0099] in This represents the action output of an agent in the j-th region.

[0100] 2. Participation Factor of Each Power Generation Unit in the Region This allows the general instructions to be precisely assigned to each unit.

[0101] The formula is as follows:

[0102] (5)

[0103] The key implementation details of the LSCMA-DMPG algorithm in this embodiment are as follows:

[0104] Counterfactual learning mechanism: To solve the problem of credit allocation among multiple agents, the algorithm constructs counterfactual benchmarks during training;

[0105] Meta-learner dynamic parameter tuning: The meta-learner (a DDQN agent) outputs optimized OU exploration noise parameters (variance σ and attenuation coefficient β) for the current perturbation scenario; its specific training process is as follows... Figure 4 As shown, the process includes steps such as noise parameter initialization, interaction with the LFC agent, experience collection, loss calculation and parameter updating. Finally, the optimized exploration parameters are dynamically injected into the basic learner to achieve adaptive exploration.

[0106] Offline Training and Online Execution: The algorithm employs a centralized training and distributed execution (CTDE) framework. In the offline phase (the process is detailed below),... Figure 5 ), utilizing historical data for large-scale centralized training. In the online phase (the process is detailed in [link to process]), Figure 6 The agent makes decisions independently based solely on its local state, without the need for real-time communication between regions. The updates of the agent network follow the policy gradient rule (the expression of which is shown in Equation (6)), while the updates of the Critic network are based on minimizing the time difference error (the expression of which is shown in Equation (7)).

[0107] The formula is as follows:

[0108] (6) (7)

[0109] In the formula, T represents the length of the episode, h t i Let G represent the implicit state of the i-th actor network, and a represent the joint action, where G... t λ Indicates time difference target.

[0110] (5) Execute control commands and cycle.

[0111] The optimal power allocation command generated in step (4) The commands are then distributed to each generating unit and energy storage system for execution. After one control cycle ends, the process returns to step (3) to begin a new round of data acquisition, decision generation, and command execution, forming a closed-loop control.

[0112] The CC-LFC method based on the LSCMA-DMPG algorithm provided by this invention has the following advantages compared with the prior art:

[0113] (1) This invention proposes a biomimetic collaborative load frequency control (CC-LFC) method, which imitates the distributed neural network structure of squid and models each area controller and power distributor as intelligent agents, realizing autonomous collaborative optimization of multi-area microgrids, overcoming the problem of inconsistent targets and difficulty in coordination between controller and distributor in traditional centralized control, and significantly improving the frequency stability and response speed of the system.

[0114] (2) This invention proposes a large-scale counterfactual multi-agent deep meta-policy gradient (LSCMA-DMPG) algorithm, which combines centralized training with distributed execution and introduces a meta-reinforcement learning mechanism to achieve multi-task collaborative learning and rapid adaptation capabilities. This algorithm utilizes global information for policy evaluation during training and relies only on local information during execution, ensuring system coordination while reducing communication burden. Compared to traditional algorithms such as MADDPG, it exhibits stronger convergence and robustness.

[0115] (3) The present invention is verified by simulation and experiment of actual four-region microgrid. The proposed method can significantly reduce frequency deviation, tie line power fluctuation and total power generation cost under various complex operating conditions such as random disturbance and random renewable energy fluctuation. The frequency regulation performance is improved by more than 95% and the power generation cost is reduced by 1.45%, which is better than 16 existing centralized LFC methods and 3 mainstream deep reinforcement learning algorithms, demonstrating excellent engineering applicability and economy.

[0116] A second aspect of the present invention provides a power grid load frequency control device based on large-scale multi-agent deep meta-reinforcement learning, comprising the following steps:

[0117] The dynamic response model construction module is used to construct dynamic response models of control agents and scheduling agents in each region of a multi-region microgrid. Each region is configured with control agents and scheduling agents to collect the local status of each region in the current control cycle.

[0118] The policy network training module defines the optimization problem for training the LSCMA-DMPG policy network and completes the offline training of the policy network based on a centralized training and distributed execution framework.

[0119] The area control error calculation module is used to collect real-time local status and calculate the area control error ACE according to the NERC BAL-001-1 standard.

[0120] The global optimal decision generation module updates the local state spaces of the control agent and the scheduling agent based on the obtained ACE, inputs the control agent and the scheduling agent into the pre-trained LSCMA-DMPG policy network, and outputs the optimal overall adjustment instruction. and unit participation factors ;

[0121] The optimal decision execution module is used to control each generating unit and energy storage system to execute the optimal overall regulation command. It then returns to the area control error calculation module when the next control cycle arrives, repeating the closed-loop process of data acquisition-decision generation-execution.

[0122] Another aspect of the present invention provides a power grid load frequency control system based on large-scale multi-agent deep meta-reinforcement learning, comprising: a computer-readable storage medium and a processor;

[0123] The computer-readable storage medium is used to store executable instructions;

[0124] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in the first aspect.

[0125] In another aspect, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in the first aspect.

[0126] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0128] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0129] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning, characterized in that, Includes the following steps: Step (1): Construct dynamic response models of control agents and scheduling agents for each region of the multi-region microgrid. Configure control agents and scheduling agents for each region to collect the local status of each region during the current control cycle. Step (2): Define the optimization problem for training the LSCMA-DMPG policy network, and complete the offline training of the policy network based on the centralized training and distributed execution framework: Step (3): Collect real-time local status and calculate the area control error ACE according to the NERC BAL-001-1 standard; Step (4): Based on the ACE obtained in step (3), update the local state spaces of the control agent and the scheduling agent, and input them into the trained LSCMA-DMPG policy network to output the optimal overall adjustment instruction. and unit participation factors ; (5) Control each unit and energy storage system to execute the optimal overall regulation command. And when the next control cycle arrives, return to step (3) to repeat the closed-loop process of data acquisition-decision generation-execution.

2. The power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in claim 1, characterized in that, In step (1), 2n control agents and scheduling agents are set up in each region to replace the traditional centralized controller. Each agent only collects local status.

3. The power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in claim 1, characterized in that, In step (1), the local state includes frequency deviation Δf and tie-line power ΔP. ij and unit output ΔP G .

4. The power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in claim 1, characterized in that, Step (2) specifically includes: a) Definition and Role of the Optimization Problem: Define the objective function and constraints of the optimization problem. The objective function is used to minimize the system frequency deviation and the total power generation cost. Its core elements are directly transformed into the reward function of the agent to guide the agent's collaborative learning. The constraints include the ramp-up rate limit of the power generation unit and the regional power balance requirement. The constraints are structurally implemented through the normalization design of the agent's action space, and the penalty term is set in the reward function to punish the constraint violation. b) Training architecture and process: A two-layer architecture including meta-learners and basic learners is adopted for centralized training; The basic learner runs in multiple parallel systems, with its control and scheduling agents acting as explorers, performing actions and collecting experience samples in the environment; the learners are trained in a centralized manner, using global information to update the critic network and the actor network based on policy gradients; by introducing a counterfactual credit allocation mechanism, the individual contribution of each agent is accurately evaluated. The meta-learner uses a deep Q-network to dynamically generate optimal exploration noise parameters for different perturbation tasks and inject them into the base learner to achieve adaptive exploration and multi-task collaborative learning. c) Policy network acquisition: After sufficient offline training under the above architecture, when the average reward of the agent converges or reaches the preset training period, a mature LSCMA-DMPG policy network is obtained.

5. The power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in claim 4, characterized in that, Set the objective function and constraints for the dynamic response model, where the objective function is in a non-quadratic form: （2）； In the formula, For the target in the j-th region, Δf j Let ΔP be the frequency deviation of the j-th region. Gi For the output of the i-th unit, ΔP ij For tie-line power, α i β i γ i Conversion factor; The constraints include the ramp rate constraint and power balance constraint of the power generation unit: （3）； Where ΔP i rate It is the ramp rate of the i-th unit, ΔP order-Σ It is the command for the i-th unit, ΔP i min It is the minimum technical output of the i-th unit, ΔP i max It is the maximum technical output of the i-th unit, ΔP Gi This is the output of the i-th unit.

6. The power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in claim 1, characterized in that, Step (4) specifically includes: Step 4.1: Initialize algorithm parameters and agent network Initialize the network parameters of all control agents and schedulers (Actor and Critic) using a random distribution, including Xavier initialization or He initialization. Set the training hyperparameters, where: the critic network learning rate α critic The value ranges from 0.001 to 0.0001; the actuator network learning rate α actor The value ranges from 0.0001 to 0.00001; the discount factor γ is 0.99; the capacity of the experience replay pool is set to 1×10⁶, 5×10⁶ or any value between these values ​​for samples. Initialize N parallel systems, where N is an integer, and load perturbation scenarios of different magnitudes are applied to each system to simulate a multi-task learning environment; Step 4.2: Exploration Phase In each parallel system, the explorer (i.e., the control agent and the scheduling agent) determines the current system state s based on the current system state. t The action a is output through its actuator network μ(·). t : （4）； Here, OU Noise is the exploration noise generated by the Ornstein-Uhlenbeck process, and its variance σ is dynamically provided by the meta-learner to achieve adaptive exploration for different perturbation scenarios; Step 4.3: Experience Collection and Storage Each explorer will generate a state transition tuple from its interaction with the environment. Stored in the shared experience replay pool, where r t For instant rewards; Step 4.4: Learning Phase The learner randomly samples a batch of data from the experience pool and updates the network according to the following steps: a) Update the Critic network: The joint action-value function is estimated using a centralized critic network Q(·) by minimizing the mean squared error loss function L(θ). Q To update network parameters: （5）； in, , and These are the target critic network and the target executor network, respectively. b) Update the Actor network: Update the actuator network parameters for each agent using policy gradient ascent: （6）； c) Counterfactual credit allocation: A counterfactual benchmark module is introduced to achieve credit allocation by constructing counterfactual scenarios: the action of agent i is replaced with a default reference action, and the Q-value in the scenario is calculated. The advantage function of agent i is then approximated as: This allows for more precise guidance in updating their strategies; Step 4.5: Meta-learner parameter injection Obtain exploration noise parameters optimized for the current perturbation scenario from the meta-learner. And dynamically injected into the explorers of each parallel system to replace the default noise parameters; Step 4.6: Policy Synchronization and Deployment The learner's updated actuator network parameters are transmitted via a parameter server or shared memory mechanism. Synchronize explorers across all parallel systems for the next round of interaction and data collection; Step 4.7: Termination Condition Determination If the number of training rounds reaches the preset maximum value, or the rate of change of the average reward for several consecutive rounds is lower than the preset threshold, the strategy is determined to have converged, training is terminated and the final network parameters are output; otherwise, return to step (2) to continue training.

7. The power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in claim 6, characterized in that, In step 4.7, the preset maximum value for the number of training rounds ranges from 5000 to 10000 rounds, and the preset threshold for the rate of change of the average reward per round is... .

8. A power grid load frequency control device based on large-scale multi-agent deep meta-reinforcement learning, characterized in that, Includes the following steps: The dynamic response model construction module is used to construct dynamic response models of control agents and scheduling agents in each region of a multi-region microgrid. Each region is configured with control agents and scheduling agents to collect the local status of each region in the current control cycle. The policy network training module defines the optimization problem for training the LSCMA-DMPG policy network and completes the offline training of the policy network based on a centralized training and distributed execution framework. The area control error calculation module is used to collect real-time local status and calculate the area control error ACE according to the NERC BAL-001-1 standard. The global optimal decision generation module updates the local state spaces of the control agent and the scheduling agent based on the obtained ACE, inputs the control agent and the scheduling agent into the pre-trained LSCMA-DMPG policy network, and outputs the optimal overall adjustment instruction. and unit participation factors ; The optimal decision execution module is used to control each generating unit and energy storage system to execute the optimal overall regulation command. It then returns to the area control error calculation module when the next control cycle arrives, repeating the closed-loop process of data acquisition-decision generation-execution.

9. A power grid load frequency control system based on large-scale multi-agent deep meta-reinforcement learning, comprising: Computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the power grid load frequency control method based on large-scale multi-agent deep meta-reinforcement learning as described in any one of claims 1-7.

Images