Multi-agent reinforcement learning multi-stability control measure distributed collaborative frequency stabilization system

Abstract

The invention discloses a multi-agent reinforcement learning multi-stability control measure distributed cooperation frequency stabilization system. The system comprises an initialization module, an interaction module, a decision module, a learning module, a fusion system frequency response model, a DC power support and load shedding agent only observes local information, and distributed cooperation is realized without interaction. Through optimization of offline training, an experience playback pool, Gaussian noise and the like, network updating is guided by a reward function containing frequency deviation, power support and load shedding cost. According to the method, the problems of traditional single control, insufficient collaboration, complex calculation and the like are solved, a control instruction can be quickly and accurately given, the control cost is reduced while the system frequency stability is ensured, and the decision accuracy and timeliness are improved.

Description

Technical Field

[0001] This invention belongs to the field of power system frequency security issues, and more specifically, relates to a distributed and collaborative frequency stabilization system with multiple agents, reinforcement learning, and multiple stabilization measures. Background Technology

[0002] Frequency stability is a crucial indicator for the safe operation of a power system, depending on the power balance between power sources and loads. In recent years, the large-scale replacement of synchronous generator units with new energy units has led to fluctuations and randomness in the system's active power output, as well as a decrease in system inertia. Once a disturbance occurs, the frequency drop rate accelerates. Simultaneously, the reverse distribution of energy and load, along with large-scale long-distance transmission of ultra-high-voltage direct current (UHVDC) lines with large single-circuit capacities, means that commutation failures, DC blocking, and subsequent DC restart faults will inevitably cause significant power surges, leading to serious frequency stability problems. Inappropriate control measures can further deteriorate system frequency stability, even causing system frequency collapse. Therefore, emergency control measures must be implemented to stabilize the power system's frequency stability in response to severe disturbances.

[0003] Traditional frequency emergency control methods include event-driven pre-planned control and response-driven low-frequency load shedding. Pre-planned control struggles to cope with complex and ever-changing operating conditions; low-frequency load shedding, as the last line of defense for power system frequency stability, is often set according to the most severe scenario, cutting off excessive loads, resulting in high stability control costs. DC emergency power support, leveraging the rapid adjustability of DC, is a more economical and efficient stability control method that can compensate for some power shortages from the power source side, alleviating the pressure of load shedding.

[0004] DC power support has relatively low control costs, while load shedding measures have ample reserves. Currently, most methods rely on single stabilization control measures to maintain frequency stability, lacking an emergency frequency control method that coordinates DC power support with load shedding. Furthermore, physical model-based and event-driven methods suffer from problems such as high computational load, complex calculation processes, and poor accuracy when dealing with the coordination challenges of these two stabilization control measures under complex and variable operating conditions. Summary of the Invention

[0005] To address the above problems, this invention proposes a distributed cooperative frequency stabilization system based on multi-agent reinforcement learning and multiple stabilization measures, comprising:

[0006] The initialization module is used to configure the value network parameters of the multi-agent system, identify the system frequency response model parameters under various operating modes and fault scenarios, set the maximum number of iterations for multi-agent interactive training, and read the preset values ​​of load shedding instructions, DC power support instruction sets, and characteristic electrical quantities of the power system.

[0007] The interaction module is used to perform the following operations: after a disturbance occurs in the power system, it reads electrical characteristic quantities as the state input of the agent; if the agent issues a stability control command, it transmits the control command to the power system for execution; it reads the electrical characteristic quantities of the system after the stability control measures are implemented; it calculates the reward based on the stability control measures implemented by all agents and the frequency information; and it evaluates the overall decision-making effect and stability control cost.

[0008] The decision module is used to process the initial electrical quantities, generate observations from multiple agents, and input them into the agent network. The DC power support agent and the load shedding agent output DC power support commands and load shedding commands, respectively, to ensure the safe and stable frequency of the system under the collaborative control of multiple agents.

[0009] The learning module is used to replace the value network of the multi-agent deep policy gradient algorithm with the system frequency response model that has been identified offline. It accurately evaluates the stability control execution effect and stability control cost in combination with the system frequency response mechanism, and guides all agents to update the network parameters in the direction of global optimum.

[0010] Preferably, the multi-agent algorithm based on the fusion system frequency response model consists of a policy network and a centralized action value function. The inputs of the multi-agent algorithm are the frequency information, voltage information, and power information of the stabilization substation after real-time disturbance. The frequency information includes the minimum frequency value, the initial rate of change of frequency, and the area formed by the 50Hz frequency curve. The voltage information includes the voltage mean and voltage variance. The power information is the DC power mean and DC power variance for the DC agent and the load shedding agent. The output is the stabilization command. The inputs of the centralized action value function are the current operating mode, the SFR model parameters under the disturbance scenario, and the amount of stabilization measures to be implemented. The output is the centralized action value used to update the policy network and evaluate the stabilization effect and cost under the current scenario. The DC power support command set is constrained by the upper and lower limits of the support measures. The upper limit of the DC power support measures is determined by the long-term overload characteristics of the DC equipment in the power system and the minimum firing angle of the converter. The lower limit of the power support is 0. The load shedding command set is constrained by the maximum amount of measures that can be shed, which is determined by the actual amount of load that can be shed from the load shedding substation.

[0011] Preferably, the multi-agent system interacts with the power system 1 second after the disturbance occurs, and each agent only observes local information and does not need to interact.

[0012] Preferably, the multi-agent system achieves distributed collaborative control by observing local information without interaction. Based on the frequency response mechanism of the multi-agent reinforcement learning multi-stabilization control measures distributed collaborative frequency stabilization system, the frequency response model parameters under all fault scenarios are identified offline. During the training process, the frequency response model is used to evaluate the system inertia center frequency maximum point under the current multi-agent strategy. The magnitude of the control measures of all agents is combined with the global control effect and control cost to achieve accurate evaluation, accurately guiding the update direction of each agent, and realizing distributed collaborative frequency emergency control of multiple agents without interaction.

[0013] Preferably, during the offline training of the multi-agent system, an experience replay pool is introduced to improve training efficiency. The system frequency response model parameters under different operating modes and disturbance scenarios are stored in the experience pool, and multiple sets of historical data are extracted to improve data utilization efficiency and enhance the decision-making ability of the agents.

[0014] Preferably, in the process of applying the frequency response model to evaluate the system inertia center frequency maximum point under the current multi-agent strategy during training, Gaussian noise is introduced to enhance the agent's ability to explore unknown strategies and avoid getting trapped in local optima during training.

[0015] Preferably, a reward function is applied to denoise and evaluate the decision-making ability of the current multi-agent policy network, retaining the network parameter with the largest reward value. The reward function consists of three parts: the first part is the frequency deviation cost, which is the square of the deviation between the lowest point of the system inertia center frequency and the first round trigger threshold of low-frequency load shedding multiplied by the weighting coefficient -100000; the second part is the power support cost, which is the global DC power support amount multiplied by the weighting coefficient -6; and the third part is the load shedding cost, which is the global load shedding measure amount multiplied by the weighting coefficient -1.

[0016] Preferably, the method for frequency stabilization includes:

[0017] Step 1: Identify the frequency response model parameters under different operating modes and fault scenarios, construct a distributed collaborative multi-agent model for frequency emergency control, and initialize the network parameters, learning rate, and Gaussian noise variance of all agent policy networks.

[0018] Step 2: Randomly select the initial state of the perturbation observation, select an action for each agent, execute the global action in the environment, obtain a new global state, calculate the global reward, and store the global state, global action, new global state, global reward, and corresponding frequency response model parameters as a set of data in the experience pool.

[0019] Step 3: Extract a batch of data from the experience pool, input the state of each agent into the agent policy network, generate a control policy, and add noise to explore and obtain new action combinations.

[0020] Step 4: Use the centralized action value function to evaluate the control effect and control cost, guide the multi-agent policy network to update the policy gradient, and generate the optimal distributed frequency emergency policy model.

[0021] Preferably, in step four, the optimal distributed frequency emergency strategy model is the policy network model in the multi-agent algorithm based on the fusion frequency response model that has been trained. The input layer of each agent is the state information of each stabilization substation, and the output is the stabilization command executed by each stabilization substation.

[0022] Beneficial effects

[0023] This invention optimizes the network through offline training, experience replay pooling, and Gaussian noise, using a reward function that incorporates frequency deviation, power support, and load shedding costs to guide network updates. This method addresses the problems of traditional single control, insufficient coordination, and computational complexity, enabling rapid and accurate control commands. It reduces control costs while ensuring system frequency stability, and improves decision-making accuracy and timeliness.

[0024] The problem of coordinated frequency emergency control of multiple stability control measures in power systems is modeled as a partially observable Markov decision process. Each stability control substation's control model is modeled as an agent with independent decision-making capabilities, enabling multi-agent collaboration to solve the frequency emergency control problem. To this end, a multi-agent algorithm for solving the partially observable Markov decision process is proposed. By analyzing the transient frequency security and stability mechanism of power systems, the transient frequency response model is integrated into the multi-agent reinforcement learning algorithm, guiding the multi-agents to update model parameters towards the global optimum. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings involved in the embodiments or the prior art are briefly described below. Obviously, these drawings illustrate several embodiments of the present invention, and those skilled in the art can derive other possible drawings based on these drawings without creative effort. The purpose of the drawings is limited to illustrating specific embodiments and does not limit the scope of the present invention.

[0026] Figure 1 This is a schematic diagram of the multi-agent reinforcement learning algorithm for the fusion frequency response model proposed in this invention, which is applied to distributed collaborative frequency emergency control with multiple stability control measures.

[0027] Figure 2 This is a 39-bus system example with multiple DC feeds used for demonstration purposes;

[0028] Figure 3 This is an example showing the average reward compared to the traditional algorithm;

[0029] Figure 4 The example shows a comparison of the recovery curves of the multi-agent algorithm with the fusion frequency response model proposed in this invention and the traditional algorithm in emergency power frequency control.

[0030] Figure 5 The example illustrates a comparison of the control costs of the multi-agent algorithm based on the fusion frequency response model proposed in this invention and traditional algorithms in emergency power frequency control. Detailed Implementation

[0031] To make the objectives, features, and advantages of the present invention more apparent and easier to understand, the technical solutions of the present invention are now clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the present invention is not limited to the embodiments described below, and specific implementation methods can be determined based on the technical solutions of the present invention and in combination with actual circumstances. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0032] This invention models the problem of coordinated frequency emergency control of multiple stability control measures in power systems as a partially observable Markov decision process. It models the control model of each stability control substation as an agent with independent decision-making capabilities, enabling multi-agent collaboration to solve the frequency emergency control problem. To this end, a multi-agent algorithm for solving the partially observable Markov decision process is proposed. By analyzing the transient frequency security and stability mechanism of power systems, the transient frequency response model is integrated into the multi-agent reinforcement learning algorithm, guiding the multi-agent to update model parameters towards the global optimum.

[0033] The power system frequency emergency control decision-making process conforms to a Markov decision process. The state is a stability-related reference quantity observed from the power grid; the action is the location and depth of the emergency control measures; and the reward function is the feedback of the agent's stability control strategy on the safe operation of the system. The system frequency reflects the global active power balance of the power grid. For a power system with multiple emergency power support substations and load shedding substations, leveraging the good dynamic synchronization between bus frequencies, each stability control substation is controlled by an agent. The emergency frequency control problem for emergency power support and load shedding is modeled as a partially observable Markov decision process. The agent continuously interacts with the environment to obtain the globally optimal strategy. The main components are as follows:

[0034] For EFC, each stability control agent only observes the local electrical information of the bus where the stability control substation is located, and its status The observations of the entire system include the states of all agents O = (o i ,…,o N ).

[0035] Actions refer to the amount of stability control measures executed by each stability control substation. The upper limit of the action space of the DC agent is constrained by the long-term overload capacity of the DC and the minimum firing angle of the rectifier side. Each agent has a continuous action space, and their joint action is represented as A = (a LS1 ,a EPS1 ,…,a EPSn ).

[0036] After all agents have performed their actions, they receive a shared reward by reducing the deviation of the inertia center frequency, reducing the DC power support, and reducing the load shedding, which is used to update the strategy.

[0037]

[0038] In the formula: P EPSi P is the DC power support measure. LSj For load shedding measures; C f C EPS C LS These are the weighting coefficients. Since DC power support does not require load loss, the control cost is lower. The relationship between the weighting coefficients is as follows:

[0039] C f <<C LS <C EPS <0 (2)

[0040] By employing a multi-agent approach to address the aforementioned issues, the dimensionality of the action and state spaces is reduced, thereby improving agent decision-making performance. After offline training, all load-cutting agents and emergency power support agents make decisions using only local information from the local stabilization substation where the agents reside, achieving distributed collaborative emergency frequency control with zero communication.

[0041] Figure 1 This is a schematic diagram of the multi-agent reinforcement learning algorithm for the fusion frequency response model applied to distributed collaborative frequency emergency control with multiple stability control measures proposed in this invention. Figure 1 As shown, the process is divided into two parts. In the offline training phase, each agent first provides a stabilizing action based on its current state. Then, a centralized action value function is used to evaluate the current states and actions of all agents. Finally, the parameters of each agent are updated with the goal of maximizing the centralized action value Q. In the online execution phase, each agent performs a stabilizing action based on its observations. i Generate stabilization commands. Distributed agents cooperate without needing to communicate, completing emergency frequency control with minimal stabilization costs.

[0042] To address the emergency frequency control problem involving coordination between multiple load shedding agents and emergency power support agents, a multi-agent deep deterministic policy gradient algorithm is applied to solve for the stability control measures. The policy network determines the control strategy, and the value network evaluates the stability control effect and cost based on the current global state information X and the actions of all agents. Since the Q value is only used during training, grid parameter information can be added to X in addition to agent observations.

[0043] Because the parameters of the value network are randomly initialized, it is difficult to accurately evaluate the control effect and stabilization cost, leading to unstable convergence training results. Therefore, by using a frequency response model, a centralized action function without a value network is derived, and the maximum frequency deviation of the system is represented by the frequency response model, as shown in the following equation:

[0044]

[0045] In the formula: parameters α, ω n ,ζ,ω r ψ are calculation parameters, R and K are model parameters, K is the mechanical power gain coefficient, and R is the equivalent droop coefficient; ΔP dist For power deficit disturbance; ΔP shed Indicates the total load shearing; ΔP EPS Indicates the total power support amount, t max Δt represents the time when the frequency is at its lowest value. a This refers to the time from the occurrence of the disturbance to the execution of emergency power support or load shedding. The frequency response model is identified based on the system's frequency response information, and the action value function is defined as follows:

[0046]

[0047] To achieve distributed and coordinated emergency frequency control, all agent parameters are updated simultaneously using global information to achieve global optima. During algorithm training, Gaussian noise is added to the policies output by the agents to construct an exploration process and avoid getting trapped in local optima.

[0048] a i =a i '+N(0,σ 2 (5)

[0049] In the formula a i 'Indicates the action output by the policy network; a i This represents the action after adding noise; N(0,σ) 2 The mean is zero and the variance is σ. 2 The noise is Gaussian and independent of the action. The gradient of both sides of equation (5) with respect to the agent parameter θ is obtained as follows:

[0050]

[0051] To maximize the value Q of the concentrated action, the Q value is calculated relative to the actual action a based on chain rules. i The gradient and policy network output action $$a_{i}$$ for parameters The gradient is used to update the parameters of each agent.

[0052]

[0053] in Here, μ is the parameter during the learning process of the i-th agent, m is the minimum batch size for experience replay, and μ is the parameter. i '(o i ) is the policy network of the i-th agent, consisting of observations O. i Given the action, η is the learning rate.

[0054] The range of actions output by the agent network is [-1, 1]. Therefore, the i-th agent network output action a i The amount of stability control measures actually implemented by the stability control substation ΔP i The following mapping relationship exists:

[0055]

[0056] In the formula: P i,max P i,min Let P represent the upper and lower bounds of the action space of the i-th agent, respectively, and let P be the upper and lower bounds of the action space of the i-th agent. i,min Set to 0.

[0057] According to equations (4), (5), and (6), the gradient of the action relative to the Q value of the i-th agent after adding noise is:

[0058]

[0059] First, the concentrated action value function Q is obtained according to equation (8) for the actual action a that adds noise. i The gradient of the input is delayed by Δt. a The step response is given by the frequency response model to obtain the system frequency response expression, denoted as λ(t-Δt). a The response frequency corresponding to the lowest system frequency at which the input step response with no delay is given by the frequency response model. Let λ(t) be the response frequency at which the system frequency is lowest. Its expression is as follows:

[0060]

[0061] Delayed by Δt aThe step response with amplitude as the global measure and the step response with amplitude as the disturbance without delay are input into the frequency response model to obtain the time t corresponding to the lowest frequency point. max Substituting equations (3), (11), (12), and (13) into equation (10), we get:

[0062]

[0063] To achieve the same network parameter update as in equation (8), a loss function is set to minimize the loss using gradient descent. The loss function of the agent policy network is shown in equation (15):

[0064]

[0065] Taking the derivative of the loss function with respect to the neural network parameters in equation (15) yields... Comparing equations (8) and (16), we can see that Therefore, the proposed algorithm ultimately achieves the same mathematical meaning as the multi-agent deep deterministic policy gradient algorithm, but improves training performance by incorporating the mechanism of the frequency response model.

[0066] The distributed cooperative frequency stability control process based on a multi-agent algorithm using a fusion frequency response model includes the following steps:

[0067] Step 1: Using post-fault frequency response data and system inertia, identify the frequency response model parameters under different fault scenarios using the least squares method;

[0068] Step 2: Initialize all agent neural network parameters Learning rate η, noise variance σ 2 Reset training interaction steps to 0;

[0069] Step 3: Read the observed state S during the initial period of the disturbance. For each agent i, select action a. i =μ(o i |θ i );

[0070] Step 4: Execute action a = (a1, ..., a N ), observe the new state S', calculate the reward r, and obtain the frequency response model parameter information corresponding to the current fault scenario, denoted as info;

[0071] Step 5: Store (S, a, r, S', info) in the experience pool, randomly draw a batch of data, and use this batch of data to update the policy network of the trained agent; Update steps:

[0072] ① Based on state S, obtain action ai =μ(o i ,θ i )+N(0,σ 2 Let a be the set of actions of all agents after adding the exploration noise;

[0073] ② Calculate the concentrated action value Q and policy gradient using the frequency response model;

[0074] ③ Update the policy network parameters of the i-th agent according to the policy gradient;

[0075] ④ Retain the multi-agent policy network parameters that have the highest concentrated action value Q;

[0076] Example Demonstration

[0077] To demonstrate the effect, a structure such as Figure 2 The example of a 39-bus system with multiple DC feeds shown has the following settings:

[0078] 1. Emergency DC power support equipment: High-voltage DC transmission system;

[0079] 2. DC power support substation capacity range: 0MW to 50MW;

[0080] 3. Load shedding substation capacity range: 0MW to 100MW;

[0081] 4. Training fault scenarios: Synchronous machine G03 shuts down, Synchronous machine G09 shuts down, Wind farm 1 shuts down, Wind farm 2 shuts down, Photovoltaic farm 1 shuts down, Photovoltaic farm 2 shuts down;

[0082] 5. Fault occurrence time: 1 second;

[0083] 6. Implementation time of stability control measures: 1.1 seconds after the fault occurs;

[0084] 7. Location of the stabilization substations: The two DC substations are located at bus 39 and bus 30 respectively; the two load shedding substations are located at bus 20 and bus 25 respectively;

[0085] 8. Number of training loops: 100,000;

[0086] 9. Reinforcement learning parameters: learning rate 0.0001, batch size 16, Gaussian noise variance 0.2, frequency low-point bias weight C of reward function and concentrated action value function. f The emergency power support cost is -100,000, C. EPS The load shedding cost is -1, and the cost of shedding the load is -6.

[0087] Figure 3To compare the average reward value of each training cycle in the multi-agent algorithm fusion frequency response model proposed in this invention with that of traditional reinforcement learning algorithms, the average reward convergence value under eight different random seeds is used for each algorithm to reflect its average convergence performance. It can be seen that, compared with traditional algorithms, the algorithm proposed in this invention can converge to a higher average reward value with a smaller confidence interval. The proposed algorithm can accurately evaluate the value of concentrated actions, avoiding the problem of agents converging to suboptimal solutions due to inaccurate evaluation of control effects and control costs, thus resulting in more stable and reliable control performance.

[0088] By utilizing the proposed multi-agent algorithm based on the fused frequency response model, emergency frequency control was performed in the power system during disturbances. The control effect and control cost are as follows: Figure 4 , Figure 5 As shown, the effectiveness of the multi-intelligent reinforcement learning algorithm with fused frequency response model proposed in this invention for emergency frequency control of power systems is demonstrated. Compared with traditional reinforcement learning methods, the proposed algorithm ensures the safety and stability of power system frequencies while incurring lower control costs.

[0089] The above technical solutions only embody the preferred technical solutions of the present invention. Any modifications that may be made by those skilled in the art to certain parts thereof embody the principles of the present invention and fall within the protection scope of the present invention.

Claims

1. A frequency stabilization system based on distributed collaborative multi-agent reinforcement learning and multiple stabilization measures, characterized in that, include: The initialization module is used to configure the value network parameters of the multi-agent system, identify the system frequency response model parameters under various operating modes and fault scenarios, set the maximum number of iterations for multi-agent interactive training, and read the preset values ​​of load shedding instructions, DC power support instruction sets, and characteristic electrical quantities of the power system. The interaction module is used to perform the following operations: after a disturbance occurs in the power system, it reads electrical characteristic quantities as the state input of the agent; if the agent gives a stabilization command, it transmits the control command to the power system for execution; it reads the electrical characteristic quantities of the system after the stabilization measures are executed; it calculates the reward based on the stabilization measures of all agents and frequency information; and it evaluates the overall decision-making effect and stabilization cost. The decision module is used to process the initial electrical quantities, generate observations of multiple agents, and input them into the agent network. The DC power support agent and the load shedding agent output DC power support commands and load shedding commands respectively, ensuring the safe and stable frequency of the system under the cooperative control of multiple agents. The learning module is used to replace the value network of the multi-agent deep policy gradient algorithm with the system frequency response model that has been identified offline. It accurately evaluates the stability control execution effect and stability control cost in combination with the system frequency response mechanism, and guides all agents to update the network parameters in the direction of global optimum.

2. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 1, characterized in that: The multi-agent algorithm based on the fusion system frequency response model consists of a policy network and a centralized action value function. The inputs of the multi-agent algorithm are the frequency, voltage, and power information of the stabilization substation after real-time disturbance. The frequency information includes the minimum frequency value, the initial rate of change of frequency, and the area formed by the 50Hz frequency curve. The voltage information includes the voltage mean and voltage variance. The power information is the DC power mean and DC power variance for the DC agent and the load shedding agent. The output is the stabilization command. The inputs of the centralized action value function are the current operating mode, the SFR model parameters under the disturbance scenario, and the amount of stabilization measures to be implemented. The output is the centralized action value used to update the policy network and evaluate the stabilization effect and cost under the current scenario. The DC power support command set is constrained by the upper and lower limits of the support measures. The upper limit of the DC power support measures is determined by the long-term overload characteristics of the DC equipment in the power system and the minimum firing angle of the converter. The lower limit of the power support is 0. The load shedding command set is constrained by the maximum amount of measures that can be shed, which is determined by the actual amount of load that can be shed from the load shedding substation.

3. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 2, characterized in that: The multiple agents interact with the power system 1 second after the disturbance occurs. Each agent only observes local information and does not need to interact.

4. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 3, characterized in that: The multi-agent system achieves distributed collaborative control by observing local information without interaction. Based on the frequency response mechanism of the multi-agent reinforcement learning multi-stabilization control measures distributed collaborative frequency stabilization system, it identifies frequency response model parameters under all fault scenarios offline. During training, it evaluates the system's inertia center frequency peak under the current multi-agent strategy using the frequency response model. It also accurately evaluates the global control effect and cost by combining the magnitude of control measures of all agents, precisely guiding the update direction of each agent and realizing distributed collaborative emergency frequency control without interaction among multiple agents.

5. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 4, characterized in that: During the offline training of the multi-agent system, an experience replay pool is introduced to improve training efficiency. The system frequency response model parameters under different operating modes and disturbance scenarios are stored in the experience pool. At the same time, multiple sets of historical data are extracted to improve data utilization efficiency and enhance the decision-making ability of the agents.

6. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 4, characterized in that: In the process of training, the frequency response model is used to evaluate the system inertia center frequency maximum point under the current multi-agent policy. Gaussian noise is introduced to enhance the agent's ability to explore unknown policies and avoid getting trapped in local optima during training.

7. The frequency stabilization system of distributed cooperation of multiple agent reinforcement learning and multiple stabilization measures according to claim 1, characterized in that: The reward function is applied to denoise and evaluate the decision-making ability of the current multi-agent policy network, retaining the network parameter with the largest reward value. The reward function consists of three parts: the first part is the frequency deviation cost, which is the square of the deviation between the lowest point of the system inertia center frequency and the first round trigger threshold of low-frequency load shedding multiplied by the weight coefficient -100000; the second part is the power support cost, which is the global DC power support amount multiplied by the weight coefficient -6; and the third part is the load shedding cost, which is the global load shedding measure amount multiplied by the weight coefficient -1.

8. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 1, characterized in that, The methods used for frequency stabilization include: Step 1: Identify the frequency response model parameters under different operating modes and fault scenarios, construct a distributed collaborative multi-agent model for frequency emergency control, and initialize the network parameters, learning rate, and Gaussian noise variance of all agent policy networks. Step 2: Randomly select the initial state of the perturbation observation, select an action for each agent, execute the global action in the environment, obtain a new global state, calculate the global reward, and store the global state, global action, new global state, global reward, and corresponding frequency response model parameters as a set of data in the experience pool. Step 3: Extract a batch of data from the experience pool, input the state of each agent into the agent policy network, generate a control policy, and add noise to explore and obtain new action combinations. Step 4: Use the centralized action value function to evaluate the control effect and control cost, guide the multi-agent policy network to update the policy gradient, and generate the optimal distributed frequency emergency policy model.

9. The frequency stabilization system based on multi-agent reinforcement learning and distributed collaborative multi-stabilization measures according to claim 8, characterized in that, In step four, the optimal distributed frequency emergency strategy model is the policy network model in the multi-agent algorithm based on the fusion frequency response model that has been trained. The input layer of each agent is the state information of each stabilization substation, and the output is the stabilization command executed by each stabilization substation.

Images