Multi-agent power system distribution event trigger control method based on PPO algorithm

Abstract

The invention discloses a multi-agent power system distribution event trigger control method based on a PPO algorithm, and the method comprises the steps: building a Simulink model based on multi-agent reinforcement learning, carrying out the analysis of a single-round reward and an average reward of an agent along with the increase of a training round, and carrying out the multi-agent reinforcement learning through the PPO algorithm; based on the Lyapunov stability and the Lasael invariance principle, an exponential decay function is adopted as a threshold form in event trigger control; sequence decisions of the agents in an uncertain environment are described by means of a Markov decision process, multi-agent event trigger control based on a PPO strategy gradient algorithm is constructed, and communication decisions and control action updating between the agents are triggered as required. According to the method, the PPO algorithm is utilized to accumulate rewards to the maximum, multiple agents are driven to learn a self-adaptive event triggering strategy, triggering is carried out in a distributed cooperative task according to needs, and the system stability is guaranteed while the communication frequency is remarkably reduced.

Description

Technical Field

[0001] This invention relates to the fields of power system automation control and artificial intelligence technology, and in particular to a multi-agent power system allocation event triggering control method based on the PPO algorithm. Background Technology

[0002] The penetration rate of distributed renewable energy sources such as wind power and photovoltaics in the power system is continuously increasing. New power systems are gradually covering the entire society's power system, characterized by high-proportion renewable energy integration, source-grid-load-storage coordination, and digital intelligent dispatching, achieving optimization across the entire chain from power generation to consumption. However, the indirectness, randomness, and disturbance of renewable energy sources pose significant challenges to the stable operation, economic dispatch, and real-time control of the power grid, necessitating dynamic adjustment and intelligent control to achieve energy supply and demand balance.

[0003] Multi-agent systems, with their unique distributed cooperation capabilities and adaptive event-triggered control design, have broad application prospects in new energy power systems and are considered one of the effective ways to solve the aforementioned challenges. They model photovoltaic, wind power, energy storage, and load units as independent intelligent agents, achieving global collaborative optimization through local information interaction. Event-triggered control is a strategy that executes control updates or communication only when the system state meets specific conditions (such as errors exceeding a threshold). Compared to traditional periodic control, it significantly reduces communication and computational burdens and improves resource utilization efficiency.

[0004] In the existing technology, the research on combining multi-agent reinforcement learning with event-triggered control still faces many challenges: 1) High model dependence and low data utilization. Traditional methods rely heavily on accurate mathematical models of the system and have poor adaptability in complex and ever-changing power environments; 2) Strong limitations of triggering conditions. Fixed triggering thresholds are difficult to balance control accuracy and communication overhead in dynamic environments; 3) Poor stability in complex environments. Competition and cooperation among multiple agents make it difficult for the strategy to converge and the system is prone to instability.

[0005] Proximal policy optimization (PPO) algorithms, as advanced policy gradient algorithms, effectively handle high-dimensional state and action spaces while ensuring training stability by limiting the magnitude of policy updates. Therefore, designing a multi-agent control method that can adaptively optimize triggering policies and ensure system stability by combining the learning advantages of PPO algorithms with the efficiency advantages of event-triggered mechanisms has become a pressing technical problem. Summary of the Invention

[0006] The problem to be solved by this invention is to provide a multi-agent power system allocation event triggering control method based on the PPO algorithm. By utilizing the maximum cumulative reward of the PPO algorithm, the multi-agent learns an adaptive event triggering strategy, which triggers on demand in distributed collaborative tasks, and ensures system stability while significantly reducing communication frequency.

[0007] This invention adopts the following technical solution: a multi-agent power system allocation event triggering control method based on the PPO algorithm, comprising the following steps:

[0008] Step 1: Create a multi-agent power system: Each agent corresponds to a distributed energy unit in the power system, including: photovoltaic power generation unit, wind power generation unit and energy storage unit. Power balance and voltage regulation are achieved through local information interaction, and the distributed energy units are coordinated to operate together on the grid.

[0009] Step 2: Construct a Simulink model based on multiple agents: Based on the multi-agent power system, build a simulation model on the Simulink simulation platform to perform multi-agent collaborative simulation, including multiple agent modules and a core environment module. The multiple agent modules adopt a distributed decision-making mechanism and use a signal bus to couple information between agents.

[0010] Step 3: Multi-agent reinforcement learning based on PPO algorithm: Bind the Simulink model to the reinforcement learning interface, create an independent policy network and value network for each agent, configure the policy network as a random policy with the probability distribution output by the PPO algorithm, set the hyperparameters of the PPO algorithm, train the actions of multiple energy unit agents, and optimize the agent policies.

[0011] Step 4, Event Trigger Settings: Based on the principles of Lyapunov stability and Lassalle invariance, an exponential decay function is used as the threshold form in event trigger control;

[0012] Step 5, Markov Decision: Describe the sequential decision-making of an agent in an uncertain environment using the Markov decision process;

[0013] Step 6: Multi-agent event-triggered control based on PPO algorithm: Distributed event-triggered control is performed based on PPO algorithm. By optimizing the policy network to learn random policies, the continuous action space and high-dimensional state space are processed. By maximizing the cumulative reward, the agents learn adaptive triggering policies. According to the event-triggered control mechanism, communication decisions and control action updates between agents are triggered as needed, so that the multi-agent power system gradually tends to a stable state.

[0014] As a preferred embodiment, in the Simulink model described in step 2, each agent module integrates a policy decision unit and is equipped with input and output ports;

[0015] The input port receives environmental status and external signals, and the output port outputs the status observation, action, immediate reward and termination condition isdone. The output port is transmitted to the core environment module through the signal bus.

[0016] The core environment module receives the joint actions of all agents and triggers a state transition, calculates the system state at the next moment, feeds back the updated environmental parameters and reward signals, and constructs an explicit communication channel to transmit policy parameters.

[0017] As a preferred embodiment, the PPO algorithm described in step 3 is as follows:

[0018] Step 301, Initialization: Initialize the policy network and value network, as well as the optimizer and related configuration parameters;

[0019] Step 302, Sampling: Using the current strategy Interact with the real environment and collect the states of multiple agents. ,action ,award and next state data;

[0020] Step 303: Calculate the dominance function: Using the dominance function For each intelligent agent In The dominance function is calculated for actions performed under certain conditions to measure the superiority or inferiority of the action compared to the average performance, and gradually leverages... The guidance strategy is continuously improved; among them, This indicates that the agent is in a certain state. Take action below Action value function; Representing state The state value function.

[0021] Step 304, Strategy Update: Use the Clip method to calculate the probability ratio between the old and new strategies and limit it to a preset range;

[0022] Step 305, Value Function Update: Using the loss function Optimize value function The value network is updated by minimizing the mean square error.

[0023] in, It represents the expected value based on experience, and calculates the average mean square error between the predicted value and the target value. This represents the total cumulative reward that the agent can actually obtain at the start of this state, as shown in the following formula. By combining immediate rewards with long-term goals, the algorithm avoids entering local optima.

[0024] ;

[0025] in, Discount factor To evaluate the correlation between current rewards and future rewards.

[0026] Step 306: Repeat steps 3.2 to 3.5, iterating through multiple rounds to gradually optimize the strategy until convergence.

[0027] As a preferred option, in step 3, the agent's policy is optimized using the PPO algorithm. The specific steps are as follows:

[0028] Step 311: Define the sampling time, total time, and maximum number of steps in the training settings;

[0029] Step 312: Start the Simulink model, set the number of agents and the size of the observation space, use discrete {0,1} to represent the action space, and create an env environment to connect the various agent modules in Simulink.

[0030] Step 313: Iteratively construct the policy network Actor and value network Critic for each agent module. The Actor network maps observations to action probability distributions through a multilayer perceptron, and the Critic network evaluates state value to calculate the advantage function.

[0031] The Actor network structure consists of an input layer, multiple fully connected layers, and ReLU activation, which outputs action probabilities via softmax; the Critic network outputs a single value to estimate the agent's state value.

[0032] Step 314: Configure the PPO algorithm, set the entropy loss weight, create the corresponding PPO agent and start training. Improve the stability of multi-agent training by limiting the magnitude of policy updates.

[0033] Step 315: After training is completed, conduct tests, simulate the environment using the sim function, and save the experience data.

[0034] As a preferred embodiment, the event triggering control described in step 4 includes triggering parameters. From the current transmission signal The last transmitted signal ,parameter , and A joint decision was made to adjust the threshold based on resource estimates, with the specific triggering conditions as follows:

[0035] ;

[0036] ;

[0037] in, It is an adjustable parameter. for Given a lower bound, and if The adaptive event triggering mechanism will then become a time-invariant communication mechanism, with the superscript T indicating transpose.

[0038] Indicates the time of the next event trigger (the (k+1)th trigger); The current event trigger time (absolute time of the kth trigger) is indicated; h represents the sampling period, and l represents the number of sampling periods to wait for, which is the number of sampling periods from the current trigger to the next trigger; State measurement error is defined as (Actual state - Last transmission state); This indicates that the positive definite weighted matrix is ​​used to define the error.

[0039] In the calculation formula, the minimum number of sampling periods l is found to minimize the error. Threshold The next update is triggered only when the accumulated error becomes large enough.

[0040] In the calculation, This indicates that the updated threshold parameter will be used for the next trigger judgment. It is a lower bound guarantee of the threshold The value should not be too small to avoid triggering too frequently; This represents the threshold parameter that is currently triggered; This represents the adaptive adjustment coefficient, which controls the magnitude of threshold changes. The time degenerates to a fixed threshold, as mentioned above, "the adaptive event triggering mechanism will become a time-invariant communication mechanism"; It is the sensitivity coefficient for state changes, used to control the impact of state changes on the threshold. The state norm at the next trigger, This represents the state norm at the time of the current trigger.

[0041] When data is transmitted between agents in the form of digital signals, the agents Whether the sampled signal propagates depends on the sampling time. Is the event triggering condition violated? The event triggering condition is as follows:

[0042] ;

[0043] in, For threshold parameters, The time when the sampled signal of the intelligent agent is propagated. Describes the first... Index of the time when the next event is triggered. Let i represent the measurement error vector of agent i. The reference signal representing agent i; This indicates that the positive definite weighted matrix is ​​used to define the error.

[0044] The weighted quadratic form representing the current measurement error indicates the magnitude of the deviation between the actual state and the previously transmitted state.

[0045] The maximum allowable error energy is determined by the difference between neighboring states. and The threshold parameter determines the outcome.

[0046] When error The event is triggered at a certain time.

[0047] As a preferred embodiment, the Markov decision process described in step 5 is as follows:

[0048] Step 501: Treat the action decision-making process of a single agent as a local Markov decision, consisting of a quintuple. definition;

[0049] in, Represents the state space set, Represents the action space set; Represents the state transition probability, defined as the probability given the current state. And satisfy the action Under the condition that the state transitions, The probability of; Let the reward function represent the actions taken by the agent. Transferred to The agent then receives an immediate reward. A discount factor used to assess immediate rewards and future impact;

[0050] Step 502: In each time step t, the agent is in the system state Choose to perform an action This allows the system to switch to... And the agent receives an immediate reward. ;

[0051] Step 503: Solve the Markov decision problem, with the goal of finding a strategy. This is manifested in the state space. To the action space Mapping of the distribution above.

[0052] As a preferred embodiment, in the multi-agent event triggering control described in step 6, the event triggering condition is expressed as follows:

[0053] ;

[0054] ;

[0055] In the formula, Represents the controlling agent Is it related to intelligent agents? communication, Represents intelligent agents Whether to receive from the intelligent agent The Group of observations , and All were calculated using a policy network. and ;

[0056] Based on the event triggering conditions, the final input is given to the intelligent agent. Observational information The elements are updated as follows:

[0057] ;

[0058] The formula for calculating the joint strategy is as follows:

[0059] ;

[0060] in, Represents control input, and Indicate control decision, It is observation information. It refers to the actions of the agent; the joint policy guides the agent to learn continuously towards the direction of maximizing reward through constant updates.

[0061] As a preferred embodiment, the distributed event-triggered control described in step 6 achieves rapid convergence of the joint reward value of the agent by optimizing the structure and parameters of the policy network;

[0062] The joint reward function is shown in the following formula. Represents the total reward obtained by the agent at time i:

[0063] ;

[0064] in, This represents the reward for the agent's motion control. This indicates the collision penalty for the intelligent agent. This indicates a penalty for agent communication.

[0065] As a preferred embodiment, in the distributed event-triggered control described in step 6, the initial trigger threshold is 15% of the maximum value in the state space, and a Lyapunov function decay rate is bound as a dynamic adjustment strategy to normalize the joint reward of the multi-agent power system to... Furthermore, cooperative rewards are multiplied by a coefficient of 0.8, and adversarial rewards are multiplied by a coefficient of 1.2 to strengthen the competitive orientation.

[0066] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0067] 1. Significantly reduced communication and computing overhead: By introducing an event-triggered mechanism, the intelligent agent only performs communication and control updates when the state error exceeds the limit, avoiding a large amount of invalid or inefficient communication under the traditional time-triggered method, which greatly saves network bandwidth and computing resources, and is particularly suitable for large-scale distributed power systems with limited communication resources.

[0068] 2. Enhanced System Adaptability and Robustness: This invention employs the PPO data-driven reinforcement learning algorithm, enabling event-triggered control to autonomously learn optimal strategies through direct interaction with the environment, rather than relying on analytical power system models. When facing uncertainties such as random fluctuations in renewable energy output and load changes, the system exhibits stronger adaptive adjustment capabilities and robustness.

[0069] 3. Balancing Control Performance and System Stability: This invention bases the design of event triggering conditions on Lyapunov stability theory and demonstrates the system's convergence based on the Lassalle invariance principle. Combined with the stable training characteristics of the PPO algorithm, this ensures that the proposed control method not only has high accuracy in steady state but also strictly guarantees the stability of the power system during dynamic transitions, preventing instability or oscillations.

[0070] 4. Achieving a balance between collaborative efficiency and individual autonomy: The method of this invention guides the agent to pursue global collaborative goals (such as power balance and economic optimization) while taking into account individual constraints (such as equipment operating limits and safe distances) and communication costs by designing a reasonable joint reward function (such as including tracking rewards, collision / limit violation penalties, and communication penalties). This achieves a balance between overall system efficiency and individual autonomous decision-making. Attached Figure Description

[0071] Figure 1 This is a flowchart of the multi-agent power system allocation event triggering control method based on the PPO algorithm of the present invention;

[0072] Figure 2 This is an equivalent circuit diagram of a photovoltaic power generation unit according to an embodiment of the present invention;

[0073] Figure 3 This is a schematic diagram of the control of the wind power generation unit energy storage system according to an embodiment of the present invention;

[0074] Figure 4 This is a diagram of the Simulink model structure based on multi-agent systems in this invention.

[0075] Figure 5 This is a progress chart of the Simulink model running in an embodiment of the present invention;

[0076] Figure 6 This is a diagram of the multi-agent distributed event triggering control architecture of the PPO algorithm of this invention;

[0077] Figure 7 This is a distribution diagram of event triggering in a multi-agent system according to an embodiment of the present invention;

[0078] Figure 8 This is a multi-agent state convergence diagram according to an embodiment of the present invention;

[0079] Figure 9 This is a comparison chart of multi-agent error and threshold in an embodiment of the present invention. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the application will be further described in detail below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments involved in this invention. All non-innovative embodiments based on these embodiments by other researchers in the art are within the protection scope of this invention. Furthermore, the step numbers in the embodiments of this invention are only set for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0081] In one embodiment of the present invention, a multi-agent power system allocation event triggering control method based on the PPO algorithm is proposed. The method utilizes the maximum cumulative reward of the PPO algorithm to drive the multi-agent to learn an adaptive event triggering strategy, which triggers on demand in distributed collaborative tasks, and ensures system stability while significantly reducing communication frequency.

[0082] The specific steps of the method in this embodiment are as follows:

[0083] (I) Constructing a multi-agent system

[0084] This embodiment of the multi-agent system includes: a photovoltaic power generation unit, a wind power generation unit, and an energy storage unit, with each agent corresponding to a distributed energy unit in the power system.

[0085] 1. Photovoltaic power generation unit

[0086] The equivalent circuit of the photovoltaic power generation unit in this embodiment is as follows: Figure 2 As shown, and These represent the number of photovoltaic cells connected in series and in parallel in the photovoltaic array, respectively. This refers to the actual output voltage of the photovoltaic cell; This refers to the actual output current of the photovoltaic cell. For photocurrent, This is the diode saturation current.

[0087] The theoretical relationship between the output voltage and current fitted by the equivalent circuit diagram of this simplified photovoltaic power generation is expressed as follows:

[0088] ;

[0089] In practical engineering applications, photogenerated current and diode saturation current Make corrections:

[0090] ;

[0091] ;

[0092] In the formula, This represents the actual light irradiance received by the photovoltaic array. The irradiance received under standard test conditions for a photovoltaic array is generally [value missing]. ; This refers to the temperature coefficient of a photovoltaic power generation system; it is usually provided by the manufacturer and has no unit. This refers to the saturation current of a non-ideal diode in a photovoltaic cell model under standard test conditions. The current of the photogenerated current source under standard test conditions; This represents the actual surface temperature of the current photovoltaic power generation system. Standard ambient temperature; The bandgap width is related to the photovoltaic cell material.

[0093] Photovoltaic irradiance is used as an evaluation index to measure the output performance and power generation efficiency of photovoltaic power generation systems.

[0094] ;

[0095] in, This indicates the actual output power of the photovoltaic power generation system; This refers to the power derating factor for photovoltaic power generation systems. The rated power of the photovoltaic power generation system under standard test conditions; The solar irradiance on the photovoltaic panel under standard test conditions. This refers to the solar irradiance reaching the photovoltaic panel in a real-world scenario. The output power-temperature conversion coefficient of a photovoltaic power generation system; This represents the actual surface temperature of the photovoltaic panel as currently measured. This refers to the surface temperature of the photovoltaic panel under standard test conditions.

[0096] 2. Wind power generation unit

[0097] In this embodiment, the wind power generation unit utilizes an energy storage system to address wind power dispatch risks and control such as... Figure 3 As shown. When the wind power available for grid connection is dispatched... Exceeding the safe operating limits of the power grid The energy storage system is charged at any time, and the energy storage system outputs power. Combined with wind power generation capacity Ultimately, it will converge into It flows into the power grid to ensure its safe operation.

[0098] The output power of a wind turbine is determined by the real-time wind speed and the turbine's output power, expressed as:

[0099] ;

[0100] in, This refers to the rated power of the fan. This represents the real-time wind speed at the wind turbine blades. , and These represent the rated wind speed of the fan, the cut-in wind speed at the fan blades, and the cut-out wind speed at the fan blades, respectively. This refers to the rated output power of the fan.

[0101] 3. Energy storage unit

[0102] In this embodiment, the energy storage unit consists of an energy storage body and an energy management system forming a closed-loop control architecture. Based on the KiBaM model, the battery is considered as having two energy storage boxes. , In a system where the current regulator operates through an equivalent resistance Controlling the charging and discharging process of the battery and maintaining energy balance;

[0103] The maximum charging current and maximum discharging current of a battery within a certain period are expressed as follows:

[0104] ;

[0105] In the formula, and They are respectively The maximum charging and discharging currents at the beginning of the time period; For the quasi-transfer rate of charge, This is the full charge ratio; For time intervals; for The maximum energy storage capacity of the battery during the specified time period; Indicates that the battery is in Available capacity at the beginning of the time period; Indicates that the energy storage system is in The theoretical total electrical energy capacity for a given period of time;

[0106] The maximum charging current and charging rate of the battery are set to limit the maximum charging and discharging power of the energy storage unit. The final battery charging and discharging power is expressed as:

[0107] ;

[0108] In the formula, This represents the upper limit of the charging power of the battery at the initial moment of a certain period of time. This indicates the upper limit of discharge power at the beginning of that period. This refers to the upper limit of the charging power of the battery in the energy storage system under the condition of charging rate limitation. This is the upper limit of charging power under the maximum charging current limit. , These represent the charging and discharging efficiency of the battery.

[0109] (II) Constructing a Simulink model based on multi-agent reinforcement learning

[0110] Build a multi-agent simulation model in Simulink, such as Figure 4 As shown, the simulation model consists of the core architecture of agent modules AgentA and AgentB, and the environment module MyEnv, achieving multi-agent collaborative simulation through modular design.

[0111] Each agent module integrates a policy decision-making unit. Its input port UY receives the environmental state and external signals, and its output ports observation, action, reward, and isdone respectively represent state observation, action output, immediate reward, and termination condition.

[0112] The environment module MyEnv, as the core of the dynamic system, receives actions from the agent and triggers a state transition mechanism. It calculates the system state at the next moment through physical equations or data-driven models and feeds back the updated environmental parameters and reward signals.

[0113] Multiple intelligent agent modules and a core environment module are coupled through a signal bus, enabling the construction of explicit communication channels to transmit strategy parameters while supporting observation-based indirect cooperation and sharing of local state information.

[0114] In this Simulink model, Agent A and Agent B employ a distributed decision-making mechanism. Each agent independently executes a policy network that computes a random policy using the probability distribution output by the PPO algorithm. The output action is transmitted to the environment module via a bus, and the policy configuration network configures it with a random policy based on the probability distribution output by the PPO algorithm.

[0115] During operation, the output ports of the intelligent agents are aggregated into environmental inputs through the Sum module, the environmental status outputs are distributed to each intelligent agent through the Mux module, and the communication links are filtered through the Bus Creator / Selector module.

[0116] In the environment module, the agent jointly updates the system state to achieve state transition and calculates the instantaneous reward based on the agent's behavior and the environment state. The isdone signal is monitored by the termination condition judge to trigger the simulation termination.

[0117] During agent interaction, agents can exchange observation data through Simulink's Data Store Memory module, and agent A can read the historical action sequence of agent B through the cross-observation channel. External deep learning algorithms are called through the MATLAB Function module for training, achieving joint training of Simulink simulation and Matlab algorithms.

[0118] (III) Multi-agent reinforcement learning based on PPO algorithm

[0119] The Proximal Policy Optimization (PPO) algorithm aims to improve training stability and efficiency by limiting the magnitude of policy updates. Its main goal is to optimize the expectation, and the core idea is to use the Clip method to limit the magnitude of policy gradient updates, deriving update methods for different reinforcement learning parameters.

[0120] In this embodiment, the PPO algorithm implements mini-batch updates through multiple training steps, which can successfully solve the step size selection problem in traditional policy gradient algorithms, and further improves the stability of training by limiting the policy update magnitude.

[0121] The objective function of the PPO algorithm is expressed as:

[0122] ;

[0123] in, It represents the probability ratio, which is the ratio of the probability of choosing an action under the same state as the new strategy and the old strategy; The dominant function is represented by the clipping operation. Will Limited to the range Internal; to prevent policy mutations The default value is 0.2. , The weighting coefficients are used to control the early strategy updates and the later computational convergence.

[0124] probability ratio Represented as:

[0125] ;

[0126] In the formula, Indicates the current state of the agent. Actions under the new strategy The probability, Represents the current state of the agent. Actions under the old strategy The probability of. This represents the degree to which the agent changes its strategy under different environments.

[0127] Advantage function Represented as:

[0128] ;

[0129] To enable Critic to more accurately estimate state values ​​and thus provide more reliable policy optimization suggestions, the value function is updated. The update method minimizes the mean square error, as shown in the following formula:

[0130] ;

[0131] In the formula, The value function prediction representing the current state can be used as the Critic network's prediction of the current state. Long-term value estimation; This represents the total cumulative reward that the agent can actually obtain at the start of this state, combining immediate rewards with long-term goals to help the algorithm escape the trap of local optima.

[0132] ;

[0133] ;

[0134] in, Discount factor To evaluate the correlation between current rewards and future rewards.

[0135] To encourage multi-agent dynamic adjustment of strategies between exploration, exploitation, and abandonment, this embodiment of the PPO algorithm further introduces an entropy regularization term in the following formula, which maintains high policy randomness by adding the negative value of policy entropy to the loss function:

[0136] ;

[0137] In the formula, It is the average of all agent samples. Is the policy network in state The probability of choosing an action is increased to improve the uniformity and uncertainty of the policy distribution. Used to measure the randomness of a strategy.

[0138] Furthermore, due to the increased complexity of action states caused by the cooperation and competition among various agents, a single loss function is insufficient to cover the optimization objective. This embodiment introduces a total loss function to avoid conflicts in multi-objective optimization.

[0139] In the total loss function of PPO, the entropy regularization term, along with the policy gradient loss and the value function loss, are set together as the total loss objective function, expressed as:

[0140] ;

[0141] in, It is the entropy regularization term Represents the policy gradient loss. This represents the loss of function value. The weighting coefficients for controlling the loss function are often set to 0.5 to avoid overfitting of the Critic network. The strength of the entropy regularization term is adjusted by setting it to a large value in the early stages of training to encourage exploration, and gradually decreasing it in the later stages of training to converge to a high-reward strategy.

[0142] The specific operation process of the PPO algorithm in this embodiment is as follows.

[0143] 1. Initialization: Initialize the policy network (actor) and value network (critic), as well as the optimizer and other related configuration parameters.

[0144] 2. Sampling: Use the current strategy Interact with the real environment and collect the states of multiple agents. ,action ,award And data such as the next state.

[0145] 3. Calculate the dominance function: Using the dominance function For each intelligent agent In The dominance function is calculated for actions performed under certain conditions to measure the performance level of that action relative to the average performance. Then, it is gradually improved by... The guidance strategy is continuously improved.

[0146] 4. Strategy Update: The Clip method is used to calculate the probability ratio between the old and new strategies and to limit it within a specific range to avoid excessive policy changes. If the change exceeds this range, a penalty will be imposed to ensure that the update magnitude is moderate.

[0147] 5. Value function update: using the loss function To optimize the value function By updating the value network by minimizing the mean square error, it can estimate the state values ​​more accurately.

[0148] 6. Repeat the above steps: After multiple iterations, the strategy will be gradually optimized until it converges.

[0149] After establishing a multi-agent model in Simulink, reinforcement learning (MARL) is used via the MATLAB Function module to train the agents' actions. After setting training parameters, building the PPO network, training it, and conducting simulation tests, the actions of the two agents are evaluated, thus combining Simulink simulation with MATLAB algorithms.

[0150] To ensure real-time interaction between the agent and the environment, this embodiment defines the environment using the demoMARL function in the Simulink model and determines the training duration using a time parameter. Specific training parameters are as follows: a sampling duration of 0.1 seconds is set, and the total simulation time is 100 seconds. Multi-agent interactive operation is achieved by binding the Simulink model with a reinforcement learning interface. An Actor (policy network) and a Critic (value network) are constructed for each agent. The Actor network maps observations to action probability distributions using a multilayer perceptron (MLP), while the Critic network evaluates state value to calculate the advantage function (GAE).

[0151] The stability mechanism of the PPO algorithm optimizes the agent's policy. It adopts ClipFactor to implement the core pruning mechanism of PPO, which limits the policy update magnitude. GAE is used to balance bias and variance, and entropy regularization is combined to prevent the policy from converging to a local optimum too early.

[0152] The specific operating steps are as follows:

[0153] 1. Define the sampling time in the training settings. Total time The maximum number of steps to calculate is 1000.

[0154] 2. Start the Simulink model, set the number of agents to 2, the observation space size to 1, use discrete {0,1} to represent the action space, and create an environment to connect AgentA and AgentB modules in Simulink.

[0155] 3. Construct Actor and Critic networks for two agents in a loop; the Actor network structure consists of an input layer, multiple fully connected layers and ReLU activation, and finally a softmax output action probability; the Critic network output is a single value to estimate the agent's state value.

[0156] 4. Configure the PPO algorithm, set the entropy loss weight to 0.2, create two PPO agents and start training.

[0157] 5. The training options include 10 episodes, each with a maximum of 1000 steps, and the training progress is displayed. After training is complete, a test is performed using the `sim` function to simulate the environment and save the experience data.

[0158] The progress graph of the Simulink model in this embodiment is as follows: Figure 5 As shown, this reflects the policy optimization process during training and the agent's control performance during testing. From Figure 5 As can be seen, the reward per round gradually increases with the increase of training rounds, indicating that the performance of the agent is constantly improving; while the average reward does not change much, indicating that the multi-agent reinforcement system is highly stable.

[0159] (iv) Event Trigger Settings

[0160] 1. Lyapunov stability theory

[0161] In Lyapunov's stability theory framework, the stability of an equilibrium point is rigorously defined mathematically: if there exists a neighborhood such that all initial-state trajectories are always confined within that neighborhood, then the equilibrium point satisfies the stability condition; otherwise, it is considered an unstable state. Building upon this, asymptotic stability further extends the stability boundary, requiring the system state to maintain local boundedness and also satisfying the condition that the trajectory converges to the equilibrium point as time approaches infinity. Lyapunov provides a sufficient nonlinear criterion for these two types of stability by constructing an energy function.

[0162] For dynamic systems with explicit time variables, Lyapunov theory establishes a universal analytical paradigm for stability criteria through the energy decay condition. Consider a self-consistent system as shown in the following equation:

[0163] ;

[0164] in, Represents from region D to Local Lipschitz mapping

[21] , assuming The system equilibrium point, i.e. .

[0165] The following conclusions can be drawn:

[0166] 1. If For any stable equilibrium point of the system, then... ,exist , making

[0167] ;

[0168] 2. If If the system is at an unstable equilibrium point, then the system is in... The situation is unstable.

[0169] 3. If For the asymptotically stable equilibrium point of the system, if It is both a stable equilibrium point and exists Make:

[0170] ;

[0171] Having defined stability and asymptotic stability, Lyapunov's stability theory is as follows:

[0172] make The equilibrium point in a self-consistent system. It includes The region. Let the function It is a continuously differentiable function that satisfies if and And satisfy ,but This is the equilibrium point.

[0173] In addition, if and And satisfy ,So This is the asymptotic equilibrium point.

[0174] If there exists a continuously differentiable function And simultaneously satisfy and If a certain function..., then the function can be called a Lyapunov function. satisfy as well as Then it is called positive definite. If the function satisfy This is called positive semi-definite. If a function It is called negative definite or negative semi-definite, then the corresponding It is called positive definite or positive semi-definite. If If a function does not fall under any of the above categories, it is called nondefinite.

[0175] Lyapunov stability theory states that if there exists a continuously differentiable positive definite function... And its derivative satisfies the condition of negative semi-definiteness, then the origin For a stable equilibrium point; if its derivative is negative definite, then the origin... This is the asymptotic equilibrium point.

[0176] According to the conditions for Lyapunov stability theory, a function can be assumed to be stable. However, if a function satisfies stability, the Lyapunov conditions may not necessarily hold. Extending this to the control system of this embodiment, since different application scenarios require different Lyapunov functions, if a certain Lyapunov function fails to satisfy the stability criteria for determining the system's equilibrium point, the corresponding equilibrium point may not be an unstable equilibrium point or an asymptotic equilibrium point. Further analysis of the equilibrium point is needed, or other Lyapunov functions can be sought to verify whether the equilibrium point is stable or asymptotically stable.

[0177] 2. Lassalle's invariance principle

[0178] Assumption For a solution of a certain self-consistent system, the point Called The positive limit point, assuming there exists a sequence ,when From time to time All The set of positive limit points is called The set of positive limits. This is called the invariant set with respect to the self-consistent system.

[0179] for If at some point a solution belongs to the set Then the solution belongs to the set at any time in the future and the past. .

[0180] like satisfy Then it is called when When it approaches infinity, It will approach a set For any ,exist And satisfy .

[0181] in, Point to the set The distance, representing the point to set The minimum distance between all points:

[0182] ;

[0183] Lassalle's principle differs from Lyapunov's stability theory; Lassalle's invariance principle does not require the function to be constant. Positive definite. Lassalle's principle is summarized as follows:

[0184] Let set Let be a positive invariant set of self-consistent systems, and let continuously differentiable functions be defined. ,exist China satisfies ,gather yes All points satisfied The set that constitutes the composition.

[0185] Assumption For set The largest invariant set in the set is, for any set derived from... The solution in, when The solutions will all approach the set. The largest invariant set in [the dataset].

[0186] 3. Event Triggering Control Mechanism

[0187] Event-triggered control differs from traditional control based on time-based conditions; instead, it determines whether a control task is executed based on event-triggered conditions. Through a dynamic threshold mechanism, event-triggered control can achieve an optimal balance between resource efficiency and system performance, consuming fewer computational resources compared to traditional control methods. By employing an exponential threshold design, it can effectively avoid unlimited triggering while maintaining system stability and computational accuracy, all while ensuring computational convergence.

[0188] In this embodiment, the event-triggered control adopts a distributed event-triggered control method. Distributed event-triggered control can reflect the information interaction characteristics of a multi-agent system. Assuming a system with... A distributed multi-agent system with [number] agents is described as follows:

[0189] ;

[0190] When data is transmitted between agents in the form of digital signals, assuming the sampling period of all agents is... intelligent agent Whether the sampled signal propagates depends on the sampling time. Is the event triggering condition violated? The event triggering condition is:

[0191] ;

[0192] in: For threshold parameters, The time when the sampled signal of the agent is propagated, and the following conditions are met:

[0193] ;

[0194] ;

[0195] In particular, when using event-triggered control, in order to avoid the Zeno phenomenon where an infinite number of events occur within a finite number of events, the minimum time interval between the execution times of any two events must be strictly greater than zero, and the joint design of feedback control and event triggering mechanism parameters must be emphasized.

[0196] (v) Markov decision-making

[0197] Markov decision-making (MDP) is a framework for reinforcement learning used to describe sequential decision-making by agents in uncertain environments. In multi-agent systems, each agent's state transitions and reward functions depend on the actions of other agents; specifically, the action decision-making process of a single agent can be viewed as a local Markov decision.

[0198] A Markov decision consists of a quintuple. Definition. In this quintuple, Represents the state space set, Represents the action space set; Represents the state transition probability, defined as the probability given the current state. And satisfy the action Under the condition that the state transitions, The probability of; Let the reward function represent the actions taken by the agent. Transferred to The agent then receives an immediate reward. A discount factor used to assess immediate rewards and future impact.

[0199] In this embodiment of the multi-agent system, Markov decision-making is used to represent the decision-making process of an agent with complete observability of the system state. At each time step t, the agent is in the system state... Choose to perform an action This allows the system to switch to... And the agent receives an immediate reward. .

[0200] Therefore, the goal of solving Markov decision problems is to find a strategy. That is, from the state space To the action space Mapping of the upper distribution. In this state Cumulative discount rewards can be defined as:

[0201] ;

[0202] Because the strategy and state transitions are random, This represents the expected value of the average over all possible trajectories; This represents the weighting coefficient of the reward at step t; The instant reward at step t; Indicates action according to strategy In state Distribution sampling under the following conditions; Indicates from state Initial cumulative rewards.

[0203] The state value function and action value under the policy are defined as follows:

[0204] ;

[0205] ;

[0206] in, , , and These represent the system states from and The expectation of initial cumulative discount rewards.

[0207] (vi) Multi-agent event-triggered control based on PPO algorithm

[0208] In multi-agent event-triggered control based on the PPO policy gradient algorithm, Markov decision-making provides the foundation for decision modeling for each agent. Based on Markov decision-making, the state space includes the agent's own state, the communication state of neighboring agents, and environmental information; the action space is defined as whether to trigger communication or update the control input; and the reward function combines consistency performance metrics and communication costs.

[0209] In this embodiment, the PPO algorithm directly learns stochastic policies through an optimized policy network to process continuous action spaces and high-dimensional state spaces, meeting the need for dynamic adjustment of triggering policies in multi-agent systems. The core of event-triggered control is to ensure that the agent updates the control state only when the state error exceeds a threshold. The PPO algorithm, by maximizing cumulative rewards, drives the agent to learn adaptive triggering policies, achieving on-demand triggering in distributed cooperative tasks. This balances control performance with communication overhead, significantly reducing communication frequency while maintaining system performance. ] .

[0210] This embodiment features a multi-agent distributed event-triggered control architecture based on the PPO algorithm, as follows: Figure 6 As shown, a data-driven approach to autonomously optimize the triggering mechanism provides an effective way to solve the communication efficiency problem in large-scale distributed control. In multi-agent event-triggered control based on the PPO policy gradient algorithm, two variables are defined in the event triggering condition: the control agent... Is it related to intelligent agents? communicative and intelligent agents Whether to receive from the intelligent agent The Group of observations of The formula for calculating the trigger condition is as follows:

[0211] ;

[0212] ;

[0213] In the formula, and , and All of these are calculated using a strategy network.

[0214] Based on the above event triggering conditions, the final input is given to the agent. Observational information The elements are updated as follows:

[0215] ;

[0216] The formula for calculating the joint strategy is as follows:

[0217] ;

[0218] in, Represents control input, and Indicate control decision, It is observation information. These are the actions of the agent. The joint policy can be continuously updated to guide the agent to learn in the direction of maximizing reward.

[0219] In this embodiment, distributed event-triggered control achieves rapid convergence of the agent's reward value through continuous optimization of the policy network's structure and parameters. To facilitate a balance between the agent's movement process and communication costs, a reward function is proposed:

[0220] ;

[0221] in, This represents the total reward the agent receives at each moment.

[0222] Rewards for the agent's motion control Designed as follows:

[0223] ;

[0224] in, It is the straight-line distance between the agent's actual position and its desired position.

[0225] To prevent collisions caused by agents being too close together during exploration, a collision penalty term is implemented. Designed as follows:

[0226] ;

[0227] in, , Let be the straight-line distance between any two different intelligent agents. This represents the straight-line distance between the agent and the i-th obstacle. Since the key terms in the calculation are all relative distances, it can be concluded that the penalty intensity is negatively correlated with both the distance between agents and the relative distance between the agent and the obstacle.

[0228] To reduce communication between agents and thus lower communication costs, this embodiment further penalizes agent communication, including a collision penalty item. Designed as follows:

[0229] ;

[0230] in, The function aims to minimize the amount of data communicated and received by agents at any given time, and adds a collision term to penalize multi-agent communication.

[0231] Furthermore, this embodiment describes the multi-agent event triggering algorithm flow based on the PPO policy gradient algorithm, as follows: Figure 1 As shown:

[0232] After setting up the multi-agent environment, the training parameters are initialized, and then the multi-agent scene is randomly initialized.

[0233] Subsequently, event-triggered control is implemented, using event triggering conditions to determine whether communication exists between agents, and jointly calculating control inputs and communication decisions via a policy network. In distributed event-triggered control, the initial trigger threshold is set to 15% of the maximum value in the state space and bound to a Lyapunov function decay rate as a dynamic adjustment policy. The joint reward of the multi-agent system needs to be normalized to... Furthermore, cooperative rewards are multiplied by a coefficient of 0.8, and adversarial rewards are multiplied by a coefficient of 1.2 to strengthen the competitive orientation.

[0234] As each agent interacts with its environment, the system updates and stores the observation information and related data of each agent in a timely manner until the maximum data processing capacity of the system is reached.

[0235] The PPO (Progressive Policies-Based Programming) gradient algorithm is used to train the agents and continuously update the network parameters and communication decisions, prompting each agent to increase its movement probability. As time increases, the multi-agent system will gradually approach a stable state, until training ends when the maximum number of training rounds is reached.

[0236] Furthermore, a multi-agent event-triggered control simulation test was conducted using six agents. First, the multi-agent system was initialized. The connection relationships among the six agents were defined using the Laplace matrix L, with its diagonal elements representing node degrees. To test the system's convergence capability under non-uniform initial conditions, the agents were distributed and their two-dimensional states were salient. The threshold function decay rate was then controlled by adjusting parameters, tolerating larger errors initially while strictly constraining error growth later. The output results are as follows: Figures 7 to 9 As shown.

[0237] Figure 7 This diagram illustrates that, under the control of event triggering conditions, the triggering times for different agents vary with time. The density of points in the diagram represents how multiple agents, through continuous strategy adjustments, avoid the system falling into the trap of excessively rapid threshold decay, ensuring system convergence in later stages. This diagram leads to the conclusion that the node triggering times for different agents are non-uniformly distributed, occurring frequently during the system adjustment phase and sparsely during the system's stable phase.

[0238] Figure 8 The different colored curves represent different agents. As the running time increases, the six agents experience a brief period of oscillation, and after the threshold of the triggering event condition decays, they enter a stable phase. Finally, the state values ​​of the six agents are... The fact that all values ​​tend to 0.5 indicates that the system has reached stability. This figure demonstrates that under the event-triggered mechanism, the multi-agent state can still converge even at non-uniform triggering times, and the multi-agent system can achieve the consensus objective.

[0239] Figure 9 It consists of 6 subgraphs, each representing the relationship between the error of each node and the dynamic threshold under event-triggered control for different agents. Figure 9 The intersection of the curves represents the event trigger point. When the error curve exceeds a set threshold, an event is triggered. Since the threshold changes exponentially over time, the initial large error before the system reaches a stable state leads to frequent triggers. As the system gradually stabilizes in the later stages of control, the trigger interval lengthens. This figure shows that the real-time error of each agent is consistently controlled below the exponentially decaying threshold, confirming that event triggering can effectively control multi-agent systems. The triggering conditions effectively constrain error growth, ensuring the effectiveness of the control input and system stability.

[0240] In summary, by comparing the multi-agent error with the threshold, it is shown that the real-time error of each agent is consistently controlled below the exponentially decaying threshold, confirming that the event triggering of this invention can effectively control multi-agent systems. The triggering condition can effectively constrain error growth, ensuring the effectiveness of control input and system stability.

[0241] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multi-agent power system distribution event triggering control method based on the PPO algorithm, characterized in that, Includes the following steps: Step 1: Create a multi-agent power system: Each agent corresponds to a distributed energy unit in the power system, including: photovoltaic power generation unit, wind power generation unit and energy storage unit. Power balance and voltage regulation are achieved through local information interaction, and the distributed energy units are coordinated to operate together on the grid. Step 2: Construct a Simulink model based on multiple agents: Based on the multi-agent power system, build a simulation model on the Simulink simulation platform to perform multi-agent collaborative simulation, including multiple agent modules and a core environment module. The multiple agent modules adopt a distributed decision-making mechanism and use a signal bus to couple information between agents. Step 3: Multi-agent reinforcement learning based on PPO algorithm: Bind the Simulink model to the reinforcement learning interface, create an independent policy network and value network for each agent, configure the policy network as a random policy with the probability distribution output by the PPO algorithm, set the hyperparameters of the PPO algorithm, train the actions of multiple energy unit agents, and optimize the agent policies. Step 4, Event Trigger Settings: Based on the principles of Lyapunov stability and Lassalle invariance, an exponential decay function is used as the threshold form in event trigger control; Step 5, Markov Decision: Describe the sequential decision-making of an agent in an uncertain environment using the Markov decision process; Step 6: Multi-agent event-triggered control based on PPO algorithm: Distributed event-triggered control is performed based on PPO algorithm. By optimizing the policy network to learn random policies, the continuous action space and high-dimensional state space are processed. By maximizing the cumulative reward, the agents learn adaptive triggering policies. According to the event-triggered control mechanism, communication decisions and control action updates between agents are triggered as needed, so that the multi-agent power system gradually tends to a stable state.

2. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, In the Simulink model described in step 2, each agent module integrates a policy decision unit and is equipped with input and output ports; The input port receives environmental status and external signals, and the output port outputs the status observation, action, immediate reward and termination condition isdone. The output port is transmitted to the core environment module through the signal bus. The core environment module receives the joint actions of all agents and triggers a state transition, calculates the system state at the next moment, feeds back the updated environmental parameters and reward signals, and constructs an explicit communication channel to transmit policy parameters.

3. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, In the PPO algorithm described in step 3, the objective function is expressed as: ； in, Indicates the probability ratio. Represents the dominance function. Indicates mutation parameters, obtained through clipping operations. Will Limited to the range Inside; , These are weighting coefficients used to control the early-stage strategy updates and the later-stage computational convergence. The value function predicts the current state. This represents the total cumulative reward actually obtained by the intelligent agent. Indicates the current state of the agent Actions under the new strategy The probability of; The total loss function is expressed as: ； in, It is the entropy regularization term. Represents the policy gradient loss. This represents the loss of function value.

4. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, The PPO algorithm described in step 3 is as follows: Step 301, Initialization: Initialize the policy network and value network, as well as the optimizer and related configuration parameters; Step 302, Sampling: Using the current strategy Interact with the real environment and collect the states of multiple agents. ,action ,award and next state data; Step 303: Calculate the dominance function: Using the dominance function For each intelligent agent In The dominance function is calculated for actions performed under certain conditions to measure the superiority or inferiority of the action compared to the average performance, and gradually leverages... The guidance strategy is continuously improved; in, This indicates that the agent is in a certain state. Take action below Action value function; Representing state State value function; Step 304, Strategy Update: Use the Clip method to calculate the probability ratio between the old and new strategies and limit it to a preset range; Step 305, Value Function Update: Using the loss function Optimize value function The value network is updated by minimizing the mean square error; where, This represents the total cumulative reward that the agent can actually obtain starting from this state. Expressing expected value based on experience; Step 306: Repeat steps 3.2 to 3.5, iterating through multiple rounds to gradually optimize the strategy until convergence.

5. The multi-agent power system distribution event triggering control method according to claim 4, characterized in that, In step 3, the agent's policy is optimized using the PPO algorithm. The specific steps are as follows: Step 311: Define the sampling time, total time, and maximum number of steps in the training settings; Step 312: Start the Simulink model, set the number of agents and the size of the observation space, use discrete {0,1} to represent the action space, and create an env environment to connect the various agent modules in Simulink. Step 313: Iteratively construct the policy network Actor and value network Critic for each agent module. The Actor network maps observations to action probability distributions through a multilayer perceptron, and the Critic network evaluates state value to calculate the advantage function. The Actor network structure consists of an input layer, multiple fully connected layers, and ReLU activation, which outputs action probabilities via softmax; the Critic network outputs a single value to estimate the agent's state value. Step 314: Configure the PPO algorithm, set the entropy loss weight, create the corresponding PPO agent and start training. Improve the stability of multi-agent training by limiting the magnitude of policy updates. Step 315: After training is completed, conduct tests, simulate the environment using the sim function, and save the experience data.

6. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, Step 4 describes the event triggering control, triggering parameters. From the current transmission signal The last transmitted signal ,parameter , and A joint decision was made to adjust the threshold based on resource estimates, with the specific triggering conditions as follows: ； ； in, It is an adjustable parameter. for Given a lower bound, and if The adaptive event triggering mechanism will then become a time-invariant communication mechanism; Indicates the time when the next event will be triggered; This indicates the current event trigger time; h represents the sampling period, and l represents the number of sampling periods to wait for, which is the number of sampling periods elapsed from the current trigger to the next trigger; Condition measurement error, The positive definite weighting matrix is ​​used to define the error; Distributed event-triggered control embodies the characteristics of information interaction in multi-agent systems and has... Distributed multi-agent system with multiple agents The description is as follows: ； in, Represents the state vector of the i-th agent. This represents the control input of the i-th agent. , It is the system state parameter matrix; is the initial state of agent i, and N is the total number of agents in the system; When data is transmitted between agents in the form of digital signals, the agents Whether the sampled signal propagates depends on the sampling time. Is the event triggering condition violated? The event triggering condition is as follows: ； in, For threshold parameters, The time when the sampled signal of the intelligent agent is propagated. Describes the first... Index of the time when the next event is triggered. Represents the measurement error vector of agent i; The reference signal representing agent i; This indicates that the positive definite weighted matrix is ​​used to define the error.

7. The multi-agent power system distribution event triggering control method according to claim 6, characterized in that, The Markov decision process described in step 5 is as follows: Step 501: Treat the action decision-making process of a single agent as a local Markov decision, consisting of a quintuple. definition; in, Represents the state space set, Represents the action space set; Represents the state transition probability, defined as the probability given the current state. And satisfy the action Under the condition that the state transitions, The probability of; Let the reward function represent the actions taken by the agent. Transferred to The agent then receives an immediate reward. A discount factor used to assess immediate rewards and future impact; Step 502: In each time step t, the agent is in the system state Choose to perform an action This allows the system to switch to... And the agent receives an immediate reward. ; Step 503: Solve the Markov decision problem, with the goal of finding a strategy. This is manifested in the state space. To the action space Mapping of the upper distribution; In this state The cumulative discount reward is defined as follows: ； in, This represents the expected value of the average over all possible trajectories. This represents the weighting coefficient of the reward at step t; The instant reward at step t; Indicates action according to strategy In state Distribution sampling under the following conditions; Indicates from state Initial cumulative rewards; The state value function under the policy and action value Defined as: ； ； in, , , and These represent the system states from and The expectation of initial cumulative discount rewards.

8. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, In the multi-agent event triggering control described in step 6, the event triggering condition is expressed as follows: ； ； In the formula, Represents the controlling agent Is it related to intelligent agents? communication, Represents intelligent agents Whether to receive from the intelligent agent The Group of observations , and All were calculated using a policy network. and ; Based on the event triggering conditions, the final input is given to the intelligent agent. Observational information The elements are updated as follows: ； The formula for calculating the joint strategy is as follows: ； in, Represents control input, and Indicate control decision, It is observation information. It refers to the actions of the agent; the joint policy guides the agent to learn continuously towards the direction of maximizing reward through constant updates.

9. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, Step 6 describes a distributed event-triggered control system that optimizes the structure and parameters of the policy network to achieve rapid convergence of the agent's joint reward value. The joint reward function is shown in the following equation. Represents the total reward obtained by the agent at time i: ； in, Represents the reward for the agent's motion control: ； in, The actual location of the intelligent agent With desired location The straight-line distance between them; Indicates the collision penalty for the intelligent agent: ； in, , Let be the straight-line distance between any two different intelligent agents. This represents the straight-line distance between the agent and the i-th obstacle; Indicates the penalty for agent communication: ； in, The function aims to minimize the amount of data communicated and received by agents at any given time, and adds a collision term to penalize multi-agent communication.

10. The multi-agent power system distribution event triggering control method according to claim 1, characterized in that, In the distributed event-triggered control described in step 6, the initial trigger threshold is 15% of the maximum value in the state space, and a Lyapunov function decay rate is bound as a dynamic adjustment strategy to normalize the joint reward of the multi-agent power system to... Furthermore, cooperative rewards are multiplied by a coefficient of 0.8, and adversarial rewards are multiplied by a coefficient of 1.2 to strengthen the competitive orientation.

Images