Power system multi-agent parallel training method based on swarm intelligence optimization

Abstract

The invention provides an electric power system multi-agent parallel training method based on swarm intelligence optimization. The method comprises the following steps: constructing a system simulation model based on a to-be-controlled novel electric power system; constructing an initial multi-agent based on the system simulation model; initializing particle swarm parameters and initializing an initial multi-agent and a plurality of training agents; carrying out distributed parallel training on the plurality of training agents until all the training agents are trained, and obtaining a current round of multi-agents; acquiring fitness of each agent in the current round of agent group and updating particle swarm parameters, and further updating basic parameters of each agent in the current round of agent group to update a plurality of training agents; the distributed parallel training of the plurality of training agents is repeated until a preset training condition is met, and an agent group is obtained; and controlling the to-be-controlled novel power system based on the intelligent agent group. Parallel training of multiple agents is realized, and the control efficiency of a novel power system can be improved.

Description

Technical Field

[0001] The present invention relates to the field of novel power system control technology, and in particular to a power system multi-agent parallel training method based on swarm intelligence optimization. Background Art

[0002] Compared with the traditional power system, the new power system has undergone profound changes in terms of technical foundation, operation mechanism and functional form, especially in terms of power supply composition, grid scale and load characteristics, which have stronger uncertainty, complexity and nonlinearity. Therefore, the actual operation and control of the new power system faces more difficulties, and a more complete and flexible control method is urgently needed.

[0003] In applications related to new power systems, deep reinforcement learning with online learning and experience replay capabilities is an effective control method. In existing reinforcement learning methods for new power systems, in order to solve the dimensionality curse anomaly caused by the ultra-high dimensional data of new power systems, multi-agent deep reinforcement learning algorithms are usually used to control new power systems. The data dimension of this type of method is often bound to the number of agents, and the size of the action space often increases exponentially with the number of agents. However, the memory of a single computing node is limited and usually cannot fully support large-scale multi-agent reinforcement learning models; in addition, the computing power of a single node also greatly constrains the training speed of large-scale multi-agent reinforcement learning models.

[0004] Although the existing multi-computing node distributed multi-agent reinforcement learning training methods can effectively solve the above two problems, the existing methods are usually prone to fall into local optimal solutions during the training process when facing multi-objective control tasks with deceptive rewards and sparse rewards. The convergence speed is extremely slow, and it is difficult to obtain the optimal policy. It is difficult to meet the requirements of large-scale multi-agent reinforcement learning training and the control requirements of new power systems. Summary of the invention

[0005] The present invention aims to provide a method for parallel training of multi-agents in power systems based on swarm intelligence optimization to solve the above-mentioned technical problems. By optimizing the network parameter adjustment process of multi-agents in distributed training scenarios in new power systems, parallel training of multi-agents can be achieved to improve the overall performance of multi-agents and the control efficiency of new power systems.

[0006] In order to solve the above technical problems and realize the enhanced and efficient training of large-scale multi-agents, the present invention provides a multi-agent parallel training method for a power system based on swarm intelligence optimization, comprising the following steps:

[0007] Construct a system simulation model based on the new power system to be controlled;

[0008] Determine the system control target based on the system simulation model and construct the initial multi-agent based on the system control target;

[0009] Initialize the particle swarm parameters, and initialize the basic parameters of each agent in the initial multi-agent according to the particle swarm parameters to obtain several training agents;

[0010] Conduct distributed parallel training on several training agents until all training agents have completed training, and obtain the current round of multi-agents;

[0011] Obtain the fitness of each agent in the current round of agent group, and update the particle swarm parameters based on the fitness, and then update the basic parameters of each agent in the current round of agent group to update several training agents;

[0012] Repeat the distributed parallel training of several training agents until the preset training conditions are met to obtain a group of agents;

[0013] Control of new power systems based on intelligent agent groups.

[0014] In the above scheme, by optimizing the network parameter adjustment process of multiple agents in the distributed training scenario of the new power system, parallel training of multiple agents is realized to improve the overall performance of multiple agents, improve the overall performance and training efficiency of multiple agents, and thus improve the control efficiency of the new power system.

[0015] It should be noted that this method can be implemented in an environment with a central node server and several sub-node servers, that is, this method realizes large-scale multi-agent distributed parallel training of the new power system on a general server cluster, effectively improving the overall performance and training efficiency of the multi-agent, and thus improving the control efficiency of the new power system.

[0016] Furthermore, the method of determining the system control target based on the system simulation model and constructing the initial multi-agent based on the system control target includes the following steps:

[0017] Determine the system topology, equipment sub-model and load sub-model based on the system simulation model;

[0018] Determine the grid stability control target, energy optimization dispatch control target and fault recovery control target based on the system topology, equipment sub-model and load sub-model;

[0019] Based on the grid stability control objectives, energy optimization dispatch control objectives and fault recovery control objectives, construct power system stability intelligent agent, energy optimization dispatch intelligent agent and fault recovery intelligent agent;

[0020] Determine the Markov decision process of the power system stability agent, energy optimization scheduling agent and fault recovery agent, and construct the initial multi-agent.

[0021] Furthermore, the Markov decision process includes:

[0022] A tuple defining a Markov decision:

[0023] <S,A,P,R,γ>

[0024] Where S is the state vector of the agent, S = {s1, s2, …, s n},s n represents the nth observation of the power system; A is the action vector of the agent, A={a1,a2,…,a m}, a m represents the mth relevant decision quantity of the power system; P is the state transition probability of the intelligent agent, which is expressed as:

[0025]

[0026] The state S of the power system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, expressed as:

[0027]

[0028] in Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

[0029] Furthermore, the Markov decision process of determining the power system stability agent, the energy optimization dispatch agent and the fault recovery agent to construct the initial multi-agent includes:

[0030] For agents with different system control objectives, the Markov decision process has different state vectors, action vectors, state transitions, and reward functions; among them:

[0031] For the power system stability agent, the state vector S1 includes the power system frequency f t , node voltage V t 、Generator speedω t , load level L t , power flow P t and energy storage state E t ,have:

[0032] S1=[f t ,V t ,ω t ,L t ,P t ,E t ] T

[0033] Action vector A1 includes the regulating power P of the generator g , load adjustment ΔL t 、Charging and discharging power of energy storage device P e , Photovoltaic discharge power P pv And the fan discharge power P wind ,have:

[0034] A1=[P g ,ΔL t ,P e ,P pv ,P wind ] T

[0035] The reward function R1 is based on the frequency deviation Δf t , voltage deviation ΔV t Design, with:

[0036]

[0037] Among them, k f , k v and k s Both represent weight coefficients; S st It is an indicator of power system stability;

[0038] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0039] S t+1 =f(S t ,A t ,w t )

[0040] Among them, w t is the system noise and external disturbance;

[0041] For the energy optimization scheduling agent, the state vector S2 includes the output power P of each power generation unit g , Energy storage device status E t , Load demand D t , Renewable Energy Forecast t and electricity price information C t ,have:

[0042] S2=[P g ,Et ,D t ,R t ,C t ] T

[0043] Action vector A2 includes the generation scheduling plan ΔP g , energy storage charging and discharging power P e , Load adjustment strategy ΔD t , Power purchase or sales strategy P b ,have:

[0044] A2=[ΔP g ,P e ,ΔD t ,P b ] T

[0045] The reward function R2 is expressed as:

[0046]

[0047] Among them, C g , C e , C b is the cost of power generation, energy storage, and electricity purchase and sale, η r Incentive factor for renewable energy utilization;

[0048] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0049] S t+1 =f(S t ,A t ,ζ t )

[0050] Among them, t Errors in load forecasting and renewable energy forecasting;

[0051] For the fault recovery agent, the state vector S3 includes the fault location F t , Fault type T t , Line Status L t , Standby power state B t , load level D t and recovery progress R t ,have:

[0052] S3=[F t ,T t ,L t ,B t ,D t ,R t ] T

[0053] Action vector A3 includes fault isolation measure I t , Fault point repair solution M t , load transfer strategy ΔS t and backup power enabled ΔB t ,have:

[0054] A3=[I t ,M t ,ΔS t ,ΔB t ] T

[0055] The reward function R3 is expressed as:

[0056]

[0057] Among them, k r and k l is the weight coefficient, R t is the recovery time and δ i is the power supply status of load node i;

[0058] The state transfer equation is expressed as:

[0059] S t+1 =f(S t ,A t ,η t )

[0060] Among them, η t This is an uncertainty factor in the fault recovery process.

[0061] Furthermore, the particle swarm parameters are initialized, and the basic parameters of each agent in the initial multi-agent are initialized according to the particle swarm parameters to obtain a plurality of training agents, including:

[0062] Define the particle swarm parameters and initialize P particles in the particle swarm. Each particle p represents a possible agent network parameter configuration and its allocation scheme. Each particle position X p Contains the basic parameters of the agent, including network parameters and allocation information; each particle speed V p Used to guide particle position updates;

[0063] Initialize the particle position, specifically expressed as:

[0064]

[0065] Among them, Θ represents the network parameter space of all agents, Π represents the agent allocation strategy space, and all particle speeds are initialized to 0, so:

[0066]

[0067] At the same time, initialize the individual optimal position of the particle swarm and the global optimal position They are:

[0068]

[0069] Among them, F p (X p ) is the fitness function, which is used to evaluate the position quality of the particle;

[0070] After completing the above initialization, several training agents are obtained.

[0071] Furthermore, the distributed parallel training of the plurality of training agents is performed until all training agents are trained to obtain the current round of multi-agents, including:

[0072] Each training agent A i With a corresponding deep neural network θ i , by defining the mapping M to map the particle positions to the network parameters of each training agent, we have:

[0073]

[0074] Where N is the total number of agents, is the training agent A corresponding to particle p i Next, several training nodes for distributed parallel training of training agents are preset, and resource evaluation is performed on each training node to obtain the memory M of each training node. m and the calculation speed C m ; m = 1, 2, ..., M, which represents the sequence number of each node;

[0075] According to the memory M of each training node m and the calculation speed C m Several training agents are adaptively allocated to different training nodes for distributed parallel training according to the resource requirements of each training agent until all training agents are trained to obtain the current round of multi-agents.

[0076] Furthermore, the memory M of each node m and the calculation speed C m According to the resource requirements of each training agent, several training agents are adaptively allocated to different training nodes for distributed parallel training until all training agents are trained, and the current round of multi-agents is obtained, including:

[0077] Determine the training agent A iResource requirements, including memory requirements m i and computational requirements c i , where i = 1, 2, …, I, represents the number of training agents;

[0078] Define the training agent allocation scheme π s As a mapping function:

[0079] π s :A→S

[0080] where π s (A i )=S m Represents training agent A i Assigned to training node S m Conduct training;

[0081] For each training agent, the minimization target is determined based on its mapping function to adaptively allocate different training agents to different training nodes for distributed parallel training until all training agents are trained and the current round of multi-agents is obtained; the minimization target is expressed as:

[0082]

[0083] Furthermore, the distributed parallel training includes:

[0084] Each training node is responsible for iterative training of the assigned training agent in the local environment to accumulate the local experience pool until the number of training rounds reaches the upper limit or the training agent reaches the convergence condition; where: Each training node S m Follow the steps below to train agent A responsible for it i The iterative training process is:

[0085] In each training round t, collect training agent A i Experience samples After that, store it in the local experience pool Right now:

[0086]

[0087] By drawing from the experience pool A small batch of experience is randomly sampled for training to achieve experience replay; at the same time, due to the limited storage resources of the training node, a priority experience replay strategy is used to manage the experience pool That is, according to the empirical priority p(τ i ), prioritize the sampling of experiences with higher importance, including:

[0088]

[0089] in, is the value network at round t and round t+1, α is the hyperparameter for adjusting the priority distribution, and γ is the discount factor;

[0090] When the number of training rounds of the training agent reaches the upper limit or the training agent reaches the convergence condition, the training of the training node is completed and the current round of multi-agents is obtained.

[0091] Furthermore, for each agent, the training objective of the distributed parallel training is expressed as:

[0092]

[0093] in, Represents the training objective for training the agent.

[0094] Furthermore, the obtaining of the fitness of each agent in the current round of agent group, and updating the particle swarm parameters based on the fitness, and then updating the basic parameters of each agent in the current round of agent group to update a number of training agents, includes:

[0095] Based on the fitness function, the fitness of each agent in the current round of agent group is obtained. The fitness function F of each agent p (X p ) is expressed as:

[0096] F p (X p )=α p Stability(X p )+β p ·Efficiency(X p )+γ p Recovery(X p )

[0097] Where: α p , β p , γ p is the weight coefficient, which is used to balance the importance of different performance indicators; Stability (X p )、Efficiency(X p )、Recovery(X p ) is the training agent A represented by the particle i Performance in maintaining power system stability, energy optimization dispatch and fault recovery;

[0098] Based on the fitness, the particle position and particle velocity of the particle are updated to guide the particle to converge to the optimal solution, specifically including: updating the particle velocity based on the particle velocity update formula, expressed as:

[0099]

[0100] in, is the velocity vector of particle p in the tth generation; w is the inertia weight, which is used to adjust the degree of dependence of the particle on its own velocity; c1 and c2 are learning factors, which control the degree of learning of the particle to the individual optimal position and the global optimal position respectively; r1 and r2 are uniformly distributed random numbers, and satisfy r1,r2∈[0,1];

[0101] Based on the updated particle velocity, the particle position is updated, expressed as:

[0102]

[0103] Based on this, the individual best position is updated according to the following formula:

[0104]

[0105] And update the global optimal position according to the following formula:

[0106]

[0107] Finally, according to the global optimal position X of the particle swarm global best , generate a new agent θ i ′, used to update the training agent, is:

[0108] θ′ i =M(X global best ,A i ). BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 A schematic diagram of a multi-agent parallel training method for a power system based on swarm intelligence optimization provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0110] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0111] Existing control methods for new power systems are usually designed for general deep learning frameworks and specific hardware environments. In addition, when faced with multi-objective control tasks with deceptive rewards and sparse rewards, the training process is prone to fall into local optimal solutions, the convergence speed is extremely slow, and it is difficult to obtain the optimal policy. These methods cannot meet the requirements of large-scale multi-agent reinforcement learning training in new power systems while adapting to general server clusters. To achieve efficient training of large-scale multi-agent reinforcement learning, please refer to Figure 1 This embodiment provides a multi-agent parallel training method for a power system based on swarm intelligence optimization, comprising the following steps:

[0112] Construct a system simulation model based on the new power system to be controlled;

[0113] Determine the system control target based on the system simulation model and construct the initial multi-agent based on the system control target;

[0114] Initialize the particle swarm parameters, and initialize the basic parameters of each agent in the initial multi-agent according to the particle swarm parameters to obtain several training agents;

[0115] Conduct distributed parallel training on several training agents until all training agents have completed training, and obtain the current round of multi-agents;

[0116] Obtain the fitness of each agent in the current round of agent group, and update the particle swarm parameters based on the fitness, and then update the basic parameters of each agent in the current round of agent group to update several training agents;

[0117] Repeat the distributed parallel training of several training agents until the preset training conditions are met to obtain a group of agents;

[0118] Control of new power systems based on intelligent agent groups.

[0119] This embodiment optimizes the network parameter adjustment process of multiple agents in the distributed training scenario of the new power system to achieve parallel training of multiple agents, thereby improving the overall performance of multiple agents, improving the overall performance and training efficiency of multiple agents, and thus improving the control efficiency of the new power system.

[0120] It should be noted that this method can be implemented in an environment with a central node server and several sub-node servers, that is, this method realizes large-scale multi-agent distributed parallel training of the new power system on a general server cluster, effectively improving the overall performance and training efficiency of the multi-agent, and thus improving the control efficiency of the new power system.

[0121] It should be noted that the process of controlling the new power system by a group of intelligent agents can be achieved by adopting the existing control method. The technical point of this embodiment is how to quickly train and obtain multiple intelligent agents to improve the overall performance of multiple intelligent agents, improve the overall performance and training efficiency of multiple intelligent agents, and then improve the control efficiency of the new power system.

[0122] In one embodiment, a system simulation model constructed based on the new power system to be controlled may include the topology, equipment model, and load model of the new power system, and construct a corresponding large signal mathematical model, thereby providing a basic environment for reinforcement learning training of multiple agents.

[0123] In this embodiment, the network topology of the power system can be determined in power system simulation software such as DSP and BPA, including the location and connection relationship of power plants, substations, transmission lines, distribution lines, and load nodes, the type of each node can be defined, and the electrical parameters of the transmission lines and distribution lines can be determined. A new power system simulation model of photovoltaic, wind turbine, synchronous generator, energy storage, transformer, interconnection line and other components is established, and then a complete large signal mathematical model of the system is established based on the obtained simulation model.

[0124] In one embodiment, determining the system control target based on the system simulation model and constructing the initial multi-agent based on the system control target includes the following steps:

[0125] Determine the system topology, equipment sub-model and load sub-model based on the system simulation model;

[0126] Determine the grid stability control target, energy optimization dispatch control target and fault recovery control target based on the system topology, equipment sub-model and load sub-model;

[0127] Based on the grid stability control objectives, energy optimization dispatch control objectives and fault recovery control objectives, construct power system stability intelligent agent, energy optimization dispatch intelligent agent and fault recovery intelligent agent;

[0128] Determine the Markov decision process of the power system stability agent, energy optimization scheduling agent and fault recovery agent, including the state, action, reward and transfer mechanism, and build the initial multi-agent.

[0129] In one embodiment, the Markov decision process includes:

[0130] A tuple defining a Markov decision:

[0131] <S,A,P,R,γ>

[0132] Where S is the state vector of the agent, S = {s1, s2, …, s n},s n represents the nth observation of the power system; A is the action vector of the agent, A={a1,a2,…,a m}, a m represents the mth relevant decision quantity of the power system; P is the state transition probability of the intelligent agent, which is expressed as:

[0133]

[0134] The state S of the power system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, expressed as:

[0135]

[0136] in Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

[0137] In one embodiment, the Markov decision process for determining the power system stability agent, the energy optimization dispatch agent, and the fault recovery agent to construct the initial multi-agent includes:

[0138] For agents with different system control objectives, the Markov decision process has different state vectors, action vectors, state transitions, and reward functions; among them:

[0139] For the power system stability agent, the state vector S1 includes the power system frequency f t , node voltage V t 、Generator speedω t , load level L t , power flow P t and energy storage state E t ,have:

[0140] S1=[f t ,V t ,ω t ,L t ,P t ,E t ] T

[0141] Action vector A1 includes the regulating power P of the generator g , load adjustment ΔL t 、Charging and discharging power of energy storage device P e , Photovoltaic discharge power Ppv And the fan discharge power P wind ,have:

[0142] A1=[P g ,ΔL t ,P e ,P pv ,P wind ] T

[0143] The reward function R1 is based on the frequency deviation Δf t , voltage deviation ΔV t Design, with:

[0144]

[0145] Among them, k f , k v and k s Both represent weight coefficients; S st It is an indicator of power system stability;

[0146] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0147] S t+1 =f(S t ,A t ,w t )

[0148] Among them, w t is the system noise and external disturbance;

[0149] For the energy optimization scheduling agent, the state vector S2 includes the output power P of each power generation unit g , Energy storage device status E t , Load demand D t , Renewable Energy Forecast t and electricity price information C t ,have:

[0150] S2=[P g ,E t ,D t ,R t ,C t ] T

[0151] Action vector A2 includes the generation scheduling plan ΔP g , energy storage charging and discharging power P e , Load adjustment strategy ΔD t , Power purchase or sales strategy P b ,have:

[0152] A2=[ΔPg ,P e ,ΔD t ,P b ] T

[0153] The reward function R2 is designed to optimize the system operation cost and energy utilization efficiency, which can be expressed as:

[0154]

[0155] Among them, C g , C e , C b is the cost of power generation, energy storage, and electricity purchase and sale, η r Incentive factor for renewable energy utilization;

[0156] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0157] S t+1 =f(S t ,A t ,ζ t )

[0158] Among them, t Errors in load forecasting and renewable energy forecasting;

[0159] For the fault recovery agent, the state vector S3 includes the fault location F t , Fault type T t , Line Status L t , Standby power state B t , load level D t and recovery progress R t ,have:

[0160] S3=[F t ,T t ,L t ,B t ,D t ,R t ] T

[0161] Action vector A3 includes fault isolation measure I t , Fault point repair solution M t , load transfer strategy ΔS t and backup power enabled ΔB t ,have:

[0162] A3=[I t ,M t ,ΔS t ,ΔB t ]T

[0163] The reward function R3 is designed to minimize the fault recovery time and the unpowered load, which can be specifically expressed as:

[0164]

[0165] Among them, k r and k l is the weight coefficient, R t is the recovery time and δ i is the power supply status of load node i;

[0166] The state transfer equation is expressed as:

[0167] S t+1 =f(S t ,A t ,η t )

[0168] Among them, η t This is an uncertainty factor in the fault recovery process.

[0169] In one embodiment, the particle swarm parameters are initialized, and the basic parameters of each agent in the initial multi-agent are initialized according to the particle swarm parameters to obtain a plurality of training agents, including:

[0170] Define the particle swarm parameters and initialize P particles in the particle swarm. Each particle p represents a possible agent network parameter configuration and its allocation scheme. Each particle position X p Contains the basic parameters of the agent, including network parameters and allocation information; each particle speed V p Used to guide particle position updates;

[0171] Initialize the particle position, specifically expressed as:

[0172]

[0173] Among them, Θ represents the network parameter space of all agents, Π represents the agent allocation strategy space, and all particle speeds are initialized to 0, so:

[0174]

[0175] At the same time, initialize the individual optimal position of the particle swarm and the global optimal position X global best , respectively:

[0176]

[0177] Among them, Fp (X p ) is the fitness function, which is used to evaluate the position quality of the particle;

[0178] After completing the above initialization, several training agents are obtained.

[0179] In one embodiment, the distributed parallel training of the plurality of training agents is performed until all training agents are trained to obtain the current round of multi-agents, including:

[0180] Each training agent A i With a corresponding deep neural network θ i , by defining the mapping M to map the particle positions to the network parameters of each training agent, we have:

[0181]

[0182] Where N is the total number of agents, is the training agent A corresponding to particle p i Next, several training nodes for distributed parallel training of training agents are preset, and resource evaluation is performed on each training node to obtain the memory M of each training node. m and the calculation speed C m ; m = 1, 2, ..., M, which represents the sequence number of each node;

[0183] According to the memory M of each training node m and the calculation speed C m Several training agents are adaptively allocated to different training nodes for distributed parallel training according to the resource requirements of each training agent until all training agents are trained to obtain the current round of multi-agents.

[0184] In one embodiment, the memory M of each node m and the calculation speed C m According to the resource requirements of each training agent, several training agents are adaptively allocated to different training nodes for distributed parallel training until all training agents are trained, and the current round of multi-agents is obtained, including:

[0185] Determine the training agent A i Resource requirements, including memory requirements m i and computational requirements c i , where i = 1, 2, …, I, represents the number of training agents;

[0186] Define the training agent allocation scheme π s As a mapping function:

[0187] πs :A→S

[0188] where π s (A i )=S m Represents training agent A i Assigned to training node S m Conduct training;

[0189] For each training agent, the minimization target is determined based on its mapping function to adaptively allocate different training agents to different training nodes for distributed parallel training until all training agents are trained and the current round of multi-agents is obtained; the minimization target is expressed as:

[0190]

[0191] In one embodiment, the distributed parallel training includes:

[0192] Each training node is responsible for iterative training of the assigned training agent in the local environment to accumulate the local experience pool until the number of training rounds reaches the upper limit or the training agent reaches the convergence condition; where: Each training node S m Follow the steps below to train agent A responsible for it i The iterative training process is:

[0193] In each training round t, collect training agent A i Experience samples After that, store it in the local experience pool Right now:

[0194]

[0195] The experience pool can be used to break the correlation between samples and improve training stability, that is, by A small batch of experience is randomly sampled for training to achieve experience replay; at the same time, due to the limited storage resources of the training node, in order to prevent the experience pool from growing infinitely, a priority experience replay strategy is used to manage the experience pool That is, according to the empirical priority p(τ i ), prioritize the sampling of experiences with higher importance, including:

[0196]

[0197] in, is the value network at round t and round t+1, α is the hyperparameter for adjusting the priority distribution, and γ is the discount factor;

[0198] When the number of training rounds of the training agent reaches the upper limit or the training agent reaches the convergence condition, the training of the training node is completed and the current round of multi-agents is obtained.

[0199] In this embodiment, the process of training the training agent by the training node can ensure that each agent efficiently learns and optimizes its strategy on a specific training node to adapt to the specific control objectives of the power system, such as grid stability, energy optimization scheduling, fault recovery, etc.

[0200] In one embodiment, for each agent, the training objective of the distributed parallel training is expressed as:

[0201]

[0202] in, represents the training goal of the training agent, which is also the optimal control strategy of the agent; V(S t ) is defined as the cumulative return value reward function of the agent, which can be specifically expressed as:

[0203]

[0204] In one embodiment, the step of obtaining the fitness of each agent in the current round of agent group, updating the particle swarm parameters based on the fitness, and then updating the basic parameters of each agent in the current round of agent group to update a plurality of training agents includes:

[0205] Based on the fitness function, the fitness of each agent in the current round of agent group is obtained. The fitness function F of each agent p (X p ) is expressed as:

[0206] F p (X p )=α p Stability(X p )+β p ·Efficiency(X p )+γ p Recovery(X p )

[0207] Where: α p , β p , γ p is the weight coefficient, which is used to balance the importance of different performance indicators; Stability (X p )、Efficiency(X p )、Recovery(X p ) is the training agent A represented by the particle iPerformance in maintaining power system stability, energy optimization dispatch and fault recovery;

[0208] In the multi-agent parallel training process, each training node (sub-node server) S m Responsible for training a part of the agent A i After completing the local iterative training, the training node (subnode server) returns the training results to the central node server. The training results include the network parameters θ of the agent i , fitness value F i and other information such as training loss and number of rounds. After the central node server waits for and receives the training results returned by all training nodes (child node servers), it optimizes and updates the training agent. The specific steps include fitness evaluation, particle position and velocity update, generation of new agents, and replacement of the lowest fitness agent. p (X p ) is the fitness function, which evaluates the position quality of the particle. The higher the fitness value, the better the agent represented by the current particle performs in the task. Subsequently, the particle position and particle velocity of the particle are updated based on the fitness to guide the particle to converge to the optimal solution, specifically including: updating the particle velocity based on the particle velocity update formula, expressed as:

[0209]

[0210] in, is the velocity vector of particle p in the tth generation; w is the inertia weight, which is used to adjust the degree of dependence of the particle on its own velocity; c1 and c2 are learning factors, which control the degree of learning of the particle to the individual optimal position and the global optimal position respectively; r1 and r2 are uniformly distributed random numbers, and satisfy r1,r2∈[0,1];

[0211] Based on the updated particle velocity, the particle position is updated, expressed as:

[0212]

[0213] Based on this, the individual best position is updated according to the following formula:

[0214]

[0215] And update the global optimal position according to the following formula:

[0216]

[0217] Finally, according to the global optimal position X of the particle swarm global best , generate a new agent θ i ′, used to update the training agent, is:

[0218] θ′ i =M(X global best ,A i ).

[0219] In this embodiment, the parameters of the particle swarm can be initialized using the central node server, and the network parameters of each agent can be initialized according to the particle swarm parameters. At the same time, according to the memory size and computing speed of each sub-node server, multiple agents are adaptively divided into different sub-node servers, and each sub-node server is responsible for the training tasks of a part of the agent. The sub-node server iteratively trains the agent locally to accumulate the local experience pool until the number of training rounds reaches the upper limit or the agent reaches the convergence condition; then, the sub-node server returns the training results to the central node server, and the central node server updates the particle swarm parameters according to the fitness of each agent, and updates the agent model parameters according to the particle swarm parameters. Finally, the central server selects the training agent corresponding to the global best particle, performs a round of training on it, and determines whether the model meets the performance requirements or whether the parameters converge. If the performance requirements are met or the model parameters converge, the training is completed, otherwise the agents corresponding to each particle are reallocated to the sub-node server, and the next round of distributed training begins.

[0220] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-agent parallel training method for power system based on swarm intelligence optimization, characterized in that: The following steps are involved: Construct a system simulation model based on the new power system to be controlled; Determine the system control target based on the system simulation model and construct the initial multi-agent based on the system control target; Initialize the particle swarm parameters, and initialize the basic parameters of each agent in the initial multi-agent according to the particle swarm parameters to obtain several training agents; Conduct distributed parallel training on several training agents until all training agents have completed training, and obtain the current round of multi-agents; Obtain the fitness of each agent in the current round of agent group, and update the particle swarm parameters based on the fitness, and then update the basic parameters of each agent in the current round of agent group to update several training agents; Repeat the distributed parallel training of several training agents until the preset training conditions are met to obtain a group of agents; Control of new power systems based on intelligent agent groups.

2. According to the method of claim 1, a multi-agent parallel training method for power system based on swarm intelligence optimization is characterized in that: The method of determining the system control target based on the system simulation model and constructing the initial multi-agent based on the system control target includes the following steps: Determine the system topology, equipment sub-model and load sub-model based on the system simulation model; Determine the grid stability control target, energy optimization dispatch control target and fault recovery control target based on the system topology, equipment sub-model and load sub-model; Based on the grid stability control objectives, energy optimization dispatch control objectives and fault recovery control objectives, construct power system stability intelligent agent, energy optimization dispatch intelligent agent and fault recovery intelligent agent; Determine the Markov decision process of the power system stability agent, energy optimization scheduling agent and fault recovery agent, and construct the initial multi-agent.

3. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 2 is characterized in that: The Markov decision process includes: A tuple defining a Markov decision: <S,A,P,R,γ> Where S is the state vector of the agent, S = {s1, s2, …, s n },s n represents the nth observation of the power system; A is the action vector of the agent, A={a1,a2,…,a m }, a m represents the mth relevant decision quantity of the power system; P is the state transition probability of the intelligent agent, which is expressed as: The state S of the power system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, expressed as: in Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

4. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 3 is characterized in that: The Markov decision process for determining the power system stability agent, the energy optimization dispatch agent and the fault recovery agent to construct the initial multi-agent includes: For agents with different system control objectives, the Markov decision process has different state vectors, action vectors, state transitions, and reward functions; among them: For the power system stability agent, the state vector S1 includes the power system frequency f t , node voltage V t 、Generator speedω t , load level L t , power flow P t and energy storage state E t ,have: S1=[f t ,V t ,ω t ,L t ,P t ,E t ] T Action vector A1 includes the regulating power P of the generator g , load adjustment ΔL t 、Charging and discharging power of energy storage device P e , Photovoltaic discharge power P pv And the fan discharge power P wind ,have: A1=[P g ,ΔL t ,P e ,P pv ,P wind ] T The reward function R1 is based on the frequency deviation Δf t , voltage deviation ΔV t Design, with: Among them, k f , k v and k s Both represent weight coefficients; S st It is an indicator of power system stability; The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is: S t+1 =f(S t ,A t ,w t ) Among them, w t is the system noise and external disturbance; For the energy optimization scheduling agent, the state vector S2 includes the output power P of each power generation unit g , Energy storage device status E t , Load demand D t , Renewable Energy Forecast t and electricity price information C t ,have: S2=[P g ,E t ,D t ,R t ,C t ] T Action vector A2 includes the generation scheduling plan ΔP g , energy storage charging and discharging power P e , Load adjustment strategy ΔD t , Power purchase or sales strategy P b ,have: A2=[ΔP g ,P e ,ΔD t ,P b ] T The reward function R2 is expressed as: Among them, C g , C e , C b is the cost of power generation, energy storage, and electricity purchase and sale, η r Incentive factor for renewable energy utilization; The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is: S t+1 =f(S t ,A t ,g t ) Among them, t Errors in load forecasting and renewable energy forecasting; For the fault recovery agent, the state vector S3 includes the fault location F t , Fault type T t , Line Status L t , Standby power state B t , load level D t and recovery progress R t ,have: S3=[F t ,T t ,L t ,B t ,D t ,R t ] T Action vector A3 includes fault isolation measure I t , Fault point repair solution M t , load transfer strategy ΔS t and backup power enabled ΔB t ,have: A3=[I t ,M t ,ΔS t ,ΔB t ] T The reward function R3 is expressed as: Among them, k r and k l is the weight coefficient, R t is the recovery time and δ i is the power supply status of load node i; The state transfer equation is expressed as: S t+1 =f(S t ,A t ,or t ) Among them, η t This is an uncertainty factor in the fault recovery process.

5. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 1 is characterized in that: The particle swarm parameters are initialized, and the basic parameters of each agent in the initial multi-agent are initialized according to the particle swarm parameters to obtain a number of training agents, including: Define the particle swarm parameters and initialize P particles in the particle swarm. Each particle p represents a possible agent network parameter configuration and its allocation scheme. Each particle position X p Contains the basic parameters of the agent, including network parameters and allocation information; each particle speed V p Used to guide particle position updates; Initialize the particle position, specifically expressed as: Among them, Θ represents the network parameter space of all agents, Π represents the agent allocation strategy space, and all particle speeds are initialized to 0, so: At the same time, initialize the individual optimal position of the particle swarm and the global optimal position X global best , respectively: Among them, F p (X p ) is the fitness function, which is used to evaluate the position quality of the particle; After completing the above initialization, several training agents are obtained.

6. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 5, characterized in that: The distributed parallel training of the plurality of training agents is performed until all the training agents are trained to obtain the current round of multi-agents, including: Each training agent A i With a corresponding deep neural network θ i , by defining the mapping M to map the particle positions to the network parameters of each training agent, we have: Where N is the total number of agents, is the training agent A corresponding to particle p i Next, several training nodes for distributed parallel training of training agents are preset, and resource evaluation is performed on each training node to obtain the memory M of each training node. m and the calculation speed C m ; m = 1, 2, ..., M, which represents the sequence number of each node; According to the memory M of each training node m and the calculation speed C m Several training agents are adaptively allocated to different training nodes for distributed parallel training according to the resource requirements of each training agent until all training agents are trained to obtain the current round of multi-agents.

7. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 6, characterized in that: According to the memory M of each node m and the calculation speed C m According to the resource requirements of each training agent, several training agents are adaptively allocated to different training nodes for distributed parallel training until all training agents are trained, and the current round of multi-agents is obtained, including: Determine the training agent A i Resource requirements, including memory requirements m i and computational requirements c i , where i = 1, 2, …, I, represents the number of training agents; Define the training agent allocation scheme π s As a mapping function: p s :A→S where π s (A i )=S m Represents training agent A i Assigned to training node S m Conduct training; For each training agent, the minimization target is determined based on its mapping function to adaptively allocate different training agents to different training nodes for distributed parallel training until all training agents are trained and the current round of multi-agents is obtained; the minimization target is expressed as:

8. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 7, characterized in that: The distributed parallel training comprises: Each training node is responsible for iterative training of the assigned training agent in the local environment to accumulate the local experience pool until the number of training rounds reaches the upper limit or the training agent reaches the convergence condition; where: Each training node S m Follow the steps below to train agent A responsible for it i The iterative training process is: In each training round t, collect training agent A i Experience samples After that, store it in the local experience pool Right now: By drawing from the experience pool A small batch of experience is randomly sampled for training to achieve experience replay; at the same time, due to the limited storage resources of the training node, a priority experience replay strategy is used to manage the experience pool That is, according to the empirical priority p(τ i ), prioritize the sampling of experiences with higher importance, including: in, is the value network at round t and round t+1, α is the hyperparameter for adjusting the priority distribution, and γ is the discount factor; When the number of training rounds of the training agent reaches the upper limit or the training agent reaches the convergence condition, the training of the training node is completed and the current round of multi-agents is obtained.

9. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 8, characterized in that: For each agent, the training objective of the distributed parallel training is expressed as: in, Represents the training objective for training the agent.

10. The method for parallel training of multi-agents in a power system based on swarm intelligence optimization according to claim 8, characterized in that: The step of obtaining the fitness of each agent in the current round of agent group, and updating the particle swarm parameters based on the fitness, and then updating the basic parameters of each agent in the current round of agent group to update a plurality of training agents, includes: Based on the fitness function, the fitness of each agent in the current round of agent group is obtained. The fitness function F of each agent p (X p ) is expressed as: F p (X p )=α p ·Stability(X p )+β p ·Efficiency(X p )+γ p ·Recovery(X p ) Where: α p , β p , γ p is the weight coefficient, which is used to balance the importance of different performance indicators; Stability (X p )、Efficiency(X p )、Recovery(X p ) is the training agent A represented by the particle i Performance in maintaining power system stability, energy optimization dispatch and fault recovery; Based on the fitness, the particle position and particle velocity of the particle are updated to guide the particle to converge to the optimal solution, specifically including: updating the particle velocity based on the particle velocity update formula, expressed as: in, is the velocity vector of particle p in the tth generation; w is the inertia weight, which is used to adjust the degree of dependence of the particle on its own velocity; c1 and c2 are learning factors, which control the degree of learning of the particle to the individual optimal position and the global optimal position respectively; r1 and r2 are uniformly distributed random numbers, and satisfy r1,r2∈[0,1]; Based on the updated particle velocity, the particle position is updated, expressed as: Based on this, the individual best position is updated according to the following formula: And update the global optimal position according to the following formula: Finally, according to the global optimal position X of the particle swarm globalbest , generate a new agent θ i ′ , used to update the training agent, there are: θ′ i =M(X global best ,A i )。

Images