Power system multi-agent parallel training method based on adaptive evolution

Abstract

The invention provides a power system multi-agent parallel training method based on adaptive evolution. The method comprises the following steps: constructing a system simulation model based on a to-be-controlled novel power system; constructing an initial multi-agent based on the system simulation model; initializing network parameters of each agent in the multiple agents to obtain a plurality of training agents; carrying out distributed parallel training on the plurality of training agents until all the training agents are trained, and obtaining a current round of multi-agents; acquiring the fitness of each agent in the agent group in the current round, performing selection, intersection and variation based on the fitness, and acquiring a new agent to update the plurality of training agents; repeatedly executing distributed parallel training on the plurality of training agents until a preset training condition is met, and obtaining an agent group; and controlling the to-be-controlled novel power system based on the intelligent agent group. According to the method, the multi-agent training speed can be greatly improved, and the control efficiency of the novel power system is effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of novel power system control technology, and in particular to a power system multi-agent parallel training method based on adaptive evolution. Background Art

[0002] Compared with the traditional power system, the new power system has undergone profound changes in terms of technical foundation, operation mechanism and functional form, especially in terms of power supply composition, grid scale and load characteristics, which have stronger uncertainty, complexity and nonlinearity. Therefore, the actual operation and control of the new power system faces more difficulties, and a more complete and flexible control method is urgently needed.

[0003] In applications related to new power systems, deep reinforcement learning with online learning and experience replay capabilities is an effective control method. In existing reinforcement learning methods for new power systems, in order to solve the dimensionality curse anomaly caused by the ultra-high dimensional data of new power systems, multi-agent deep reinforcement learning algorithms are usually used to control new power systems. The data dimension of this type of method is often bound to the number of agents, and the size of the action space often increases exponentially with the number of agents. However, the memory of a single computing node is limited and usually cannot fully support large-scale multi-agent reinforcement learning models; in addition, the computing power of a single node also greatly constrains the training speed of large-scale multi-agent reinforcement learning models.

[0004] Although the existing multi-computing node distributed multi-agent reinforcement learning training methods can effectively solve the above two problems, the existing methods are usually prone to fall into local optimal solutions during the training process when facing multi-objective control tasks with deceptive rewards and sparse rewards. The convergence speed is extremely slow, and it is difficult to obtain the optimal policy. It is difficult to meet the requirements of large-scale multi-agent reinforcement learning training and the control requirements of new power systems. Summary of the invention

[0005] The present invention aims to provide a method for parallel training of multi-agents in power systems based on adaptive evolution to solve the above-mentioned technical problems and realize distributed parallel training of multi-agent reinforcement learning models of new power systems on a general server cluster. While effectively coping with various multi-objective engineering tasks with deceptive rewards and sparse rewards in new power systems, it significantly reduces the memory and computing pressure of a single computing node, greatly improves the training speed of multi-agents, and effectively improves the control efficiency of new power systems.

[0006] In order to solve the above technical problems and realize efficient training of large-scale multi-agent reinforcement learning, the present invention provides a multi-agent parallel training method for a power system based on adaptive evolution, comprising the following steps:

[0007] Construct a system simulation model based on the new power system to be controlled;

[0008] Determine the system control target based on the system simulation model and construct the initial multi-agent based on the system control target;

[0009] Initialize the network parameters of each agent in the multi-agent system to obtain several training agents;

[0010] Conduct distributed parallel training on several training agents until all training agents have completed training, and obtain the current round of multi-agents;

[0011] Obtain the fitness of each agent in the current round of agent group, and select, crossover and mutate based on the fitness to obtain new agents to update several training agents;

[0012] Repeat the distributed parallel training of several training agents until the preset training conditions are met to obtain a group of agents;

[0013] Control of new power systems based on intelligent agent groups.

[0014] The above scheme can be applied to general server clusters for distributed parallel training of multi-agent reinforcement learning models of new power systems. While effectively coping with various multi-objective engineering tasks with deceptive rewards and sparse rewards in new power systems, it significantly reduces the memory and computing pressure of a single computing node, greatly improves the training speed of multi-agents, and effectively improves the control efficiency of new power systems.

[0015] Furthermore, building a system simulation model based on the new power system to be controlled may include obtaining the topological structure, equipment model and load model of the new power system to be controlled, and building a corresponding large signal mathematical model to provide a basic environment for multi-agent reinforcement learning training.

[0016] Furthermore, the method of determining the system control target based on the system simulation model and constructing the initial multi-agent based on the system control target includes the following steps:

[0017] Determine the system topology, equipment sub-model and load sub-model based on the system simulation model;

[0018] Determine the grid stability control target, energy optimization dispatch control target and fault recovery control target based on the system topology, equipment sub-model and load sub-model;

[0019] Based on the grid stability control objectives, energy optimization dispatch control objectives and fault recovery control objectives, construct power system stability intelligent agent, energy optimization dispatch intelligent agent and fault recovery intelligent agent;

[0020] Determine the Markov decision process of the power system stability agent, energy optimization scheduling agent and fault recovery agent, including state, action, reward and transfer mechanism, and construct the initial multi-agent.

[0021] Furthermore, the Markov decision process includes:

[0022] A tuple defining a Markov decision:

[0023] <S,A,P,R,γ>

[0024] Among them, S is the state vector of the agent, S = {s 1 ,s 2 ,…,s n},s n represents the nth observation of the power system; A is the action vector of the agent, A = {a 1 ,a 2 ,…,a m}, a m represents the mth relevant decision quantity of the power system; P is the state transition probability of the intelligent agent, which is expressed as:

[0025]

[0026] The state S of the power system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, expressed as:

[0027]

[0028] in Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

[0029] Furthermore, the Markov decision process of determining the power system stability agent, the energy optimization dispatch agent and the fault recovery agent to construct the initial multi-agent includes:

[0030] For agents with different system control objectives, the Markov decision process has different state vectors, action vectors, state transitions, and reward functions; among them:

[0031] For the power system stability agent, the state vector S 1 Including power system frequency f t , node voltage V t 、Generator speedω t, load level L t , power flow P t and energy storage state E t ,have:

[0032] S 1 =[f t ,V t ,ω t ,L t ,P t ,E t ] T

[0033] Action vector A 1 Including the generator's regulating power P g , load adjustment ΔL t 、Charging and discharging power of energy storage device P e , Photovoltaic discharge power P pv And the fan discharge power P wind ,have:

[0034] A 1 =[P g ,ΔL t ,P e ,P pv ,P wind ] T

[0035] Reward function R 1 According to the frequency deviation Δf t , voltage deviation ΔV t Design, with:

[0036]

[0037] Among them, k f , k v and k s Both represent weight coefficients; S st It is an indicator of power system stability;

[0038] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0039] S t+1 =f(S t ,A t ,w t )

[0040] Among them, w t is the system noise and external disturbance;

[0041] For the energy-optimized scheduling agent, the state vector S 2 Including the output power P of each power generation unitg , Energy storage device status E t , Load demand D t , Renewable Energy Forecast t and electricity price information C t ,have:

[0042] S 2 =[P g ,E t ,D t ,R t ,C t ] T

[0043] Action vector A 2 Including the generation dispatch plan ΔP g , energy storage charging and discharging power P e , Load adjustment strategy ΔD t , Power purchase or sales strategy P b ,have:

[0044] A 2 =[ΔP g ,P e ,ΔD t ,P b ] T

[0045] Reward function R 2 It is expressed as:

[0046]

[0047] Among them, C g , C e , C b is the cost of power generation, energy storage, and electricity purchase and sale, η r Incentive factor for renewable energy utilization;

[0048] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0049] S t+1 =f(S t ,A t ,ζ t )

[0050] Among them, t Errors in load forecasting and renewable energy forecasting;

[0051] For a fault-recovery agent, the state vector S 3 Including fault location F t , Fault type T t , Line Status L t , Standby power state Bt , load level D t and recovery progress R t ,have:

[0052] S 3 =[F t ,T t ,L t ,B t ,D t ,R t ] T

[0053] Action vector A 3 Including fault isolation measures I t , Fault point repair solution M t , load transfer strategy ΔS t and backup power enabled ΔB t ,have:

[0054] A 3 =[I t ,M t ,ΔS t ,ΔB t ] T

[0055] Reward function R 3 It is expressed as:

[0056]

[0057] Among them, k r and k l is the weight coefficient, R t is the recovery time and δ i is the power supply status of load node i;

[0058] The state transfer equation is expressed as:

[0059] S t+1 =f(S t ,A t ,η t )

[0060] Among them, η t This is an uncertainty factor in the fault recovery process.

[0061] Furthermore, the network parameters of each agent in the multi-agent are initialized to obtain a plurality of training agents, including:

[0062] Each agent A i Have a deep neural network for policy learning and decision making, whose network parameters include the weight matrix W i and the bias vector bi ; According to the following formula, the weight matrix W i To initialize:

[0063]

[0064] Among them, n in and n out are the number of input and output neurons of the weight matrix respectively; the bias vector b is calculated according to the following formula i To initialize:

[0065]

[0066] Among them, σ is the preset standard deviation parameter;

[0067] After completing the above initialization, several training agents are obtained.

[0068] Furthermore, the distributed parallel training of the plurality of training agents is performed until all training agents are trained to obtain the current round of multi-agents, including:

[0069] Preset several training nodes for distributed parallel training of training agents, evaluate the resources of each training node, and obtain the memory M of each training node m and the calculation speed C m ; m = 1, 2, ..., M, which represents the sequence number of each node;

[0070] According to the memory M of each training node m and the calculation speed C m Several training agents are adaptively allocated to different training nodes for distributed parallel training according to the resource requirements of each training agent until all training agents are trained to obtain the current round of multi-agents.

[0071] In the above solution, the training node can be executed by a sub-node server.

[0072] Further, the memory M of each training node m and the calculation speed C m According to the resource requirements of each training agent, several training agents are adaptively allocated to different training nodes for distributed parallel training until all training agents are trained, and the current round of multi-agents is obtained, including:

[0073] Determine the training agent A i Resource requirements, including memory requirements m i and computational requirements c i , where i = 1, 2, …, I, represents the number of training agents;

[0074] Define the training agent allocation scheme π s As a mapping function:

[0075] π s :A→S

[0076] where π s (A i )=S m Represents training agent A i Assigned to training node S m Conduct training;

[0077] For each training agent, the minimization target is determined based on its mapping function to adaptively allocate different training agents to different training nodes for distributed parallel training until all training agents are trained and the current round of multi-agents is obtained; the minimization target is expressed as:

[0078]

[0079] Furthermore, the distributed parallel training includes:

[0080] Each training node is responsible for iterative training of the assigned training agent in the local environment to accumulate the local experience pool until the number of training rounds reaches the upper limit or the training agent reaches the convergence condition; where: Each training node S m Follow the steps below to train agent A responsible for it i The iterative training process is:

[0081] In each training round t, collect training agent A i Experience samples After that, store it in the local experience pool Right now:

[0082]

[0083] The experience pool is used to break the correlation between samples and improve the training stability, that is, by extracting i A small batch of experience is randomly sampled for training to achieve experience replay; at the same time, due to the limited storage resources of the training node, in order to prevent the experience pool from growing infinitely, a priority experience replay strategy can be used to manage the experience pool D i , that is, according to the empirical priority p(τ i ), prioritize the sampling of experiences with higher importance, including:

[0084]

[0085] in, is the value network at round t and round t+1, α is the hyperparameter for adjusting the priority distribution, and γ is the discount factor;

[0086] When the number of training rounds of the training agent reaches the upper limit or the training agent reaches the convergence condition, the training of the training node is completed and the current round of multi-agents is obtained.

[0087] The distributed parallel training process provided by the above scheme can ensure that each training agent can efficiently learn and optimize its strategy on a specific training node to adapt to the specific control objectives of the new power system, such as grid stability, energy optimization scheduling, fault recovery, etc.

[0088] Furthermore, for each agent, the training objective of the distributed parallel training is expressed as:

[0089]

[0090] in, represents the training goal of the training agent; V(S t ) represents the cumulative return value reward function of the training agent, expressed as:

[0091]

[0092] Furthermore, the step of obtaining the fitness of each agent in the current round of agent group, and performing selection, crossover and mutation based on the fitness to obtain a new agent to update a plurality of training agents includes:

[0093] Based on the fitness function, the fitness of each agent in the current round of agent group is obtained. The fitness function F(A i ) is expressed as:

[0094] F(A i )=α·Stability(A i )+β·Efficiency(A i )+γ·Recovery(A i )

[0095] Among them, the fitness function F(A i ) is used to evaluate each training agent A i The higher the fitness value, the better the training agent performs in the current task; α, β, γ are weight coefficients used to balance the importance of different performance indicators; Stability (A i )、Efficiency(A i )、Recovery(A i ) is the training agent A iPerformance in maintaining power system stability, energy optimization dispatch and fault recovery;

[0096] The training agents for crossover and mutation are selected based on fitness, expressed as:

[0097] Calculate the total fitness F of all training agents total :

[0098]

[0099] Calculate the selection probability P of each training agent i :

[0100]

[0101] Select agents for reproduction via cumulative probability distribution:

[0102] Assume a random number r∈[0,1), select a training agent A that satisfies the following conditions k :

[0103]

[0104] After completing the training agent selection, the selected training agents are randomly crossed. Assume that two parent training agents A i and A j The network parameters are θ i and θ j , select a crossover point c and split the network parameters of the parent training agent into two sub-agents:

[0105]

[0106] Cross-combine the two sub-agents to obtain two sub-agents A i′ and A j′ , and its network parameters are expressed as:

[0107]

[0108] After obtaining the offspring agents after the crossover, the network parameters can be fine-tuned to increase the population diversity; after obtaining the offspring agents after the crossover, the offspring agents A k′ The network parameters θ k′ To mutate:

[0109]

[0110] in, The mean is 0 and the variance is σ 2Gaussian noise; then, remove the training agent with the lowest fitness among the parent training agents and transfer the child agent A k′ as a new training agent.

[0111] In the above scheme, the selection, crossover, mutation and replacement processes are repeated until a new generation of population is generated; through multiple generations of genetic algorithm iterations, the overall fitness of the multi-agents is gradually optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 A schematic flow chart of a method for parallel training of multi-agents in a power system based on adaptive evolution is provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0113] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0114] Existing multi-computing node distributed multi-agent reinforcement learning training methods are usually designed for general deep learning frameworks and specific hardware environments. In the face of multi-objective control tasks with deception rewards and sparse rewards, the training process is prone to fall into local optimal solutions, the convergence speed is extremely slow, and it is difficult to obtain the optimal policy. These methods cannot meet the requirements of large-scale multi-agent reinforcement learning training in new power systems while adapting to general server clusters.

[0115] In order to solve the above technical problems, Figure 1 As shown, this embodiment provides a method for parallel training of multi-agents in a power system based on adaptive evolution, comprising the following steps:

[0116] Construct a system simulation model based on the new power system to be controlled;

[0117] Determine the system control target based on the system simulation model and construct the initial multi-agent based on the system control target;

[0118] Initialize the network parameters of each agent in the multi-agent system to obtain several training agents;

[0119] Conduct distributed parallel training on several training agents until all training agents have completed training, and obtain the current round of multi-agents;

[0120] Obtain the fitness of each agent in the current round of agent group, and select, crossover and mutate based on the fitness to obtain new agents to update several training agents;

[0121] Repeat the distributed parallel training of several training agents until the preset training conditions are met to obtain a group of agents;

[0122] Control of new power systems based on intelligent agent groups.

[0123] This embodiment can be applied to a general-purpose server cluster for distributed parallel training of multi-agent reinforcement learning models of new power systems. While effectively coping with various multi-objective engineering tasks with deceptive rewards and sparse rewards in new power systems, it significantly reduces the memory and computing pressure of a single computing node, greatly improves the training speed of multi-agents, and effectively improves the control efficiency of new power systems.

[0124] It should be noted that this method can be implemented in an environment with a central node server and several sub-node servers, that is, this method realizes large-scale multi-agent distributed parallel training of the new power system on a general server cluster, effectively improving the overall performance and training efficiency of the multi-agent, and thus improving the control efficiency of the new power system.

[0125] It should be noted that the process of controlling the new power system by a group of intelligent agents can be achieved by adopting the existing control method. The technical point of this embodiment is how to quickly train and obtain multiple intelligent agents to improve the overall performance of multiple intelligent agents, improve the overall performance and training efficiency of multiple intelligent agents, and then improve the control efficiency of the new power system.

[0126] In one embodiment, a system simulation model constructed based on the new power system to be controlled may include the topology, equipment model, and load model of the new power system, and construct a corresponding large signal mathematical model, thereby providing a basic environment for reinforcement learning training of multiple agents.

[0127] In this embodiment, the network topology of the power system can be determined in power system simulation software such as DSP and BPA, including the location and connection relationship of power plants, substations, transmission lines, distribution lines, and load nodes, the type of each node can be defined, and the electrical parameters of the transmission lines and distribution lines can be determined. A new power system simulation model of photovoltaic, wind turbine, synchronous generator, energy storage, transformer, interconnection line and other components is established, and then a complete large signal mathematical model of the system is established based on the obtained simulation model.

[0128] In one embodiment, determining the system control target based on the system simulation model and constructing the initial multi-agent based on the system control target includes the following steps:

[0129] Determine the system topology, equipment sub-model and load sub-model based on the system simulation model;

[0130] Determine the grid stability control target, energy optimization dispatch control target and fault recovery control target based on the system topology, equipment sub-model and load sub-model;

[0131] Based on the grid stability control objectives, energy optimization dispatch control objectives and fault recovery control objectives, construct power system stability intelligent agent, energy optimization dispatch intelligent agent and fault recovery intelligent agent;

[0132] Determine the Markov decision process of the power system stability agent, energy optimization scheduling agent and fault recovery agent, including state, action, reward and transfer mechanism, and construct the initial multi-agent.

[0133] In one embodiment, the Markov decision process includes:

[0134] A tuple defining a Markov decision:

[0135] <S,A,P,R,γ>

[0136] Among them, S is the state vector of the agent, S = {s 1 ,s 2 ,…,s n},s n represents the nth observation of the power system; A is the action vector of the agent, A = {a 1 ,a 2 ,…,a m}, a m represents the mth relevant decision quantity of the power system; P is the state transition probability of the intelligent agent, which is expressed as:

[0137]

[0138] The state S of the power system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, expressed as:

[0139]

[0140] in Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

[0141] In one embodiment, the Markov decision process for determining the power system stability agent, the energy optimization dispatch agent, and the fault recovery agent to construct the initial multi-agent includes:

[0142] For agents with different system control objectives, the Markov decision process has different state vectors, action vectors, state transitions, and reward functions; among them:

[0143] For the power system stability agent, the state vector S 1 Including power system frequency f t , node voltage V t 、Generator speedω t , load level L t , power flow P t and energy storage state E t ,have:

[0144] S 1 =[f t ,V t ,ω t ,L t ,P t ,E t ] T

[0145] Action vector A 1 Including the generator's regulating power P g , load adjustment ΔL t 、Charging and discharging power of energy storage device P e , Photovoltaic discharge power P pv And the fan discharge power P wind ,have:

[0146] A 1 =[P g ,ΔL t ,P e ,P pv ,P wind ] T

[0147] Reward function R 1 According to the frequency deviation Δf t , voltage deviation ΔV t Design, with:

[0148]

[0149] Among them, k f , k v and k s Both represent weight coefficients; S st It is an indicator of power system stability;

[0150] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0151] S t+1 =f(S t ,A t ,w t )

[0152] Among them, w t is the system noise and external disturbance;

[0153] For the energy-optimized scheduling agent, the state vector S 2 Including the output power P of each power generation unit g , Energy storage device status E t , Load demand D t , Renewable Energy Forecast t and electricity price information C t ,have:

[0154] S 2 =[P g ,E t ,D t ,R t ,C t ] T

[0155] Action vector A 2 Including the generation dispatch plan ΔP g , energy storage charging and discharging power P e , Load adjustment strategy ΔD t , Power purchase or sales strategy P b ,have:

[0156] A 2 =[ΔP g ,P e ,ΔD t ,P b ] T

[0157] Reward function R 2 It is expressed as:

[0158]

[0159] Among them, C g , C e , C b is the cost of power generation, energy storage, and electricity purchase and sale, η r Incentive factor for renewable energy utilization;

[0160] The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is:

[0161] S t+1 =f(S t ,A t ,ζ t )

[0162] Among them, t Errors in load forecasting and renewable energy forecasting;

[0163] For a fault-recovery agent, the state vector S 3 Including fault location F t , Fault type T t , Line Status L t , Standby power state B t , load level D t and recovery progress R t ,have:

[0164] S 3 =[F t ,T t ,L t ,B t ,D t ,R t ] T

[0165] Action vector A 3 Including fault isolation measures I t , Fault point repair solution M t , load transfer strategy ΔS t and backup power enabled ΔB t ,have:

[0166] A 3 =[I t ,M t ,ΔS t ,ΔB t ] T

[0167] Reward function R 3 It is expressed as:

[0168]

[0169] Among them, k r and k l is the weight coefficient, R t is the recovery time and δ i is the power supply status of load node i;

[0170] The state transfer equation is expressed as:

[0171] S t+1 =f(S t ,At ,η t )

[0172] Among them, η t This is an uncertainty factor in the fault recovery process.

[0173] In one embodiment, the network parameters of each agent in the multi-agent are initialized to ensure that the agent can effectively explore and utilize environmental information, thereby obtaining a plurality of training agents, including:

[0174] Each agent A i Have a deep neural network for policy learning and decision making, whose network parameters include the weight matrix W i and the bias vector b i ; According to the following formula, the weight matrix W i To initialize:

[0175]

[0176] Among them, n in and n out are the number of input and output neurons of the weight matrix respectively; the bias vector b is calculated according to the following formula i To initialize:

[0177]

[0178] Among them, σ is the preset standard deviation parameter;

[0179] After completing the above initialization, several training agents are obtained.

[0180] In one embodiment, the distributed parallel training of the plurality of training agents is performed until all training agents are trained to obtain the current round of multi-agents, including:

[0181] Preset several training nodes for distributed parallel training of training agents, evaluate the resources of each training node, and obtain the memory M of each training node m and the calculation speed C m ; m = 1, 2, ..., M, which represents the sequence number of each node;

[0182] According to the memory M of each training node m and the calculation speed C m Several training agents are adaptively allocated to different training nodes for distributed parallel training according to the resource requirements of each training agent until all training agents are trained to obtain the current round of multi-agents.

[0183] In this embodiment, the training node may be executed by a sub-node server.

[0184] Further, the memory M of each training node m and the calculation speed C m According to the resource requirements of each training agent, several training agents are adaptively allocated to different training nodes for distributed parallel training until all training agents are trained, and the current round of multi-agents is obtained, including:

[0185] Determine the training agent A i Resource requirements, including memory requirements m i and computational requirements c i , where i = 1, 2, …, I, represents the number of training agents;

[0186] Define the training agent allocation scheme π s As a mapping function:

[0187] π s :A→S

[0188] where π s (A i )=S m Represents training agent A i Assigned to training node S m Conduct training;

[0189] For each training agent, the minimization target is determined based on its mapping function to adaptively allocate different training agents to different training nodes for distributed parallel training until all training agents are trained and the current round of multi-agents is obtained; the minimization target is expressed as:

[0190]

[0191] In one embodiment, the distributed parallel training includes:

[0192] Each training node is responsible for iterative training of the assigned training agent in the local environment to accumulate the local experience pool until the number of training rounds reaches the upper limit or the training agent reaches the convergence condition; where: Each training node S m Follow the steps below to train agent A responsible for it i The iterative training process is:

[0193] In each training round t, collect training agent A i Experience samples After that, store it in the local experience pool Right now:

[0194]

[0195] The experience pool is used to break the correlation between samples and improve the training stability, that is, by extracting i A small batch of experience is randomly sampled for training to achieve experience replay; at the same time, due to the limited storage resources of the training node, in order to prevent the experience pool from growing infinitely, a priority experience replay strategy can be used to manage the experience pool That is, according to the empirical priority p(τ i ), prioritize the sampling of experiences with higher importance, including:

[0196]

[0197] in, is the value network at round t and round t+1, α is the hyperparameter for adjusting the priority distribution, and γ is the discount factor;

[0198] When the number of training rounds of the training agent reaches the upper limit or the training agent reaches the convergence condition, the training of the training node is completed and the current round of multi-agents is obtained.

[0199] The distributed parallel training process provided in this embodiment can ensure that each training agent efficiently learns and optimizes its strategy on a specific training node to adapt to the specific control objectives of the new power system, such as grid stability, energy optimization scheduling, fault recovery, etc.

[0200] In this embodiment, for each agent, the training objective of the distributed parallel training is expressed as:

[0201]

[0202] in, represents the training goal of the training agent; V(S t ) represents the cumulative return value reward function of the training agent, expressed as:

[0203]

[0204] In this embodiment, the step of obtaining the fitness of each agent in the current round of agent group, and performing selection, crossover and mutation based on the fitness to obtain a new agent to update a plurality of training agents includes:

[0205] Based on the fitness function, the fitness of each agent in the current round of agent group is obtained. The fitness function F(A i ) is expressed as:

[0206] F(A i )=α·Stability(A i )+β·Efficiency(A i)+γ·Recovery(A i )

[0207] Among them, the fitness function F(A i ) is used to evaluate each training agent A i The higher the fitness value, the better the training agent performs in the current task; α, β, γ are weight coefficients used to balance the importance of different performance indicators; Stability (A i )、Efficiency(A i )、Recovery(A i ) is the training agent A i Performance in maintaining power system stability, energy optimization dispatch and fault recovery;

[0208] In the multi-agent parallel training process, each training node (sub-node server) S m Responsible for training a part of the agent A i After completing the local iterative training, the training node (subnode server) will send the training results, including the network parameters θ of the agent i , fitness value F i The central node server waits for and receives the training results returned by all the child node servers, and then executes the genetic algorithm operation to optimize and update the agent population. Specifically, the training agent for crossover and mutation is selected based on fitness, which is expressed as:

[0209] Calculate the total fitness F of all training agents total :

[0210]

[0211] Calculate the selection probability P of each training agent i :

[0212]

[0213] Select agents for reproduction via cumulative probability distribution:

[0214] Assume a random number r∈[0,1), select a training agent A that satisfies the following conditions k :

[0215]

[0216] After completing the training agent selection, the selected training agents are randomly crossed. Assume that two parent training agents A i and A j The network parameters are θi and θ j , select a crossover point c and split the network parameters of the parent training agent into two sub-agents:

[0217]

[0218] Cross-combine the two sub-agents to obtain two sub-agents A i′ and A j′ , and its network parameters are expressed as:

[0219]

[0220] After obtaining the offspring agents after the crossover, the network parameters can be fine-tuned to increase the population diversity; after obtaining the offspring agents after the crossover, the offspring agents A k′ The network parameters θ k′ To mutate:

[0221]

[0222] in, The mean is 0 and the variance is σ 2 Gaussian noise; then, remove the training agent with the lowest fitness among the parent training agents and transfer the child agent A k′ as a new training agent.

[0223] In this embodiment, the selection, crossover, mutation and replacement processes are repeated until a new generation of population is generated; through multiple generations of genetic algorithm iterations, the overall fitness of the multi-agents is gradually optimized.

[0224] In this embodiment, the central node server can be used to initialize the network parameters of each agent to ensure that the agent can effectively explore and utilize environmental information. At the same time, according to the memory size and computing speed of each sub-node server, multiple agents are adaptively divided into different sub-node servers, and each sub-node server is responsible for the training tasks of a part of the agent. The sub-node server iteratively trains the agent locally to accumulate the local experience pool until the number of training rounds reaches the upper limit or the agent reaches the convergence condition; then, the sub-node server returns the training results to the central node server, and the central node server selects, crosses, and mutates according to the fitness of each agent to obtain a new agent, and removes the agent with the lowest fitness in the original model, and adds the new agent to the model. Finally, the central server selects the training agent corresponding to the global best particle, performs a round of training on it, and determines whether the model meets the performance requirements or whether the parameters converge. If the performance requirements are met or the model parameters converge, the training is completed, otherwise the agents corresponding to each particle are reallocated to the sub-node server, and the next round of distributed training begins.

[0225] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-agent parallel training method for power systems based on adaptive evolution, characterized in that: The following steps are involved: Construct a system simulation model based on the new power system to be controlled; Determine the system control target based on the system simulation model and construct the initial multi-agent based on the system control target; Initialize the network parameters of each agent in the multi-agent system to obtain several training agents; Conduct distributed parallel training on several training agents until all training agents have completed training, and obtain the current round of multi-agents; Obtain the fitness of each agent in the current round of agent group, and select, crossover and mutate based on the fitness to obtain new agents to update several training agents; Repeat the distributed parallel training of several training agents until the preset training conditions are met to obtain a group of agents; Control of new power systems based on intelligent agent groups.

2. According to claim 1, a method for parallel training of multi-agents in power systems based on adaptive evolution is characterized in that: The method of determining the system control target based on the system simulation model and constructing the initial multi-agent based on the system control target includes the following steps: Determine the system topology, equipment sub-model and load sub-model based on the system simulation model; Determine the grid stability control target, energy optimization dispatch control target and fault recovery control target based on the system topology, equipment sub-model and load sub-model; Based on the grid stability control objectives, energy optimization dispatch control objectives and fault recovery control objectives, construct power system stability intelligent agent, energy optimization dispatch intelligent agent and fault recovery intelligent agent; Determine the Markov decision process of the power system stability agent, energy optimization scheduling agent and fault recovery agent, and construct the initial multi-agent.

3. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 2, characterized in that: The Markov decision process includes: A tuple defining a Markov decision: <S,A,P,R,γ> Where S is the state vector of the agent, S = {s1, s2, …, s n },s n represents the nth observation of the power system; A is the action vector of the agent, A={a1,a2,…,a m }, a m represents the mth relevant decision quantity of the power system; P is the state transition probability of the intelligent agent, which is expressed as: The state S of the power system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, expressed as: in Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

4. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 3 is characterized in that: The Markov decision process for determining the power system stability agent, the energy optimization dispatch agent and the fault recovery agent to construct the initial multi-agent includes: For agents with different system control objectives, the Markov decision process has different state vectors, action vectors, state transitions, and reward functions; among them: For the power system stability agent, the state vector S1 includes the power system frequency f t , node voltage V t 、Generator speedω t , load level L t , power flow P t and energy storage state E t ,have: S1=[f t ,V t ,ω t ,L t ,P t ,E t ] T Action vector A1 includes the regulating power P of the generator g , load adjustment ΔL t 、Charging and discharging power of energy storage device P e , Photovoltaic discharge power P pv And the fan discharge power P wind ,have: A1=[P g ,ΔL t ,P e ,P pv ,P wind ] T The reward function R1 is based on the frequency deviation Δf t , voltage deviation ΔV t Design, with: Among them, k f , k v and k s Both represent weight coefficients; S st It is an indicator of power system stability; The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is: S t+1 =f(S t ,A t ,w t ) Among them, w t is the system noise and external disturbance; For the energy optimization scheduling agent, the state vector S2 includes the output power P of each power generation unit g , Energy storage device status E t , Load demand D t , Renewable Energy Forecast t and electricity price information C t ,have: S2=[P g ,E t ,D t ,R t ,C t ] T Action vector A2 includes the generation scheduling plan ΔP g , energy storage charging and discharging power P e , Load adjustment strategy ΔD t , Power purchase or sales strategy P b ,have: A2=[ΔP g ,P e ,ΔD t ,P b ] T The reward function R2 is expressed as: Among them, C g , C e , C b is the cost of power generation, energy storage, and electricity purchase and sale, η r Incentive factor for renewable energy utilization; The state transfer equation is determined based on the dynamic behavior and control strategy of the power system, which is: S t+1 =f(S t ,A t ,g t ) Among them, t Errors in load forecasting and renewable energy forecasting; For the fault recovery agent, the state vector S3 includes the fault location F t , Fault type T t , Line Status L t , Standby power state B t , load level D t and recovery progress R t ,have: S3=[F t ,T t ,L t ,B t ,D t ,R t ] T Action vector A3 includes fault isolation measure I t , Fault point repair solution M t , load transfer strategy ΔS t and backup power enabled ΔB t ,have: A3=[I t ,M t ,ΔS t ,ΔB t ] T The reward function R3 is expressed as: Among them, k r and k l is the weight coefficient, R t is the recovery time and δ i is the power supply status of load node i; The state transfer equation is expressed as: S t+1 =f(S t ,A t ,or t ) Among them, η t This is an uncertainty factor in the fault recovery process.

5. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 1, characterized in that: The network parameters of each agent in the multi-agent are initialized to obtain a plurality of training agents, including: Each agent A i Have a deep neural network for policy learning and decision making, whose network parameters include the weight matrix W i and the bias vector b i ; According to the following formula, the weight matrix W i To initialize: Among them, n in and n out are the number of input and output neurons of the weight matrix respectively; the bias vector b is calculated according to the following formula i To initialize: Among them, σ is the preset standard deviation parameter; After completing the above initialization, several training agents are obtained.

6. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 5, characterized in that: The distributed parallel training of the plurality of training agents is performed until all the training agents are trained to obtain the current round of multi-agents, including: Preset several training nodes for distributed parallel training of training agents, evaluate the resources of each training node, and obtain the memory M of each training node m and the calculation speed C m ; m = 1, 2, ..., M, which represents the sequence number of each node; According to the memory M of each training node m and the calculation speed C m Several training agents are adaptively allocated to different training nodes for distributed parallel training according to the resource requirements of each training agent until all training agents are trained to obtain the current round of multi-agents.

7. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 6, characterized in that: According to the memory M of each training node m and the calculation speed C m According to the resource requirements of each training agent, several training agents are adaptively allocated to different training nodes for distributed parallel training until all training agents are trained, and the current round of multi-agents is obtained, including: Determine the training agent A i Resource requirements, including memory requirements m i and computational requirements c i , where i = 1, 2, …, I, represents the number of training agents; Define the training agent allocation scheme π s As a mapping function: p s :A→S where π s (A i )=S m Represents training agent A i Assigned to training node S m Conduct training; For each training agent, the minimization target is determined based on its mapping function to adaptively allocate different training agents to different training nodes for distributed parallel training until all training agents are trained and the current round of multi-agents is obtained; the minimization target is expressed as:

8. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 7, characterized in that: The distributed parallel training comprises: Each training node is responsible for iterative training of the assigned training agent in the local environment to accumulate the local experience pool until the number of training rounds reaches the upper limit or the training agent reaches the convergence condition; where: Each training node S m Follow the steps below to train agent A responsible for it i The iterative training process is: In each training round t, collect training agent A i Experience samples After that, store it in the local experience pool Right now: By drawing from the experience pool A small batch of experience is randomly sampled for training to achieve experience replay; at the same time, due to the limited storage resources of the training node, a priority experience replay strategy is used to manage the experience pool That is, according to the empirical priority p(τ i ), prioritize the sampling of experiences with higher importance, including: in, is the value network at round t and round t+1, α is the hyperparameter for adjusting the priority distribution, and γ is the discount factor; When the number of training rounds of the training agent reaches the upper limit or the training agent reaches the convergence condition, the training of the training node is completed and the current round of multi-agents is obtained.

9. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 8, characterized in that: For each agent, the training objective of the distributed parallel training is expressed as: in, Represents the training objective for training the agent.

10. The method for parallel training of multi-agents in a power system based on adaptive evolution according to claim 8, characterized in that: The step of obtaining the fitness of each agent in the current round of agent group, and performing selection, crossover and mutation based on the fitness to obtain a new agent to update a plurality of training agents includes: Based on the fitness function, the fitness of each agent in the current round of agent group is obtained. The fitness function F(A i ) is expressed as: F(A i )=α·Stability(A i )+β·Efficiency(A i )+γ·Recovery(A i ) Among them: α, β, γ are weight coefficients used to balance the importance of different performance indicators; Stability (A i )、Efficiency(A i )、Recovery(A i ) is the training agent A i Performance in maintaining power system stability, energy optimization dispatch and fault recovery; The training agents for crossover and mutation are selected based on fitness, expressed as: Calculate the total fitness F of all training agents total : Calculate the selection probability P of each training agent i : Select agents for reproduction via cumulative probability distribution: Assume a random number r∈[0,1), select a training agent A that satisfies the following conditions k : After completing the training agent selection, the selected training agents are randomly crossed. Assume that two parent training agents A i and A j The network parameters are θ i and θ j , select a crossover point c and split the network parameters of the parent training agent into two sub-agents: Cross-combine the two sub-agents to obtain two sub-agents A i′ and A j′ , and its network parameters are expressed as: After obtaining the offspring agent after crossover, k′ The network parameters θ k′ To mutate: in, The mean is 0 and the variance is σ 2 Gaussian noise; then, remove the training agent with the lowest fitness among the parent training agents and transfer the child agent A k′ as a new training agent.

Images