Multi-agent adaptive hybrid parallel training method of novel power system

Abstract

The invention discloses a novel multi-agent adaptive hybrid parallel training method for a power system, and the method comprises the steps: obtaining a topological structure and network parameters of the system through building a physical simulation model of the power system; establishing tuples of a Markov decision process according to the topological structure and the network parameters to obtain a plurality of tuples; establishing a target function according to the tuples; and distributing the agents with high coupling degree to the same sub-node server or adjacent sub-node servers for training to obtain a multi-agent training result. According to the multi-agent self-adaptive hybrid parallel training method of the novel power system, the agents with high coupling degree are distributed to the same or adjacent sub-node servers for training, so that the memory and the calculation pressure of a single calculation node can be reduced; the problem that it is difficult to carry out training under the condition of adapting to a universal server cluster so as to relieve the memory and calculation pressure of a single calculation node and improve the training speed of an intelligent agent can be solved.

Description

Technical Field

[0001] The invention relates to the technical field of power system control, and in particular to a novel multi-agent adaptive hybrid parallel training method for power systems. Background Art

[0002] Compared with traditional power systems, new power systems have undergone profound changes in terms of technical foundation, operation mechanism and functional form, especially in terms of power supply composition, grid scale and load characteristics, which have more prominent uncertainty, complexity and nonlinearity. Therefore, the actual operation and control of new power systems face more difficulties, and more perfect and flexible control methods are urgently needed. In the applications related to new power systems, reinforcement learning methods can effectively guide actual production activities. In the existing reinforcement learning methods for new power systems, in order to solve the dimensionality disaster anomaly caused by the ultra-high dimensional data of new power systems, multi-agent deep reinforcement learning algorithms are usually used; the data dimension of this type of method is often bound to the number of agents, and the size of the action space also tends to increase exponentially with the increase in the number of agents.

[0003] However, the memory of a single computing node is limited and usually cannot fully support large-scale multi-agent reinforcement learning models. In addition, the computing power of a single node also greatly constrains the training speed of large-scale multi-agent reinforcement learning models. Although the existing multi-computing node distributed multi-agent reinforcement learning training methods can effectively solve the above two problems, the existing methods are usually designed for general deep learning frameworks and specific hardware environments. These methods are difficult to meet the requirements of large-scale multi-agent reinforcement learning training in new power systems while adapting to general server clusters. Summary of the invention

[0004] The present invention provides a novel multi-agent adaptive hybrid parallel training method for power systems to solve the problem that it is difficult to perform training in a general server cluster, so as to reduce the memory and computing pressure of a single computing node and improve the training speed of the agent.

[0005] By establishing a physical simulation model of the power system, the topology structure and network parameters of the power system are obtained;

[0006] Based on a plurality of preset target requirements, a tuple of a Markov decision process is established according to the topological structure and the network parameters to obtain a plurality of tuples;

[0007] Establishing an objective function according to the plurality of tuples;

[0008] Based on the objective function, agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training until the parameters of multiple agents in several sub-node servers converge to obtain the training results of the multiple agents.

[0009] The present invention establishes a tuple of a Markov decision process by presetting several target requirements, so that the training process has a clear goal orientation. By constructing an objective function, the agent can be guided to continuously optimize its behavior during the training process to maximize the overall performance of the system. The agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training, which can significantly reduce the communication delay and data transmission cost between them, which helps to optimize the utilization rate of system resources, reduce unnecessary resource waste, and improve the overall efficiency of the system. In addition, the agents trained in the same sub-node server or adjacent sub-node servers can more easily achieve collaborative learning and knowledge sharing. Through this parameter interaction mechanism, the agents can learn from each other's useful experience and knowledge, thereby accelerating the training process and improving the generalization ability of the model. In addition, the convergence check mechanism helps to avoid problems such as overfitting and underfitting.

[0010] Compared with the prior art, the present invention realizes distributed processing of training by allocating intelligent agents with high coupling degree to the same sub-node server or adjacent sub-node servers for training. This distributed architecture can significantly reduce the memory and computing pressure of a single computing node; by reasonably allocating intelligent agents to sub-node servers, it can avoid overload of a single node and ensure the overall stability and reliability of the system; and, the intelligent agent can make the best decision based on the current state and reward function to maximize the overall performance of the system, thereby further improving the training speed. Therefore, it can solve the problem of difficulty in training while adapting to a general server cluster, reduce the memory and computing pressure of a single computing node, and improve the training speed of the intelligent agent.

[0011] As a preferred solution, based on a number of preset target requirements, a tuple of a Markov decision process is established according to the topological structure and network parameters to obtain a number of tuples, specifically:

[0012] Establishing an intelligent agent set according to the topological structure and network parameters of the power system; wherein the intelligent agent set includes a power system stability intelligent agent, an energy optimization scheduling intelligent agent and a fault recovery intelligent agent;

[0013] Define the tuple of the Markov decision process and get the initial tuple;

[0014] According to the intelligent agent set and the initial tuple, the plurality of tuples are established by defining the state vectors, action vectors, state transition probabilities and reward functions required by the plurality of objectives.

[0015] This preferred solution applies the Markov decision process to power system management, and realizes intelligent processing of complex decision-making problems in the power system by establishing a tuple including state vectors, action vectors, state transition probabilities and reward functions.

[0016] As a preferred solution, according to the intelligent body set and the initial tuple, the several tuples are established by defining the state vectors, action vectors, state transition probabilities and reward functions required by the several objectives, specifically:

[0017] Based on the set of agents and the initial ancestor, a first state vector and a first action vector are established according to a system frequency and a load adjustment amount;

[0018] According to the frequency deviation and system noise, a first state transfer equation and a first reward function are established;

[0019] A power system stability tuple is formed by the first state vector, the first action vector, the first state transfer equation and the first reward function;

[0020] According to the output power and load demand of each power generation unit, an energy optimization scheduling tuple and a fault recovery tuple are established;

[0021] The plurality of tuples are composed of the power system stability tuple, the energy optimization scheduling tuple and the fault recovery tuple.

[0022] In this preferred solution, each tuple is designed for a specific aspect of the power system, which allows each agent to focus on solving its corresponding problem, thereby improving the efficiency and accuracy of problem solving. By establishing a first state vector and a first action vector based on the system frequency and load adjustment, the system can sense and respond to dynamic changes in the power grid in real time. Frequency deviation and system noise are taken into account when establishing the first state transfer equation, which makes the tuple closer to actual operating conditions. In addition, the establishment of an energy optimization scheduling tuple helps to reduce operating costs and improve energy efficiency; the establishment of a fault recovery tuple helps to improve the reliability and stability of the power system and reduce power outages and economic losses caused by faults. Different types of agents and tuples can work together to achieve overall optimization of agents in the power system.

[0023] As a preferred solution, the first reward function is specifically:

[0024] r t =-(k f |Δf t |+k v ||ΔV t ||+k s Sst )

[0025] Among them, k f , k v and k s is the weight coefficient, Δf t is the frequency deviation, ΔV t is the voltage deviation, S st It is an indicator of system stability.

[0026] As a preferred solution, based on the objective function, agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training, specifically:

[0027] Based on the physical simulation model, a coupling feature matrix is ​​constructed according to the topological coupling degree between the multiple agents;

[0028] Calculating a feature vector set according to the coupling feature matrix and the non-coupling features between the multi-agents;

[0029] Based on the objective function and according to the similarity of features in the feature vector set, agents with high coupling degree are allocated to the same sub-node server or adjacent sub-node servers for training.

[0030] This preferred solution can accurately identify which agents interact more frequently or have higher dependencies by analyzing the topological coupling between multiple agents. Allocating these agents to the same subnode server or adjacent subnode servers for training can significantly reduce the communication delay and cost between them, thereby improving the overall training efficiency.

[0031] As a preferred solution, based on the physical simulation model, a coupling feature matrix is ​​constructed according to the topological coupling degree between multiple agents, specifically:

[0032] Based on the physical simulation model, a topological coupling matrix is ​​established according to the connection status of the state vectors between the multiple agents;

[0033] Establishing a control variable correlation matrix according to the connection status of the control vectors between the multiple agents;

[0034] Establishing a resource sharing matrix according to the resource sharing degree among the multiple agents;

[0035] The topological coupling matrix, the control variable correlation matrix and the resource sharing matrix are weightedly calculated to obtain the coupling characteristic matrix.

[0036] This preferred solution not only considers the connection of state vectors between multiple agents, but also considers the connection of control vectors and resource sharing. This multi-dimensional consideration makes the constructed coupling feature matrix more comprehensive and accurate, and can more truly reflect the complex relationship between multiple agents. In addition, by weighted calculation of the topological coupling matrix, the control variable correlation matrix, and the resource sharing matrix, the weights of different factors can be adjusted according to actual conditions, so that the results are more in line with actual needs and the accuracy of the coupling feature matrix is ​​improved.

[0037] As a preferred solution, based on the objective function and according to the similarity of features in the feature vector set, agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training, specifically:

[0038] According to the similarity of the features in the feature vector set, the agents with high coupling degree are allocated to the same server or adjacent sub-node servers;

[0039] Based on the objective function, controlling the plurality of sub-node servers to train the multi-agent to obtain local model parameters of the plurality of sub-node servers;

[0040] In the form of parameter interaction, the local model parameters of the first sub-node server in the local model parameters are merged with the local model parameters of the first adjacent sub-node server to obtain the first sub-node server after parameter fusion and update;

[0041] The sub-node servers in the plurality of sub-node servers are traversed to obtain the plurality of sub-node servers after parameter fusion and update.

[0042] This preferred solution uses parameter interaction to integrate and update the local model parameters of the sub-node servers, so that each sub-node server can learn useful information and knowledge from other servers. This knowledge sharing mechanism helps to enhance the generalization ability of the server and enables the trained servers to better adapt to different scenarios and tasks.

[0043] The present invention also provides a novel multi-agent adaptive hybrid parallel training device for power systems, comprising a data module, a tuple module, a function module and a training module;

[0044] The data module is used to obtain the topology and network parameters of the power system by establishing a physical simulation model of the power system;

[0045] The tuple module is used to establish a tuple of a Markov decision process according to the topological structure and network parameters based on a plurality of preset target requirements to obtain a plurality of tuples;

[0046] The function module is used to establish an objective function according to the plurality of tuples;

[0047] The training module is used to distribute highly coupled agents to the same sub-node server or adjacent sub-node servers for training based on the objective function until the parameters of multiple agents in several sub-node servers converge to obtain the training results of the multiple agents.

[0048] As a preferred solution, the tuple module includes a model unit, a definition unit and a tuple unit;

[0049] The model unit is used to establish an intelligent agent set according to the topological structure and network parameters of the power system; wherein the intelligent agent set includes a power system stability intelligent agent, an energy optimization scheduling intelligent agent and a fault recovery intelligent agent;

[0050] The definition unit is used to define a tuple of a Markov decision process to obtain an initial tuple;

[0051] The tuple unit is used to establish the plurality of tuples according to the intelligent agent set and the initial tuple by defining the state vectors, action vectors, state transition probabilities and reward functions required by the plurality of objectives.

[0052] As a preferred solution, the tuple unit includes a vector subunit, a function subunit, a stability subunit, a fault subunit and a tuple subunit;

[0053] The vector subunit is used to establish a first state vector and a first action vector according to the system frequency and load adjustment amount based on the intelligent body set and the initial ancestor;

[0054] The function subunit is used to establish a first state transfer equation and a first reward function according to the frequency deviation and the system noise;

[0055] The stabilization subunit is configured to form a power system stability tuple by using the first state vector, the first action vector, the first state transfer equation and the first reward function;

[0056] The fault subunit is used to establish an energy optimization scheduling tuple and a fault recovery tuple according to the output power and load demand of each power generation unit;

[0057] The tuple subunit is used to form the plurality of tuples from the power system stability tuple, the energy optimization scheduling tuple and the fault recovery tuple.

[0058] As a preferred solution, the first reward function is specifically:

[0059] r t =-(kf |Δf t |+k v ||ΔV t ||+k s S st )

[0060] Among them, k f , k v and k s is the weight coefficient, Δf t is the frequency deviation, ΔV t is the voltage deviation, S st It is an indicator of system stability.

[0061] As a preferred solution, the training module includes a matrix unit, a feature unit and an allocation unit;

[0062] Wherein, the matrix unit is used to construct a coupling feature matrix according to the topological coupling degree between multiple agents based on the physical simulation model;

[0063] The feature unit is used to calculate a feature vector set according to the coupling feature matrix and the non-coupling features between the multi-agents;

[0064] The allocation unit is used to allocate intelligent agents with high coupling degree to the same sub-node server or adjacent sub-node servers for training based on the objective function and the similarity of features in the feature vector set.

[0065] As a preferred solution, the matrix unit includes a topology subunit, a variable subunit, a resource subunit and a coupling subunit;

[0066] Wherein, the topology subunit is used to establish a topology coupling matrix based on the physical simulation model and according to the connection status of the state vectors between the multiple agents;

[0067] The variable subunit is used to establish a control variable correlation matrix according to the connection status of the control vectors between the multiple agents;

[0068] The resource subunit is used to establish a resource sharing matrix according to the resource sharing degree among the multiple agents;

[0069] The coupling subunit is used to perform weighted calculation on the topological coupling matrix, the control variable correlation matrix and the resource sharing matrix to obtain the coupling characteristic matrix.

[0070] As a preferred solution, the allocation unit includes a similarity subunit, a training subunit, a fusion subunit and an update subunit;

[0071] Wherein, the similarity subunit is used to allocate agents with high coupling degree to the same server or adjacent subnode servers according to the similarity of features in the feature vector set;

[0072] The training subunit is used to control the plurality of sub-node servers to train the multi-agent based on the objective function to obtain local model parameters of the plurality of sub-node servers;

[0073] The fusion subunit is used to fuse the local model parameters of the first subnode server and the local model parameters of the first adjacent subnode server in the local model parameters in the form of parameter interaction to obtain the first subnode server after parameter fusion and update;

[0074] The updating sub-unit is used to traverse the sub-node servers among the plurality of sub-node servers to obtain the plurality of sub-node servers after parameter fusion and update.

[0075] The present application also provides a storage medium having a computer program stored thereon, wherein the computer program is called and executed by a computer to implement the multi-agent adaptive hybrid parallel training method for a novel power system as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of a novel multi-agent adaptive hybrid parallel training method for a power system provided in an embodiment of the present application;

[0077] Figure 2 is a training flow chart provided in an embodiment of the present application;

[0078] Figure 3 It is a structural schematic diagram of a novel multi-agent adaptive hybrid parallel training device for a power system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] In the description of the present application, it should be understood that the terms "first", "second" and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first", "second" and "third" may explicitly or implicitly include one or more of the features. In the description of the present application, unless otherwise specified, "several" means two or more.

[0081] The embodiment of the present application provides a multi-agent adaptive hybrid parallel training method for a new power system, which is mainly used in situations where distributed parallel training of a large-scale multi-agent reinforcement learning model of the new power system is required to reduce the memory and computing pressure of a single computing node and improve the training speed of the model.

[0082] Embodiment 1:

[0083] See also Figure 1 The embodiment of the present application provides a novel multi-agent adaptive hybrid parallel training method for a power system, including S1 to S4, and the specific implementation steps are as follows:

[0084] S1. Obtain the topological structure and network parameters of the power system by establishing a physical simulation model of the power system.

[0085] Step S1 of the embodiment of the present application is specifically as follows:

[0086] Determine the network topology in the power system, and determine the electrical parameters of the transmission and distribution lines by defining the types of each node; the network topology includes the location and connection relationship of power plants, substations, transmission lines, distribution lines and load nodes;

[0087] According to the network topology and electrical parameters, a new power system simulation model corresponding to photovoltaic, wind turbine, synchronous generator, energy storage, transformer and interconnection line components is established in DSP, BPA and other power system simulation software;

[0088] According to the new power system simulation model of the power system, a corresponding complete large signal model is established to provide a basic environment for subsequent multi-agent reinforcement learning training.

[0089] It should be noted that the power system in the first embodiment is a new power system.

[0090] S2. Based on a number of preset target requirements, a tuple of a Markov decision process is established according to the topological structure and network parameters to obtain a number of tuples.

[0091] Step S2 of the embodiment of the present application includes S2.1 to S2.4; wherein S2.1 is a process of establishing an initial tuple, S2.2 is a process of establishing a power system stability tuple, S2.3 is a process of establishing an energy optimization scheduling tuple, and S2.4 is a process of establishing a fault recovery tuple and obtaining a plurality of tuples, specifically:

[0092] S2.1. According to the new power system simulation model, the topology structure and network parameters of the power system are obtained; wherein the topology structure and network parameters include the topology structure, equipment model and load model of the power system;

[0093] An intelligent agent set is established according to the topological structure and network parameters of the power system; wherein the intelligent agent set includes the power system stability agent, the energy optimization dispatching agent and the fault recovery agent; and the agents with different objectives have different state definitions, action definitions, transfer mechanisms and reward mechanisms.

[0094] Define the tuple of the Markov decision process and get the initial tuple;

[0095] Among them, the initial ancestor is:

[0096] <S,A,P,R,γ>

[0097] Where S is the state vector of the current agent, S = {s1, s2, ..., s n},s n represents the nth observation of the system; A is the action vector of the current agent, A = {a1, a2, ..., a m}, a m represents the mth relevant decision quantity of the system; P is the state transition probability of the agent, which can be expressed as Represents the state S of the system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, which can be expressed as Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

[0098] S2.2, based on the intelligent body set and the initial ancestor, in the power grid stability control target requirement, a first state vector and a first action vector are established according to the system frequency and load adjustment amount;

[0099] According to the frequency deviation and system noise, a first state transfer equation and a first reward function are established;

[0100] A power system stability tuple is formed by a first state vector, a first action vector, a first state transfer equation and a first reward function;

[0101] Among them, the first state vector is:

[0102] S=[f t , V t ,ω t , L t , P t , E t ] T

[0103] The first motion vector is:

[0104] A=[P g , ΔL t , P e , P gv , P wind ] T

[0105] The first state transfer equation is:

[0106] S t+1 =f(S t , A t , w t )

[0107] The first reward function is:

[0108] r t =-(k f |Δf t |+k v ||ΔV t ||+k s S st )

[0109] Among them, f t is the system frequency, V t is the node voltage, ω t is the generator speed, L t is the load level, P t is the power flow, E t is the energy storage state, T is the total time;

[0110] P g is the regulated power, ΔL t is the load adjustment, P e is the charging and discharging power of the energy storage device, P gv is the photovoltaic discharge power, P wind is the fan discharge power;

[0111] wt is the system noise and external disturbance, S t is the first state vector at time t, A t is the first action vector at time t;

[0112] k f , k v and k s is the weight coefficient, Δf t is the frequency deviation, ΔV t is the voltage deviation, S st It is an indicator of system stability.

[0113] In this embodiment S2.2, by establishing the first state vector and the first action vector according to the system frequency and the load adjustment amount, the system can perceive and respond to the dynamic changes of the power grid in real time. When establishing the first state transfer equation, the frequency deviation and system noise are considered, which makes the tuple closer to the actual operation.

[0114] S2.3, based on the intelligent body set and the initial ancestor, in the energy optimization scheduling control target requirement, a second state vector and a second action vector are established according to the output power of each power generation unit and the state of the energy storage device;

[0115] According to the cost of power generation, energy storage and purchase and sale points, a second state transfer equation and a second reward function are established;

[0116] An energy optimization scheduling tuple is formed by a second state vector, a second action vector, a second state transfer equation, and a second reward function;

[0117] Among them, the second state vector is:

[0118] S=[P g , E t , D t , R t , C t ] T

[0119] The second motion vector is:

[0120] A=[ΔP g , P e , ΔD t , P b ] T

[0121] The second state transfer equation is:

[0122] S t+1 =f(S t , A t , t )

[0123] The second reward function is:

[0124]

[0125] Among them, P g is the output power of each power generation unit, E t is the state of the energy storage device, D t is the load demand, R t is the renewable energy forecast, C t It is the electricity price information;

[0126] ΔP g is the generation dispatch plan, P e is the energy storage charging and discharging power, ΔD t is the load adjustment strategy, P b Is it a strategy to buy or sell electricity?

[0127] ζ t is the error between load forecast and renewable energy forecast, S t is the second state vector at time t, A t is the second action vector at time t;

[0128] C g , C e and C b are the costs of power generation, energy storage and point of purchase and sale, η r is the renewable energy utilization reward coefficient, renewable represents renewable energy, P gi is the output power of the ith power generation unit.

[0129] S2.4, based on the intelligent body set and the initial ancestor, in the fault recovery target requirement, establish a third state vector and a third action vector according to the fault location and the fault type;

[0130] According to the power supply status of the load node and the uncertain factors of the fault recovery process, a third state transfer equation and a third reward function are established;

[0131] A fault recovery tuple is formed by a third state vector, a third action vector, a third state transfer equation and a third reward function;

[0132] A plurality of tuples are formed by a power system stability tuple, an energy optimization dispatch tuple and a fault recovery tuple;

[0133] Among them, the third state vector is:

[0134] S=[F t , T t , L t , B t , D t , Rt ] T

[0135] The third action vector is:

[0136] A=[I t , M t , ΔS t , ΔB t ] T

[0137] The third state transfer equation is:

[0138] S t+1 =f(S t , A t , η t )

[0139] The third reward function is:

[0140]

[0141] Among them, F t is the fault location, T t is the fault type, L t is the line status, B t It is the standby power state, D t is the load level, R t It is the recovery progress;

[0142] I t is a fault isolation measure, M t is the fault point repair solution, ΔS t is the load transfer strategy, ΔB t The backup power supply is enabled;

[0143] η t It is an uncertain factor in the fault recovery process. t is the third state vector at time t, A t is the third action vector at time t;

[0144] k r and k l is the weight coefficient, R t is the recovery time, δ i is the power supply status of load node i, D i is the load level of load node i.

[0145] In embodiments S2.2 to S2.4, each tuple is designed for a specific aspect of the power system, which enables each agent to focus on solving its corresponding problem, thereby improving the efficiency and accuracy of problem solving. The establishment of the power system stability tuple helps to promote the safe and stable operation of the power system; the establishment of the energy optimization scheduling tuple helps to reduce operating costs and improve energy efficiency; the establishment of the fault recovery tuple helps to improve the reliability and stability of the power system and reduce the power outage time and economic losses caused by faults.

[0146] S3. Establish an objective function based on several tuples.

[0147] Step S3 of the embodiment of the present application is specifically as follows:

[0148] Establish a cumulative reward value reward function based on a number of tuples;

[0149] The objective function is obtained by maximizing the cumulative return value reward function;

[0150] Among them, the cumulative return value reward function is:

[0151]

[0152] The objective function is:

[0153]

[0154] in, is the optimal control strategy of the agent to be obtained, A t is the action vector at time t, R is the reward function, S t and S t+1 are the state vectors at time t and time t+1 respectively, and Pr(S t+1 |S t , A t ) represent the state S of the system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; γ is the discount factor, its value is between 0 and 1; V(S t+1 ) means in the next state S t+1 The optimal state value, that is, the expected return that can be obtained by continuing to execute the optimal strategy in a certain state in the future. This is a recursive call, and the cumulative return reward function is V(S t ).

[0155] S4. Based on the objective function, the highly coupled agents are assigned to the same sub-node server or adjacent sub-node servers for training until the parameters of the multiple agents in several sub-node servers converge and the training results of the multiple agents are obtained.

[0156] Step S4 of the embodiment of the present application includes S4.1 to S4.3; wherein S4.1 is the process of establishing a coupling feature matrix, S4.2 is the process of executing a training task, and S4.3 is the process of obtaining a training result, specifically:

[0157] S4.1. Based on the physical simulation model, topological structure, network parameters, and the connection status of state vectors between multiple agents, a topological coupling matrix T is established;

[0158] According to the connection of control vectors between multiple agents, establish the control variable correlation matrix V;

[0159] According to the resource sharing degree among multiple agents, a resource sharing matrix R is established. R ;

[0160] For the topological coupling matrix T, the control variable correlation matrix V and the resource sharing matrix R R Perform weighted calculation to obtain the coupling characteristic matrix C ij ;

[0161] Among them, for the topological coupling matrix, we have:

[0162] T ij =λ ij Adj ij

[0163] For the control variable correlation matrix, we have:

[0164] V ij =ρ ij Bj ij

[0165] For the resource sharing matrix, we have:

[0166] R Rij =δ ij

[0167] The coupling characteristic matrix is:

[0168] C ij =α1T ij +α2V ij +α3R Rij

[0169] Among them, T ij Represents agent A i and Agent A jThe topological coupling between ij is the topological connection weight, which is determined by the reactance characteristics of the interconnection line of the new power system; Adj ij is the adjacency matrix element, representing agent A i and Agent A j Is the state vector of connected?

[0170] V ij Represents agent A i and Agent A j Correlation of control variables between ij is the control variable coupling weight, which is determined by the reactance characteristics of the new power system interconnection line and the characteristics of the control variable; Bdj ij is the adjacency matrix element, representing agent A i and Agent A j Whether the control vector is connected;

[0171] R Rji Represents agent A i and Agent A j The resource sharing degree between ij is the shared resource weight;

[0172] α1, α2 and α3 are weight coefficients.

[0173] It should be noted that the coupling characteristics are mainly divided into three categories: topological coupling, control variable correlation, and resource sharing; among them, topological coupling is jointly determined by the physical topological structure of the new power system and the state vector of the intelligent agent, control variable correlation is used to analyze the correlation between the control variables of the intelligent agent (such as power, voltage, frequency), and resource sharing is used to identify intelligent agents that share the same resources (such as transmission lines and energy storage devices).

[0174] By analyzing the topological coupling degree between multiple agents, this embodiment S4.1 can accurately identify which agents interact more frequently or have a higher degree of dependence; and the establishment of the coupling feature matrix not only considers the connection of the state vectors between multiple agents, but also considers the connection of the control vectors and the resource sharing degree. This multi-dimensional consideration makes the constructed coupling feature matrix more comprehensive and accurate, and can more truly reflect the complex relationship between multiple agents. In addition, by weighted calculation of the topological coupling matrix, the control variable correlation matrix, and the resource sharing matrix, the weights of different factors can be adjusted according to the actual situation, so that the results are more in line with actual needs, and the accuracy of the coupling feature matrix is ​​improved.

[0175] S4.2. According to the coupling characteristic matrix C ij and the decoupling feature between multiple agents D ij , calculate and obtain the feature vector set;

[0176] According to the similarity of the features in the feature vector set, the agents in each cluster are initially assigned to the sub-node servers, ensuring that agents with high coupling degree are assigned to the same server or adjacent sub-node servers as much as possible;

[0177] In the case of providing a basic training environment through a complete large signal model, based on the objective function, several sub-node servers are controlled to train the multi-agents to obtain the local model parameters of several sub-node servers; wherein each sub-node server N m Responsible for k agents (A m1 , A m2 , ..., A mk ) training task; and in the local training process, the subnode server performs reinforcement learning training according to the specific tasks of the power system to obtain the local model parameters (θ m1 ,θ m2 , ..., θ mk );

[0178] Based on the parameter fusion formula, the first child node server N in the local model parameters is transformed into m The local model parameters (θ m1 ,θ m2 , ..., θ mk ) and the first adjacent child node server N n The local model parameters (θ n1 ,θ n2 , ..., θ nj ) to obtain the first child node server N after parameter fusion update m ;

[0179] Traversing the sub-node servers in the plurality of sub-node servers to obtain the plurality of sub-node servers after parameter fusion and update;

[0180] Among them, the uncoupled characteristics are:

[0181] D ij =1-C ij

[0182] The eigenvectors in the eigenvector set are:

[0183]

[0184] The parameter fusion formula is:

[0185]

[0186] Where C represents the coupling characteristic matrix C ij , D represents the uncoupled feature Dij ;

[0187] β∈[0,1] is the fusion coefficient, which controls the weight ratio of parameter interaction; ζ is the variable of the summation symbol, which changes from 0 to j, indicating that All items Add together.

[0188] In this embodiment S4.2, these agents are assigned to the same subnode server or adjacent subnode servers for training, which can significantly reduce the communication delay and cost between them, thereby improving the overall training efficiency;

[0189] In addition, by integrating and updating the local model parameters of the sub-node servers in the form of parameter interaction, each sub-node server can learn useful information and knowledge from other servers. This knowledge sharing mechanism helps to enhance the generalization ability of the server and enables the trained servers to better adapt to different scenarios and tasks.

[0190] S4.3, communicating with all the sub-node servers through the control center node server, and obtaining the intelligent agent model parameters trained in all the sub-node servers;

[0191] If the parameters of the multi-agents in several sub-node servers converge, the training is completed and the training results of the multi-agents are obtained;

[0192] If the parameters of the multi-agents in several sub-node servers have not converged, the training tasks of the multi-agents are reallocated to each sub-node server and the next round of training begins until the parameters of the multi-agents converge and the training results of the multi-agents are obtained;

[0193] Among them, the training results of multi-agents refer to the behavior control strategies of multi-agents.

[0194] To apply the embodiments of the present invention, please refer to Figure 2 , Figure 2 : is a training flow chart provided in this embodiment, which shows the method of performing multi-agent adaptive hybrid parallel training in this embodiment 1, which is generally as follows:

[0195] First, a simulation model of a new power system is established, including the topological structure, equipment model, load model, etc. of the power system, and a corresponding large-signal mathematical model is constructed to provide a basic environment for multi-agent reinforcement learning training.

[0196] Subsequently, the Markov decision process (MDP) of each reinforcement learning agent, including state, action, reward, and transfer mechanism, is determined according to the control objectives of the power system, such as grid stability, energy optimization dispatch, and fault recovery;

[0197] Next, the central node server initializes the network parameters of each agent to ensure that the agent can effectively explore and utilize environmental information;

[0198] Subsequently, according to the coupling and non-coupling characteristics of the agents, multiple agents are adaptively divided into different sub-node servers, and each sub-node server is responsible for the training tasks of a part of the agents; after completing a round of local environment training, the sub-node server will perform an iterative parameter update with the adjacent sub-node server;

[0199] Finally, the child node server returns the training results to the central node server. After the central node server updates the model parameters, it determines whether the multi-agent model training has converged. If it has converged, the training is completed. Otherwise, the agents are reallocated to the child node servers and the next round of distributed training begins.

[0200] Overall, this embodiment has the following beneficial effects:

[0201] The present invention establishes a tuple of the Markov decision process through a number of preset target requirements, so that the training process has a clear goal orientation. By constructing the objective function, the intelligent agent can be guided to continuously optimize its own behavior during the training process to maximize the overall performance of the system. Assigning intelligent agents with high coupling to the same sub-node server or adjacent sub-node servers for training can significantly reduce the communication delay and data transmission cost between them, which helps to optimize the utilization of system resources, reduce unnecessary resource waste, and improve the overall efficiency of the system. In addition, intelligent agents trained in the same sub-node server or adjacent sub-node servers can more easily achieve collaborative learning and knowledge sharing. Through this parameter interaction mechanism, intelligent agents can draw on and learn from each other's useful experience and knowledge, thereby accelerating the training process and improving the generalization ability of the model. In addition, the convergence check mechanism helps to avoid problems such as overfitting and underfitting;

[0202] Therefore, the present invention realizes distributed parallel training of large-scale multi-agent reinforcement learning models of new power systems on general server clusters, significantly reduces the memory and computing pressure of a single computing node, and greatly improves the training speed of the model.

[0203] Embodiment 2:

[0204] See also Figure 3 , the embodiment of the present application provides a novel multi-agent adaptive hybrid parallel training device for a power system, comprising a data module 10, a tuple module 20, a function module 30 and a training module 40;

[0205] The data module 10 is used to obtain the topological structure and network parameters of the power system by establishing a physical simulation model of the power system;

[0206] The tuple module 20 is used to establish a tuple of a Markov decision process according to a topological structure and network parameters based on a plurality of preset target requirements, so as to obtain a plurality of tuples;

[0207] A function module 30, for establishing a cumulative reward value reward function according to a plurality of tuples;

[0208] The training module 40 is used to distribute highly coupled agents to the same sub-node server or adjacent sub-node servers for training based on the cumulative return value reward function until the parameters of multiple agents in several sub-node servers converge to obtain the training results of the multiple agents.

[0209] In one embodiment, the data module 10 is specifically:

[0210] Determine the network topology in the power system, and determine the electrical parameters of the transmission and distribution lines by defining the types of each node; the network topology includes the location and connection relationship of power plants, substations, transmission lines, distribution lines and load nodes;

[0211] According to the network topology and electrical parameters, a new power system simulation model corresponding to photovoltaic, wind turbine, synchronous generator, energy storage, transformer and interconnection line components is established in DSP, BPA and other power system simulation software;

[0212] According to the new power system simulation model of the power system, a corresponding complete large signal model is established to provide a basic environment for subsequent multi-agent reinforcement learning training.

[0213] It should be noted that the power system in the second embodiment is a new power system.

[0214] In one embodiment, the tuple module 20 includes a model unit, an intelligent unit, a definition unit, a vector subunit, a function subunit, a stability subunit, a scheduling subunit, a fault subunit and a tuple subunit; wherein the model unit, the intelligent unit and the definition unit are processes for establishing an initial tuple, the vector subunit, the function subunit and the stability subunit are processes for establishing a power system stability tuple, the scheduling subunit is a process for establishing an energy optimization scheduling tuple, the fault subunit and the tuple subunit are processes for establishing a fault recovery tuple and obtaining a plurality of tuples, specifically:

[0215] A model unit, used for obtaining the topological structure and network parameters of the power system according to the new power system simulation model; wherein the topological structure and network parameters include the topological structure, equipment model and load model of the power system;

[0216] The intelligent unit is used to establish an intelligent agent set according to the topological structure and network parameters of the power system; wherein the intelligent agent set includes a power system stability agent, an energy optimization scheduling agent and a fault recovery agent; and agents with different goals have different state definitions, action definitions, transfer mechanisms and reward mechanisms.

[0217] Definition unit, used to define the tuple of Markov decision process and obtain the initial tuple;

[0218] Among them, the initial ancestor is:

[0219] <S,A,P,R,γ>

[0220] Where S is the state vector of the current agent, S = {s1, s2, ..., s n},s n represents the nth observation of the system; A is the action vector of the current agent, A = {a1, a2, ..., a m}, a m represents the mth relevant decision quantity of the system; P is the state transition probability of the agent, which can be expressed as Represents the state S of the system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of; R is the reward function of the agent, which can be expressed as Indicates that in state S t Next, execute action A t The reward value that can be obtained; γ is the discount factor, and its value is between 0 and 1.

[0221] A vector subunit, for establishing a first state vector and a first action vector according to a system frequency and a load adjustment amount in a power grid stability control target requirement based on the intelligent body set and the initial element;

[0222] A function subunit, used for establishing a first state transfer equation and a first reward function according to the frequency deviation and the system noise;

[0223] a stabilization subunit, configured to form a power system stability tuple from a first state vector, a first action vector, a first state transfer equation, and a first reward function;

[0224] Among them, the first state vector is:

[0225] S=[f t , V t ,ω t , L t , P t , Et ] f

[0226] The first motion vector is:

[0227] A=[P g , ΔL t , P e , P pv , P wind ] T

[0228] The first state transfer equation is:

[0229] S t+1 =f(S t , A t , w t )

[0230] The first reward function is:

[0231] r t =-(k f |Δf t |+k v ||ΔV t ||+k s S st )

[0232] Among them, f t is the system frequency, V t is the node voltage, ω t is the generator speed, L t is the load level, P t is the power flow, E t is the energy storage state, T is the total time;

[0233] P g is the regulated power, ΔL t is the load adjustment, P e is the charging and discharging power of the energy storage device, P pv is the photovoltaic discharge power, P wind is the fan discharge power;

[0234] w t is the system noise and external disturbance, S t is the first state vector at time t, A t is the first action vector at time t;

[0235] k f , k v and k s is the weight coefficient, Δf t is the frequency deviation, ΔV t is the voltage deviation, S stIt is an indicator of system stability.

[0236] In the vector subunit, function subunit and stability subunit of this embodiment, by establishing the first state vector and the first action vector according to the system frequency and load adjustment amount, the system can perceive and respond to the dynamic changes of the power grid in real time. When establishing the first state transfer equation, the frequency deviation and system noise are considered, which makes the tuple closer to the actual operation.

[0237] The scheduling subunit is used to establish a second state vector and a second action vector according to the output power of each power generation unit and the state of the energy storage device in the energy optimization scheduling control target requirement based on the intelligent body set and the initial element;

[0238] The scheduling subunit is also used to establish a second state transfer equation and a second reward function based on the cost of power generation, energy storage and purchase and sale points;

[0239] The scheduling subunit is further used to form an energy optimization scheduling tuple from the second state vector, the second action vector, the second state transfer equation and the second reward function;

[0240] Among them, the second state vector is:

[0241] S=[P g , E t , D t , R t , C t ] T

[0242] The second motion vector is:

[0243] A=[ΔP g , P e , ΔD t , P b ] T

[0244] The second state transfer equation is:

[0245] S t+1 =f(S t , A t , t )

[0246] The second reward function is:

[0247]

[0248] Among them, P g is the output power of each power generation unit, E t is the state of the energy storage device, D t is the load demand, R t is the renewable energy forecast, Ct It is the electricity price information;

[0249] ΔP g is the generation dispatch plan, P e is the energy storage charging and discharging power, ΔD t is the load adjustment strategy, P b Is it a strategy to buy or sell electricity?

[0250] ζ t is the error between load forecast and renewable energy forecast, S t is the second state vector at time t, A t is the second action vector at time t;

[0251] C g , C e and C b are the costs of power generation, energy storage and point of purchase and sale, η r is the renewable energy utilization reward coefficient, renewable represents renewable energy, P gi is the output power of the ith power generation unit.

[0252] A fault subunit, for establishing a third state vector and a third action vector according to a fault location and a fault type in a fault recovery target requirement based on the intelligent body set and the initial ancestor;

[0253] The fault subunit is further used to establish a third state transfer equation and a third reward function according to the power supply state of the load node and the uncertainty factors of the fault recovery process;

[0254] The fault subunit is further used to form a fault recovery tuple from a third state vector, a third action vector, a third state transfer equation and a third reward function;

[0255] A tuple subunit, used for forming a plurality of tuples from a power system stability tuple, an energy optimization dispatch tuple, and a fault recovery tuple;

[0256] Among them, the third state vector is:

[0257] S=[F t , T t , L t , B t , D t , R t ] T

[0258] The third action vector is:

[0259] A=[I t , M t , ΔS t , ΔB t ]T

[0260] The third state transfer equation is:

[0261] S t+1 =f(S t , A t , η t )

[0262] The third reward function is:

[0263]

[0264] Among them, F t is the fault location, T t is the fault type, L t is the line status, B t It is the standby power state, D t is the load level, R t It is the recovery progress;

[0265] I t is a fault isolation measure, M t is the fault point repair solution, ΔS t is the load transfer strategy, ΔB t The backup power supply is enabled;

[0266] η t It is an uncertain factor in the fault recovery process. t is the third state vector at time t, A t is the third action vector at time t;

[0267] k r and k l is the weight coefficient, R t is the recovery time, δ i is the power supply status of load node i, D i is the load level of load node i.

[0268] In the embodiment vector subunit, function subunit and stability subunit, scheduling subunit, fault subunit and tuple subunit, each tuple is designed for a specific aspect of the power system, which enables each intelligent agent to focus on solving its corresponding problem, thereby improving the efficiency and accuracy of problem solving. The establishment of the power system stability tuple helps to promote the safe and stable operation of the power system; the establishment of the energy optimization scheduling tuple helps to reduce operating costs and improve energy efficiency; the establishment of the fault recovery tuple helps to improve the reliability and stability of the power system and reduce the power outage time and economic losses caused by faults.

[0269] In one embodiment, the function module 30 is specifically:

[0270] Establish a cumulative reward value reward function based on a number of tuples;

[0271] The objective function is obtained by maximizing the cumulative return value reward function;

[0272] Among them, the cumulative return value reward function is:

[0273]

[0274] The objective function is:

[0275]

[0276] in, is the optimal control strategy of the agent to be obtained, A t is the action vector at time t, R is the reward function, S t and S t+1 are the state vectors at time t and time t+1 respectively, and Pr(S t+1 |S t , A t ) represent the state S of the system at time t t By performing action A t Arrive at the next moment state S t+1 The probability of V is the system voltage; γ is the discount factor, which is between 0 and 1; V(S t+1 ) means in the next state S t+1 The optimal state value, that is, the expected return that can be obtained by continuing to execute the optimal strategy in a certain state in the future. This is a recursive call, and the cumulative return reward function is V(S t ).

[0277] In one embodiment, the training module 40 includes a topology subunit, a variable subunit, a resource subunit, a coupling subunit, a feature unit, a similarity subunit, a training subunit, a fusion subunit, an update subunit and a convergence unit; wherein the topology subunit, the variable subunit, the resource subunit and the coupling subunit are the process of establishing a coupling feature matrix, the feature unit, the similarity subunit, the training subunit, the fusion subunit and the update subunit are the process of executing the training task, and the convergence unit is the process of obtaining the training result, specifically:

[0278] The topology subunit is used to establish a topological coupling matrix T based on the physical simulation model, the topological structure, and the network parameters according to the connection of the state vectors between the multiple agents;

[0279] The variable subunit is used to establish the control variable correlation matrix V according to the connection of the control vectors between multiple agents;

[0280] The resource subunit is used to establish the resource sharing matrix R according to the resource sharing degree among multiple agents. R ;

[0281] The coupling subunit is used to adjust the topological coupling matrix T, the control variable correlation matrix V and the resource sharing matrix R R Perform weighted calculation to obtain the coupling characteristic matrix C ij ;

[0282] Among them, for the topological coupling matrix, we have:

[0283] T ij =λ ij Adj ij

[0284] For the control variable correlation matrix, we have:

[0285] V ij =ρ ij Bj ij

[0286] For the resource sharing matrix, we have:

[0287] R Rij =δ ij

[0288] The coupling characteristic matrix is:

[0289] C ij =α1T ij +α2V ij +α3R Rij

[0290] Among them, T ij Represents agent A i and Agent A j The topological coupling between ij is the topological connection weight, which is determined by the reactance characteristics of the interconnection line of the new power system; Adj ij is the adjacency matrix element, representing agent A i and Agent A j Is the state vector of connected?

[0291] V ij Represents agent A i and Agent A j Correlation of control variables between ij is the control variable coupling weight, which is determined by the reactance characteristics of the new power system interconnection line and the characteristics of the control variable; Bdj ij is the adjacency matrix element, representing agent A i and Agent Aj Whether the control vector is connected;

[0292] R Rji Represents agent A i and Agent A j The resource sharing degree between ij is the shared resource weight;

[0293] α1, α2 and α3 are weight coefficients.

[0294] It should be noted that the coupling characteristics are mainly divided into three categories: topological coupling, control variable correlation, and resource sharing; among them, topological coupling is jointly determined by the physical topological structure of the new power system and the state vector of the intelligent agent, control variable correlation is used to analyze the correlation between the control variables of the intelligent agent (such as power, voltage, frequency), and resource sharing is used to identify intelligent agents that share the same resources (such as transmission lines and energy storage devices).

[0295] By analyzing the topological coupling degree between multiple agents, this embodiment S4.1 can accurately identify which agents interact more frequently or have a higher degree of dependence; and the establishment of the coupling feature matrix not only considers the connection of the state vectors between multiple agents, but also considers the connection of the control vectors and the resource sharing degree. This multi-dimensional consideration makes the constructed coupling feature matrix more comprehensive and accurate, and can more truly reflect the complex relationship between multiple agents. In addition, by weighted calculation of the topological coupling matrix, the control variable correlation matrix, and the resource sharing matrix, the weights of different factors can be adjusted according to the actual situation, so that the results are more in line with actual needs, and the accuracy of the coupling feature matrix is ​​improved.

[0296] Characteristic unit, used to calculate the coupling characteristic matrix C ij and the decoupling feature between multiple agents D ij , calculate and obtain the feature vector set;

[0297] The similarity subunit is used to preliminarily allocate the agents in each cluster to the subnode servers according to the similarity of the features in the feature vector set, ensuring that agents with high coupling degree are allocated to the same server or adjacent subnode servers as much as possible;

[0298] The training subunit is used to control several sub-node servers to train the multi-agent based on the objective function under the condition that the basic training environment is provided by the complete large signal model, and obtain the local model parameters of several sub-node servers; wherein each sub-node server N m Responsible for k agents (A m1 , A m2 , ..., A mk) training task; and in the local training process, the subnode server performs reinforcement learning training according to the specific tasks of the power system to obtain the local model parameters (θ m1 ,θ m2 , ..., θ mk );

[0299] The fusion subunit is used to combine the first subnode server N in the local model parameters in the form of parameter interaction based on the parameter fusion formula. m The local model parameters (θ m1 ,θ m2 , ..., θ mk ) and the first adjacent child node server N n The local model parameters (θ n1 ,θ n2 , ..., θ nj ) to obtain the first child node server N after parameter fusion update m ;

[0300] An updating sub-unit, used for traversing the sub-node servers among the plurality of sub-node servers, and obtaining the plurality of sub-node servers after parameter fusion and update;

[0301] Among them, the uncoupled characteristics are:

[0302] D ij =1-C ij

[0303] The eigenvectors in the eigenvector set are:

[0304]

[0305] The parameter fusion formula is:

[0306]

[0307] Where C represents the coupling characteristic matrix C ij , D represents the uncoupled feature D ij ;

[0308] β∈[0,1] is the fusion coefficient, which controls the weight ratio of parameter interaction; ζ is the variable of the summation sign, which changes from 0 to j, indicating that All items Add together.

[0309] In this embodiment, the feature unit, similarity subunit, training subunit, fusion subunit and update subunit assign these agents to the same subnode server or adjacent subnode servers for training, which can significantly reduce the communication delay and cost between them, thereby improving the overall training efficiency;

[0310] In addition, by integrating and updating the local model parameters of the sub-node servers in the form of parameter interaction, each sub-node server can learn useful information and knowledge from other servers. This knowledge sharing mechanism helps to enhance the generalization ability of the server and enables the trained servers to better adapt to different scenarios and tasks.

[0311] A convergence unit is used to communicate with all the sub-node servers through the control center node server to obtain the intelligent agent model parameters trained in all the sub-node servers;

[0312] The convergence unit is also used to complete the training if the parameters of the multi-agents in several sub-node servers converge, and obtain the training results of the multi-agents;

[0313] The convergence unit is also used to reallocate the training tasks of the multi-agent to each sub-node server if the parameters of the multi-agent in several sub-node servers have not converged, and start the next round of training until the parameters of the multi-agent converge, thereby obtaining the training results of the multi-agent;

[0314] Among them, the training results of multi-agents refer to the behavior control strategies of multi-agents.

[0315] To apply the embodiments of the present invention, please refer to Figure 2 , Figure 2 is a training flow chart provided in this embodiment, which shows the method of performing multi-agent adaptive hybrid parallel training in this embodiment 2, which is generally as follows:

[0316] First, a simulation model of a new power system is established, including the topological structure, equipment model, load model, etc. of the power system, and a corresponding large-signal mathematical model is constructed to provide a basic environment for multi-agent reinforcement learning training.

[0317] Subsequently, the Markov decision process (MDP) of each reinforcement learning agent, including state, action, reward, and transfer mechanism, is determined according to the control objectives of the power system, such as grid stability, energy optimization dispatch, and fault recovery;

[0318] Next, the central node server initializes the network parameters of each agent to ensure that the agent can effectively explore and utilize environmental information;

[0319] Subsequently, according to the coupling and non-coupling characteristics of the agents, multiple agents are adaptively divided into different sub-node servers, and each sub-node server is responsible for the training tasks of a part of the agents; after completing a round of local environment training, the sub-node server will perform an iterative parameter update with the adjacent sub-node server;

[0320] Finally, the child node server returns the training results to the central node server. After the central node server updates the model parameters, it determines whether the multi-agent model training has converged. If it has converged, the training is completed. Otherwise, the agents are reallocated to the child node servers and the next round of distributed training begins.

[0321] Overall, this embodiment has the following beneficial effects:

[0322] The present invention establishes a tuple of the Markov decision process through a number of preset target requirements, so that the training process has a clear goal orientation. By constructing the objective function, the intelligent agent can be guided to continuously optimize its own behavior during the training process to maximize the overall performance of the system. Assigning intelligent agents with high coupling to the same sub-node server or adjacent sub-node servers for training can significantly reduce the communication delay and data transmission cost between them, which helps to optimize the utilization of system resources, reduce unnecessary resource waste, and improve the overall efficiency of the system. In addition, intelligent agents trained in the same sub-node server or adjacent sub-node servers can more easily achieve collaborative learning and knowledge sharing. Through this parameter interaction mechanism, intelligent agents can draw on and learn from each other's useful experience and knowledge, thereby accelerating the training process and improving the generalization ability of the model. In addition, the convergence check mechanism helps to avoid problems such as overfitting and underfitting;

[0323] Therefore, the present invention realizes distributed parallel training of large-scale multi-agent reinforcement learning models of new power systems on general server clusters, significantly reduces the memory and computing pressure of a single computing node, and greatly improves the training speed of the model.

[0324] Embodiment three:

[0325] The embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the multi-agent adaptive hybrid parallel training method for a novel power system;

[0326] Wherein, the multi-agent adaptive hybrid parallel training method of the novel power system, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0327] The above are preferred embodiments of the present invention. It should be noted that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A novel multi-agent adaptive hybrid parallel training method for power systems, characterized in that: include: By establishing a physical simulation model of the power system, the topology structure and network parameters of the power system are obtained; Based on a plurality of preset target requirements, a tuple of a Markov decision process is established according to the topological structure and the network parameters to obtain a plurality of tuples; Establishing an objective function according to the plurality of tuples; Based on the objective function, agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training until the parameters of multiple agents in several sub-node servers converge to obtain the training results of the multiple agents.

2. A novel multi-agent adaptive hybrid parallel training method for power system according to claim 1, characterized in that: Based on a number of preset target requirements, a tuple of a Markov decision process is established according to the topological structure and the network parameters to obtain a number of tuples, specifically: Establishing an intelligent agent set according to the topological structure and network parameters of the power system; wherein the intelligent agent set includes a power system stability intelligent agent, an energy optimization scheduling intelligent agent and a fault recovery intelligent agent; Define the tuple of the Markov decision process and get the initial tuple; According to the intelligent agent set and the initial tuple, the plurality of tuples are established by defining the state vectors, action vectors, state transition probabilities and reward functions required by the plurality of objectives.

3. A novel multi-agent adaptive hybrid parallel training method for power system according to claim 2, characterized in that: According to the set of intelligent agents and the initial tuple, the several tuples are established by defining the state vectors, action vectors, state transition probabilities and reward functions required by the several objectives, specifically: Based on the set of agents and the initial ancestor, a first state vector and a first action vector are established according to a system frequency and a load adjustment amount; According to the frequency deviation and system noise, a first state transfer equation and a first reward function are established; A power system stability tuple is formed by the first state vector, the first action vector, the first state transfer equation and the first reward function; According to the output power and load demand of each power generation unit, an energy optimization scheduling tuple and a fault recovery tuple are established; The plurality of tuples are composed of the power system stability tuple, the energy optimization scheduling tuple and the fault recovery tuple.

4. A novel multi-agent adaptive hybrid parallel training method for power system according to claim 3, characterized in that: The first reward function is specifically: r t =-(k f |Δf t |+k v ||ΔV t ||+k s S st ) Among them, k f , k v and k s is the weight coefficient, Δf t is the frequency deviation, ΔV t is the voltage deviation, S st It is an indicator of system stability.

5. A novel multi-agent adaptive hybrid parallel training method for power system according to claim 1, characterized in that: Based on the objective function, agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training, specifically: Based on the physical simulation model, a coupling feature matrix is ​​constructed according to the topological coupling degree between the multiple agents; Calculating a feature vector set according to the coupling feature matrix and the non-coupling features between the multi-agents; Based on the objective function and according to the similarity of features in the feature vector set, agents with high coupling degree are allocated to the same sub-node server or adjacent sub-node servers for training.

6. A novel multi-agent adaptive hybrid parallel training method for power system according to claim 5, characterized in that: Based on the physical simulation model, a coupling feature matrix is ​​constructed according to the topological coupling degree between multiple agents, specifically: Based on the physical simulation model, a topological coupling matrix is ​​established according to the connection status of the state vectors between the multiple agents; Establishing a control variable correlation matrix according to the connection status of the control vectors between the multiple agents; Establishing a resource sharing matrix according to the resource sharing degree among the multiple agents; The topological coupling matrix, the control variable correlation matrix and the resource sharing matrix are weightedly calculated to obtain the coupling characteristic matrix.

7. A novel multi-agent adaptive hybrid parallel training method for power system according to claim 5, characterized in that: Based on the objective function and according to the similarity of the features in the feature vector set, the agents with high coupling degree are assigned to the same sub-node server or adjacent sub-node servers for training, specifically: According to the similarity of the features in the feature vector set, the agents with high coupling degree are allocated to the same server or adjacent sub-node servers; Based on the objective function, controlling the plurality of sub-node servers to train the multi-agent to obtain local model parameters of the plurality of sub-node servers; In the form of parameter interaction, the local model parameters of the first sub-node server in the local model parameters are merged with the local model parameters of the first adjacent sub-node server to obtain the first sub-node server after parameter fusion and update; The sub-node servers in the plurality of sub-node servers are traversed to obtain the plurality of sub-node servers after parameter fusion and update.

Images