Multi-agent deep reinforcement learning-based power grid power failure arrangement system and method, and medium

Abstract

The invention relates to the field of power systems, and discloses a multi-agent deep reinforcement learning-based power grid power failure arrangement system, which comprises an arrangement problem conversion module and a decision solving module, the arrangement problem conversion module is used for converting a power grid power-cut plan arrangement problem into a multi-agent Markov decision process model; and the decision solving module is used for carrying out training solving on the multi-agent Markov decision process model by utilizing a multi-agent depth deterministic strategy gradient algorithm under set relevant constraints of power grid power failure arrangement to obtain a power grid power failure arrangement strategy. The scheme is inspired by a single-agent reinforcement learning thought, a power grid power-cut plan arrangement problem is converted into a multi-agent Markov decision process, power equipment is regarded as an independent agent, equipment with an intelligent power-cut plan arrangement function is trained by adopting a multi-agent depth deterministic strategy gradient algorithm, and the power-cut plan arrangement problem is solved. And an optimal power-cut plan arrangement strategy is learned, so that safe operation of a power grid and equipment thereof is ensured.

Description

Technical Field

[0001] The present invention relates to the field of power systems, and in particular to a power grid power outage scheduling system and method and a medium based on multi-agent deep reinforcement learning. Background Art

[0002] As the scale of the power grid continues to expand, the number of new energy sources such as wind power and photovoltaic power grids continues to increase, and the number of devices in the power grid is gradually increasing, resulting in increasingly complex power grid structures and increasingly complex and changeable power grid operation states, which poses a huge challenge to the safe operation of power grid equipment. In order to achieve the goal of "carbon peak and carbon neutrality" as soon as possible, the proportion of new energy sources such as wind power and photovoltaic power grids connected to the grid is gradually increasing. The uncertainty, volatility and indirectness of new energy output have brought severe challenges to the safe and reliable operation of the power grid. In order to ensure the safe operation of the power grid and the reliability of power supply, the scheduling of power outage maintenance plans for power equipment has become an urgent problem to be solved. In the power outage maintenance plan for power grid equipment, it is necessary to focus on power supply guarantee, that is, the number of power outage devices should be as small as possible to reduce the impact of power outages on users and improve the reliability and continuity of power supply. However, due to the complexity and cumbersomeness of the power outage plan business, it is based on manual experience to schedule, resulting in low scheduling efficiency and easy to be affected by human subjective factors, and the strategy of power supply guarantee is sometimes difficult to implement.

[0003] For solving the power grid outage scheduling model, most of the existing methods are heuristic algorithms based on swarm intelligence optimization, such as particle swarm optimization, dragonfly algorithm and ant colony algorithm. Although these algorithms avoid the emergence of local optimal solutions to a certain extent, they lack continuous interaction with the environment and cannot adapt to the real-time changing power grid environment. At the same time, with the gradual increase in power grid equipment, the power grid outage scheduling model has become more complex, resulting in an increased computational burden on the heuristic algorithm. A large number of iterations are required when finding the optimal solution of the model, resulting in problems such as slow convergence speed and low computational efficiency.

[0004] Power outage scheduling is actually a Markov decision process, and reinforcement learning is good at solving this type of decision-making problem. The basic idea of ​​reinforcement learning is how an agent makes decisions in an uncertain environment to maximize its own rewards. When scheduling power outages based on single-agent reinforcement learning, the power equipment agent is usually regarded as a unified whole. However, due to limitations such as communication burden and control delay, treating the agent as a unified whole can easily lead to problems such as low efficiency and poor scheduling effect of power outage scheduling. Summary of the invention

[0005] The purpose of the present invention is to overcome the lack of continuous interaction with the environment in solving the power grid outage scheduling model of the existing heuristic algorithm, and it is difficult to adapt to the real-time changes of the power grid. At the same time, due to the existence of communication burden and control delay, the power grid outage scheduling based on single-agent reinforcement learning is prone to low scheduling efficiency and poor scheduling effect. The system, method and medium of power grid outage scheduling based on multi-agent deep reinforcement learning are provided. The scheme is inspired by the single-agent reinforcement learning idea, and the power grid outage scheduling problem is converted into a multi-agent Markov decision process. The power equipment, such as: generator sets, lines, and transformers are regarded as separate agents, and the multi-agent deep deterministic policy gradient algorithm is used to train the equipment with intelligent power outage scheduling. The agent interacts continuously with the power grid environment and learns the optimal power outage scheduling strategy, thereby ensuring the safe operation of the power grid and its equipment.

[0006] In order to achieve the above-mentioned object, the present invention provides, on one hand, a multi-agent deep reinforcement learning power grid outage scheduling system, including a scheduling problem conversion module and a decision solving module;

[0007] The scheduling problem conversion module is used to convert the power grid outage scheduling problem into a multi-agent Markov decision process model;

[0008] The decision-solving module is used to train and solve the multi-agent Markov decision process model using a multi-agent deep deterministic policy gradient algorithm under the relevant constraints of the set power grid power outage scheduling, so as to obtain the power grid power outage scheduling strategy.

[0009] Preferably, the process of converting the power outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N,S,a1,a2,…,a N ,r1,r2,…,r N ,F T ,γ}; where N is the number of agents; S is the set of power grid states; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…,r N ] represents the corresponding reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transfer function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' means the agent performs a power outage action A=[a1,a2,…,a N] and then transferred to the grid state set at the next moment; γ represents the discount rate.

[0010] Preferably, the specific operations of converting the power grid outage scheduling problem into a multi-agent Markov decision process model include:

[0011] Determine the objective function F(P) in the power grid outage scheduling problem, expressed as:

[0012]

[0013] C i =Σc t

[0014]

[0015] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ]; p i represents the ith power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, 2, ..., n; T u represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily power outage plans; Indicates the power outage plan i The date on which the execution of Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object on day t;

[0016] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0017] The expression of the state variable is:

[0018]

[0019] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed sign;

[0020] The state set of the generator set includes:

[0021]

[0022] Where P v (t) represents the daily average active output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t);

[0023] The state set of a line includes:

[0024]

[0025] Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t);

[0026] The state set of a transformer includes:

[0027]

[0028] In the formula, R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v(t) represents the average load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t);

[0029] The expression of the action variable is:

[0030]

[0031] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, represents the transposed matrix of D,v(t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t);

[0032] The expression of the reward function is:

[0033]

[0034] Where r represents the reward function; ω1 represents the weight coefficient of the total outage quantity; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage quantity.

[0035] Preferably, the set constraints related to the power grid blackout scheduling include safety constraints and rationality constraints;

[0036] The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line flow constraints, transformer voltage ratio constraints, winding temperature constraints and load rate constraints;

[0037] The expression of power balance constraint is:

[0038]

[0039] Where D(t) represents the daily average load demand of the system on day t; N G Indicates the number of generator sets;

[0040] The expression of unit output constraint is:

[0041]

[0042] In the formula, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; It represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day;

[0043] The expression of node voltage constraint is:

[0044]

[0045] In the formula, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day;

[0046] The expression of line power flow constraint is:

[0047]

[0048] In the formula, It represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day;

[0049] The expression of transformer voltage ratio constraint is:

[0050]

[0051] In the formula, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day;

[0052] The expression of winding temperature constraint is:

[0053]

[0054] In the formula, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day;

[0055] The expression of load rate constraint is:

[0056]

[0057] In the formula, It represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0058] Preferably, the rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints;

[0059] The expression of the unchangeable plan constraint is:

[0060]

[0061] In the formula, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an irreversible planned power outage; Indicates the start time of the power outage of the irreversible plan; Indicates the end time of the power outage of the irreversible plan;

[0062] The expression of the mutual exclusion constraint is:

[0063] z p (t)·z q (t) = 0

[0064] In the formula, z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t);

[0065] The expression of the same-stop relation constraint is:

[0066] z h (t)·z l (t) = 1

[0067] In the formula, z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h(t)∈z i (t), z l (t)∈z i (t);

[0068] The expression of maintenance decision variable constraint is:

[0069] 1≤X v ≤N dm And X v ∈N dm

[0070] Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed;

[0071] The expression of maintenance human resource constraint is:

[0072]

[0073] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0074] The expression for overhauling the special event constraint is:

[0075]

[0076] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0077] Preferably, under the relevant constraints of the set power grid outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include:

[0078] Determine the policy gradient of the kth agent, expressed as:

[0079]

[0080] In the formula, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ krepresents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, …, a N ]~B indicates power failure action [a1,a2,…,a N ] obeys the distribution of B;

[0081] Define the loss function of the evaluation network, the expression is:

[0082]

[0083] In the formula, L(θ k ) is the loss function of the evaluation network of the kth agent; Indicates k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′ k (s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network;

[0084] The parameters of the target network are updated using soft update, and the expression is:

[0085]

[0086] θ' k =τθ k +(1-τ)θ' k

[0087] In the formula, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0088] A second aspect of the present invention provides a multi-agent deep reinforcement learning power grid outage scheduling method, comprising:

[0089] The power grid outage scheduling problem is transformed into a multi-agent Markov decision process model;

[0090] Under the relevant constraints of the set power outage scheduling, the multi-agent deep deterministic policy gradient algorithm is used to train and solve the multi-agent Markov decision process model to obtain the power outage scheduling strategy.

[0091] Preferably, the process of converting the power grid outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N is the number of agents; S is the set of power grid states; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…r N ] represents the corresponding reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transfer function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' means the agent performs a power outage action A=[a1,a2,…,a N ] and then transferred to the grid state set at the next moment; γ represents the discount rate.

[0092] Preferably, the specific operations of converting the power grid outage scheduling problem into a multi-agent Markov decision process model include:

[0093] Determine the objective function F(P) in the power grid outage scheduling problem, expressed as:

[0094]

[0095] C i =∑c t

[0096]

[0097] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ]; p i represents the ith power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, 2, ..., n; T u represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily power outage plans; Indicates the power outage plan p i The date on which the execution of Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object on day t;

[0098] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0099] The expression of the state variable is:

[0100]

[0101] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v(t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed sign;

[0102] The state set of the generator set includes:

[0103]

[0104] Where P v (T) represents the daily average active output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t);

[0105] The state set of a line includes:

[0106]

[0107] Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t);

[0108] The state set of a transformer includes:

[0109]

[0110] In the formula, R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t);

[0111] The expression of the action variable is:

[0112]

[0113] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t);

[0114] The expression of the reward function is:

[0115]

[0116] Where r represents the reward function; ω1 represents the weight coefficient of the total outage quantity; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage quantity.

[0117] Preferably, the set constraints related to the power grid blackout scheduling include safety constraints and rationality constraints;

[0118] The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line flow constraints, transformer voltage ratio constraints, winding temperature constraints and load rate constraints;

[0119] The expression of power balance constraint is:

[0120]

[0121] Where D(t) represents the daily average load demand of the system on day t; N GIndicates the number of generator sets;

[0122] The expression of unit output constraint is:

[0123]

[0124] In the formula, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; It represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day;

[0125] The expression of the node voltage constraint is:

[0126]

[0127] In the formula, It represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day;

[0128] The expression of line power flow constraint is:

[0129]

[0130] In the formula, It represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day;

[0131] The expression of transformer voltage ratio constraint is:

[0132]

[0133] In the formula, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day;

[0134] The expression of winding temperature constraint is:

[0135]

[0136] In the formula, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day;

[0137] The expression of load rate constraint is:

[0138]

[0139] In the formula, It represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0140] Preferably, the rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints;

[0141] The expression of the unchangeable plan constraint is:

[0142]

[0143] In the formula, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an irreversible planned power outage; Indicates the start time of the power outage of the irreversible plan; Indicates the end time of the power outage of the irreversible plan;

[0144] The expression of the mutual exclusion constraint is:

[0145] z p (t)·z q (t) = 0

[0146] In the formula, z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t);

[0147] The expression of the same-stop relation constraint is:

[0148] z h (t)·z l (t) = 1

[0149] In the formula, z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t);

[0150] The expression of maintenance decision variable constraint is:

[0151] 1≤X v ≤N dm And X v ∈N dm

[0152] In the formula, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed;

[0153] The expression of maintenance human resource constraint is:

[0154]

[0155] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0156] The expression for overhauling the special event constraint is:

[0157]

[0158] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0159] Preferably, under the relevant constraints of the set power grid outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include:

[0160] Determine the policy gradient of the kth agent, expressed as:

[0161]

[0162] In the formula, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s krepresents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, …, a N ]~B indicates power failure action [a1,a2,…,a N ] obeys the distribution of B;

[0163] Define the loss function of the evaluation network, the expression is:

[0164]

[0165] In the formula, L(θ k ) is the loss function of the evaluation network of the kth agent; Indicates k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′ k (s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network;

[0166] The parameters of the target network are updated using soft update, and the expression is:

[0167]

[0168] θ' k =τθ k +(1-τ)θ' k

[0169] In the formula, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0170] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0171] Compared with the prior art, the present invention mainly realizes the task of power grid outage scheduling by adjusting the number and duration of power outages of the power equipment intelligent body to ensure the safe and stable operation of the power grid. First, the power grid outage scheduling problem is transformed into a multi-agent Markov decision process model, which includes power grid state, power outage action, reward, joint state transfer function and discount rate; at the same time, in order to ensure the safety of power grid operation and the rationality of power outage scheduling, relevant constraints are introduced; further, the multi-agent deep deterministic policy gradient algorithm is used to solve the Markov decision model constructed above. After solving, the power outage time and number of power outages of generator sets, lines and transformers can be obtained. Each intelligent agent generates a power outage scheduling strategy by sensing the power grid environment and executes corresponding actions. The fully trained intelligent agent can make the optimal action strategy (i.e., the number and duration of power outages of the intelligent agent) in the current environment, so as to obtain the maximum reward value and improve the efficiency of power grid outage scheduling.

[0172] In the present invention, each intelligent agent is in a completely cooperative relationship. Although the actions of each intelligent agent only act on its own environment, all intelligent agents share the same reward function. The intelligent agents cooperate with each other, and the MADDPG algorithm is used to solve the power outage time and number of power outages of each intelligent agent, realizing the power grid power outage planning. Using this method, intelligent power grid power outage planning can be realized, which effectively reduces the occurrence of unexpected failures, helps to timely discover abnormalities of power equipment, prolong the service life of equipment, and ensure the safe and stable operation of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Figure 1 It is a structural schematic diagram of the power grid outage scheduling system of the multi-agent deep reinforcement learning of the present invention;

[0174] Figure 2It is a flowchart of the multi-agent deep deterministic policy gradient algorithm;

[0175] Figure 3 It is a flow chart of the power grid power outage scheduling method based on multi-agent deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0176] The specific implementation of the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific implementation described here is only used to illustrate and explain the present invention, and is not used to limit the present invention.

[0177] Example 1

[0178] like Figure 1 The multi-agent deep reinforcement learning power grid outage scheduling system shown includes a scheduling problem conversion module and a decision solving module;

[0179] The scheduling problem conversion module is used to convert the power grid outage scheduling problem into a multi-agent Markov decision process model;

[0180] The decision-solving module is used to train and solve the multi-agent Markov decision process model under the relevant constraints of the set power grid outage scheduling using the multi-agent deep deterministic policy gradient algorithm to obtain the power grid outage scheduling strategy.

[0181] Since the power outage scheduling strategy of power equipment directly affects the reliability and economy of the power system, the present invention realizes the safe and stable operation of the power grid by scheduling the power outage of power equipment, wherein the power outage scheduling refers to the reasonable arrangement of the power outage sequence and time of power equipment under certain constraints, so that the power system can minimize the impact on system reliability, economy and safety when undergoing maintenance, overhaul or expansion.

[0182] Among them, in the problem of power grid outage planning, each generator set, each line and each transformer in the power grid are regarded as their own separate intelligent agent, that is, there will be multiple intelligent agents in the power grid.

[0183] In the preferred case, the process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N is the number of agents; S is the set of power grid states; [a1,a2,…,a N ] represents the power outage action set of all agents, where a1 represents the power outage action set of the first agent, a2 represents the power outage action set of the second agent, ..., a Nrepresents the set of all power outage actions of the Nth agent (usually the set of all power outage actions of an agent will include multiple power outage actions); [r1, r2, … r N ] represents the corresponding reward value set obtained by all agents after executing the power outage action, r1 represents the reward value obtained by the first agent after executing the power outage action a1, r2 represents the reward value obtained by the second agent after executing the power outage action a2, …, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transfer function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' means the agent performs a power outage action A=[a1,a2,…,a N ] and then transfer to the grid state set at the next moment; γ represents the discount rate, which indicates the attention paid to future rewards. The larger γ is, the more attention is paid to future rewards (rewards include current rewards and future rewards: current rewards refer to the rewards that the power grid environment will give back to the agent after the agent acts on the power grid environment; future rewards refer to the rewards that the agent may get in the future (the rewards that the agent may get in the future depend on the impact of its current actions on the environment and the dynamic evolution characteristics of the environment. The actions of the agent will affect the entire future reward sequence, not the immediate reward), but this reward is not certain whether it is good or bad. The purpose of introducing the discount rate is to indicate whether the agent pays more attention to current rewards or has higher expectations for future rewards; in specific cases, rewards refer to the negative value of the objective function, including power outage frequency, economic losses caused by power outages, and power outage balance. The agent will be rewarded after executing the power outage plan action).

[0184] In the present invention, the grid state is composed of the states of the intelligent entities in the grid; the state refers to the output of the generator set, the operating temperature of the line or the transformation ratio of the transformer, etc.; the power outage action refers to the power outage duration and the number of power outages, for example, one generator set needs to be shut down for 30 minutes, etc.

[0185] The state transfer function in the present invention refers to the intelligent agent performing a power outage action A=[a1, a2, ..., a N ] is the probability distribution of the power grid state to be transferred to the next moment after the state transfer; the state transfer refers to the generator set, line and transformer (i.e., intelligent agent), according to the current environmental state (i.e., the current power grid state), according to the power outage scheduling strategy to adjust its own power outage duration and number, which will change the power flow distribution of the power grid, and the daily average output of the unit, the daily average voltage of the node and the daily average transformation ratio of the transformer will be transferred to the state of the next moment.

[0186] Preferably, the specific operations of converting the power grid outage scheduling problem into a multi-agent Markov decision process model include:

[0187] Determine the objective function F(P) in the power grid outage scheduling problem (in the power grid outage scheduling, power supply guarantee is the primary consideration, that is, the fewer the power outage devices, the better, so as to reduce the impact of power outages on users and improve the reliability and continuity of power supply; at the same time, the economic cost of power outages should be reduced as much as possible; finally, in order to improve the stability of the power grid and avoid the risks of power outages to the operation of the power grid, the power outage plans for each period should be as balanced as possible; the present invention takes the above three aspects into consideration and determines the objective function in the power grid outage scheduling problem based on this), the expression is:

[0188]

[0189] C i =∑c t

[0190]

[0191] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ]; p i represents the ith power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, ..., n, that is, all outage plans {p1, ..., p n There are n total (these power outage plans are pending power outage plans that have not yet been implemented); T u represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period (because power outages will affect the daily power consumption of electricity users, which will cause economic losses to users; in addition, when the equipment is out of power, it will be repaired and maintained, which will also generate certain economic consumption. Therefore, power outages will cause certain economic losses); c t represents the economic cost of each day in the power outage cycle (for example, a power outage plan p i The daily economic loss during the power outage period is {c1, c2, ...}. The power outage period is a period of time, such as 1 month. In this month, there will be an economic loss of c every day within 30 days. t , t=1,2,…,30, by multiple c tThe total economic loss during the power outage period (economic loss refers to economic cost); T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the daily power outage plan number, which is used to measure the balance of power outages within the time period; Indicates the power outage plan p i The date on which the execution of Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object (i.e., agent) on day t.

[0192] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0193] The expression of the state variable is:

[0194]

[0195] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v (t) is the transposed matrix; T is the sign of the transposed matrix.

[0196] The state set of the generator set includes:

[0197]

[0198] Where P v (t) represents the daily average active output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t).

[0199] The state set of a line includes:

[0200]

[0201] Where: Vv (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the average daily power transmission of the vth line on the tth day (i.e., the average daily power transmission of the vth line on the tth day, i.e., the current on the transmission line), Indicates F v The transposed matrix of (t).

[0202] The state set of a transformer includes:

[0203]

[0204] In the formula, R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average load rate of the vth transformer on the tth day (the ratio of the current load to the rated capacity), Indicates H v The transposed matrix of (t).

[0205] The expression of the action variable is:

[0206]

[0207] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, S GE,v (t)=1 means the vth generator set is out of power on the tth day, otherwise SCE,v(t)=0, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, S LI,v (t) = 1 means that the vth line is out of power on the tth day, otherwise S LI,v(t) = 0, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, S TR,v (t) = 1 means that the vth transformer is out of power on the tth day, otherwise S TR,v (t) = 0, Indicates S TR,v The transposed matrix of (t).

[0208] The expression of the reward function is:

[0209]

[0210] Where r represents the reward function (the reward value is calculated based on the reward function); ω1 represents the weight coefficient of the total outage power; ω2 represents the weight coefficient of the economic cost of the outage; ω3 represents the weight coefficient of the balance degree of the outage power.

[0211] In the present invention, each intelligent agent is in a fully cooperative relationship, and the number and duration of power outages of each intelligent agent are determined through mutual coordination to ensure that the objective function is minimized, so the rewards are shared among the intelligent agents.

[0212] In the problem of power grid outage scheduling, the safety of power grid operation and the rationality of power outage scheduling must be considered, thereby introducing relevant constraints. In the optimal case, the relevant constraints set for power grid outage scheduling include safety constraints and rationality constraints.

[0213] The role of the relevant constraints of power grid power outage scheduling is mainly to constrain the intelligent agent. The safety constraints are used to ensure the safety of the intelligent agent's work, such as the output of the generator set cannot be too high, the line operating temperature cannot be too high, etc.; the rationality constraints are mainly the rationality of the intelligent agent's power outage, such as two generator sets cannot be shut down at the same time, etc. By introducing constraints, the safety of the intelligent agent and the rationality of its actions can be guaranteed.

[0214] Furthermore, the safety constraints include power balance constraints, unit output constraints, node voltage constraints, line flow constraints, transformer voltage ratio constraints, winding temperature constraints, and load factor constraints.

[0215] The expression of power balance constraint is:

[0216]

[0217] Where D(t) represents the daily average load demand of the system on day t; NG Indicates the number of generator sets;

[0218] The expression of unit output constraint is:

[0219]

[0220] In the formula, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; It represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day.

[0221] The expression of node voltage constraint is:

[0222]

[0223] In the formula, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day.

[0224] The expression of line power flow constraint is:

[0225]

[0226] In the formula, It represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day.

[0227] The expression of transformer voltage ratio constraint is:

[0228]

[0229] In the formula, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day.

[0230] The expression of winding temperature constraint is:

[0231]

[0232] In the formula, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day.

[0233] The expression of load rate constraint is:

[0234]

[0235] In the formula, It represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0236] In a preferred embodiment of the present invention, rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, co-stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints.

[0237] The unchangeable plan refers to the situation that when it comes to major events and power supply needs to be guaranteed, the relevant power outage plan cannot be changed. The time of such power outage plan will be directly determined according to the subjective will of the scheduler. Usually, the unchangeable plan will be included in the power outage plan P.

[0238] Furthermore, the expression of the unchangeable plan constraint is:

[0239]

[0240] In the formula, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an irreversible planned power outage; Indicates the start time of the power outage of the irreversible plan; Indicates the end time of the power outage for this irreversible plan.

[0241] To avoid islanding in the power grid system, the planned combination of characteristics cannot be executed on the same day. p and p q All of them belong to the power outage plan P, where the corresponding power outage objects on day t are z p (t), z q (t), then

[0242] The expression of the mutual exclusion constraint is:

[0243] z p (t)·z q (t) = 0

[0244] In the formula, z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day (z p (t) and z q(t) represents two different power outage objects, that is, to indicate that the two power outage objects cannot be shut down at the same time, for example, generator set 1 and line 2 cannot be shut down at the same time, and transformer 1 and transformer 2 cannot be shut down at the same time, etc.), z p (t)∈z i (t), z q (t)∈z i (t).

[0245] According to the specified power outage plan association relationship, the execution time windows of the power outage plans that need to be executed simultaneously should overlap as much as possible. h 、p l The same power outage plan P, where the corresponding power outage object on day t is z h (t), z l (t), then the expression of the same-stop relation constraint is:

[0246] z h (t)·z l (t) = 1

[0247] In the formula, z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t), the same z h (t) and z l (t) refers to two different power outage objects, which need to be shut down for maintenance at the same time, such as generator set 1 and transformer 1 need to be shut down for maintenance at the same time, line 2 and line 3 need to be shut down for maintenance at the same time, and so on.

[0248] The above mutually exclusive relationship constraints and simultaneous shutdown relationship constraints are two different constraints. Since there are many power equipment, some equipment cannot be shut down at the same time, while some equipment can be shut down for maintenance at the same time, so relevant constraints are given.

[0249] In order to reduce the number of power outages and improve user satisfaction, the equipment power outage duration must not exceed the maximum number of days in the month, so the expression of the maintenance decision variable constraint is:

[0250] 1≤X v ≤N dm And X v ∈N dm

[0251] In the formula, X v is the power outage duration of the vth device (i.e., the power outage object); N dmIndicates the number of days in the month to be analyzed (the month with a planned power outage).

[0252] Since the number of maintenance personnel is limited, the number of maintenance of the same type of equipment on the same day is limited, and human resource constraints need to be considered in the power outage plan decision. Therefore, the expression of maintenance human resource constraints is:

[0253]

[0254] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0255] Considering that the occurrence of special events (bad weather, special holidays) will result in a period of time during the planning cycle where power outages for maintenance are not allowed, in order to improve the feasibility of the power outage plan, the expression of the maintenance special event constraint is:

[0256]

[0257] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0258] In this invention, the multi-agent deep deterministic policy gradient (MADDPG) algorithm used is an extension of the deep deterministic policy gradient (DDPG) algorithm in single-agent deep reinforcement learning in a multi-agent environment. The MADDPG algorithm adopts a "centralized training, decentralized execution" framework and can adapt to complex multi-agent environments.

[0259] The MADDPG algorithm is based on the Actor-Critic framework. In the MADDPG algorithm, each agent has its own strategy (Actor) network and evaluation (Critic) network. The Actor network is used to select power outage actions, and the Critic network is used to evaluate the value of the selected power outage actions.

[0260] The current state of the power grid at time t is s tThe input is sent to the Actor network of each agent. The Actor network will explore and obtain the power outage action of the agent. After the power outage action acts on the power grid environment, a new power grid state and the current reward will be obtained (after the power outage action of the agent acts on the power grid environment, the environmental state (power grid state) will definitely change, so the new environmental state will be fed back to the agent, and the reward will also be fed back to the agent. The agent can judge the quality of its own power outage action based on the quality of the reward). The state, power outage action, new state and reward of the agent are stored as experience in its own experience replay pool; therefore, when using the MADDPG algorithm to train and solve the multi-agent Markov decision process model, the current power grid state s at time t is used. t Input into an agent to obtain the power outage action a at the current time t t (i.e., the duration and number of power outages of the agent), execute the power outage action a at the current time t t After that, the power grid status changes from s t Transfer to s', that is, the power grid state is transferred from the current state at time t to the state at the next time (time t+1), and the reward r of the feedback under the current environment at time t is obtained t , and feed back the grid state transfer and reward to the agent. The agent continuously interacts with the grid environment and t ,a t ,r t ,s'} is stored in its own experience replay pool (in the MADDPG algorithm, the number of agents is N, and each agent has its own experience replay pool, so N agents correspond to N experience replay pools, and each experience replay pool will also store the corresponding agent's {s t ,a t ,r t ,s'} data), before updating the parameters in the network each time, a set of {s t ,a t ,r t ,s'} data to train the neural network of the agent (the policy network and the evaluation network are composed of neural networks. Neural networks refer to fully connected neural networks in deep learning. When training the neural network of each agent, a set of {s t ,a t ,r t ,s'} data for training), and the new {s t ,a t ,r t,s'} back into the experience replay pool, and then randomly selected for training), until the reward fed back in the current power grid environment is high and tends to be stable, stop training (in the process of training all agents, if an agent reaches the optimal state, you can stop training it, and other agents continue training. In the centralized training stage, if an agent stops training, you can freeze the strategy of the agent to avoid affecting other agents. The present invention takes the overall reward optimization as the optimization goal, and all agents cooperate with each other. When the overall reward (the sum of the rewards of each agent) of each centralized training is large and will not change significantly, the reward at this time can be considered to be optimal and stable. At this time, stop training all agents and the training ends). After the training, the strategy network of each agent will output an action value, which is the optimal strategy (these action values ​​together constitute the power grid power outage scheduling strategy). This action value will be sent to the agent to let the agent perform the action, such as stopping 5 generators, each for 30 minutes.

[0261] The multi-agent deep deterministic policy gradient algorithm process in the present invention is as follows: Figure 2 shown.

[0262] The ultimate goal of deep reinforcement learning in this invention is to find the final joint control strategy to maximize the cumulative reward.

[0263] In the MADDPG algorithm, there are N agents, so it contains N Actor networks and N Critic networks, where the parameter set of the N Actor networks is θ = {θ1, θ2, …, θ N}, θ represents the set of policy network parameters; θ1 represents the parameters of the policy network of the first agent, θ2 represents the parameters of the policy network of the second agent, ..., θ N represents the parameters of the policy network of the Nth agent. During the deep neural network training process, the general policy gradient of the kth agent can be expressed as (the following formula is applicable to random strategies):

[0264]

[0265] In the formula, represents the policy gradient of the kth agent; logμ k Represents the strategy μ k Take the logarithm calculation; Q k (s k ,a k1 ,a k2 ,…,a km) is the kth centralized value function (centralization means that this value function depends not only on the local information of a certain agent, but also on the state and action information of all agents), and its input is the action taken by the agent based on the policy (Actor) network (the action taken by the policy network is the power outage action) a k and the current status k , whose output is the Q value of the kth agent;

[0266] In addition to the Actor network and the Critic network, the intelligent agent also has a target-Actor network (the strategy (Actor) network of the target network) and a target-Critic network. k 'Input into the target-Actor network of the current agent k to get action a k '(s k ' represents the state of the kth agent at the next moment. The kth agent is based on the current state s k , execute power outage action a k After that, the state will be changed from s k Transformed into k '), and S'=[s'1,s'2,…,s' N ](s'1 represents the state of the first agent at the next moment. After the first agent performs the power outage action a1 according to the current state s1, the state will change from s1 to s1'; s'2 represents the state of the second agent at the next moment. After the second agent performs the power outage action a2 according to the current state s2, the state will change from s2 to s2', ..., s' N Indicates the state of the Nth agent at the next moment. The Nth agent is based on the current state s N , execute power outage action a N After that, the state will be changed from s N Transformed into N ') and A'=[a'1,a'2,...,a' N ] is input into the target-Critic network to obtain the Q at the next moment k ';A' represents the set of power outage actions of all agents at the next moment, a'1 represents the set of all power outage actions of the first agent at the next moment, a'2 represents the set of all power outage actions of the second agent at the next moment, ..., a'B represents the set of all power outage actions of the Nth agent at the next moment. After the agent executes the power outage action A according to the current power grid state S, the state will transfer from S to S', and the power outage action A' will be executed in state S';Q k' represents the evaluation of the target-critic network on the power outage action A'. Because the power outage action A is performed under the power grid state S, the value Q of the power outage action A is calculated by the target-critic network. Then, at the next moment of the power grid state S', the evaluation of the next moment of the power outage action A' is Q k ', Q k ' refers to the action value of the kth agent in the grid state set S' and the power outage action set A'.

[0267] Based on this, under the relevant constraints of the set power grid outage scheduling, the specific operations of using the multi-agent deep deterministic policy gradient algorithm to train and solve the multi-agent Markov decision process model include:

[0268] Determine the policy gradient of the kth agent, expressed as (in the present invention, since the MADDPG algorithm uses the stochastic gradient descent method to update the Actor network of each agent, a deterministic strategy is used, so the actual calculation uses the following formula):

[0269]

[0270] In the formula, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent, a k There are m power outage actions in total; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, …, a N ]~B indicates power failure action [a1,a2,…,a N ] obeys the distribution of B (the experience replay pool contains strategies).

[0271] Define the loss function of the evaluation network, the expression is:

[0272]

[0273] In the formula, L(θ k ) is the loss function of the evaluation network of the kth agent; Indicates k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′k(s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy (μ′) of the target-strategy network k represents the policy of the goal-policy network of the kth agent).

[0274] The parameters of the target network are updated using soft update, and the expression is:

[0275]

[0276] θ' k =τθ k +(1-τ)θ' k

[0277] In the formula, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0278] The MADDPG algorithm stabilizes the training and solution of the Actor network and the Critic network by introducing the target network. The Actor network representing the target network, The Actor network representing the target network of the first agent, Actor network representing the target network of the second agent, …, represents the Actor network of the target network of the Nth agent, θ'1 represents the Actor network parameters of the target network of the first agent, θ'2 represents the Actor network parameters of the target network of the second agent, ..., θ' N represents the Actor network parameters of the target network of the Nth agent, and the Actor network parameters of the target network of the kth agent are θ' k The parameters of the critic network are continuously updated during training, and the target network does not participate in the training process.

[0279] In the present invention, each agent is in a fully cooperative relationship, and the agents minimize the objective function through mutual coordination, and each agent shares a global reward function. Therefore, the actions of all agents will affect the overall reward function, and the global reward function will guide the strategy optimization of each agent.

[0280] After training and solving with the MADDPG algorithm, the strategy network of each agent will obtain the corresponding power outage action value (obtained by the output of the Actor network). The power grid power outage scheduling strategy is the combination of these power outage action values, and these power outage action values ​​are sent to the agent. The agent then executes the corresponding power outage action. By adjusting the power outage duration and number of power outages of generator sets, lines and transformers, the power grid power outage plan is scheduled, ensuring the safe and stable operation of the power grid and its equipment, and also helping to extend the service life of the equipment.

[0281] Example 2

[0282] like Figure 3 The multi-agent deep reinforcement learning power grid outage scheduling method shown includes:

[0283] The power grid outage scheduling problem is transformed into a multi-agent Markov decision process model;

[0284] Under the relevant constraints of the set power outage scheduling, the multi-agent deep deterministic policy gradient algorithm is used to train and solve the multi-agent Markov decision process model to obtain the power outage scheduling strategy.

[0285] Since the power outage scheduling strategy of power equipment directly affects the reliability and economy of the power system, the present invention realizes the safe and stable operation of the power grid by scheduling the power outage of power equipment, wherein the power outage scheduling refers to the reasonable arrangement of the power outage sequence and time of power equipment under certain constraints, so that the power system can minimize the impact on system reliability, economy and safety when undergoing maintenance, overhaul or expansion.

[0286] Among them, in the problem of power grid outage planning, each generator set, each line and each transformer in the power grid are regarded as their own separate intelligent agent, that is, there will be multiple intelligent agents in the power grid.

[0287] In the preferred case, the process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N is the number of agents; S is the set of power grid states; [a1,a2,…,a N ] represents the power outage action set of all agents, where a1 represents the power outage action set of the first agent, a2 represents the power outage action set of the second agent, ..., a N represents the set of all power outage actions of the Nth agent (usually the set of all power outage actions of an agent will include multiple power outage actions); [r1, r2, … r N ] represents the corresponding reward value set obtained by all agents after executing the power outage action, r1 represents the reward value obtained by the first agent after executing the power outage action a1, r2 represents the reward value obtained by the second agent after executing the power outage action a2, …, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transfer function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' means the agent performs a power outage action A=[a1,a2,…,a N] and then transfer to the grid state set at the next moment; γ represents the discount rate, which indicates the attention paid to future rewards. The larger γ is, the more attention is paid to future rewards (rewards include current rewards and future rewards: current rewards refer to the rewards that the power grid environment will give back to the agent after the agent acts on the power grid environment; future rewards refer to the rewards that the agent may get in the future (the rewards that the agent may get in the future depend on the impact of its current actions on the environment and the dynamic evolution characteristics of the environment. The actions of the agent will affect the entire future reward sequence, not the immediate reward), but this reward is not certain whether it is good or bad. The purpose of introducing the discount rate is to indicate whether the agent pays more attention to current rewards or has higher expectations for future rewards; in specific cases, rewards refer to the negative value of the objective function, including power outage frequency, economic losses caused by power outages, and power outage balance. The agent will be rewarded after executing the power outage plan action).

[0288] In the present invention, the grid state is composed of the states of the intelligent entities in the grid; the state refers to the output of the generator set, the operating temperature of the line or the transformation ratio of the transformer, etc.; the power outage action refers to the power outage duration and the number of power outages, for example, one generator set needs to be shut down for 30 minutes, etc.

[0289] The state transfer function in the present invention refers to the intelligent agent performing a power outage action A=[a1, a2, ..., a N ] is the probability distribution of the power grid state to be transferred to the next moment after the state transfer; the state transfer refers to the generator set, line and transformer (i.e., intelligent agent), according to the current environmental state (i.e., the current power grid state), according to the power outage scheduling strategy to adjust its own power outage duration and number, which will change the power flow distribution of the power grid, and the daily average output of the unit, the daily average voltage of the node and the daily average transformation ratio of the transformer will be transferred to the state of the next moment.

[0290] Preferably, the specific operations of converting the power grid outage scheduling problem into a multi-agent Markov decision process model include:

[0291] Determine the objective function F(P) in the power grid outage scheduling problem (in the power grid outage scheduling, power supply guarantee is the primary consideration, that is, the fewer the power outage devices, the better, so as to reduce the impact of power outages on users and improve the reliability and continuity of power supply; at the same time, the economic cost of power outages should be reduced as much as possible; finally, in order to improve the stability of the power grid and avoid the risks of power outages to the operation of the power grid, the power outage plans for each period should be as balanced as possible; the present invention takes the above three aspects into consideration and determines the objective function in the power grid outage scheduling problem based on this), the expression is:

[0292]

[0293] C i=∑c t

[0294]

[0295] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ]; p i represents the ith power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, ..., n, that is, all outage plans {p1, ..., p n There are n total (these power outage plans are pending power outage plans that have not yet been implemented); T u represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period (because power outages will affect the daily power consumption of electricity users, which will cause economic losses to users; in addition, when the equipment is out of power, it will be repaired and maintained, which will also generate certain economic consumption. Therefore, power outages will cause certain economic losses); c t represents the economic cost of each day in the power outage cycle (for example, a power outage plan p i The daily economic loss during the power outage period is {c1, c2, ...}. The power outage period is a period of time, such as 1 month. In this month, there will be an economic loss of c every day within 30 days. t , t=1,2,…,30, by multiple c t The total economic loss during the power outage period (economic loss refers to economic cost); T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the daily power outage plan number, which is used to measure the balance of power outages within the time period; Indicates the power outage plan p i The date on which the execution of Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object (i.e., agent) on day t.

[0296] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0297] The expression of the state variable is:

[0298]

[0299] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v (t) is the transposed matrix; T is the sign of the transposed matrix.

[0300] The state set of the generator set includes:

[0301]

[0302] Where P v (t) represents the daily average active output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t).

[0303] The state set of a line includes:

[0304]

[0305] Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the average daily power transmission of the vth line on the tth day (i.e., the average daily power transmission of the vth line on the tth day, i.e., the current on the transmission line), Indicates F v The transposed matrix of (t).

[0306] The state set of a transformer includes:

[0307]

[0308] In the formula, R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents Rv The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average load rate of the vth transformer on the tth day (the ratio of the current load to the rated capacity), Indicates H v The transposed matrix of (t).

[0309] The expression of the action variable is:

[0310]

[0311] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, S GE,v (t) = 1 means that the vth generator set is out of power on the tth day, otherwise S GE,v (t) = 0, Indicates S GE, The transposed matrix of v(t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, S LI,v (t) = 1 means that the vth line is out of power on the tth day, otherwise S LI,v (t) = 0, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, S TR,v (T) = 1 means that the vth transformer is out of power on the tth day, otherwise S TR,v (t) = 0, Indicates S TR,v The transposed matrix of (t).

[0312] The expression of the reward function is:

[0313]

[0314] Where r represents the reward function (the reward value is calculated based on the reward function); ω1 represents the weight coefficient of the total outage power; ω2 represents the weight coefficient of the economic cost of the outage; and ω3 represents the weight coefficient of the balance degree of the outage power.

[0315] In the present invention, each intelligent agent is in a fully cooperative relationship, and the number and duration of power outages of each intelligent agent are determined through mutual coordination to ensure that the objective function is minimized, so the rewards are shared among the intelligent agents.

[0316] In the problem of power grid outage scheduling, the safety of power grid operation and the rationality of power outage scheduling must be considered, thereby introducing relevant constraints. In the optimal case, the relevant constraints set for power grid outage scheduling include safety constraints and rationality constraints.

[0317] The role of the relevant constraints of power grid power outage scheduling is mainly to constrain the intelligent agent. The safety constraints are used to ensure the safety of the intelligent agent's work, such as the output of the generator set cannot be too high, the line operating temperature cannot be too high, etc.; the rationality constraints are mainly the rationality of the intelligent agent's power outage, such as two generator sets cannot be shut down at the same time, etc. By introducing constraints, the safety of the intelligent agent and the rationality of its actions can be guaranteed.

[0318] Furthermore, the safety constraints include power balance constraints, unit output constraints, node voltage constraints, line flow constraints, transformer voltage ratio constraints, winding temperature constraints, and load factor constraints.

[0319] The expression of power balance constraint is:

[0320]

[0321] Where D(t) represents the daily average load demand of the system on day t; N G Indicates the number of generator sets;

[0322] The expression of unit output constraint is:

[0323]

[0324] In the formula, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; It represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day.

[0325] The expression of node voltage constraint is:

[0326]

[0327] In the formula, It represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day.

[0328] The expression of line power flow constraint is:

[0329]

[0330] In the formula, It represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day.

[0331] The expression of transformer voltage ratio constraint is:

[0332]

[0333] In the formula, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day.

[0334] The expression of winding temperature constraint is:

[0335]

[0336] In the formula, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day.

[0337] The expression of load rate constraint is:

[0338]

[0339] In the formula, It represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0340] In a preferred embodiment of the present invention, rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, co-stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints.

[0341] The unchangeable plan refers to the situation that when it comes to major events and power supply needs to be guaranteed, the relevant power outage plan cannot be changed. The time of such power outage plan will be directly determined according to the subjective will of the scheduler. Usually, the unchangeable plan will be included in the power outage plan P.

[0342] Furthermore, the expression of the unchangeable plan constraint is:

[0343]

[0344] In the formula, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an irreversible planned power outage; Indicates the start time of the power outage of the irreversible plan; Indicates the end time of the power outage for this irreversible plan.

[0345] To avoid islanding in the power grid system, the planned combination of characteristics cannot be executed on the same day. p and p q All of them belong to the power outage plan P, where the corresponding power outage objects on day t are z p (t), z q (t), then

[0346] The expression of the mutual exclusion constraint is:

[0347] z p (t)·z q (t) = 0

[0348] In the formula, z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day (z p (t) and z q (t) represents two different power outage objects, that is, to indicate that the two power outage objects cannot be shut down at the same time, for example, generator set 1 and line 2 cannot be shut down at the same time, and transformer 1 and transformer 2 cannot be shut down at the same time, etc.), z p (t)∈z i (t), z q (t)∈z i (t).

[0349] According to the specified power outage plan association relationship, the execution time windows of the power outage plans that need to be executed simultaneously should overlap as much as possible. h 、p l The same power outage plan P, where the corresponding power outage object on day t is z h(t), z l (t), then the expression of the same-stop relation constraint is:

[0350] z h (t)·z l (t) = 1

[0351] In the formula, z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t), the same as z h (t) and z l (t) refers to two different power outage objects, which need to be shut down for maintenance at the same time, such as generator set 1 and transformer 1 need to be shut down for maintenance at the same time, line 2 and line 3 need to be shut down for maintenance at the same time, and so on.

[0352] The above mutually exclusive relationship constraints and simultaneous shutdown relationship constraints are two different constraints. Since there are many power equipment, some equipment cannot be shut down at the same time, while some equipment can be shut down for maintenance at the same time, so relevant constraints are given.

[0353] In order to reduce the number of power outages and improve user satisfaction, the equipment power outage duration must not exceed the maximum number of days in the month, so the expression of the maintenance decision variable constraint is:

[0354] 1≤X v ≤N dm And X v ∈N dm

[0355] Where, X v is the power outage duration of the vth device (i.e., the power outage object); N dm Indicates the number of days in the month to be analyzed (the month with a planned power outage).

[0356] Since the number of maintenance personnel is limited, the number of maintenance of the same type of equipment on the same day is limited, and human resource constraints need to be considered in the power outage plan decision. Therefore, the expression of maintenance human resource constraints is:

[0357]

[0358] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LIIndicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0359] Considering that the occurrence of special events (bad weather, special holidays) will result in a period of time during the planning cycle where power outages for maintenance are not allowed, in order to improve the feasibility of the power outage plan, the expression of the maintenance special event constraint is:

[0360]

[0361] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0362] In this invention, the multi-agent deep deterministic policy gradient (MADDPG) algorithm used is an extension of the deep deterministic policy gradient (DDPG) algorithm in single-agent deep reinforcement learning in a multi-agent environment. The MADDPG algorithm adopts a "centralized training, decentralized execution" framework and can adapt to complex multi-agent environments.

[0363] The MADDPG algorithm is based on the Actor-Critic framework. In the MADDPG algorithm, each agent has its own strategy (Actor) network and evaluation (Critic) network. The Actor network is used to select power outage actions, and the Critic network is used to evaluate the value of the selected power outage actions.

[0364] The current state of the power grid at time t is s t The input is sent to the Actor network of each agent. The Actor network will explore and obtain the power outage action of the agent. After the power outage action acts on the power grid environment, a new power grid state and the current reward will be obtained (after the power outage action of the agent acts on the power grid environment, the environmental state (power grid state) will definitely change, so the new environmental state will be fed back to the agent, and the reward will also be fed back to the agent. The agent can judge the quality of its own power outage action based on the quality of the reward). The state, power outage action, new state and reward of the agent are stored as experience in its own experience replay pool; therefore, when using the MADDPG algorithm to train and solve the multi-agent Markov decision process model, the current power grid state s at time t is used. t Input into an agent to obtain the power outage action a at the current time t t (i.e., the duration and number of power outages of the agent), execute the power outage action a at the current time t t After that, the power grid status changes from st Transfer to s', that is, the power grid state is transferred from the current state at time t to the state at the next time (time t+1), and the reward r of the feedback under the current environment at time t is obtained t , and feed back the grid state transfer and reward to the agent. The agent continuously interacts with the grid environment and t ,a t ,r t ,s'} is stored in its own experience replay pool (in the MADDPG algorithm, the number of agents is N, and each agent has its own experience replay pool, so N agents correspond to N experience replay pools, and each experience replay pool will also store the corresponding agent's {s t ,a t ,r t ,s'} data), before updating the parameters in the network each time, a set of {s t ,a t ,r t ,s'} data to train the neural network of the agent (the policy network and the evaluation network are composed of neural networks. Neural networks refer to fully connected neural networks in deep learning. When training the neural network of each agent, a set of {s t ,a t ,r t ,s'} data for training), and the new {s t ,a t ,r t ,s'} back into the experience replay pool, and then randomly selected for training), until the reward fed back in the current power grid environment is high and tends to be stable, stop training (in the process of training all agents, if an agent reaches the optimal state, you can stop training it, and other agents continue training. In the centralized training stage, if an agent stops training, you can freeze the strategy of the agent to avoid affecting other agents. The present invention takes the overall reward optimization as the optimization goal, and all agents cooperate with each other. When the overall reward (the sum of the rewards of each agent) of each centralized training is large and will not change significantly, the reward at this time can be considered to be optimal and stable. At this time, stop training all agents and the training ends). After the training, the strategy network of each agent will output an action value, which is the optimal strategy (these action values ​​together constitute the power grid power outage scheduling strategy). This action value will be sent to the agent to let the agent perform the action, such as stopping 5 generators, each for 30 minutes.

[0365] The multi-agent deep deterministic policy gradient algorithm process in the present invention is as follows: Figure 2 shown.

[0366] The ultimate goal of deep reinforcement learning in this invention is to find the final joint control strategy to maximize the cumulative reward.

[0367] In the MADDPG algorithm, there are N agents, so it contains N Actor networks and N Critic networks, where the parameter set of the N Actor networks is θ = {θ1, θ2, …, θ N}, θ represents the set of policy network parameters; θ1 represents the parameters of the policy network of the first agent, θ2 represents the parameters of the policy network of the second agent, ..., θ N represents the parameters of the policy network of the Nth agent. During the deep neural network training process, the general policy gradient of the kth agent can be expressed as (the following formula is applicable to random strategies):

[0368]

[0369] In the formula, represents the policy gradient of the kth agent; logμ k Represents the strategy μ k Take the logarithm calculation; Qk(s k ,a k1 ,a k2 ,…,a km ) is the kth centralized value function (centralization means that this value function depends not only on the local information of a certain agent, but also on the state and action information of all agents), and its input is the action taken by the agent based on the policy (Actor) network (the action taken by the policy network is the power outage action) a k and the current status k , whose output is the Q value of the kth agent;

[0370] In addition to the Actor network and the Critic network, the intelligent agent also has a target-Actor network (the strategy (Actor) network of the target network) and a target-Critic network. k 'Input into the target-Actor network of the current agent k to get action a k '(s k ' represents the state of the kth agent at the next moment. The kth agent is based on the current state s k , execute power outage action a k After that, the state will be changed from s k Transformed into k '), and S'=[s'1,s'2,…,s' N](s'1 represents the state of the first agent at the next moment. After the first agent performs the power outage action a1 according to the current state s1, the state will change from s1 to s1'; s'2 represents the state of the second agent at the next moment. After the second agent performs the power outage action a2 according to the current state s2, the state will change from s2 to s2', ..., s' N Indicates the state of the Nth agent at the next moment. The Nth agent is based on the current state s N , execute power outage action a N After that, the state will be changed from s N Transformed into N ') and A'=[a'1,a'2,...,a' N ] is input into the target-Critic network to obtain the Q at the next moment k '; A' represents the power outage action set of all agents at the next moment, a'1 represents the power outage action set of the first agent at the next moment, a'2 represents the power outage action set of the second agent at the next moment, ..., a' N represents the set of all power outage actions of the Nth agent at the next moment. After the agent executes the power outage action A according to the current grid state S, the state will transfer from S to S', and the power outage action A' will be executed in state S'; Q k ' represents the evaluation of the target-critic network on the power outage action A'. Because the power outage action A is performed under the power grid state S, the value Q of the power outage action A is calculated by the target-critic network. Then, at the next moment of the power grid state S', the evaluation of the next moment of the power outage action A' is Q k ', Q k ' refers to the action value of the kth agent in the grid state set S' and the power outage action set A'.

[0371] Based on this, under the relevant constraints of the set power grid outage scheduling, the specific operations of using the multi-agent deep deterministic policy gradient algorithm to train and solve the multi-agent Markov decision process model include:

[0372] Determine the policy gradient of the kth agent, expressed as (in the present invention, since the MADDPG algorithm uses the stochastic gradient descent method to update the Actor network of each agent, a deterministic strategy is used, so the actual calculation uses the following formula):

[0373]

[0374] In the formula, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a krepresents the set of all power outage actions of the kth agent, a k There are m power outage actions in total; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, …, a N ]~B indicates power failure action [a1,a2,…,a N ] obeys the distribution of B (the experience replay pool contains strategies).

[0375] Define the loss function of the evaluation network, the expression is:

[0376]

[0377] In the formula, L(θ k ) is the loss function of the evaluation network of the kth agent; Indicates k ,a k ,r k ,s, k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′k(s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k) represents the state s' of the kth agent at the next moment k The strategy (μ′) of the target-strategy network k represents the policy of the goal-policy network of the kth agent).

[0378] The parameters of the target network are updated using soft update, and the expression is:

[0379]

[0380] θ' k =τθ k +(1-τ)θ' k

[0381] In the formula, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0382] The MADDPG algorithm stabilizes the training and solution of the Actor network and the Critic network by introducing the target network. The Actor network representing the target network, The Actor network representing the target network of the first agent, Actor network representing the target network of the second agent, …, represents the Actor network of the target network of the Nth agent, θ'1 represents the Actor network parameters of the target network of the first agent, θ'2 represents the Actor network parameters of the target network of the second agent, ..., θ' N represents the Actor network parameters of the target network of the Nth agent, and the Actor network parameters of the target network of the kth agent are θ' k The parameters of the critic network are continuously updated during training, and the target network does not participate in the training process.

[0383] In the present invention, each agent is in a fully cooperative relationship, and the agents minimize the objective function through mutual coordination, and each agent shares a global reward function. Therefore, the actions of all agents will affect the overall reward function, and the global reward function will guide the strategy optimization of each agent.

[0384] After training and solving with the MADDPG algorithm, the strategy network of each agent will obtain the corresponding power outage action value (obtained by the output of the Actor network). The power grid power outage scheduling strategy is the combination of these power outage action values, and these power outage action values ​​are sent to the agent. The agent then executes the corresponding power outage action. By adjusting the power outage duration and number of power outages of generator sets, lines and transformers, the power grid power outage plan is scheduled, ensuring the safe and stable operation of the power grid and its equipment, and also helping to extend the service life of the equipment.

[0385] Example 3

[0386] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of Example 2 are implemented.

[0387] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0388] The preferred embodiments of the present invention are described in detail above, but the present invention is not limited thereto. Within the technical concept of the present invention, the technical solution of the present invention can be subjected to a variety of simple modifications, including the combination of various technical features in any other suitable manner, and these simple modifications and combinations should also be regarded as the contents disclosed by the present invention and belong to the protection scope of the present invention.

Claims

1. A multi-agent deep reinforcement learning power grid outage scheduling system, characterized in that: It includes the choreography problem transformation module and the decision-making solution module; The scheduling problem conversion module is used to convert the power grid outage scheduling problem into a multi-agent Markov decision process model; The decision-solving module is used to train and solve the multi-agent Markov decision process model using a multi-agent deep deterministic policy gradient algorithm under the relevant constraints of the set power grid power outage scheduling, so as to obtain the power grid power outage scheduling strategy.

2. The multi-agent deep reinforcement learning power grid outage scheduling system according to claim 1 is characterized in that: The process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of a tuple: {N,S,a1,a2,…,a N ,r1,r2,…,r N ,F T ,γ}; where N is the number of agents; S is the set of power grid states; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…,r N ] represents the corresponding reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transfer function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' means the agent performs a power outage action A=[a1,a2,…,a N ] and then transferred to the grid state set at the next moment; γ represents the discount rate.

3. The multi-agent deep reinforcement learning power grid outage scheduling system according to claim 1 or 2, characterized in that: The specific operations of transforming the power grid outage scheduling problem into a multi-agent Markov decision process model include: Determine the objective function F(P) in the power grid outage scheduling problem, expressed as: C i =∑c t Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ]; p i represents the ith power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, 2, ..., n; T u represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily power outage plans; Indicates the power outage plan i The execution start date of Indicates the power outage plan i The execution termination date; i (t) represents the power outage object on day t; The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions; The expression of the state variable is: In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed sign; The state set of the generator set includes: Where P v (t) represents the daily average active output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t); The state set of a line includes: LI v (t)=[V v T (t),F v T (t)[ T Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t); The state set of a transformer includes: In the formula, R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t); The expression of the action variable is: Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t); The expression of the reward function is: Where r represents the reward function; ω1 represents the weight coefficient of the total outage quantity; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage quantity.

4. The multi-agent deep reinforcement learning power grid outage scheduling system according to claim 1 is characterized in that: The relevant constraints of the set power grid blackout scheduling include safety constraints and rationality constraints; The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line flow constraints, transformer voltage ratio constraints, winding temperature constraints and load rate constraints; The expression of power balance constraint is: Where D(t) represents the daily average load demand of the system on day t; N G Indicates the number of generator sets; The expression of unit output constraint is: In the formula, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; It represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day; The expression of node voltage constraint is: In the formula, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day; The expression of line power flow constraint is: In the formula, It represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day; The expression of transformer voltage ratio constraint is: In the formula, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day; The expression of winding temperature constraint is: In the formula, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day; The expression of load rate constraint is: In the formula, It represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

5. The multi-agent deep reinforcement learning power grid outage scheduling system according to claim 4 is characterized in that: The rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints; The expression of the unchangeable plan constraint is: In the formula, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an irreversible planned power outage; Indicates the start time of the power outage of the irreversible plan; Indicates the end time of the power outage of the irreversible plan; The expression of the mutual exclusion constraint is: z p (t)·z q (t)=0 In the formula, z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t); The expression of the same-stop relation constraint is: z h (t)·z l (t)=1 In the formula, z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t); The expression of maintenance decision variable constraint is: 1≤X v ≤N dm And X v ∈N dm Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed; The expression of maintenance human resource constraint is: Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day; The expression for overhauling the special event constraint is: Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

6. The multi-agent deep reinforcement learning power grid outage scheduling system according to claim 1, characterized in that: Under the constraints of the set power outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include: Determine the policy gradient of the kth agent, expressed as: In the formula, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, …, a N ]~B indicates power failure action [a1,a2,…,a N ] obeys the distribution of B; Define the loss function of the evaluation network, the expression is: In the formula, L(θ k ) is the loss function of the evaluation network of the kth agent; Indicates (s k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′ k (s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network; The parameters of the target network are updated using soft update, and the expression is: I will k =tθ k +(1-τ)θ' k In the formula, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

7. A multi-agent deep reinforcement learning method for power grid outage scheduling, characterized in that: include: The power grid outage scheduling problem is transformed into a multi-agent Markov decision process model; Under the relevant constraints of the set power outage scheduling, the multi-agent deep deterministic policy gradient algorithm is used to train and solve the multi-agent Markov decision process model to obtain the power outage scheduling strategy.

8. The method for power outage scheduling based on multi-agent deep reinforcement learning according to claim 7, characterized in that: The process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of a tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N is the number of agents; S is the set of power grid states; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…r N ] represents the corresponding reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transfer function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' means the agent performs a power outage action A=[a1,a2,…,a N ] and then transferred to the grid state set at the next moment; γ represents the discount rate.

9. The method for power outage scheduling based on multi-agent deep reinforcement learning according to claim 7 or 8, characterized in that: The specific operations of transforming the power outage scheduling problem into a multi-agent Markov decision process model include: Determine the objective function F(P) in the power grid outage scheduling problem, expressed as: C i =∑c t Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ]; p i represents the ith power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, 2, ..., n; T u represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily power outage plans; Indicates the power outage plan i The execution start date of Indicates the power outage plan i The execution termination date; i (t) represents the power outage object on day t; The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions; The expression of the state variable is: In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed sign; The state set of the generator set includes: Where P v (t) represents the daily average active output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t); The state set of a line includes: LI v (t)=[V v T (t),F v T (t)] T Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t); The state set of a transformer includes: In the formula, R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t); The expression of the action variable is: Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t); The expression of the reward function is: Where r represents the reward function; ω1 represents the weight coefficient of the total outage quantity; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage quantity.

10. The method for power grid outage scheduling based on multi-agent deep reinforcement learning according to claim 7, characterized in that: The relevant constraints of the set power grid blackout scheduling include safety constraints and rationality constraints; The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line flow constraints, transformer voltage ratio constraints, winding temperature constraints and load rate constraints; The expression of power balance constraint is: Where D(t) represents the daily average load demand of the system on day t; N G Indicates the number of generator sets; The expression of unit output constraint is: In the formula, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; It represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day; The expression of node voltage constraint is: In the formula, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day; The expression of line power flow constraint is: In the formula, It represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day; The expression of transformer voltage ratio constraint is: In the formula, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day; The expression of winding temperature constraint is: In the formula, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day; The expression of load rate constraint is: In the formula, It represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

11. The method for power grid outage scheduling based on multi-agent deep reinforcement learning according to claim 10, characterized in that: The rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints; The expression of the unchangeable plan constraint is: In the formula, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an irreversible planned power outage; Indicates the start time of the power outage of the irreversible plan; Indicates the end time of the power outage of the irreversible plan; The expression of the mutual exclusion constraint is: z p (t)·z q (t)=0 In the formula, z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t); The expression of the same-stop relation constraint is: z h (t)·z l (t)=1 In the formula, z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t); The expression of maintenance decision variable constraint is: 1≤X v ≤N dm And X v ∈N dm Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed; The expression of maintenance human resource constraint is: Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day; The expression for overhauling the special event constraint is: Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

12. The method for power grid outage scheduling based on multi-agent deep reinforcement learning according to claim 7, characterized in that: Under the constraints of the set power outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include: Determine the policy gradient of the kth agent, expressed as: In the formula, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, …, a N ]~B indicates power failure action [a1,a2,…,a N ] obeys the distribution of B; Define the loss function of the evaluation network, the expression is: In the formula, L(θ k ) is the loss function of the evaluation network of the kth agent; Indicates (s k ,a k ,r k ,s, k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′ k (s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network; The parameters of the target network are updated using soft update, and the expression is: I will k =tθ k +(1-τ)θ' k In the formula, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 7 to 12 are implemented.

Images