Power grid outage scheduling system and method using multi-agent deep reinforcement learning and medium

Abstract

The application relates to the field of power systems and discloses a power grid outage scheduling system based on multi-agent deep reinforcement learning, which comprises a scheduling problem conversion module and a decision solving module; the scheduling problem conversion module is used for converting a power grid outage scheduling problem into a multi-agent Markov decision process model; and the decision solving module is used for training and solving the multi-agent Markov decision process model by using a multi-agent deep deterministic policy gradient algorithm under the set constraints of the power grid outage scheduling, so as to obtain a power grid outage scheduling strategy. The scheme is inspired by the single-agent reinforcement learning idea, converts the power grid outage scheduling problem into a multi-agent Markov decision process, regards power equipment as a single agent, trains the equipment with intelligent outage scheduling by using the multi-agent deep deterministic policy gradient algorithm, and learns the optimal outage scheduling strategy, so as to guarantee the safe operation of the power grid and the equipment.

Description

Technical Field

[0001] The present invention relates to the field of power systems, and in particular to a power grid outage scheduling system and method based on multi-agent deep reinforcement learning, as well as a medium. Background Art

[0002] As the scale of the power grid continues to expand, the number of renewable energy sources such as wind and photovoltaic power connected to the grid continues to increase, and the number of devices in the grid is gradually increasing, resulting in an increasingly complex grid structure and increasingly complex and variable grid operating conditions, posing significant challenges to the safe operation of grid equipment. To achieve the goals of "carbon peak and carbon neutrality" as soon as possible, the proportion of renewable energy sources such as wind and photovoltaic power connected to the grid is gradually increasing. The uncertainty, volatility, and indirectness of renewable energy output pose severe challenges to the safe and reliable operation of the grid. To ensure the safe operation of the grid and the reliability of power supply, the scheduling of power outages and maintenance plans for power equipment has become a pressing issue. When planning power outages for power grid equipment, it is important to prioritize power supply maintenance, meaning that as few devices as possible are outaged to minimize the impact of outages on users and improve the reliability and continuity of power supply. However, due to the complexity and cumbersome nature of power outage planning, which relies on manual experience for scheduling, the scheduling is inefficient and easily influenced by subjective factors, making power supply maintenance strategies sometimes difficult to implement.

[0003] Existing methods for solving power grid outage scheduling models are mostly based on heuristic algorithms using swarm intelligence optimization, such as particle swarm optimization, dragonfly algorithm, and ant colony algorithm. While these algorithms can avoid local optimal solutions to a certain extent, they lack continuous interaction with the environment and are unable to adapt to the ever-changing power grid environment. Furthermore, as the number of grid devices increases, the power grid outage scheduling model becomes more complex, increasing the computational burden of heuristic algorithms. Finding the optimal solution requires numerous iterations, resulting in slow convergence and low computational efficiency.

[0004] Power outage scheduling is essentially a Markov decision process, a type of decision-making problem that reinforcement learning excels at solving. The fundamental idea of ​​reinforcement learning is how intelligent agents make decisions to maximize their own rewards in uncertain environments. When implementing power outage scheduling based on single-agent reinforcement learning, the power equipment agents are typically treated as a unified entity. However, due to limitations such as communication overhead and control latency, treating the agents as a unified entity can easily lead to problems such as low efficiency and poor performance in power outage scheduling. Summary of the Invention

[0005] The purpose of the present invention is to overcome the lack of continuous interaction with the environment when solving the power grid outage scheduling model in existing heuristic algorithms, making it difficult to adapt to real-time changes in the power grid. At the same time, due to the existence of communication burden and control delay, the scheduling of power grid outages based on single-agent reinforcement learning is prone to problems such as low scheduling efficiency and poor scheduling results. Therefore, a multi-agent deep reinforcement learning power grid outage scheduling system, method, and medium are provided. Inspired by the concept of single-agent reinforcement learning, this solution transforms the power grid outage scheduling problem into a multi-agent Markov decision process. Power equipment such as generators, lines, and transformers are treated as separate agents. A multi-agent deep deterministic policy gradient algorithm is used to train equipment with intelligent outage scheduling. The agents continuously interact with the power grid environment to learn the optimal outage scheduling strategy, thereby ensuring the safe operation of the power grid and its equipment.

[0006] In order to achieve the above objectives, the present invention provides, on one hand, a multi-agent deep reinforcement learning power grid outage scheduling system, comprising a scheduling problem conversion module and a decision-solving module;

[0007] The scheduling problem conversion module is used to convert the power grid outage scheduling problem into a multi-agent Markov decision process model;

[0008] The decision-solving module is used to train and solve the multi-agent Markov decision process model using a multi-agent deep deterministic policy gradient algorithm under the relevant constraints of the set power grid outage scheduling to obtain the power grid outage scheduling strategy.

[0009] Preferably, the process of converting the power outage scheduling problem into a multi-agent Markov decision process model consists of a tuple: {N, S, a1, a2, …, a N ,r1,r2,…,r N ,F T ,γ}; where N represents the number of agents; S represents the grid state set; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…,r N ] represents the reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transition function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' represents the power outage action A=[a1,a2,…,a N] and then transfer to the grid state set at the next moment; γ represents the discount rate.

[0010] Preferably, the specific operations of converting the power grid outage planning problem into a multi-agent Markov decision process model include:

[0011] Determine the objective function F(P) in the power grid outage scheduling problem, which is expressed as:

[0012]

[0013] C i =Σc t

[0014]

[0015] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ];p i represents the i-th power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan scheme, i = 1, 2, ..., n; T u Represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily scheduled power outages; Indicates the power outage plan p i The execution start date; Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object on day t;

[0016] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0017] Among them, the expression of the state variable is:

[0018]

[0019] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed symbol;

[0020] The state set of the generator set includes:

[0021]

[0022] Where, P v (t) represents the daily average active power output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t);

[0023] The state set of a line includes:

[0024]

[0025] Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t);

[0026] The state set of a transformer includes:

[0027]

[0028] Where R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v(t) represents the average load rate of the vth transformer at the tth day, represents H v the transpose matrix of (t) ;

[0029] The expression of the action variable is:

[0030]

[0031] wherein A(t) represents the action variable space at the tth day, D GE,v (t) represents the outage duration of the vth generator set starting at the tth day, represents D GE,v the transpose matrix of (t) ; S GE,v (t) represents the outage state of the vth generator set at the tth day, represents S GE,v the transpose matrix of (t) ; D LI,v (t) represents the outage duration of the vth line starting at the tth day, represents D LI,v the transpose matrix of (t) ; S LI,v (t) represents the outage state of the vth line at the tth day, represents S LI,v the transpose matrix of (t) ; D TR,v (t) represents the outage duration of the vth transformer starting at the tth day, represents D, the transpose matrix of v(t) ; S TR,v (t) represents the outage state of the vth transformer at the tth day, represents S TR,v the transpose matrix of (t) ;

[0032] The expression of the reward function is:

[0033]

[0034] wherein r represents the reward function; ω1 represents the weight coefficient of the total outage amount; ω2 represents the weight coefficient of the outage economic cost; and ω3 represents the weight coefficient of the outage amount balance degree.

[0035] Preferably, the related constraints of the set power grid outage scheduling include safety constraints and rationality constraints;

[0036] The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line power flow constraints, transformer voltage ratio constraints, winding temperature constraints and load rate constraints;

[0037] The expression of the power balance constraint is:

[0038]

[0039] where D(t) denotes the daily average load demand of the system at day t; N G denotes the number of generators;

[0040] The expression of the generator output constraint is:

[0041]

[0042] where denotes the daily average active power output upper limit of the vth generator at day t; denotes the daily average active power output lower limit of the vth generator at day t; denotes the daily average reactive power output upper limit of the vth generator at day t; denotes the daily average reactive power output lower limit of the vth generator at day t;

[0043] The expression of the node voltage constraint is:

[0044]

[0045] where denotes the daily average terminal voltage upper limit of the vth line at day t; denotes the daily average terminal voltage lower limit of the vth line at day t;

[0046] The expression of the line power flow constraint is:

[0047]

[0048] where denotes the daily average power flow value upper limit of the vth line at day t; denotes the daily average power flow value lower limit of the vth line at day t;

[0049] The expression of the transformer voltage ratio constraint is:

[0050]

[0051] where denotes the daily average voltage ratio upper limit of the vth transformer at day t; denotes the daily average voltage ratio lower limit of the vth transformer at day t;

[0052] The expression of the winding temperature constraint is:

[0053]

[0054] where denotes the daily average winding temperature upper limit of the vth transformer at day t; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day;

[0055] The expression of load rate constraint is:

[0056]

[0057] Where, represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0058] Preferably, the rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints;

[0059] The expression of the unchangeable plan constraint is:

[0060]

[0061] Where, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an unchangeable planned power outage; Indicates the start time of the power outage of this irreversible plan; Indicates the end time of the power outage of this irreversible plan;

[0062] The expression of the mutual exclusion constraint is:

[0063] z p (t)·z q (t) = 0

[0064] Where z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t);

[0065] The expression of the same-stop relation constraint is:

[0066] z h (t)·z l (t)=1

[0067] Where z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h(t)∈z i (t), z l (t)∈z i (t);

[0068] The expression of maintenance decision variable constraint is:

[0069] 1≤X v ≤N dm And X v ∈N dm

[0070] Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed;

[0071] The expression of maintenance human resource constraint is:

[0072]

[0073] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0074] The expression for overhauling the special event constraint is:

[0075]

[0076] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0077] Preferably, under the relevant constraints of the set power grid outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include:

[0078] Determine the policy gradient of the kth agent, expressed as:

[0079]

[0080] Where, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ krepresents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, ..., a N ]~B represents power failure action [a1,a2,…,a N ] obeys the distribution of B;

[0081] Define the loss function of the evaluation network, the expression is:

[0082]

[0083] Where, L(θ k ) is the loss function of the evaluation network of the k-th agent; Indicates the k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′ k (s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network;

[0084] The parameters of the target network are updated using the soft update method, and the expression is:

[0085]

[0086] θ' k =τθ k +(1-τ)θ' k

[0087] Where, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0088] A second aspect of the present invention provides a multi-agent deep reinforcement learning method for power grid outage scheduling, comprising:

[0089] The power grid outage scheduling problem is transformed into a multi-agent Markov decision process model;

[0090] Under the relevant constraints of the set power grid outage scheduling, the multi-agent deep deterministic policy gradient algorithm is used to train and solve the multi-agent Markov decision process model to obtain the power grid outage scheduling strategy.

[0091] Preferably, the process of converting the power outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N, S, a1, a2, ... a N ,r1,r2,…r N ,F T ,γ}; where N represents the number of agents; S represents the grid state set; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…r N ] represents the reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs power outage action a N The reward value obtained after T represents the state transition function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' represents the power outage action A=[a1,a2,…,a N ] and then transfer to the grid state set at the next moment; γ represents the discount rate.

[0092] Preferably, the specific operations of converting the power grid outage planning problem into a multi-agent Markov decision process model include:

[0093] Determine the objective function F(P) in the power grid outage scheduling problem, which is expressed as:

[0094]

[0095] C i =∑c t

[0096]

[0097] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ];p i represents the i-th power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan scheme, i = 1, 2, ..., n; T u Represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily scheduled power outages; Indicates the power outage plan p i The execution start date; Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object on day t;

[0098] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0099] Among them, the expression of the state variable is:

[0100]

[0101] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v(t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed symbol;

[0102] The state set of the generator set includes:

[0103]

[0104] Where, P v (T) represents the average daily active power output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Indicates Q v The transposed matrix of (t);

[0105] The state set of a line includes:

[0106]

[0107] Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t);

[0108] The state set of a transformer includes:

[0109]

[0110] Where R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average daily load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t);

[0111] The expression of the action variable is:

[0112]

[0113] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t);

[0114] The expression of the reward function is:

[0115]

[0116] Where r represents the reward function; ω1 represents the weight coefficient of the total outage amount; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage amount.

[0117] Preferably, the set constraints related to the power grid outage scheduling include safety constraints and rationality constraints;

[0118] The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line power flow constraints, transformer voltage ratio constraints, winding temperature constraints, and load rate constraints;

[0119] The expression of power balance constraint is:

[0120]

[0121] Where D(t) represents the daily average load demand of the system on day t; N GIndicates the number of generator sets;

[0122] The expression of unit output constraint is:

[0123]

[0124] Where, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; represents the upper limit of the daily average reactive output of the vth generator set on day t; It represents the lower limit of the daily average reactive output of the vth generator set on the tth day;

[0125] The expression of the node voltage constraint is:

[0126]

[0127] Where, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; represents the lower limit of the daily average terminal voltage of the vth line on the tth day;

[0128] The expression of line power flow constraint is:

[0129]

[0130] Where, represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day;

[0131] The expression of transformer voltage ratio constraint is:

[0132]

[0133] Where, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day;

[0134] The expression of winding temperature constraint is:

[0135]

[0136] Where, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day;

[0137] The expression of load rate constraint is:

[0138]

[0139] Where, represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0140] Preferably, the rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints;

[0141] The expression of the unchangeable plan constraint is:

[0142]

[0143] Where, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an unchangeable planned power outage; Indicates the start time of the power outage of this irreversible plan; Indicates the end time of the power outage of this irreversible plan;

[0144] The expression of the mutual exclusion constraint is:

[0145] z p (t)·z q (t) = 0

[0146] Where z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t);

[0147] The expression of the same-stop relation constraint is:

[0148] z h (t)·z l (t)=1

[0149] Where z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t);

[0150] The expression of maintenance decision variable constraint is:

[0151] 1≤X v ≤N dm And X v ∈N dm

[0152] Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed;

[0153] The expression of maintenance human resource constraint is:

[0154]

[0155] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0156] The expression for overhauling the special event constraint is:

[0157]

[0158] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0159] Preferably, under the relevant constraints of the set power grid outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include:

[0160] Determine the policy gradient of the kth agent, expressed as:

[0161]

[0162] Where, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s krepresents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, ..., a N ]~B represents power failure action [a1,a2,…,a N ] obeys the distribution of B;

[0163] Define the loss function of the evaluation network, the expression is:

[0164]

[0165] Where, L(θ k ) is the loss function of the evaluation network of the k-th agent; Indicates the k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′ k (s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network;

[0166] The parameters of the target network are updated using the soft update method, and the expression is:

[0167]

[0168] θ' k =τθ k +(1-τ)θ' k

[0169] Where, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0170] A third aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0171] Compared with the existing technology, the present invention mainly realizes the task of power grid outage scheduling by adjusting the number and duration of power outages of the power equipment intelligent body to ensure the safe and stable operation of the power grid. First, the power grid outage scheduling problem is transformed into a multi-agent Markov decision process model, which includes the power grid state, power outage action, reward, joint state transfer function and discount rate; at the same time, in order to ensure the safety of power grid operation and the rationality of power outage scheduling, relevant constraints are introduced; further, the multi-agent deep deterministic policy gradient algorithm is used to solve the above-mentioned Markov decision model. After solving, the power outage time and number of power outages of generator sets, lines and transformers can be obtained. Each intelligent agent generates a power outage scheduling strategy by sensing the power grid environment and executes the corresponding action. The fully trained intelligent agent can make the optimal action strategy (i.e. the number and duration of power outages of the intelligent agent) under the current environment, thereby obtaining the maximum reward value and improving the efficiency of power grid outage scheduling.

[0172] In the present invention, each intelligent agent is in a fully cooperative relationship. Although the actions of each intelligent agent only act on its own environment, all intelligent agents share the same reward function. The intelligent agents cooperate with each other and use the MADDPG algorithm to solve the power outage time and number of power outages of each intelligent agent, realizing the power grid power outage plan scheduling. Using this method, intelligent power grid power outage plan scheduling can be realized, effectively reducing the occurrence of unexpected failures, helping to timely detect abnormalities in power equipment, extending the service life of equipment, and ensuring the safe and stable operation of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0173] Figure 1 This is a schematic diagram of the structure of the power grid outage scheduling system based on multi-agent deep reinforcement learning of the present invention;

[0174] Figure 2This is a flowchart of the multi-agent deep deterministic policy gradient algorithm;

[0175] Figure 3 This is a flow chart of the power grid outage scheduling method based on multi-agent deep reinforcement learning of the present invention. DETAILED DESCRIPTION

[0176] The following is a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings and specific examples. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not intended to limit the present invention.

[0177] Example 1

[0178] like Figure 1 The multi-agent deep reinforcement learning power grid outage scheduling system shown includes a scheduling problem transformation module and a decision-solving module;

[0179] The scheduling problem conversion module is used to convert the power grid outage scheduling problem into a multi-agent Markov decision process model;

[0180] The decision-solving module is used to train and solve the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm under the relevant constraints of the set power grid outage scheduling to obtain the power grid outage scheduling strategy.

[0181] Since the power outage planning strategy of power equipment directly affects the reliability and economy of the power system, the present invention realizes the safe and stable operation of the power grid by scheduling the power outage of power equipment. The power outage planning refers to the reasonable arrangement of the power outage sequence and time of power equipment under certain constraints, so that the power system can minimize the impact on system reliability, economy and safety when undergoing maintenance, overhaul or expansion.

[0182] Among them, in the problem of power grid outage planning, each generator set, each line and each transformer in the power grid is regarded as a separate intelligent agent, that is, there will be multiple intelligent agents in the power grid.

[0183] In the optimal case, the process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N represents the number of agents; S represents the grid state set; [a1,a2,…,a N ] represents the power outage action set of all agents, where a1 represents the power outage action set of the first agent, a2 represents the power outage action set of the second agent, ..., a Nrepresents the set of all outage actions of the Nth agent (usually one agent's set of all outage actions will include multiple outage actions) ; [a1, a2, …, a N ] represents the set of all rewards corresponding to the set of all outage actions performed by all agents, r1 represents the reward value obtained by the 1st agent after performing the outage action a1, r2 represents the reward value obtained by the 2nd agent after performing the outage action a2, …, r N represents the reward value obtained by the Nth agent after performing the outage action a N ; F T represents the state transition function, F T : S x a1 x a2, …, x a N x S' → [0, 1], S' represents the set of power grid states to which the agents are transferred to the next time after performing the outage action A = [a1, a2, …, a N ] in the power grid state S; γ represents the discount rate, which represents the degree of attention to future rewards, the greater the γ, the more attention is paid to future rewards (the rewards include current rewards and future rewards: the current rewards refer to the rewards that the power grid environment can feed back to the agent after the agent applies the outage action to the power grid environment; the future rewards refer to the rewards that the agent may obtain in the future (the rewards that the agent may obtain in the future depend on the influence of the current action of the agent on the environment and the dynamic evolution characteristics of the environment. The action of the agent will affect the entire future reward sequence, rather than the current instant reward), but this reward is not certain to be good or bad, and the purpose of introducing the discount rate is to represent whether the agent pays more attention to the current reward or has higher expectations for the future reward; in specific cases, the rewards refer to the negative values of the objective functions, including the outage frequency, the economic loss of outage and the outage balance degree, and the agent will obtain the rewards after performing the outage plan action).

[0184] In the present application, the power grid state is composed of the states of the agents in the power grid; wherein the state refers to the output of the generator set, the working temperature of the line or the transformation ratio of the transformer, etc.; the outage action refers to the outage duration and the outage quantity, for example, the generator set needs to be outaged for 1 unit and for 30 minutes, etc..

[0185] The state transition function in the present application refers to the probability distribution of the power grid state to which the agents are transferred to the next time after performing the outage action A = [a1, a2, …, a N ] in the current power grid state; wherein the state transition refers to that the generator set, the line and the transformer (i.e. the agent) adjust the outage duration and the outage quantity according to the current environment state (i.e. the current power grid state) and the outage scheduling strategy, which will change the power flow distribution, the daily average output of the generator set, the daily average voltage of the node and the daily average transformation ratio of the transformer, and thus the state is transferred to the state of the next time.

[0186] Preferably, the specific operations of converting the power grid outage planning problem into a multi-agent Markov decision process model include:

[0187] Determine the objective function F(P) in the power grid outage scheduling problem (in power grid outage scheduling, power supply security is the primary consideration, that is, the fewer devices that are out of power, the better, to reduce the impact of power outages on users and improve the reliability and continuity of power supply; at the same time, the economic cost of power outages should be reduced as much as possible; finally, to improve power grid stability and avoid the risks of power outages to power grid operation, the power outage plans for each time period should be as balanced as possible; the present invention takes these three aspects into consideration and determines the objective function in the power grid outage scheduling problem based on them). The expression is:

[0188]

[0189] C i =∑c t

[0190]

[0191] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ];p i represents the i-th power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, ..., n, that is, all outage plans {p1, ..., p n There are n power outage plans in total (these power outage plans are pending power outage plans that have not yet been implemented); T u Represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period (because power outages affect the daily electricity consumption of electricity users, which will cause economic losses to users; in addition, when the equipment is out of power, it will be repaired and maintained, which will also generate certain economic consumption. Therefore, power outages will cause certain economic losses); c t represents the economic cost of each day in the power outage cycle (for example, a power outage plan p i The daily economic loss during the power outage period is {c1, c2, ...}. The power outage period is a period of time, such as 1 month. Within the 30 days of this month, there will be an economic loss of c every day. t , t=1,2,…,30, by multiple c tThe total economic loss during the power outage period (economic loss refers to economic cost); T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily power outage plans, which is used to measure the balance of power outages within the time period; Indicates the power outage plan p i The execution start date; Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object (i.e., intelligent agent) on day t.

[0192] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0193] The expression of the state variable is:

[0194]

[0195] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the sign of the transposed matrix.

[0196] The state set of the generator set includes:

[0197]

[0198] Where, P v (t) represents the daily average active power output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Represents Q v The transposed matrix of (t).

[0199] The state set of a line includes:

[0200]

[0201] Where: Vv (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the average daily power transmission of the vth line on the tth day (i.e., the average daily power transmission of the vth line on the tth day, i.e., the current on the transmission line), Indicates F v The transposed matrix of (t).

[0202] The state set of a transformer includes:

[0203]

[0204] Where R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average daily load rate of the vth transformer on the tth day (the ratio of the current load to the rated capacity), Indicates H v The transposed matrix of (t).

[0205] The expression of the action variable is:

[0206]

[0207] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, S GE,v (t)=1 means the vth generator set is out of power on the tth day, otherwise SCE,v(t)=0, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, S LI,v (t)=1 means that the vth line has power outage on the tth day, otherwise S LI,v(t) = 0, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, S TR,v (t)=1 means the vth transformer is out of power on the tth day, otherwise S TR,v (t) = 0, Indicates S TR,v The transposed matrix of (t).

[0208] The expression of the reward function is:

[0209]

[0210] Where r represents the reward function (the reward value is calculated based on the reward function); ω1 represents the weight coefficient of the total outage quantity; ω2 represents the weight coefficient of the economic cost of the outage; and ω3 represents the weight coefficient of the balance degree of the outage quantity.

[0211] In the present invention, each intelligent agent is in a fully cooperative relationship and determines the number and duration of power outages for each intelligent agent through mutual coordination to ensure that the objective function is minimized, so the rewards are shared among the intelligent agents.

[0212] In the problem of power grid outage planning, the safety of power grid operation and the rationality of power outage planning must be considered, thereby introducing relevant constraints. In the optimal case, the relevant constraints set for power grid outage planning include safety constraints and rationality constraints.

[0213] The main function of the relevant constraints for power grid outage scheduling is to constrain the intelligent agent. Safety constraints are used to ensure the safety of the intelligent agent's operation, such as the output of the generator set cannot be too high, and the operating temperature of the line cannot be too high. Rationality constraints mainly focus on the rationality of the intelligent agent's power outage, such as two generator sets cannot be shut down at the same time. By introducing constraints, the safety of the intelligent agent and the rationality of its actions can be guaranteed.

[0214] Furthermore, safety constraints include power balance constraints, unit output constraints, node voltage constraints, line power flow constraints, transformer voltage ratio constraints, winding temperature constraints, and load factor constraints.

[0215] The expression of power balance constraint is:

[0216]

[0217] Where D(t) represents the daily average load demand of the system on day t; NG Indicates the number of generator sets;

[0218] The expression of unit output constraint is:

[0219]

[0220] Where, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; represents the upper limit of the daily average reactive output of the vth generator set on day t; It represents the lower limit of the daily average reactive power output of the vth generator set on the tth day.

[0221] The expression of the node voltage constraint is:

[0222]

[0223] Where, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day.

[0224] The expression of line power flow constraint is:

[0225]

[0226] Where, represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day.

[0227] The expression of transformer voltage ratio constraint is:

[0228]

[0229] Where, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day.

[0230] The expression of winding temperature constraint is:

[0231]

[0232] Where, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of transformer v on day t.

[0233] The expression of load rate constraint is:

[0234]

[0235] Where, represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0236] In a preferred embodiment of the present invention, rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints.

[0237] An unchangeable plan refers to situations where the power outage schedule cannot be altered due to major events, such as those requiring guaranteed power supply. The schedule is determined based on the subjective wishes of the planner. This unchangeable plan is typically included in the power outage plan P.

[0238] Furthermore, the expression of the unchangeable plan constraint is:

[0239]

[0240] Where, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an unchangeable planned power outage; Indicates the start time of the power outage of this irreversible plan; Indicates the end time of the power outage for this irreversible plan.

[0241] To avoid islanding in the power grid system, the planned combination of features cannot be executed on the same day. p and p q All of them belong to the power outage plan P, where the corresponding power outage objects on day t are z p (t), z q (t), then

[0242] The expression of the mutual exclusion constraint is:

[0243] z p (t)·z q (t) = 0

[0244] Where z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day (z p (t) and z q(t) represents two different power outage objects, that is, in order to represent that two power outage objects cannot be powered off at the same time, for example, generator set 1 and line 2 cannot be powered off at the same time, and for example, transformer 1 and transformer 2 cannot be powered off at the same time, and the like), z p (t)∈z i (t), z q (t)∈z i (t).

[0245] According to the specified power outage plan association relationship, for the power outage plans that need to be executed at the same time, the execution time windows should be overlapped as much as possible. If two power outage plans p h , p l belong to the same power outage plan scheme P, and the power outage objects corresponding to the t-th day are z h (t), z l (t), then the expression of the same stop relationship constraint is:

[0246] z h (t)·z l (t)=1

[0247] In the formula, z h (t) represents the h-th power outage object of the t-th day, z l (t) represents the l-th power outage object of the t-th day; z h (t)∈z i (t), z l (t)∈z i (t), and z h (t) and z l (t) refer to two different power outage objects, and the two objects need to be powered off at the same time for maintenance, such as generator set 1 and transformer 1 needing to be powered off at the same time for maintenance, line 2 and line 3 needing to be powered off at the same time for maintenance, and the like.

[0248] The above-mentioned mutual exclusion relationship constraint and the same stop relationship constraint are two different constraints. Since there are many power equipment, some equipment cannot be powered off at the same time, and some equipment can be powered off at the same time for maintenance, so the related constraints are given.

[0249] In order to reduce the number of power outages and improve user satisfaction, the equipment power outage duration should not exceed the maximum number of days in the month, so the expression of the maintenance decision variable constraint is:

[0250] 1≤X v ≤N dm and X v ∈N dm

[0251] In the formula, X v is the power outage duration of the v-th equipment (i.e. power outage object); N dmIndicates the number of days in the month to be analyzed (the month with a planned power outage).

[0252] Since the number of maintenance personnel is limited and the number of maintenance operations for the same type of equipment on the same day is limited, human resource constraints need to be considered in the power outage plan decision-making. Therefore, the expression of the maintenance human resource constraint is:

[0253]

[0254] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0255] Considering that the occurrence of special events (bad weather, special holidays) will result in a period of time during the planning cycle where power outage maintenance is not allowed, in order to improve the feasibility of the power outage plan, the expression of the maintenance special event constraint is:

[0256]

[0257] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0258] The multi-agent deep deterministic policy gradient (MADDPG) algorithm used in this paper extends the deep deterministic policy gradient (DDPG) algorithm used in single-agent deep reinforcement learning to multi-agent environments. MADDPG employs a "centralized training, decentralized execution" framework, making it adaptable to complex multi-agent environments.

[0259] The MADDPG algorithm is based on the Actor-Critic framework. In the MADDPG algorithm, each agent has its own policy (Actor) network and evaluation (Critic) network. The Actor network is used to select power outage actions, and the Critic network is used to evaluate the value of the selected power outage actions.

[0260] The current state of the power grid at time t s tInput into the Actor network of each agent, the Actor network will explore and obtain the power outage action of the agent. After the power outage action acts on the power grid environment, a new power grid state and the current reward will be obtained (after the power outage action of the agent acts on the power grid environment, the environmental state (power grid state) will definitely change, so the new environmental state will be fed back to the agent, and the reward will also be fed back to the agent. The agent can judge the quality of its own power outage action based on the quality of the reward). The state, power outage action, new state and reward of the agent are stored as experience in its own experience replay pool; therefore, when using the MADDPG algorithm to train and solve the multi-agent Markov decision process model, the current power grid state s at time t is used. t Input into an agent to obtain the power outage action a at the current time t t (i.e., the duration and number of power outages of the agent), execute the power outage action a at the current time t t After that, the grid status is changed from s t Transfer to s', that is, the power grid state is transferred from the current state at time t to the state at the next time (time t+1), and the reward r of the feedback in the current environment at time t is obtained t , and feed back the grid state transfer and reward to the agent. The agent continuously interacts with the grid environment and t ,a t ,r t ,s'} is stored in its own experience replay pool (in the MADDPG algorithm, the number of agents is N, and each agent has an experience replay pool, so N agents correspond to N experience replay pools, and each experience replay pool also stores the corresponding agent's {s t ,a t ,r t ,s'} data), before updating the parameters in the network each time, a set of {s t ,a t ,r t ,s'} data to train the neural network of the agent (the policy network and the evaluation network are composed of neural networks. The neural network refers to the fully connected neural network in deep learning. When training the neural network of each agent, a set of {s t ,a t ,r t ,s'} data to train), and the new {s t ,a t ,r t,s'} back into the experience replay pool, and then randomly selected for training) until the reward feedback under the current power grid environment is high and tends to be stable, stop training (in the process of training all agents, if an agent reaches the optimal state, you can stop training for it, and other agents continue to train. In the centralized training stage, if an agent stops training, you can freeze the strategy of the agent to avoid affecting other agents. The present invention takes the optimal overall reward as the optimization goal, and all agents cooperate with each other. When the overall reward of each centralized training (the sum of the rewards of each agent) is large and does not change significantly, the reward at this time can be considered to be optimal and stable. At this time, stop training for all agents and the training ends). After the training is completed, the strategy network of each agent will output an action value, which is the optimal strategy (these action values ​​together constitute the power grid outage scheduling strategy). This action value will be sent to the agent to let the agent perform the action, such as stopping 5 generators, each for 30 minutes.

[0261] The multi-agent deep deterministic policy gradient algorithm process in this invention is as follows: Figure 2 shown.

[0262] The ultimate goal of deep reinforcement learning in this invention is to find the final joint control strategy to maximize the cumulative reward.

[0263] In the MADDPG algorithm, there are N agents, so it contains N Actor networks and N Critic networks, where the parameter set of N Actor networks is θ = {θ1, θ2, …, θ N}, θ represents the set of policy network parameters; θ1 represents the parameters of the policy network of the first agent, θ2 represents the parameters of the policy network of the second agent, ..., θ N represents the parameters of the policy network of the Nth agent. During deep neural network training, the general policy gradient of the kth agent can be expressed as (the following formula is applicable to random policies):

[0264]

[0265] Where, represents the policy gradient of the kth agent; logμ k Represents the strategy μ k Take the logarithm calculation; Q k (s k ,a k1 ,a k2 ,…,a km) is the kth centralized (centralized means that this value function not only depends on the local information of an agent, but also depends on the state and action information of all agents) value function, whose input is the action (the action taken by the policy network is the power-off action) a taken by the agent based on the policy (Actor) network k and the current state s k , and the output is the Q value of the kth agent;

[0266] In addition to the Actor network and the Critic network, the agent also has a target (target network) Actor network (the policy (Actor) network of the target network) and a target Critic network. The s k ’ is input into the target Actor network of the current agent k to obtain the action a k ’(s k ’ represents the state of the kth agent at the next time, and after the kth agent performs the power-off action a k based on the current state s k , the state will change from s k to s k ’), and S’ = [s’1, s’2, …, s’ N ] (s’1 represents the state of the 1st agent at the next time, and after the 1st agent performs the power-off action a1 based on the current state s1, the state will change from s1 to s1’; s’2 represents the state of the 2nd agent at the next time, and after the 2nd agent performs the power-off action a2 based on the current state s2, the state will change from s2 to s2’, …, s’ N represents the state of the Nth agent at the next time, and after the Nth agent performs the power-off action a N based on the current state s N , the state will change from s N to s N ’), and A’ = [a’1, a’2, …, a’ N ] are input into the target Critic network to obtain the Q k ’ at the next time; A’ represents the power-off action set of all agents at the next time, a’1 represents the power-off action set of the 1st agent at the next time, a’2 represents the power-off action set of the 2nd agent at the next time, …, a’B represents the power-off action set of the Nth agent at the next time. After the agent performs the power-off action A based on the current power grid state S, the state will change from S to S’, and the power-off action A’ will be performed under the state S’; Q k' represents the target-critic network's evaluation of the power outage action A'. Since the power outage action A is executed under the grid state S, the target-critic network calculates the value Q of the power outage action A. Then, at the next grid state S', the evaluation of the power outage action A' at the next moment is Q k ', Q k ' refers to the action value of the kth agent in the grid state set S' and the power outage action set A'.

[0267] Based on this, under the constraints of the set power outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include:

[0268] Determine the policy gradient of the kth agent, expressed as (in this invention, since the MADDPG algorithm uses stochastic gradient descent to update the Actor network of each agent, a deterministic strategy is used, so the actual calculation uses the following formula):

[0269]

[0270] Where, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent, a k There are m power outage actions in total; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, ..., a N ]~B represents power failure action [a1,a2,…,a N ] obeys the distribution of B (the experience replay pool contains strategies).

[0271] Define the loss function of the evaluation network, the expression is:

[0272]

[0273] Where, L(θ k ) is the loss function of the evaluation network of the k-th agent; Indicates the k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′k(s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network (μ′ k represents the policy of the goal-policy network of the k-th agent).

[0274] The parameters of the target network are updated using the soft update method, and the expression is:

[0275]

[0276] θ' k =τθ k +(1-τ)θ' k

[0277] Where, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0278] The MADDPG algorithm stabilizes the training solution of the Actor network and the Critic network by introducing the target network. The Actor network representing the target network, an Actor network representing a target network of the first agent, an Actor network representing a target network of the second agent, an Actor network representing a target network of the Nth agent, θ'1 represents an Actor network parameter of the target network of the first agent, θ'2 represents an Actor network parameter of the target network of the second agent, and θ' N an Actor network parameter of the target network of the Nth agent, the Actor network parameter of the target network of the kth agent being θ' k The parameters of the Critic network are constantly updated during training, and the target network does not participate in the training process.

[0279] In the present application, each agent belongs to a complete cooperative relationship, and each agent minimizes the objective function through mutual coordination, and each agent shares a global reward function. Therefore, the actions of all agents will affect the overall reward function, and the global reward function will guide the policy optimization of each agent.

[0280] After being trained and solved by the MADDPG algorithm, each agent's policy network will obtain a corresponding power outage action value (obtained from the Actor network output), and the power grid power outage scheduling strategy is a combination of these power outage action values. These power outage action values are issued to the agent, and the agent corresponds to execute the corresponding power outage action, adjust the power outage duration and the number of power outage of the generator set, line and transformer, realize the power grid power outage plan scheduling, and ensure the safe and stable operation of the power grid and its equipment. At the same time, it also helps to prolong the service life of the equipment.

[0281] Embodiment 2

[0282] As shown in the power grid power outage scheduling method of multi-agent deep reinforcement learning, Figure 3 comprises:

[0283] convert the power grid power outage plan scheduling problem into a multi-agent Markov decision process model;

[0284] Under the set constraints of the power grid power outage scheduling, the multi-agent deep deterministic policy gradient algorithm is used to train and solve the multi-agent Markov decision process model, and the power grid power outage scheduling strategy is obtained.

[0285] Since the power outage planning strategy of power equipment directly affects the reliability and economy of the power system, the present invention realizes the safe and stable operation of the power grid by scheduling the power outage of power equipment. The power outage planning refers to the reasonable arrangement of the power outage sequence and time of power equipment under certain constraints, so that the power system can minimize the impact on system reliability, economy and safety when undergoing maintenance, overhaul or expansion.

[0286] Among them, in the problem of power grid outage planning, each generator set, each line and each transformer in the power grid is regarded as a separate intelligent agent, that is, there will be multiple intelligent agents in the power grid.

[0287] In the optimal case, the process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of the tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N represents the number of agents; S represents the grid state set; [a1,a2,…,a N ] represents the power outage action set of all agents, where a1 represents the power outage action set of the first agent, a2 represents the power outage action set of the second agent, ..., a N represents the set of all power outage actions of the Nth agent (usually the set of all power outage actions of an agent will include multiple power outage actions); [r1, r2, ... r N ] represents the set of reward values ​​obtained by all agents after executing the power outage action, r1 represents the reward value obtained by the first agent after executing the power outage action a1, r2 represents the reward value obtained by the second agent after executing the power outage action a2, ..., r N Indicates that the Nth agent performs power outage action a N The reward value obtained after T represents the state transition function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' represents the power outage action A=[a1,a2,…,a N] and then transfer to the grid state set at the next moment; γ represents the discount rate, which indicates the attention paid to future rewards. The larger γ is, the more attention is paid to future rewards (rewards include current rewards and future rewards: current rewards refer to the rewards that the power grid environment will give back to the agent after the agent acts on the power outage environment; future rewards refer to the rewards that the agent may get in the future (the rewards that the agent may get in the future depend on the impact of its current actions on the environment and the dynamic evolution characteristics of the environment. The actions of the agent will affect the entire future reward sequence, not the immediate reward), but this reward is not certain whether it is good or bad. The purpose of introducing the discount rate is to indicate whether the agent pays more attention to current rewards or has higher expectations for future rewards; in specific cases, rewards refer to the negative value of the objective function, including power outage frequency, economic losses from power outages, and power outage balance. The agent will be rewarded after executing the power outage plan action).

[0288] In this invention, the grid state is composed of the states of the intelligent agents in the grid; the state refers to the output of the generator set, the operating temperature of the line, or the transformation ratio of the transformer, etc. The power outage action refers to the duration and number of power outages, for example, one generator set needs to be shut down for 30 minutes.

[0289] The state transition function in this invention refers to the power outage action A=[a1,a2,…,a N ] is the probability distribution of the power grid state at the next moment after the state transition; the state transition refers to the generator set, line and transformer (i.e., intelligent body), which adjusts its own power outage duration and number according to the power outage scheduling strategy based on the current environmental state (i.e., the current power grid state), which will change the power flow distribution of the power grid, and the daily average output of the unit, the daily average voltage of the node and the daily average transformation ratio of the transformer will be transferred to the state at the next moment.

[0290] Preferably, the specific operations of converting the power grid outage planning problem into a multi-agent Markov decision process model include:

[0291] Determine the objective function F(P) in the power grid outage scheduling problem (in power grid outage scheduling, power supply security is the primary consideration, that is, the fewer devices that are out of power, the better, to reduce the impact of power outages on users and improve the reliability and continuity of power supply; at the same time, the economic cost of power outages should be reduced as much as possible; finally, to improve power grid stability and avoid the risks of power outages to power grid operation, the power outage plans for each time period should be as balanced as possible; the present invention takes these three aspects into consideration and determines the objective function in the power grid outage scheduling problem based on them). The expression is:

[0292]

[0293] C i=∑c t

[0294]

[0295] Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ];p i represents the i-th power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan, i = 1, ..., n, that is, all outage plans {p1, ..., p n There are n power outage plans in total (these power outage plans are pending power outage plans that have not yet been implemented); T u Represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period (because power outages affect the daily electricity consumption of electricity users, which will cause economic losses to users; in addition, when the equipment is out of power, it will be repaired and maintained, which will also generate certain economic consumption. Therefore, power outages will cause certain economic losses); c t represents the economic cost of each day in the power outage cycle (for example, a power outage plan p i The daily economic loss during the power outage period is {c1, c2, ...}. The power outage period is a period of time, such as 1 month. Within the 30 days of this month, there will be an economic loss of c every day. t , t=1,2,…,30, by multiple c t The total economic loss during the power outage period (economic loss refers to economic cost); T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily power outage plans, which is used to measure the balance of power outages within the time period; Indicates the power outage plan p i The execution start date; Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object (i.e., intelligent agent) on day t.

[0296] The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions;

[0297] The expression of the state variable is:

[0298]

[0299] In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the sign of the transposed matrix.

[0300] The state set of the generator set includes:

[0301]

[0302] Where, P v (t) represents the daily average active power output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Represents Q v The transposed matrix of (t).

[0303] The state set of a line includes:

[0304]

[0305] Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the average daily power transmission of the vth line on the tth day (i.e., the average daily power transmission of the vth line on the tth day, i.e., the current on the transmission line), Indicates F v The transposed matrix of (t).

[0306] The state set of a transformer includes:

[0307]

[0308] Where R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents Rv The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average daily load rate of the vth transformer on the tth day (the ratio of the current load to the rated capacity), Indicates H v The transposed matrix of (t).

[0309] The expression of the action variable is:

[0310]

[0311] Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, S GE,v (t)=1 means the vth generator set is out of power on the tth day, otherwise S GE,v (t) = 0, Indicates S GE, The transposed matrix of v(t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, S LI,v (t)=1 means that the vth line has power outage on the tth day, otherwise S LI,v (t) = 0, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, S TR,v (T)=1 means the vth transformer is out of power on the tth day, otherwise S TR,v (t) = 0, Indicates S TR,v The transposed matrix of (t).

[0312] The expression of the reward function is:

[0313]

[0314] In the formula, r represents a reward function (a reward value is calculated according to the reward function); ω1 represents a weight coefficient of total power outage; ω2 represents a weight coefficient of power outage economic cost; and ω3 represents a weight coefficient of power outage balance degree.

[0315] In the application, each agent is in a complete cooperation relationship, and the power outage quantity and power outage duration of each agent are determined through mutual coordination to ensure minimization of the target function, so that the agents share the reward.

[0316] In the power grid power outage scheduling problem, the safety of power grid operation and the rationality of power outage scheduling are considered, and related constraints are introduced. In an optimal case, the related constraints of power outage scheduling of the power grid include safety constraints and rationality constraints.

[0317] The related constraints of power outage scheduling of the power grid mainly serve to constrain the agents. The safety constraints are used to ensure the safety of the agents, such as the output of the generator set cannot be too high, the working temperature of the line cannot be too high, etc. The rationality constraints are mainly the rationality of the power outage of the agents, such as two generator sets cannot be powered off at the same time, etc. Through the introduction of the constraint conditions, the safety of the agents and the rationality of the actions can be ensured.

[0318] Further, the safety constraints include power balance constraints, unit output constraints, node voltage constraints, line power flow constraints, transformer voltage ratio constraints, winding temperature constraints and load rate constraints.

[0319] The expression of the power balance constraint is:

[0320]

[0321] In the formula, D(t) represents the daily average load demand of the system on the tth day; N G represents the number of generator sets;

[0322] The expression of the unit output constraint is:

[0323]

[0324] In the formula, represents the upper limit of the daily average active power output of the vth generator set on the tth day; represents the lower limit of the daily average active power output of the vth generator set on the tth day; represents the upper limit of the daily average reactive power output of the vth generator set on the tth day; represents the lower limit of the daily average reactive power output of the vth generator set on the tth day.

[0325] The expression of the node voltage constraint is:

[0326]

[0327] Where, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; It represents the lower limit of the daily average terminal voltage of the vth line on the tth day.

[0328] The expression of line power flow constraint is:

[0329]

[0330] Where, represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day.

[0331] The expression of transformer voltage ratio constraint is:

[0332]

[0333] Where, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day.

[0334] The expression of winding temperature constraint is:

[0335]

[0336] Where, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of transformer v on day t.

[0337] The expression of load rate constraint is:

[0338]

[0339] Where, represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

[0340] In a preferred embodiment of the present invention, rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints, and maintenance special event constraints.

[0341] An unchangeable plan refers to situations where the power outage schedule cannot be altered due to major events, such as those requiring guaranteed power supply. The schedule is determined based on the subjective wishes of the planner. This unchangeable plan is typically included in the power outage plan P.

[0342] Furthermore, the expression of the unchangeable plan constraint is:

[0343]

[0344] Where, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an unchangeable planned power outage; Indicates the start time of the power outage of this irreversible plan; Indicates the end time of the power outage for this irreversible plan.

[0345] To avoid islanding in the power grid system, the planned combination of features cannot be executed on the same day. p and p q All of them belong to the power outage plan P, where the corresponding power outage objects on day t are z p (t), z q (t), then

[0346] The expression of the mutual exclusion constraint is:

[0347] z p (t)·z q (t) = 0

[0348] Where z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day (z p (t) and z q (t) represents two different power outage objects, that is, to indicate that the two power outage objects cannot be shut down at the same time, for example, generator set 1 and line 2 cannot be shut down at the same time, or transformer 1 and transformer 2 cannot be shut down at the same time, etc.), z p (t)∈z i (t), z q (t)∈z i (t).

[0349] According to the specified power outage plan association relationship, the execution time windows of the power outage plans that need to be executed at the same time should overlap as much as possible. h 、p l The same power outage plan P, where the corresponding power outage object on day t is z h(t), z l (t), then the expression of the same-stop relation constraint is:

[0350] z h (t)·z l (t)=1

[0351] Where z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t), same as z h (t) and z l (t) refers to two different power outage objects, which need to be shut down for maintenance at the same time, such as generator set 1 and transformer 1 need to be shut down for maintenance at the same time, line 2 and line 3 need to be shut down for maintenance at the same time, etc.

[0352] The above-mentioned mutually exclusive constraint and simultaneous shutdown constraint are two different constraints. Since there are many power equipment, some equipment cannot be shut down at the same time, while some equipment can be shut down for maintenance at the same time, so relevant constraints are given.

[0353] To reduce the number of power outages and improve user satisfaction, the equipment outage duration must not exceed the maximum number of days in the month. Therefore, the expression for the maintenance decision variable constraint is:

[0354] 1≤X v ≤N dm And X v ∈N dm

[0355] Where, X v is the power outage duration of the vth device (i.e., the power outage object); N dm Indicates the number of days in the month to be analyzed (the month with a planned power outage).

[0356] Since the number of maintenance personnel is limited and the number of maintenance operations for the same type of equipment on the same day is limited, human resource constraints need to be considered in the power outage plan decision-making. Therefore, the expression of the maintenance human resource constraint is:

[0357]

[0358] Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LIIndicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day;

[0359] Considering that the occurrence of special events (bad weather, special holidays) will result in a period of time during the planning cycle where power outage maintenance is not allowed, in order to improve the feasibility of the power outage plan, the expression of the maintenance special event constraint is:

[0360]

[0361] Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

[0362] The multi-agent deep deterministic policy gradient (MADDPG) algorithm used in this paper extends the deep deterministic policy gradient (DDPG) algorithm used in single-agent deep reinforcement learning to multi-agent environments. MADDPG employs a "centralized training, decentralized execution" framework, making it adaptable to complex multi-agent environments.

[0363] The MADDPG algorithm is based on the Actor-Critic framework. In the MADDPG algorithm, each agent has its own policy (Actor) network and evaluation (Critic) network. The Actor network is used to select power outage actions, and the Critic network is used to evaluate the value of the selected power outage actions.

[0364] The current state of the power grid at time t s t Input into the Actor network of each agent, the Actor network will explore and obtain the power outage action of the agent. After the power outage action acts on the power grid environment, a new power grid state and the current reward will be obtained (after the power outage action of the agent acts on the power grid environment, the environmental state (power grid state) will definitely change, so the new environmental state will be fed back to the agent, and the reward will also be fed back to the agent. The agent can judge the quality of its own power outage action based on the quality of the reward). The state, power outage action, new state and reward of the agent are stored as experience in its own experience replay pool; therefore, when using the MADDPG algorithm to train and solve the multi-agent Markov decision process model, the current power grid state s at time t is used. t Input into an agent to obtain the power outage action a at the current time t t (i.e., the duration and number of power outages of the agent), execute the power outage action a at the current time t t After that, the grid status is changed from st Transfer to s', that is, the power grid state is transferred from the current state at time t to the state at the next time (time t+1), and the reward r of the feedback in the current environment at time t is obtained t , and feed back the grid state transfer and reward to the agent. The agent continuously interacts with the grid environment and t ,a t ,r t ,s'} is stored in its own experience replay pool (in the MADDPG algorithm, the number of agents is N, and each agent has an experience replay pool, so N agents correspond to N experience replay pools, and each experience replay pool also stores the corresponding agent's {s t ,a t ,r t ,s'} data), before updating the parameters in the network each time, a set of {s t ,a t ,r t ,s'} data to train the neural network of the agent (the policy network and the evaluation network are composed of neural networks. The neural network refers to the fully connected neural network in deep learning. When training the neural network of each agent, a set of {s t ,a t ,r t ,s'} data to train), and the new {s t ,a t ,r t ,s'} back into the experience replay pool, and then randomly selected for training) until the reward feedback under the current power grid environment is high and tends to be stable, stop training (in the process of training all agents, if an agent reaches the optimal state, you can stop training for it, and other agents continue to train. In the centralized training stage, if an agent stops training, you can freeze the strategy of the agent to avoid affecting other agents. The present invention takes the optimal overall reward as the optimization goal, and all agents cooperate with each other. When the overall reward of each centralized training (the sum of the rewards of each agent) is large and does not change significantly, the reward at this time can be considered to be optimal and stable. At this time, stop training for all agents and the training ends). After the training is completed, the strategy network of each agent will output an action value, which is the optimal strategy (these action values ​​together constitute the power grid outage scheduling strategy). This action value will be sent to the agent to let the agent perform the action, such as stopping 5 generators, each for 30 minutes.

[0365] The multi-agent deep deterministic policy gradient algorithm process in this invention is as follows: Figure 2 shown.

[0366] The ultimate goal of deep reinforcement learning in this invention is to find the final joint control strategy to maximize the cumulative reward.

[0367] In the MADDPG algorithm, there are N agents, so it contains N Actor networks and N Critic networks, where the parameter set of N Actor networks is θ = {θ1, θ2, …, θ N}, θ represents the set of policy network parameters; θ1 represents the parameters of the policy network of the first agent, θ2 represents the parameters of the policy network of the second agent, ..., θ N represents the parameters of the policy network of the Nth agent. During deep neural network training, the general policy gradient of the kth agent can be expressed as (the following formula is applicable to random policies):

[0368]

[0369] Where, represents the policy gradient of the kth agent; logμ k Represents the strategy μ k Take the logarithm calculation; Qk(s k ,a k1 ,a k2 ,…,a km ) is the kth centralized value function (centralization means that this value function depends not only on the local information of a certain agent, but also on the state and action information of all agents), and its input is the action taken by the agent based on the policy (Actor) network (the action taken by the policy network is the power outage action) a k and the current state s k , whose output is the Q value of the kth agent;

[0370] In addition to the Actor network and the Critic network, the intelligent agent also has a target-Actor network (the strategy (Actor) network of the target network) and a target-Critic network. k 'Input into the target-Actor network of the current agent k to get action a k '(s k 'Indicates the state of the kth agent at the next moment. The kth agent is based on the current state s k , execute power outage action a k After that, the state will be changed from s k Transformed into s k '), and S'=[s'1,s'2,…,s' N](s'1 represents the state of the first agent at the next moment. After the first agent executes the power outage action a1 based on the current state s1, the state will change from s1 to s1'; s'2 represents the state of the second agent at the next moment. After the second agent executes the power outage action a2 based on the current state s2, the state will change from s2 to s2', ..., s' N Indicates the state of the Nth agent at the next moment. The Nth agent is based on the current state s N , execute power outage action a N After that, the state will be changed from s N Transformed into s N ') and A'=[a'1,a'2,...,a' N ] is input into the target-Critic network to obtain the Q at the next moment k '; A' represents the power outage action set of all agents at the next moment, a'1 represents the power outage action set of the first agent at the next moment, a'2 represents the power outage action set of the second agent at the next moment, ..., a' N represents the set of all power outage actions of the Nth agent at the next moment. After the agent executes the power outage action A according to the current grid state S, the state will transfer from S to S', and the power outage action A' will be executed in state S'; Q k ' represents the target-critic network's evaluation of the power outage action A'. Since the power outage action A is executed under the grid state S, the target-critic network calculates the value Q of the power outage action A. Then, at the next grid state S', the evaluation of the power outage action A' at the next moment is Q k ', Q k ' refers to the action value of the kth agent in the grid state set S' and the power outage action set A'.

[0371] Based on this, under the constraints of the set power outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include:

[0372] Determine the policy gradient of the kth agent, expressed as (in this invention, since the MADDPG algorithm uses stochastic gradient descent to update the Actor network of each agent, a deterministic strategy is used, so the actual calculation uses the following formula):

[0373]

[0374] Where, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a krepresents the set of all power outage actions of the kth agent, a k There are m power outage actions in total; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, ..., a N ]~B represents power failure action [a1,a2,…,a N ] obeys the distribution of B (the experience replay pool contains strategies).

[0375] Define the loss function of the evaluation network, the expression is:

[0376]

[0377] Where, L(θ k ) is the loss function of the evaluation network of the k-th agent; Indicates the k ,a k ,r k ,s, k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a′ k1 represents the first power outage action performed by the kth agent at the next moment; a′ k2 represents the second power outage action performed by the kth agent at the next moment; a′ km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q′k(s k ,a′ k1 ,a′ k2 ,…,a′ km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network (μ′ k represents the policy of the goal-policy network of the k-th agent).

[0378] The parameters of the target network are updated using the soft update method, and the expression is:

[0379]

[0380] θ' k =τθ k +(1-τ)θ' k

[0381] Where, represents the parameters of the target-critic network of the kth agent; represents the parameters of the evaluation network of the kth agent; θ′ k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

[0382] The MADDPG algorithm stabilizes the training solution of the Actor network and the Critic network by introducing the target network. The Actor network representing the target network, The Actor network representing the target network of the first agent, Actor network representing the target network of the second agent, ..., represents the Actor network of the target network of the Nth agent, θ'1 represents the Actor network parameters of the target network of the first agent, θ'2 represents the Actor network parameters of the target network of the second agent, ..., θ' N The Actor network parameters of the target network of the Nth agent are θ', and the Actor network parameters of the target network of the kth agent are θ' k The parameters of the critic network are continuously updated during training, and the target network does not participate in the training process.

[0383] In this paper, each agent is in a fully cooperative relationship, coordinating to minimize the objective function, and each agent shares a global reward function. Therefore, the actions of all agents affect the overall reward function, and the global reward function guides the optimization of each agent's strategy.

[0384] After training and solving the MADDPG algorithm, each agent's strategy network will obtain the corresponding power outage action value (obtained by the Actor network output). The power grid outage scheduling strategy is the combination of these power outage action values. These power outage action values ​​are sent to the agents, and the agents will execute the corresponding power outage actions. By adjusting the power outage duration and number of power outages for generators, lines, and transformers, the power grid outage plan is implemented, ensuring the safe and stable operation of the power grid and its equipment, while also helping to extend the service life of the equipment.

[0385] Example 3

[0386] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of embodiment 2 when executed by a processor.

[0387] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0388] The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited thereto. Within the technical concept of the present invention, various simple variations of the technical solution of the present invention may be made, including combining the various technical features in any other appropriate manner. These simple variations and combinations should also be regarded as disclosed in the present invention and fall within the scope of protection of the present invention.

Claims

1. A multi-agent deep reinforcement learning power outage scheduling system, characterized by: Includes choreography problem transformation module and decision-making solution module; The scheduling problem conversion module is used to convert the power grid outage scheduling problem into a multi-agent Markov decision process model; The decision-solving module is used to train and solve the multi-agent Markov decision process model using a multi-agent deep deterministic policy gradient algorithm under the relevant constraints of the set power grid outage scheduling to obtain the power grid outage scheduling strategy; The relevant constraints of the set power grid outage scheduling include safety constraints and rationality constraints; The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line power flow constraints, transformer voltage ratio constraints, winding temperature constraints, and load rate constraints; The rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints and maintenance special event constraints.

2. The multi-agent deep reinforcement learning power outage scheduling system according to claim 1 is characterized in that: The process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of a tuple: {N,S,a1,a2,…,a N ,r1,r2,…,r N ,F T ,γ}; where N represents the number of agents; S represents the grid state set; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…,r N ] represents the reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transition function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' represents the power outage action A=[a1,a2,…,a N ] and transfers to the grid state set at the next moment; γ represents the discount rate.

3. The multi-agent deep reinforcement learning power outage scheduling system according to claim 1 or 2, characterized in that: The specific operations of transforming the power outage scheduling problem into a multi-agent Markov decision process model include: Determine the objective function F(P) in the power grid outage scheduling problem, which is expressed as: C i =∑c t Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ];p i represents the i-th power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan scheme, i = 1, 2, ..., n; T u Represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily scheduled power outages; Indicates the power outage plan p i The execution start date; Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object on day t; The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions; The expression of the state variable is: In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed symbol; The state set of the generator set includes: Where, P v (t) represents the daily average active power output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Represents Q v The transposed matrix of (t); The state set of a line includes: Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t); The state set of a transformer includes: Where R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average daily load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t); The expression of the action variable is: Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t); The expression of the reward function is: Where r represents the reward function; ω1 represents the weight coefficient of the total outage amount; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage amount.

4. The multi-agent deep reinforcement learning power outage scheduling system according to claim 1, characterized in that: The expression of power balance constraint is: Where D(t) represents the daily average load demand of the system on day t; N G Indicates the number of generator sets; The expression of unit output constraint is: Where, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; represents the upper limit of the daily average reactive output of the vth generator set on day t; It represents the lower limit of the daily average reactive output of the vth generator set on the tth day; The expression of the node voltage constraint is: Where, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; represents the lower limit of the daily average terminal voltage of the vth line on the tth day; The expression of line power flow constraint is: Where, represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day; The expression of transformer voltage ratio constraint is: Where, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day; The expression of winding temperature constraint is: Where, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day; The expression of load rate constraint is: Where, represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

5. The multi-agent deep reinforcement learning power grid outage scheduling system according to claim 4 is characterized in that: The expression of the unchangeable plan constraint is: Where, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an unchangeable planned power outage; Indicates the start time of the power outage of this irreversible plan; Indicates the end time of the power outage of this irreversible plan; The expression of the mutual exclusion constraint is: z p (t)·z q (t)=0 Where z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t); The expression of the same-stop relation constraint is: z h (t)·z l (t)=1 Where z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t); The expression of maintenance decision variable constraint is: 1≤X v ≤N dm And X v ∈N dm Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed; The expression of maintenance human resource constraint is: Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day; The expression for overhauling the special event constraint is: Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

6. The multi-agent deep reinforcement learning power outage scheduling system according to claim 1, characterized in that: Under the constraints of the set power outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include: Determine the policy gradient of the kth agent, expressed as: Where, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, ..., a N ]~B represents power failure action [a1,a2,…,a N ] obeys the distribution of B; Define the loss function of the evaluation network, the expression is: Where, L(θ k ) is the loss function of the evaluation network of the k-th agent; Indicates the k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a' k1 represents the first power outage action performed by the kth agent at the next moment; a' k2 represents the second power outage action performed by the kth agent at the next moment; a' km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q' k (s k ,a' k1 ,a' k2 ,…,a' km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network; The parameters of the target network are updated using the soft update method, and the expression is: I will k =tθ k +(1-τ)θ' k Where, represents the parameters of the target-critic network of the kth agent; Represents the parameters of the evaluation network of the kth agent; θ' k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

7. A multi-agent deep reinforcement learning method for power grid outage scheduling, characterized by: include: The power grid outage scheduling problem is transformed into a multi-agent Markov decision process model; Under the constraints of the set power outage scheduling, the multi-agent Markov decision process model is trained and solved using the multi-agent deep deterministic policy gradient algorithm to obtain the power outage scheduling strategy. The relevant constraints of the set power grid outage scheduling include safety constraints and rationality constraints; The safety constraints include power balance constraints, unit output constraints, node voltage constraints, line power flow constraints, transformer voltage ratio constraints, winding temperature constraints, and load rate constraints; The rationality constraints include unchangeable plan constraints, mutually exclusive relationship constraints, simultaneous stop relationship constraints, maintenance decision variable constraints, maintenance human resource constraints and maintenance special event constraints.

8. The multi-agent deep reinforcement learning power outage scheduling method according to claim 7 is characterized in that: The process of transforming the power outage scheduling problem into a multi-agent Markov decision process model consists of a tuple: {N,S,a1,a2,…a N ,r1,r2,…r N ,F T ,γ}; where N represents the number of agents; S represents the grid state set; [a1,a2,…,a N ] represents the power outage action set of all agents, a N represents the set of all power outage actions of the Nth agent; [r1,r2,…r N ] represents the reward value set obtained by all agents after executing the power outage action, r N Indicates that the Nth agent performs the power outage action a N The reward value obtained after T represents the state transition function, F T :S×a1×a2,…,×a N ×S'→[0,1], S' represents the power outage action A=[a1,a2,…,a N ] and transfers to the grid state set at the next moment; γ represents the discount rate.

9. The multi-agent deep reinforcement learning power outage scheduling method according to claim 7 or 8, characterized in that: The specific operations of transforming the power outage scheduling problem into a multi-agent Markov decision process model include: Determine the objective function F(P) in the power grid outage scheduling problem, which is expressed as: C i =∑c t Where P represents the power outage plan, all power outage plans p i The power outage plan P = [p1,…,p n ];p i represents the i-th power outage plan, p n represents the nth outage plan, n represents the total number of outage plans in the outage plan scheme, i = 1, 2, ..., n; T u Represents the total number of users; j represents the user number, j = 1, 2, ..., T u ; represents the total power outage power after the jth user executes the power outage plan; T otg Indicates the duration of power outage; C i represents the total economic cost during the power outage period; c t represents the economic cost of each day during the power outage period; T L Indicates the length of the entire power outage period; M represents the power grid; represents the variance of the number of daily scheduled power outages; Indicates the power outage plan p i The execution start date; Indicates the power outage plan p i The execution termination date; i (t) represents the power outage object on day t; The relevant variables in the multi-agent Markov decision process model include state variables, action variables, reward functions, and state transition functions; The expression of the state variable is: In the formula, GE v (t) represents the state set of the vth generator on the tth day, Indicates GE v The transposed matrix of (t); LI v (t) represents the state set of the vth line on the tth day, Indicates LI v The transposed matrix of (t); TR v (t) represents the state set of the v-th transformer on the t-th day, Represents TR v The transposed matrix of (t); T is the transposed symbol; The state set of the generator set includes: Where, P v (t) represents the daily average active power output of the vth generator set on the tth day, Indicates P v The transposed matrix of (t); Q v (t) represents the daily average reactive power output of the vth generator set on the tth day, Represents Q v The transposed matrix of (t); The state set of a line includes: Where: V v (t) represents the daily average terminal voltage of the vth line on the tth day, Indicates V v The transposed matrix of (t); F v (t) represents the daily average power transmission of the vth line on the tth day, Indicates F v The transposed matrix of (t); The state set of a transformer includes: Where R v (t) represents the daily average voltage transformation ratio of the vth transformer on the tth day, Represents R v The transposed matrix of (t); W v (t) represents the daily average winding temperature of the vth transformer on the tth day, W v The transposed matrix of (t); H v (t) represents the average daily load rate of the vth transformer on the tth day, Indicates H v The transposed matrix of (t); The expression of the action variable is: Where A(t) represents the action variable space of day t, D GE,v (t) represents the duration of the power outage of the vth generator set starting on the tth day, Indicates D GE,v The transposed matrix of (t); S GE,v (t) represents the power outage status of the vth generator set on the tth day, Indicates S GE,v The transposed matrix of (t); D LI,v (t) represents the duration of the power outage of the vth line starting on the tth day, Indicates D LI,v The transposed matrix of (t); S LI,v (t) represents the power outage status of the vth line on the tth day, Indicates S LI,v The transposed matrix of (t); D TR,v (t) represents the duration of the power outage of the vth transformer starting on the tth day, Indicates D TR,v The transposed matrix of (t); S TR,v (t) represents the power outage status of the vth transformer on the tth day, Indicates S TR,v The transposed matrix of (t); The expression of the reward function is: Where r represents the reward function; ω1 represents the weight coefficient of the total outage amount; ω2 represents the weight coefficient of the economic cost of outage; and ω3 represents the weight coefficient of the balance degree of outage amount.

10. The multi-agent deep reinforcement learning power outage scheduling method according to claim 7, characterized in that: The expression of power balance constraint is: Where D(t) represents the daily average load demand of the system on day t; N G Indicates the number of generator sets; The expression of unit output constraint is: Where, It represents the upper limit of the daily average active output of the vth generator set on the tth day; It represents the lower limit of the daily average active output of the vth generator set on the tth day; represents the upper limit of the daily average reactive output of the vth generator set on day t; It represents the lower limit of the daily average reactive output of the vth generator set on the tth day; The expression of the node voltage constraint is: Where, represents the upper limit of the daily average terminal voltage of the vth line on the tth day; represents the lower limit of the daily average terminal voltage of the vth line on the tth day; The expression of line power flow constraint is: Where, represents the upper limit of the daily average power flow value of the vth line on the tth day; It represents the lower limit of the daily average power flow value of the vth line on the tth day; The expression of transformer voltage ratio constraint is: Where, It represents the upper limit of the daily average voltage transformation ratio of the vth transformer on the tth day; It represents the lower limit of the daily average voltage transformation ratio of the vth transformer on the tth day; The expression of winding temperature constraint is: Where, It represents the upper limit of the daily average winding temperature of the vth transformer on the tth day; It represents the lower limit of the daily average winding temperature of the vth transformer on the tth day; The expression of load rate constraint is: Where, represents the upper limit of the daily average load rate of the v-th transformer on the t-th day; It represents the lower limit of the daily average load rate of the vth transformer on the tth day.

11. The multi-agent deep reinforcement learning power outage scheduling method according to claim 10, characterized in that: The expression of the unchangeable plan constraint is: Where, NC0(t d ) indicates an irreversible plan; d Indicates the duration of an unchangeable planned power outage; Indicates the start time of the power outage of this irreversible plan; Indicates the end time of the power outage of this irreversible plan; The expression of the mutual exclusion constraint is: z p (t)·z q (t)=0 Where z p (t) represents the pth power outage object on the tth day, z q (t) represents the qth power outage object on the tth day; z p (t)∈z i (t), z q (t)∈z i (t); The expression of the same-stop relation constraint is: z h (t)·z l (t)=1 Where z h (t) represents the hth power outage object on the tth day, z l (t) represents the lth power outage object on the tth day; z h (t)∈z i (t), z l (t)∈z i (t); The expression of maintenance decision variable constraint is: 1≤X v ≤N dm And X v ∈N dm Where, X v is the power outage duration of the vth device; N dm Indicates the day of the month to be analyzed; The expression of maintenance human resource constraint is: Where M GE Indicates the number of maintenance of the generator set; M LI Indicates the number of line inspections; M TR Indicates the number of transformer inspections; n GE Indicates the maximum number of generator sets that can be overhauled on the same day; n LI Indicates the maximum number of lines that can be repaired on the same day; n TR Indicates the maximum number of transformers that can be repaired on the same day; The expression for overhauling the special event constraint is: Where X represents the time set corresponding to the power outage maintenance plan; Y represents the time set corresponding to the special event.

12. The multi-agent deep reinforcement learning power outage scheduling method according to claim 7, characterized in that: Under the constraints of the set power outage scheduling, the specific operations of training and solving the multi-agent Markov decision process model using the multi-agent deep deterministic policy gradient algorithm include: Determine the policy gradient of the kth agent, expressed as: Where, represents the policy gradient of the kth agent, k∈N; μ k represents the strategy of the kth agent; a k represents the set of all power outage actions of the kth agent; θ k represents the parameters of the policy network of the kth agent; represents the gradient of the policy network parameters of the kth agent; s k represents the current state of the kth agent; a k1 represents the first power outage action performed by the kth agent; a k2 represents the second power outage action performed by the kth agent; a km represents the mth power outage action performed by the kth agent; μ k (s k ) indicates that the kth agent is in the current state s k The strategy under; B represents the experience replay pool; represents the value function of the evaluation network of the kth agent; E represents the mathematical expectation; [a1, a2, ..., a N ]~B represents power failure action [a1,a2,…,a N ] obeys the distribution of B; Define the loss function of the evaluation network, the expression is: Where, L(θ k ) is the loss function of the evaluation network of the k-th agent; Indicates the k ,a k ,r k ,s' k ) is weighted average of the joint distribution of k represents the reward of the kth agent; a k ' represents the set of all power outage actions of the kth agent at the next moment; a' k1 represents the first power outage action performed by the kth agent at the next moment; a' k2 represents the second power outage action performed by the kth agent at the next moment; a' km represents the mth power outage action performed by the kth agent at the next moment; y represents the target value; Q' k (s k ,a' k1 ,a' k2 ,…,a' km ) represents the value function of the target-evaluation network of the kth agent; s' k represents the state of the kth agent at the next moment; μ' k (s' k ) represents the state s' of the kth agent at the next moment k The strategy of the target-strategy network; The parameters of the target network are updated using the soft update method, and the expression is: I will k =tθ k +(1-τ)θ' k Where, represents the parameters of the target-critic network of the kth agent; Represents the parameters of the evaluation network of the kth agent; θ' k represents the parameters of the target-policy network of the kth agent; τ represents the update step size.

13. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 7 to 12 are implemented.

Images