Prevention control method and device based on imitation and reinforcement learning, medium and equipment

Abstract

The invention discloses a prevention and control method and device based on imitation and reinforcement learning, a medium and equipment. According to the method, the real-time state information of the power system is acquired and input to the third agent, the third agent is obtained by training the second agent through the simulation environment, and the second agent is pre-trained from the first agent based on expert data imitation learning. And the control signal output by the third agent is utilized to dynamically adjust the power system equipment, so that the safe and stable operation of the system is ensured, and the reliability of a prevention and control strategy based on simulation and reinforcement learning is improved.

Description

Technical Field

[0001] The present invention relates to the field of preventive control based on imitation and reinforcement learning, and in particular to a preventive control method, device, medium and equipment based on imitation and reinforcement learning. Background Art

[0002] In recent years, the scale of electricity has continued to expand at a high speed, the demand for electricity consumption and the demand for power stability have continued to increase, the grid structure has become increasingly complex, and the traditional power system dominated by synchronous machines is gradually transforming into a new power system dominated by distributed renewable energy. The controllable objects of the new power system have expanded from the source to all aspects of source-grid-load-storage. At present, most of the traditional methods for preventive control based on imitation and reinforcement learning are based on the optimal power flow method considering transient stability. Compared with the traditional power system, the control difficulty and scale of the new power system have increased exponentially, and the traditional methods cannot be directly used due to low computational efficiency. On the other hand, the penetration rate of new multi-distributed power sources such as solar energy and wind energy in the new power system is increasing, but there is a large gap in the support capacity of renewable energy compared with conventional power sources. Under the high penetration of renewable energy and power electronic equipment, the power system presents low inertia and low damping characteristics, which reshapes the stable domain of safe and stable operation of the power system, and faces severe risks and challenges in safe and stable operation. Therefore, it is urgent to propose a real-time and efficient preventive control method based on imitation and reinforcement learning.

[0003] At the same time, in recent years, artificial intelligence technology has developed rapidly and has made certain theoretical breakthroughs in the field of power systems. Compared with traditional transient stability preventive control methods that rely on physical modeling and analysis, artificial intelligence such as deep reinforcement learning technology can perform dimensional calculations and analysis on massive data of power systems, get rid of solving complex nonlinear differential equations, reveal the physical characteristics of transient stability of power systems from a data perspective, and transform large-scale data sets into efficient decision-making solutions. At the moment when the power industry is developing rapidly, it is considered by the industry to be the key to meeting the real-time challenges of preventive control based on imitation and reinforcement learning. However, deep neural networks have problems such as opaque models, unclear meanings of network parameters, and inability to deduce decision-making processes through physical knowledge, which hinder the engineering application of deep reinforcement learning in the field of preventive control based on imitation and reinforcement learning. Summary of the invention

[0004] The present invention provides a preventive control method, device, medium and terminal equipment based on imitation and reinforcement learning to improve the reliability of the preventive control strategy based on imitation and reinforcement learning.

[0005] In a first aspect, the present application provides a preventive control method based on imitation and reinforcement learning, comprising:

[0006] Obtain real-time operating status information of the power system;

[0007] Inputting the real-time operation status information into a third deep reinforcement learning agent so that the third deep reinforcement learning agent outputs a control signal for preventive control;

[0008] The third deep reinforcement learning agent is obtained by interactive training between the second deep reinforcement learning agent and the environment constructed by the power system simulation model; the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set;

[0009] According to the control signal, the output power and switching of the generator and reactive power source in the power system are adjusted to make the power system operate in a safe and stable working state.

[0010] This application ensures the timeliness and accuracy of the control strategy by obtaining the operating status information of the power system in real time. Secondly, this information is input into the first deep reinforcement learning agent pre-trained with expert experience data. The agent masters the decision-making mode of experts through imitation learning, which improves the reliability and efficiency of decision-making. Then, the first agent interacts with the environment constructed by the power system simulation model to form a second agent, and further optimizes and adjusts its decision-making strategy to adapt to the actual operating environment. Finally, the control signal output by the second agent is used to guide the third agent, which integrates real-time status information and trained strategies to output accurate preventive control signals. These signals guide the generators and reactive power sources in the power system to adjust the output power and switching, ensuring that the power system can maintain a safe and stable working state under various operating conditions, thereby effectively preventing the occurrence of power system failures such as large-scale power outages, and improving the stability and reliability of the entire power system.

[0011] As a preferred embodiment of the first aspect, the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on an expert experience data set, specifically:

[0012] According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints, solve the objective function with economic benefits as the optimization goal under multiple different states to obtain multiple expert strategies; wherein N is a positive integer greater than 1;

[0013] Obtaining an expert experience data set by calculating according to the multiple expert strategies;

[0014] Performing imitation learning pre-training on the first deep reinforcement learning agent according to the expert experience data set with the goal of minimizing action strategy errors;

[0015] When the minimized action strategy error converges, the training is stopped to obtain a second deep reinforcement learning agent.

[0016] In this preferred embodiment, the first deep reinforcement learning agent of the present application uses expert experience data sets through imitation learning. These data sets are based on the basic constraints of power system safety and stability, safety and stability constraints under N-1 faults, and transient stability constraints. A plurality of expert strategies are obtained by solving the objective function with economic benefits as the optimization goal. This pre-training method enables the agent to learn efficient decision-making patterns in a complex power system environment. By minimizing the action strategy error through imitation learning, the agent further optimizes its decision-making ability until the error converges, ensuring the decision-making accuracy and reliability of the agent. Ultimately, the trained second deep reinforcement learning agent can output high-quality control signals, which can guide the generators and reactive power sources in the power system to make precise adjustments, ensuring that the system can maintain a safe and stable operating state under various working conditions, effectively preventing the instability and failure of the power system, and improving the operating efficiency and safety of the entire system.

[0017] As a preferred embodiment of the first aspect, according to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults, and the transient stability constraints, the objective function with economic benefits as the optimization target under multiple different states is solved to obtain multiple expert strategies; the expert experience data set is calculated according to the multiple expert strategies, specifically:

[0018] Taking the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints as optimal power flow constraints, the generators, lines, buses and other components in the optimization objective are limited to operate within a specified range;

[0019] According to the optimal power flow constraint conditions, solving the objective function with economic benefit as the optimization target, and obtaining multiple optimal power flow results of the power system under multiple operating states;

[0020] Calculate the multi-step decision actions of the multiple optimal power flow results through a difference sampling algorithm to obtain multiple expert strategy trajectories; wherein each expert strategy trajectory includes a state sequence and an action sequence;

[0021] The plurality of expert trajectories constitute the expert experience data set.

[0022] In this preferred embodiment, the present application comprehensively considers the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults, and the transient stability constraints as the constraints of the optimal power flow, ensuring that the various components of the power system such as generators, lines, and buses operate within a safe and stable working range. Then, under these constraints, the objective function with economic benefits as the optimization goal is solved to obtain the optimal power flow results of the power system under different operating conditions. Next, the difference sampling algorithm is used to calculate the multi-step decision actions of these optimal power flow results to form multiple expert strategy trajectories, each of which contains a detailed state sequence and action sequence. These expert strategy trajectories are summarized to form an expert experience data set, which provides a rich learning sample based on the actual operation of the power system for deep reinforcement learning agents. By imitating these expert strategies, the agent can learn how to make the best control decision under various complex working conditions, thereby improving the operating efficiency and stability of the power system in practical applications and reducing the risk of system failures caused by improper operation.

[0023] As a preferred embodiment of the first aspect, the third deep reinforcement learning agent is obtained by interactive training of the second deep reinforcement learning agent with an environment constructed by a power system simulation model, specifically:

[0024] Acquire experience replay component information; wherein the experience replay component information is obtained by initializing the expert experience data set;

[0025] Training a second deep learning agent based on the experience playback component information and optimizing a loss function of the initial deep learning agent;

[0026] When the loss function of the second deep learning agent converges to a preset target value, training is stopped to obtain a third deep learning reinforcement agent.

[0027] In this preferred embodiment, the present application provides a rich data basis close to the actual operation situation for the training of the second deep learning agent by obtaining the experience playback component information initialized by the expert experience data set. Then, using these experience playback information, the second deep learning agent is trained, and its loss function is optimized at the same time, so that the agent can learn the optimal control strategy under various power system conditions more quickly and accurately. During the training process, the agent continuously adjusts and optimizes its decision model until the loss function converges to the preset target value. At this time, the training of the agent reaches the optimal state and forms a third deep learning enhanced agent. This well-trained agent can quickly output accurate preventive control signals according to the real-time operating status of the power system, guide the generators and reactive power supplies in the system to make necessary adjustments, and ensure that the power system can maintain a safe and stable operating state under various complex conditions, effectively prevent potential power system failures, and improve the reliability and economic operation efficiency of the power system.

[0028] In a second aspect, the present application provides a preventive control device based on imitation and reinforcement learning, wherein the preventive control device based on imitation and reinforcement learning includes an acquisition module, an input-output module, and an adjustment module;

[0029] The acquisition module is used to obtain the real-time operation status information of the power system;

[0030] The input-output module is used to input the real-time operation status information into the third deep reinforcement learning agent, so that the third deep reinforcement learning agent outputs a control signal for preventive control;

[0031] The third deep reinforcement learning agent is obtained by interactive training between the second deep reinforcement learning agent and the environment constructed by the power system simulation model; the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set;

[0032] The adjustment module is used to adjust the output power and switching of the generator and reactive power supply in the power system according to the control signal, so that the power system operates in a safe and stable working state.

[0033] This device uses three modules to divide the work and coordinate work to better provide preventive control strategies for the power system. This application ensures the timeliness and accuracy of the control strategy by obtaining the operating status information of the power system in real time. Secondly, this information is input into the first deep reinforcement learning agent pre-trained with expert experience data. The agent masters the decision-making mode of the expert through imitation learning, which improves the reliability and efficiency of the decision. Then, the first agent interacts with the environment constructed by the power system simulation model to form a second agent, and further optimizes and adjusts its decision-making strategy to adapt to the actual operating environment. Finally, the control signal output by the second agent is used to guide the third agent, which integrates the real-time status information and the trained strategy to output accurate preventive control signals. These signals guide the generators and reactive power sources in the power system to adjust the output power and switch, ensuring that the power system can maintain a safe and stable working state under various operating conditions, thereby effectively preventing the occurrence of power system failures such as large-scale power outages, and improving the stability and reliability of the entire power system.

[0034] As a preferred embodiment of the second aspect, the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on an expert experience data set, specifically:

[0035] According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints, solve the objective function with economic benefits as the optimization goal under multiple different states to obtain multiple expert strategies; wherein N is a positive integer greater than 1;

[0036] Obtaining an expert experience data set by calculating according to the multiple expert strategies;

[0037] Performing imitation learning pre-training on the first deep reinforcement learning agent according to the expert experience data set with the goal of minimizing action strategy errors;

[0038] When the minimized action strategy error converges, the training is stopped to obtain a second deep reinforcement learning agent.

[0039] In this preferred embodiment, the first deep reinforcement learning agent of the present application uses expert experience data sets through imitation learning. These data sets are based on the basic constraints of power system safety and stability, safety and stability constraints under N-1 faults, and transient stability constraints. A plurality of expert strategies are obtained by solving the objective function with economic benefits as the optimization goal. This pre-training method enables the agent to learn efficient decision-making patterns in a complex power system environment. By minimizing the action strategy error through imitation learning, the agent further optimizes its decision-making ability until the error converges, ensuring the decision-making accuracy and reliability of the agent. Ultimately, the trained second deep reinforcement learning agent can output high-quality control signals, which can guide the generators and reactive power sources in the power system to make precise adjustments, ensuring that the system can maintain a safe and stable operating state under various working conditions, effectively preventing the instability and failure of the power system, and improving the operating efficiency and safety of the entire system.

[0040] As a preferred embodiment of the second aspect, according to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults, and the transient stability constraints, solving the objective function with economic benefits as the optimization target under multiple different states, and obtaining multiple expert strategies; according to the multiple expert strategies, calculating the expert experience data set, specifically:

[0041] Taking the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints as optimal power flow constraints, the generators, lines, buses and other components in the optimization objective are limited to operate within a specified range;

[0042] According to the optimal power flow constraint conditions, solving the objective function with economic benefit as the optimization target, and obtaining multiple optimal power flow results of the power system under multiple operating states;

[0043] Calculate the multi-step decision actions of the multiple optimal power flow results through a difference sampling algorithm to obtain multiple expert strategy trajectories; wherein each expert strategy trajectory includes a state sequence and an action sequence;

[0044] The plurality of expert trajectories constitute the expert experience data set.

[0045] In this preferred embodiment, the present application comprehensively considers the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults, and the transient stability constraints as the constraints of the optimal power flow, ensuring that the various components of the power system such as generators, lines, and buses operate within a safe and stable working range. Then, under these constraints, the objective function with economic benefits as the optimization goal is solved to obtain the optimal power flow results of the power system under different operating conditions. Next, the difference sampling algorithm is used to calculate the multi-step decision actions of these optimal power flow results to form multiple expert strategy trajectories, each of which contains a detailed state sequence and action sequence. These expert strategy trajectories are summarized to form an expert experience data set, which provides a rich learning sample based on the actual operation of the power system for deep reinforcement learning agents. By imitating these expert strategies, the agent can learn how to make the best control decision under various complex working conditions, thereby improving the operating efficiency and stability of the power system in practical applications and reducing the risk of system failures caused by improper operation.

[0046] As a preferred embodiment of the second aspect, the third deep reinforcement learning agent is obtained by interactive training of the second deep reinforcement learning agent with an environment constructed by a power system simulation model, specifically:

[0047] Acquire experience replay component information; wherein the experience replay component information is obtained by initializing the expert experience data set;

[0048] Training a second deep learning agent based on the experience playback component information and optimizing a loss function of the initial deep learning agent;

[0049] When the loss function of the second deep learning agent converges to a preset target value, training is stopped to obtain a third deep learning reinforcement agent.

[0050] In this preferred embodiment, the present application provides a rich data basis close to the actual operation situation for the training of the second deep learning agent by obtaining the experience playback component information initialized by the expert experience data set. Then, using these experience playback information, the second deep learning agent is trained, and its loss function is optimized at the same time, so that the agent can learn the optimal control strategy under various power system conditions more quickly and accurately. During the training process, the agent continuously adjusts and optimizes its decision model until the loss function converges to the preset target value. At this time, the training of the agent reaches the optimal state and forms a third deep learning enhanced agent. This well-trained agent can quickly output accurate preventive control signals according to the real-time operating status of the power system, guide the generators and reactive power supplies in the system to make necessary adjustments, and ensure that the power system can maintain a safe and stable operating state under various complex conditions, effectively prevent potential power system failures, and improve the reliability and economic operation efficiency of the power system.

[0051] In a third aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium including a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the preventive control method based on imitation and reinforcement learning as described. Its beneficial effects are the same as those of the preventive control method based on imitation and reinforcement learning provided in the first aspect of the present application.

[0052] In a fourth aspect, the present application provides a terminal device comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements any one of the preventive control methods based on imitation and reinforcement learning as described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 : A flow chart of an embodiment of a preventive control method based on imitation and reinforcement learning provided in the present application;

[0054] Figure 2 : Schematic diagram of the process flow generated for the preventive control strategy provided in this application;

[0055] Figure 3 : A structural diagram of an embodiment of a preventive control device based on imitation and reinforcement learning provided in the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Embodiment 1

[0058] Please refer to Figure 1 , which is a preventive control method based on imitation and reinforcement learning provided in an embodiment of the present invention.

[0059] In this embodiment, the process of the preventive control method based on imitation and reinforcement learning in this application is described in detail through steps S01-S03.

[0060] S01: Obtain real-time operation status information of the power system;

[0061] S02: inputting the real-time operation status information into a third deep reinforcement learning agent, so that the third deep reinforcement learning agent outputs a control signal for preventive control;

[0062] The third deep reinforcement learning agent is obtained by interactive training between the second deep reinforcement learning agent and the environment constructed by the power system simulation model; the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set;

[0063] More specifically, the whole process of inputting the real-time operation status information into the third deep reinforcement learning agent so that the third deep reinforcement learning agent outputs the control signal of the preventive control is as follows: Figure 2 As shown;

[0064] More specifically, the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on an expert experience data set, specifically:

[0065] According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints, solve the objective function with economic benefits as the optimization goal under multiple different states to obtain multiple expert strategies; wherein N is a positive integer greater than 1;

[0066] Obtaining an expert experience data set by calculating according to the multiple expert strategies;

[0067] Performing imitation learning pre-training on the first deep reinforcement learning agent according to the expert experience data set with the goal of minimizing action strategy errors;

[0068] When the minimized action strategy error converges, the training is stopped to obtain a second deep reinforcement learning agent.

[0069] More specifically, the basic framework of the first deep reinforcement learning agent is built according to the Markov process, and the actual power system or simulation software is used as the environment for interacting with the first deep reinforcement learning agent. The first deep reinforcement learning agent optimizes and adjusts the value function according to the state and reward feedback from the environment and takes actions on the environment according to the value function. The environment forms a corresponding state and reward according to the input action and feeds back to the agent. The agent stores the current state, the action taken, the reward feedback from the environment, and the previous state of the environment in the experience playback module. Among them:

[0070] The reward function of the environment:

[0071]

[0072] ΔF(S t )=F(S t )-F(S t+1 ) (2)

[0073]

[0074] F PF (S t )=-F P (S t )-F V (S t ) (4)

[0075]

[0076] Where, the power flow constraint is the equation (2-12); F(S t ) is the S in the environment t The trend economic function under the state of is the change value of the trend economic function after the agent takes action; p, r, α are positive weight coefficients; G PF (S t ) is the power flow constraint penalty function, R P (S t ) is the active constraint penalty function, R V (S t ) is the voltage constraint penalty function, P G,i,t is the active output of generator set i at time t, P G,max,i and P G,min,i They are the upper and lower limits of the generator output, V i,t is the voltage amplitude of node i at time t, V i,max and V i,minis the upper and lower limits of the amplitude of node i; G TS (S t ) is the transient stability penalty function, Ω instability It is the unstable region of the system.

[0077] The value function is built by a deep neural network, which is divided into a value network and a target value network. The input of the value network is the current state S t and action A i , the output is the value Q predicted by the neural network under this action s,i ; The target value network input is the environment state S t The next state S t+1 , predict S t+1 The action with the highest value in the state A i And the corresponding value Q t+1,i .

[0078] For the value network, the loss function is:

[0079] L=[Q(S t ,A t )-(R+γmaxQ(S t+1 ,A t+1 )] 2 (9)

[0080] In the formula, Q(S t ,A t ) is the value network in S t Status, A t The predicted value of the action, R is the reward of the environment after the action, and γ is the decay coefficient of the reward. Based on the Bayesian principle, the value network updates itself after calculating the loss based on the target value network and the reward fed back by the environment:

[0081] Q'(S t ,A t )=Q(S t ,A t )+α[(R+γmaxQ(S t+1 ,A t+1 )-Q(S t ,A t )] (10)

[0082] In the formula, Q'(S t ,A t ) is the updated value network, and α is the learning rate of the agent. After T updates, the agent copies the value network to the target value network; the agent calculates the value Q provided by the value function s,i Act on the environment, that is, control each controllable state of the power system's generators, loads, etc.

[0083] For the target policy neural network, its purpose is to solve the optimal strategy:

[0084] μ * = arg maxQ(s t ,μ(s t )) (11)

[0085] Among them, μ * is the optimal strategy; arg max is the optimal function, the purpose is to calculate in state S t The policy parameter μ at the maximum Q value.

[0086] The policy network uses the policy parameter gradient of the policy expected return to update the parameters of the policy network:

[0087]

[0088] in, is the gradient operator about the parameter θ, θ is the weight parameter of the policy network; N s is the number of experiences, and η is the learning rate of the network.

[0089] For the experience replay component, the agent learns the information from each interaction with the environment, including S t , Q t,i , r, S t+1 Stored in the experience playback component in the form of data groups.

[0090] In summary, the first deep reinforcement learning agent framework suitable for safe and stable preventive control of power systems can be formed.

[0091] Furthermore, the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set, and the training process of pre-training through imitation learning based on the expert experience data set is:

[0092] Based on the N-1 rule, the optimal power flow model considering safety and stability preventive control constraints is established:

[0093] The optimal power flow model constructs the objective function with the optimal economic benefit:

[0094]

[0095] Where F is the objective function; NT is the total number of time periods in the dispatch cycle, NG is the number of generator sets in the system, and P is G,i,t is the power generation of generator set i in time period t; a i is the coefficient of the quadratic term of fuel cost, b i Fuel cost primary coefficient, c i It is the fixed cost of fuel;i,t is the operating status logic parameter of the generator set, I i,t =1 means that at time t, generator set i is in operation state, I i,t =0 means that generator set i is in shutdown state at time t; Y i,t is the number of startups of generator set i up to time t, c ST,t The fixed cost of starting the unit; Z i,t is the number of shutdowns of generator set i up to time t, c SD,t Fixed cost for shutting down the unit; VOLL d P is the penalty fee for load shedding at the power user end. LS,d,t It is the amount of active load that is forced to be removed at the power consumption terminal d at time t.

[0096] The basic constraints on the operation of the power system include power balance constraints, upper and lower limit constraints on unit output, unit climbing constraints, load shedding upper limit constraints, and power flow safety constraints.

[0097] The power balance constraint is:

[0098]

[0099] Where ND is the total number of power consumption terminals, P D,d,t is the original active load at the power consumption terminal d at time t, S T A collection of time periods.

[0100] Unit output upper and lower limit constraints:

[0101]

[0102] Where P G,min,i is the lower limit of active power output of generator set i in operation, P G,max,i It is the upper limit of active power output of generator set i in running state.

[0103] Unit climbing constraints:

[0104]

[0105] In the formula, R U,i is the maximum climbing value of generator set i per hour, R D,i is the maximum downhill slope of generator set i per hour, P ST,i is the minimum active power output of generator set i when it is started, P SD,i It is the minimum active power output when generator set i is shut down.

[0106] Load shedding upper limit constraint:

[0107]

[0108] Power flow node equality constraints:

[0109]

[0110] In the formula, S B is the set of all nodes in the power network; KL is a NB*NL order matrix, and the element KL b,l Represents the connection relationship between node b and branch l, KL b,l =1 means that there is a connection between node b and branch l, KL b,l = 0, otherwise; KG is a NB*NG order matrix, and the element KG b,i Represents the connection relationship between node b and generator set i, KG b,i =1 indicates that node b is connected to generator set i, KG b,i On the contrary; KD is a NB*ND order matrix, and the element KD b,d Characterizes the connection relationship between node b and load d, KD b,d =1 indicates that there is a connection between node b and load d, KD b,d =0, otherwise; P L,l,t is the active power flow of line l at time t.

[0111] Power flow safety inequality constraints:

[0112] V i,min,t ≤V i,t ≤V i,max,t ,i∈S B ,t∈S T (twenty one)

[0113] P L,l,min,t ≤P L,l,t ≤P L,l,max,t ,l∈S L ,t∈S T (twenty two)

[0114] In the formula, S L is the collection of all branches of the power system; V i is the voltage amplitude of node i, V i,max 、V i,min They correspond to the maximum and minimum values ​​of the voltage amplitude of node i during safe and stable operation; P L,l,max , P L,l,min They correspond to the maximum and minimum values ​​of the useful power of the power flow in branch l respectively.

[0115] Power flow node constraints under N-1 expected failure:

[0116]

[0117] In the formula, S cis the fault set, is the matrix that represents the connection relationship between lines and nodes in the power network under the expected fault state c, is the power flow of line l under the expected fault state c at time t.

[0118] Power flow safety inequality constraints under N-1 expected faults:

[0119] V i,min,t ≤V i c ≤V i,max,t ,i∈S B ,t∈S T ,c∈S c (twenty four)

[0120]

[0121] In the formula, is the voltage amplitude of node i at time t under the expected fault state c, is the power flow active power of branch l in fault state c at time t.

[0122] Transient stability constraints:

[0123]

[0124] Where, d (c) (t) is the dot product of the power system potential energy at time t under fault state c, is the acceleration power of the generator dynamic group i in the dynamic process, θ i is the angle of generator set i relative to the center of inertia of the power system, is the mechanical input power of generator set i, is the electromagnetic output power of generator set i, M i is the inertia time constant of generator set i, M T is the inertia time constant of the system inertia center, N g is the number of generators in the system.

[0125] The above constraints are used as the optimal power flow restriction conditions, and the generators, lines, buses and other components in the optimization problem are limited to work within the specified range. Combined with the power flow economic function, an optimal power flow problem optimization model considering safety, stability and preventive control constraints can be formed. Further, according to the optimization model, the optimal power flow results under m different operating conditions of the power system are calculated, and the multi-step decision actions of each result are calculated through the difference sampling algorithm to obtain m corresponding expert strategy trajectories μ m , each expert trajectory μ m Contains the corresponding state and action sequence:

[0126]

[0127] in, is the nth state and nth action of the mth expert trajectory.

[0128] Collect m state-action pairs of expert trajectories An expert experience data set D can be constructed.

[0129] This application comprehensively considers the basic constraints of power system safety and stability, safety and stability constraints under N-1 faults, and transient stability constraints as the constraints of the optimal power flow, ensuring that the various components of the power system such as generators, lines, and buses operate within a safe and stable working range. Then, under these constraints, the objective function with economic benefits as the optimization goal is solved to obtain the optimal power flow results of the power system under different operating conditions. Next, the difference sampling algorithm is used to calculate the multi-step decision actions of these optimal power flow results to form multiple expert strategy trajectories, each of which contains a detailed state sequence and action sequence. These expert strategy trajectories are summarized to form an expert experience data set, which provides a rich learning sample based on the actual operation of the power system for deep reinforcement learning agents. By imitating these expert strategies, the agent can learn how to make the best control decisions under various complex working conditions, thereby improving the operating efficiency and stability of the power system in practical applications and reducing the risk of system failures caused by improper operation.

[0130] More specifically, the present application pre-trains the agent's policy network and initializes the experience replay component through imitation learning, wherein the imitation learning pre-training uses the expert experience data set D as labeled data to perform supervised learning on the policy network.

[0131] Loss function for imitation learning:

[0132]

[0133] In the formula, θ is the parameter of the policy network, is the Laplace operator, is the labeled data transformed from expert experience, Y ψ is the decision result of the policy function.

[0134] Imitation learning updates the policy network:

[0135]

[0136] In the formula, α is the update learning rate.

[0137] The initialization experience playback component is composed of the agent interacting with the environment according to the expert trajectory to obtain the reward r of each action, and using the reward r to construct the experience group (st ,a t ,r,s t+1 ), storing several experience groups in the experience playback component.

[0138] In summary, a second deep reinforcement learning agent pre-trained by imitation learning and an initialized experience replay component can be formed.

[0139] The first deep reinforcement learning agent of the present application uses expert experience data sets through imitation learning. These data sets are based on the basic constraints of power system safety and stability, safety and stability constraints under N-1 faults, and transient stability constraints. Multiple expert strategies are obtained by solving the objective function with economic benefits as the optimization goal. This pre-training method enables the agent to learn efficient decision-making patterns in a complex power system environment. By minimizing the action strategy error through imitation learning, the agent further optimizes its decision-making ability until the error converges, ensuring the decision-making accuracy and reliability of the agent. Ultimately, the trained second deep reinforcement learning agent can output high-quality control signals, which can guide the generators and reactive power sources in the power system to make precise adjustments, ensuring that the system can maintain a safe and stable operating state under various working conditions, effectively preventing instability and failures in the power system, and improving the operating efficiency and safety of the entire system.

[0140] Furthermore, the third deep reinforcement learning agent is obtained by interactive training of the second deep reinforcement learning agent with the environment constructed by the power system simulation model, specifically:

[0141] Acquire experience replay component information; wherein the experience replay component information is obtained by initializing the expert experience data set;

[0142] Training a second deep learning agent based on the experience playback component information and optimizing a loss function of the initial deep learning agent;

[0143] When the loss function of the second deep learning agent converges to a preset target value, training is stopped to obtain a third deep learning reinforcement agent.

[0144] More specifically, the training process of the third deep reinforcement learning agent being interactively trained by the second deep reinforcement learning agent and the environment constructed by the power system simulation model is specifically as follows:

[0145] 1. The strategy network of the second deep reinforcement learning agent outputs the probability of each action in the adjustment strategy for the controllable variables in the environment, namely the active output power and reactive output power of the generator set and the load removal amount, and further uses the probability to output the control signal;

[0146] 2. After the environment receives the control signal, the state, i.e., the voltage of each node, the power transmitted by each line, etc., changes accordingly, and accordingly generates corresponding rewards and state signals to be transmitted to the agent; the agent receives the rewards and state signals and stores them as a set of data in the experience playback component;

[0147] 3. The experience replay component uses experience to continuously train the second deep reinforcement learning agent until the value network and the target value network converge to unity and the policy network reaches the optimal state. The experience replay component continuously trains the agent including: 1) the experience replay component samples a data and inputs them into the value network, the target value network and the policy network in turn; 2) the policy network outputs the corresponding action strategy, and the value network and the target value network estimate the corresponding value according to the action; 3) the value network updates its own parameters according to the loss function after each data input, and the policy network updates its own parameters according to the gradient of the value output by the value network; 4) after repeating steps 1) and 2) for 10 times, copy the value network parameters to the target value network; 5) repeating steps 1) to 4) until the value network and the target value network converge, at which point the two network parameters are exactly the same and the policy network reaches the optimal decision result, and the third deep reinforcement learning agent is trained.

[0148] This application provides a rich data foundation close to the actual operation situation for the training of the second deep learning agent by obtaining the experience playback component information initialized by the expert experience data set. Then, using this experience playback information, the second deep learning agent is trained, and its loss function is optimized at the same time, so that the agent can learn the optimal control strategy under various power system conditions more quickly and accurately. During the training process, the agent continuously adjusts and optimizes its decision model until the loss function converges to the preset target value. At this time, the training of the agent reaches the optimal state and forms a third deep learning enhanced agent. This well-trained agent can quickly output accurate preventive control signals according to the real-time operating status of the power system, guide the generators and reactive power supplies in the system to make necessary adjustments, and ensure that the power system can maintain a safe and stable operating state under various complex conditions, effectively preventing potential power system failures and improving the reliability and economic operation efficiency of the power system.

[0149] S03: According to the control signal, the output power and switching of the generator and reactive power source in the power system are adjusted to make the power system operate in a safe and stable working state.

[0150] This application ensures the timeliness and accuracy of the control strategy by obtaining the operating status information of the power system in real time. Secondly, this information is input into the first deep reinforcement learning agent pre-trained with expert experience data. The agent masters the decision-making mode of experts through imitation learning, which improves the reliability and efficiency of decision-making. Then, the first agent interacts with the environment constructed by the power system simulation model to form a second agent, and further optimizes and adjusts its decision-making strategy to adapt to the actual operating environment. Finally, the control signal output by the second agent is used to guide the third agent, which integrates real-time status information and trained strategies to output accurate preventive control signals. These signals guide the generators and reactive power sources in the power system to adjust the output power and switching, ensuring that the power system can maintain a safe and stable working state under various operating conditions, thereby effectively preventing the occurrence of power system failures such as large-scale power outages, and improving the stability and reliability of the entire power system.

[0151] Embodiment 2

[0152] Please refer to Figure 3 , which is a preventive control device based on imitation and reinforcement learning provided in an embodiment of the present application.

[0153] In this embodiment, the preventive control device based on imitation and reinforcement learning includes an acquisition module 10 , an input-output module 20 and an adjustment module 30 .

[0154] The acquisition module 10 is used to obtain real-time operation status information of the power system;

[0155] The input-output module 20 is used to input the real-time operation status information into the third deep reinforcement learning agent, so that the third deep reinforcement learning agent outputs a control signal for preventive control;

[0156] The third deep reinforcement learning agent is obtained by interactive training between the second deep reinforcement learning agent and the environment constructed by the power system simulation model; the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set;

[0157] More specifically, the whole process of inputting the real-time operation status information into the third deep reinforcement learning agent so that the third deep reinforcement learning agent outputs the control signal of the preventive control is as follows: Figure 2 As shown;

[0158] More specifically, the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on an expert experience data set, specifically:

[0159] According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints, solve the objective function with economic benefits as the optimization goal under multiple different states to obtain multiple expert strategies; wherein N is a positive integer greater than 1;

[0160] Obtaining an expert experience data set by calculating according to the multiple expert strategies;

[0161] Performing imitation learning pre-training on the first deep reinforcement learning agent according to the expert experience data set with the goal of minimizing action strategy errors;

[0162] When the minimized action strategy error converges, the training is stopped to obtain a second deep reinforcement learning agent.

[0163] More specifically, the basic framework of the first deep reinforcement learning agent is built according to the Markov process, and the actual power system or simulation software is used as the environment for interacting with the first deep reinforcement learning agent. The first deep reinforcement learning agent optimizes and adjusts the value function according to the state and reward feedback from the environment and takes actions on the environment according to the value function. The environment forms a corresponding state and reward according to the input action and feeds back to the agent. The agent stores the current state, the action taken, the reward feedback from the environment, and the previous state of the environment in the experience playback module. Among them:

[0164] The reward function of the environment:

[0165]

[0166] ΔF(S t )=F(S t )-F(S t+1 ) (32)

[0167]

[0168] F PF (S t )=-F P (S t )-F V (S t ) (34)

[0169]

[0170] Where, the power flow constraint is the equation (2-12); F(S t ) is the S in the environment t The trend economic function under the state of is the change value of the trend economic function after the agent takes action; p, r, α are positive weight coefficients; G PF (S t) is the power flow constraint penalty function, R P (S t ) is the active constraint penalty function, R V (S t ) is the voltage constraint penalty function, P G,i,t is the active output of generator set i at time t, P G,max,i and P G,min,i They are the upper and lower limits of the generator output, V i,t is the voltage amplitude of node i at time t, V i,max and V i,min is the upper and lower limits of the amplitude of node i; G TS (S t ) is the transient stability penalty function, Ω instability It is the unstable region of the system.

[0171] The value function is built by a deep neural network, which is divided into a value network and a target value network. The input of the value network is the current state S t and action A i , the output is the value Q predicted by the neural network under this action s,i ; The target value network input is the environment state S t The next state S t+1 , predict S t+1 The action with the highest value in the state A i And the corresponding value Q t+1,i .

[0172] For the value network, the loss function is:

[0173] L=[Q(S t ,A t )-(R+γmaxQ(S t+1 ,A t+1 )] 2 (39)

[0174] In the formula, Q(S t ,A t ) is the value network in S t Status, A t The predicted value of the action, R is the reward of the environment after the action, and γ is the decay coefficient of the reward. Based on the Bayesian principle, the value network updates itself after calculating the loss based on the target value network and the reward fed back by the environment:

[0175] Q'(S t ,A t )=Q(S t ,A t )+α[(R+γmaxQ(S t+1 ,A t+1)-Q(S t ,A t )] (40)

[0176] In the formula, Q'(S t ,A t ) is the updated value network, and α is the learning rate of the agent. After T updates, the agent copies the value network to the target value network; the agent calculates the value Q provided by the value function s,i Act on the environment, that is, control each controllable state of the power system's generators, loads, etc.

[0177] For the target policy neural network, its purpose is to solve the optimal strategy:

[0178] μ * = arg maxQ(s t ,μ(s t )) (41)

[0179] Among them, μ * is the optimal strategy; arg max is the optimal function, the purpose is to calculate in state S t The policy parameter μ at the maximum Q value.

[0180] The policy network uses the policy parameter gradient of the policy expected return to update the parameters of the policy network:

[0181]

[0182] in, is the gradient operator about the parameter θ, θ is the weight parameter of the policy network; N s is the number of experiences, and η is the learning rate of the network.

[0183] For the experience replay component, the agent learns the information from each interaction with the environment, including S t , Q t,i , r, S t+1 Stored in the experience playback component in the form of data groups.

[0184] In summary, the first deep reinforcement learning agent framework suitable for safe and stable preventive control of power systems can be formed.

[0185] Furthermore, the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set, and the training process of pre-training through imitation learning based on the expert experience data set is:

[0186] Based on the N-1 rule, the optimal power flow model considering safety and stability preventive control constraints is established:

[0187] The optimal power flow model constructs the objective function with the optimal economic benefit:

[0188]

[0189] Where F is the objective function; NT is the total number of time periods in the dispatch cycle, NG is the number of generator sets in the system, and P is G,i,t is the power generation of generator set i in time period t; a i is the coefficient of the quadratic term of fuel cost, b i Fuel cost primary coefficient, c i It is the fixed cost of fuel; i,t is the operating status logic parameter of the generator set, I i,t =1 means that at time t, generator set i is in operation state, I i,t =0 means that generator set i is in shutdown state at time t; Y i,t is the number of startups of generator set i up to time t, c ST,t The fixed cost of starting the unit; Z i,t is the number of shutdowns of generator set i up to time t, c SD,t Fixed cost for shutting down the unit; VOLL d P is the penalty fee for load shedding at the power user end. LS,d,t It is the amount of active load that is forced to be removed at the power consumption terminal d at time t.

[0190] The basic constraints on the operation of the power system include power balance constraints, upper and lower limit constraints on unit output, unit climbing constraints, load shedding upper limit constraints, and power flow safety constraints.

[0191] The power balance constraint is:

[0192]

[0193] Where ND is the total number of power consumption terminals, P D,d,t is the original active load at the power consumption terminal d at time t, S T A collection of time periods.

[0194] Unit output upper and lower limit constraints:

[0195]

[0196] Where P G,min,i is the lower limit of active power output of generator set i in operation, P G,max,i It is the upper limit of active power output of generator set i in running state.

[0197] Unit climbing constraints:

[0198]

[0199] In the formula, R U,i is the maximum climbing value of generator set i per hour, R D,i is the maximum downhill slope of generator set i per hour, P ST,i is the minimum active power output of generator set i when it is started, P SD,i It is the minimum active power output when generator set i is shut down.

[0200] Load shedding upper limit constraint:

[0201]

[0202] Power flow node equality constraints:

[0203]

[0204] In the formula, S B is the set of all nodes in the power network; KL is a NB*NL order matrix, and the element KL b,l Represents the connection relationship between node b and branch l, KL b,l =1 means that there is a connection between node b and branch l, KL b,l = 0, otherwise; KG is a NB*NG order matrix, and the element KG b,i Represents the connection relationship between node b and generator set i, KG b,i =1 indicates that node b is connected to generator set i, KG b,i On the contrary; KD is a NB*ND order matrix, and the element KD b,d Characterizes the connection relationship between node b and load d, KD b,d =1 indicates that there is a connection between node b and load d, KD b,d =0, otherwise; P L,l,t is the active power flow of line l at time t.

[0205] Power flow safety inequality constraints:

[0206] V i,min,t ≤V i,t ≤V i,max,t ,i∈S B ,t∈S T (51)

[0207] P L,l,min,t ≤P L,l,t ≤P L,l,max,t ,l∈S L ,t∈S T (52)

[0208] In the formula, S L is the collection of all branches of the power system; V i is the voltage amplitude of node i, Vi,max 、V i,min They correspond to the maximum and minimum values ​​of the voltage amplitude of node i during safe and stable operation; P L,l,max , P L,l,min They correspond to the maximum and minimum values ​​of the useful power of the power flow in branch l respectively.

[0209] Power flow node constraints under N-1 expected failure:

[0210]

[0211] In the formula, S c is the fault set, is the matrix that represents the connection relationship between lines and nodes in the power network under the expected fault state c, is the power flow of line l under the expected fault state c at time t.

[0212] Power flow safety inequality constraints under N-1 expected faults:

[0213] V i,min,t ≤V i c ≤V i,max,t ,i∈S B ,t∈S T ,c∈S c (54)

[0214]

[0215] In the formula, is the voltage amplitude of node i at time t under the expected fault state c, is the power flow active power of branch l in fault state c at time t.

[0216] Transient stability constraints:

[0217]

[0218] Where, d (c) (t) is the dot product of the power system potential energy at time t under fault state c, is the acceleration power of the generator dynamic group i in the dynamic process, θ i is the angle of generator set i relative to the center of inertia of the power system, is the mechanical input power of generator set i, is the electromagnetic output power of generator set i, M i is the inertia time constant of generator set i, M T is the inertia time constant of the system inertia center, N g is the number of generators in the system.

[0219] The above constraints are used as the optimal power flow restriction conditions, and the generators, lines, buses and other components in the optimization problem are limited to work within the specified range. Combined with the power flow economic function, an optimal power flow problem optimization model considering safety, stability and preventive control constraints can be formed. Further, according to the optimization model, the optimal power flow results under m different operating conditions of the power system are calculated, and the multi-step decision actions of each result are calculated through the difference sampling algorithm to obtain m corresponding expert strategy trajectories μ m , each expert trajectory μ m Contains the corresponding state and action sequence:

[0220]

[0221] in, is the nth state and nth action of the mth expert trajectory.

[0222] Collect m state-action pairs of expert trajectories An expert experience data set D can be constructed.

[0223] This application comprehensively considers the basic constraints of power system safety and stability, safety and stability constraints under N-1 faults, and transient stability constraints as the constraints of the optimal power flow, ensuring that the various components of the power system such as generators, lines, and buses operate within a safe and stable working range. Then, under these constraints, the objective function with economic benefits as the optimization goal is solved to obtain the optimal power flow results of the power system under different operating conditions. Next, the difference sampling algorithm is used to calculate the multi-step decision actions of these optimal power flow results to form multiple expert strategy trajectories, each of which contains a detailed state sequence and action sequence. These expert strategy trajectories are summarized to form an expert experience data set, which provides a rich learning sample based on the actual operation of the power system for deep reinforcement learning agents. By imitating these expert strategies, the agent can learn how to make the best control decisions under various complex working conditions, thereby improving the operating efficiency and stability of the power system in practical applications and reducing the risk of system failures caused by improper operation.

[0224] More specifically, the present application pre-trains the agent's policy network and initializes the experience replay component through imitation learning, wherein the imitation learning pre-training uses the expert experience data set D as labeled data to perform supervised learning on the policy network.

[0225] Loss function for imitation learning:

[0226]

[0227] In the formula, θ is the parameter of the policy network, is the Laplace operator, is the labeled data transformed from expert experience, Y ψ is the decision result of the policy function.

[0228] Imitation learning updates the policy network:

[0229]

[0230] In the formula, α is the update learning rate.

[0231] The initialization experience playback component is composed of the agent interacting with the environment according to the expert trajectory to obtain the reward r of each action, and using the reward r to construct the experience group (s t ,a t ,r,s t+1 ), storing several experience groups in the experience playback component.

[0232] In summary, a second deep reinforcement learning agent pre-trained by imitation learning and an initialized experience replay component can be formed.

[0233] The first deep reinforcement learning agent of the present application uses expert experience data sets through imitation learning. These data sets are based on the basic constraints of power system safety and stability, safety and stability constraints under N-1 faults, and transient stability constraints. Multiple expert strategies are obtained by solving the objective function with economic benefits as the optimization goal. This pre-training method enables the agent to learn efficient decision-making patterns in a complex power system environment. By minimizing the action strategy error through imitation learning, the agent further optimizes its decision-making ability until the error converges, ensuring the decision-making accuracy and reliability of the agent. Ultimately, the trained second deep reinforcement learning agent can output high-quality control signals, which can guide the generators and reactive power sources in the power system to make precise adjustments, ensuring that the system can maintain a safe and stable operating state under various working conditions, effectively preventing instability and failures in the power system, and improving the operating efficiency and safety of the entire system.

[0234] Furthermore, the third deep reinforcement learning agent is obtained by interactive training of the second deep reinforcement learning agent with the environment constructed by the power system simulation model, specifically:

[0235] Acquire experience replay component information; wherein the experience replay component information is obtained by initializing the expert experience data set;

[0236] Training a second deep learning agent based on the experience playback component information and optimizing a loss function of the initial deep learning agent;

[0237] When the loss function of the second deep learning agent converges to a preset target value, training is stopped to obtain a third deep learning reinforcement agent.

[0238] More specifically, the training process of the third deep reinforcement learning agent being interactively trained by the second deep reinforcement learning agent and the environment constructed by the power system simulation model is specifically as follows:

[0239] 1. The strategy network of the second deep reinforcement learning agent outputs the probability of each action in the adjustment strategy for the controllable variables in the environment, namely the active output power and reactive output power of the generator set and the load removal amount, and further uses the probability to output the control signal;

[0240] 2. After the environment receives the control signal, the state, i.e., the voltage of each node, the power transmitted by each line, etc., changes accordingly, and accordingly generates corresponding rewards and state signals to be transmitted to the agent; the agent receives the rewards and state signals and stores them as a set of data in the experience playback component;

[0241] 3. The experience replay component uses experience to continuously train the second deep reinforcement learning agent until the value network and the target value network converge to unity and the policy network reaches the optimal state. The experience replay component continuously trains the agent including: 1) the experience replay component samples a data and inputs them into the value network, the target value network and the policy network in turn; 2) the policy network outputs the corresponding action strategy, and the value network and the target value network estimate the corresponding value according to the action; 3) the value network updates its own parameters according to the loss function after each data input, and the policy network updates its own parameters according to the gradient of the value output by the value network; 4) after repeating steps 1) and 2) for 10 times, copy the value network parameters to the target value network; 5) repeating steps 1) to 4) until the value network and the target value network converge, at which point the two network parameters are exactly the same and the policy network reaches the optimal decision result, and the third deep reinforcement learning agent is trained.

[0242] This application provides a rich data foundation close to the actual operation situation for the training of the second deep learning agent by obtaining the experience playback component information initialized by the expert experience data set. Then, using this experience playback information, the second deep learning agent is trained, and its loss function is optimized at the same time, so that the agent can learn the optimal control strategy under various power system conditions more quickly and accurately. During the training process, the agent continuously adjusts and optimizes its decision model until the loss function converges to the preset target value. At this time, the training of the agent reaches the optimal state and forms a third deep learning enhanced agent. This well-trained agent can quickly output accurate preventive control signals according to the real-time operating status of the power system, guide the generators and reactive power supplies in the system to make necessary adjustments, and ensure that the power system can maintain a safe and stable operating state under various complex conditions, effectively preventing potential power system failures and improving the reliability and economic operation efficiency of the power system.

[0243] The adjustment module 30 is used to adjust the output power and switching of the generator and reactive power source in the power system according to the control signal, so that the power system operates in a safe and stable working state.

[0244] This device uses three modules to divide the work and coordinate work to better provide preventive control strategies for the power system. This application ensures the timeliness and accuracy of the control strategy by obtaining the operating status information of the power system in real time. Secondly, this information is input into the first deep reinforcement learning agent pre-trained with expert experience data. The agent masters the decision-making mode of the expert through imitation learning, which improves the reliability and efficiency of the decision. Then, the first agent interacts with the environment constructed by the power system simulation model to form a second agent, and further optimizes and adjusts its decision-making strategy to adapt to the actual operating environment. Finally, the control signal output by the second agent is used to guide the third agent, which integrates the real-time status information and the trained strategy to output accurate preventive control signals. These signals guide the generators and reactive power sources in the power system to adjust the output power and switch, ensuring that the power system can maintain a safe and stable working state under various operating conditions, thereby effectively preventing the occurrence of power system failures such as large-scale power outages, and improving the stability and reliability of the entire power system.

[0245] Embodiment three:

[0246] The embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the prevention and control method based on imitation and reinforcement learning;

[0247] Wherein, the preventive control method based on imitation and reinforcement learning, if implemented in the form of a software functional unit and used as an independent product, can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0248] Embodiment 4

[0249] The present application provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, any one of the preventive control methods based on imitation and reinforcement learning as described in Example 1 is implemented.

[0250] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A preventive control method based on imitation and reinforcement learning, characterized in that: include: Obtain real-time operating status information of the power system; Inputting the real-time operation status information into a third deep reinforcement learning agent so that the third deep reinforcement learning agent outputs a control signal for preventive control; The third deep reinforcement learning agent is obtained by interactive training between the second deep reinforcement learning agent and the environment constructed by the power system simulation model; the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set; According to the control signal, the output power and switching of the generator and reactive power source in the power system are adjusted to make the power system operate in a safe and stable working state.

2. The preventive control method based on imitation and reinforcement learning according to claim 1, characterized in that: The second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set, specifically: According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints, solve the objective function with economic benefits as the optimization goal under multiple different states to obtain multiple expert strategies; wherein N is a positive integer greater than 1; Obtaining an expert experience data set by calculating according to the multiple expert strategies; Performing imitation learning pre-training on the first deep reinforcement learning agent according to the expert experience data set with the goal of minimizing action strategy errors; When the minimized action strategy error converges, the training is stopped to obtain a second deep reinforcement learning agent.

3. The preventive control method based on imitation and reinforcement learning according to claim 2, characterized in that: According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults, and the transient stability constraints, the objective function with economic benefits as the optimization target under multiple different states is solved to obtain multiple expert strategies; the expert experience data set is calculated according to the multiple expert strategies, specifically: Taking the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints as optimal power flow constraints, the generators, lines, buses and other components in the optimization objective are limited to operate within a specified range; According to the optimal power flow constraint conditions, solving the objective function with economic benefit as the optimization target, and obtaining multiple optimal power flow results of the power system under multiple operating states; Calculate the multi-step decision actions of the multiple optimal power flow results through a difference sampling algorithm to obtain multiple expert strategy trajectories; wherein each expert strategy trajectory includes a state sequence and an action sequence; The plurality of expert trajectories constitute the expert experience data set.

4. The preventive control method based on imitation and reinforcement learning according to claim 1, characterized in that: The third deep reinforcement learning agent is obtained by interactive training of the second deep reinforcement learning agent and the environment constructed by the power system simulation model, specifically: Acquire experience replay component information; wherein the experience replay component information is obtained by initializing the expert experience data set; Training a second deep learning agent based on the experience playback component information and optimizing a loss function of the initial deep learning agent; When the loss function of the second deep learning agent converges to a preset target value, training is stopped to obtain a third deep learning reinforcement agent.

5. A preventive control device based on imitation and reinforcement learning, characterized in that: It includes an acquisition module, an input and output module, and an adjustment module; The acquisition module is used to obtain the real-time operation status information of the power system; The input-output module is used to input the real-time operation status information into the third deep reinforcement learning agent, so that the third deep reinforcement learning agent outputs a control signal for preventive control; The third deep reinforcement learning agent is obtained by interactive training between the second deep reinforcement learning agent and the environment constructed by the power system simulation model; the second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set; The adjustment module is used to adjust the output power and switching of the generator and reactive power supply in the power system according to the control signal, so that the power system operates in a safe and stable working state.

6. The preventive control device based on imitation and reinforcement learning according to claim 5, characterized in that: The second deep reinforcement learning agent is obtained by pre-training the first deep reinforcement learning agent through imitation learning based on the expert experience data set, specifically: According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints, solve the objective function with economic benefits as the optimization goal under multiple different states to obtain multiple expert strategies; wherein N is a positive integer greater than 1; Obtaining an expert experience data set by calculating according to the multiple expert strategies; Performing imitation learning pre-training on the first deep reinforcement learning agent according to the expert experience data set with the goal of minimizing action strategy errors; When the minimized action strategy error converges, the training is stopped to obtain a second deep reinforcement learning agent.

7. The preventive control device based on imitation and reinforcement learning according to claim 6, characterized in that: According to the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults, and the transient stability constraints, the objective function with economic benefits as the optimization target under multiple different states is solved to obtain multiple expert strategies; the expert experience data set is calculated according to the multiple expert strategies, specifically: Taking the basic constraints of power system safety and stability, the safety and stability constraints under N-1 faults and the transient stability constraints as optimal power flow constraints, the generators, lines, buses and other components in the optimization objective are limited to operate within a specified range; According to the optimal power flow constraint conditions, solving the objective function with economic benefit as the optimization target, and obtaining multiple optimal power flow results of the power system under multiple operating states; Calculate the multi-step decision actions of the multiple optimal power flow results through a difference sampling algorithm to obtain multiple expert strategy trajectories; wherein each expert strategy trajectory includes a state sequence and an action sequence; The plurality of expert trajectories constitute the expert experience data set.

8. The preventive control device based on imitation and reinforcement learning according to claim 5, characterized in that: The third deep reinforcement learning agent is obtained by interactive training of the second deep reinforcement learning agent and the environment constructed by the power system simulation model, specifically: Acquire experience replay component information; wherein the experience replay component information is obtained by initializing the expert experience data set; Training a second deep learning agent based on the experience playback component information and optimizing a loss function of the initial deep learning agent; When the loss function of the second deep learning agent converges to a preset target value, training is stopped to obtain a third deep learning reinforcement agent.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the preventive control method based on imitation and reinforcement learning as described in any one of claims 1 to 4.

10. A terminal device, characterized in that: It comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the preventive control method based on imitation and reinforcement learning as described in any one of claims 1 to 4.

Images