Reinforcement learning prevention and control method and device for loss function embedded knowledge

Abstract

The invention discloses a reinforcement learning prevention and control method and device for loss function embedded knowledge. According to the method, the running state information of the power system is acquired in real time and input to the deep reinforcement learning agent, the agent is subjected to interactive training with the power system simulation model, the agent is trained by using the loss function embedded with physical knowledge, and accurate prevention and control signals can be output. According to the signals, the intelligent agent dynamically adjusts power output and load switching of the generator and the reactive power supply so as to ensure that the power system is maintained in a safe and stable working state, and the reliability of a prevention and control strategy is improved.

Description

Technical Field

[0001] The present invention relates to the field of power system preventive control, and in particular to a reinforcement learning preventive control method and device with loss function embedded knowledge. Background Art

[0002] In recent years, the scale of power, power consumption, and complexity of power grid structure in various provinces across the country have increased significantly. Regional power systems divided by provinces often operate at the limit of safety and stability, resulting in insufficient safety margins for the system. The risk of power outages caused by system collapse due to accidental serious failures is increasing. Therefore, the problem of safe and stable operation of the power system is particularly prominent, and online and high-precision requirements are put forward for the safety and stability prevention and control strategies of the power system.

[0003] There are two widely used safety and stability preventive control methods based on physical mechanisms: the optimal power flow preventive control method based on satisfying preventive control constraints requires solving a large number of nonlinear constraints and algebraic equation constraints, which has high solution dimensions and long time, is difficult to solve online, and has the defect of falling into local optimality; while the preventive control method based on sensitivity trajectory obtains preventive control strategies through trial adjustment and repeated verification, which has low computational efficiency and does not consider the control cost. Therefore, it is urgent to propose a safe and stable preventive control strategy formulation method that can meet the requirements of high efficiency, onlineness, and high precision.

[0004] On the other hand, the development of digitalization of power systems in recent years has brought new challenges to the formulation of safe and stable preventive control strategies for power systems. The relevant technologies of safe and stable preventive control of power systems are constantly transforming towards intelligent modes. Although the emerging deep reinforcement learning with DDPG, A3C and other mainstream models has the advantages of high precision and fast calculation speed compared with traditional preventive control methods in this field, deep reinforcement learning is a decision-making process of dimensional analysis and is unexplainable. The reliability of the control strategy it makes cannot be proven by physical knowledge. Summary of the invention

[0005] The present invention provides a reinforcement learning preventive control method and device with loss function embedded knowledge to improve the reliability of preventive control strategies.

[0006] In a first aspect, the present application provides a reinforcement learning preventive control method with loss function embedded knowledge, comprising:

[0007] Obtain real-time operating status information of the power system;

[0008] The real-time operation status information is input into a deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control; wherein the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent and an environment constructed by a power system simulation model; and the loss function of the initial deep learning agent is constructed based on embedded physical knowledge;

[0009] According to the control signal, the output power and switching of the generator and reactive power source in the power system are adjusted to make the power system operate in a safe and stable working state.

[0010] This application obtains and analyzes the operating status information of the power system in real time, and uses the intelligent agent trained through interaction with the simulation environment to make rapid decisions. The intelligent agent is also trained using a loss function embedded with physical knowledge, and outputs control signals to dynamically adjust the output of generators and reactive power sources, effectively improving the reliability of preventive control strategies, thereby ensuring that the power system can maintain a safe and stable operating state under various operating conditions.

[0011] As a preferred embodiment of the first aspect, the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent and an environment constructed by a power system simulation model, specifically:

[0012] Acquire experience replay component information; wherein the experience replay component information is obtained by collecting rewards and state signal data generated during the interaction between the initial deep reinforcement learning agent and the environment constructed by the power system simulation model;

[0013] According to the experience playback component information, the initial deep learning agent is trained and the loss function of the initial deep learning agent is optimized; wherein the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge;

[0014] When the loss function of the initial deep learning agent converges to the preset target value, the training is stopped and a deep learning reinforcement agent is obtained.

[0015] In this preferred embodiment, the present application forms experience replay component information by collecting reward and state signal data generated by the interaction between the agent and the environment. Using this information, the loss function of the agent is trained and optimized, in which key physical knowledge is embedded to guide the learning process. As the training progresses, the loss function continues to converge until the preset performance target value is reached, and the training stops. The deep learning enhanced agent obtained can efficiently output control signals to ensure that the power system can maintain safe and stable operation under various operating conditions, while also improving the intelligence level of preventive control of the power system.

[0016] As a preferred embodiment of the first aspect, the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge, specifically:

[0017] Calculate the optimization result according to the optimal power flow calculation model, and use the optimization result as the first embedded physical knowledge of the loss function of the initial deep learning agent;

[0018] Calculating a transient stability margin according to a transient energy function, wherein the transient stability margin is used as second embedded physical knowledge of a loss function of an initial deep learning agent;

[0019] The first embedded physical knowledge and the second embedded physical knowledge are embedded into the loss function of the initial deep learning agent.

[0020] In this preferred embodiment, the present application embeds two sets of key physical knowledge, the optimization results obtained by the optimal power flow calculation model and the transient stability margin obtained by calculating the transient energy function, into the loss function of the initial deep learning intelligent agent. This method ensures that the intelligent agent not only pursues the maximization of economic benefits during the learning process, but also fully considers the safe and stable operation of the power system. This method of embedding physical knowledge improves the accuracy and reliability of the intelligent agent's decision-making, allowing the intelligent agent to generate more scientific and effective control strategies. Therefore, the intelligent agent can comprehensively consider the various needs of the power system and optimize the preventive control strategy, thereby improving the stability and economic operation efficiency of the power system when facing various operating conditions and potential faults.

[0021] As a preferred embodiment of the first aspect, the optimization result is calculated according to the optimal power flow calculation model, and the optimization result is used as the first embedded physical knowledge of the loss function of the initial deep learning agent, specifically:

[0022] According to the basic constraints of power system safety and stability and the safety and stability constraints under N-1 faults, the objective function with economic benefits as the optimization goal is solved to obtain the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source; where N is a positive integer greater than 1;

[0023] The active power and reactive power of the generator under the optimal power flow and the reactive power of the reactive power source are the first embedded physical knowledge of the loss function of the initial deep learning agent.

[0024] In this preferred embodiment, the present application solves the objective function with the goal of maximizing economic benefits while satisfying the basic constraints of power system safety and stability and N-1 fault conditions, thereby obtaining the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source. These optimization results, as the first set of embedded physical knowledge, provide an economic benefit-oriented reference benchmark for the intelligent agent, ensuring that the intelligent agent can generate control strategies that both meet the safety and stability requirements of the power system and optimize economic benefits during the learning process. This embedding method enables the intelligent agent to internalize the economic operation characteristics of the power system in the early stages of training, thereby more efficiently exploring and optimizing the control strategy in the subsequent learning process, improving the overall performance and economic benefits of the power system preventive control method.

[0025] As a preferred embodiment of the first aspect, the transient stability margin is calculated according to the transient energy function, and the transient stability margin is used as the second embedded physical knowledge of the initial deep learning agent loss function, specifically:

[0026] According to the transient energy function method, the transient stability kinetic energy function and the transient stability potential energy function are calculated to form a transient energy function;

[0027] According to the instability limit, calculating the critical transient energy of the transient energy function when a preset fault occurs, and calculating the transient stability margin according to the critical transient energy;

[0028] The transient stability margin is the second embedded physical knowledge of the loss function of the initial deep learning agent.

[0029] In this preferred embodiment, the present application first constructs a transient energy function by calculating a transient stability kinetic energy function and a transient stability potential energy function, and then uses the instability limit of the system to determine the critical transient energy under preset fault conditions, and then calculates the transient stability margin of the system. The embedding of this transient stability margin enables the intelligent agent to not only consider economic benefits during the learning process, but also pay more attention to the dynamic stability of the system. The intelligent agent can make more accurate control decisions based on the stability margin of the system, so that when the power system faces transient disturbances, it can respond quickly and take preventive measures to effectively avoid system instability and ensure the safe operation and reliability of the power system.

[0030] In a second aspect, the present application provides a reinforcement learning prevention control device with loss function embedded in knowledge. The reinforcement learning prevention control device with loss function embedded in knowledge includes an acquisition module, an input and output module, and an adjustment module;

[0031] The acquisition module is used to obtain the real-time operation status information of the power system;

[0032] The input-output module is used to input the real-time operation status information into the deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control; wherein the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent and an environment constructed by a power system simulation model; and the loss function of the initial deep learning agent is constructed based on embedded physical knowledge;

[0033] The adjustment module is used to adjust the output power and switching of the generator and reactive power supply in the power system according to the control signal, so that the power system operates in a safe and stable working state.

[0034] This device uses three modules to divide the work and coordinate work to better output preventive control strategies according to the power system status. This application obtains and analyzes the operating status information of the power system in real time, and uses the intelligent agent obtained by interactive training with the simulation environment to make quick decisions. The intelligent agent also uses the loss function embedded with physical knowledge for training, and outputs control signals to dynamically adjust the output of generators and reactive power sources, effectively improving the reliability of preventive control strategies, thereby ensuring that the power system can maintain a safe and stable operating state under various working conditions.

[0035] As a preferred embodiment of the second aspect, the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent and an environment constructed by a power system simulation model, specifically:

[0036] Acquire experience replay component information; wherein the experience replay component information is obtained by collecting rewards and state signal data generated during the interaction between the initial deep reinforcement learning agent and the environment constructed by the power system simulation model;

[0037] According to the experience playback component information, the initial deep learning agent is trained and the loss function of the initial deep learning agent is optimized; wherein the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge;

[0038] When the loss function of the initial deep learning agent converges to the preset target value, the training is stopped and a deep learning reinforcement agent is obtained.

[0039] In this preferred embodiment, the present application forms experience replay component information by collecting reward and state signal data generated by the interaction between the agent and the environment. Using this information, the loss function of the agent is trained and optimized, in which key physical knowledge is embedded to guide the learning process. As the training progresses, the loss function continues to converge until the preset performance target value is reached, and the training stops. The deep learning enhanced agent obtained can efficiently output control signals to ensure that the power system can maintain safe and stable operation under various operating conditions, while also improving the intelligence level of preventive control of the power system.

[0040] As a preferred embodiment of the second aspect, the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge, specifically:

[0041] Calculate the optimization result according to the optimal power flow calculation model, and use the optimization result as the first embedded physical knowledge of the loss function of the initial deep learning agent;

[0042] Calculating a transient stability margin according to a transient energy function, wherein the transient stability margin is used as second embedded physical knowledge of a loss function of an initial deep learning agent;

[0043] The first embedded physical knowledge and the second embedded physical knowledge are embedded into the loss function of the initial deep learning agent.

[0044] In this preferred embodiment, the present application embeds two sets of key physical knowledge, the optimization results obtained by the optimal power flow calculation model and the transient stability margin obtained by calculating the transient energy function, into the loss function of the initial deep learning intelligent agent. This method ensures that the intelligent agent not only pursues the maximization of economic benefits during the learning process, but also fully considers the safe and stable operation of the power system. This method of embedding physical knowledge improves the accuracy and reliability of the intelligent agent's decision-making, allowing the intelligent agent to generate more scientific and effective control strategies. Therefore, the intelligent agent can comprehensively consider the various needs of the power system and optimize the preventive control strategy, thereby improving the stability and economic operation efficiency of the power system when facing various operating conditions and potential faults.

[0045] As a preferred embodiment of the second aspect, the optimization result is calculated according to the optimal power flow calculation model, and the optimization result is used as the first embedded physical knowledge of the loss function of the initial deep learning agent, specifically:

[0046] According to the basic constraints of power system safety and stability and the safety and stability constraints under N-1 faults, the objective function with economic benefits as the optimization goal is solved to obtain the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source; where N is a positive integer greater than 1;

[0047] The active power and reactive power of the generator under the optimal power flow and the reactive power of the reactive power source are the first embedded physical knowledge of the loss function of the initial deep learning agent.

[0048] In this preferred embodiment, the present application solves the objective function with the goal of maximizing economic benefits while satisfying the basic constraints of power system safety and stability and N-1 fault conditions, thereby obtaining the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source. These optimization results, as the first set of embedded physical knowledge, provide an economic benefit-oriented reference benchmark for the intelligent agent, ensuring that the intelligent agent can generate control strategies that both meet the safety and stability requirements of the power system and optimize economic benefits during the learning process. This embedding method enables the intelligent agent to internalize the economic operation characteristics of the power system in the early stages of training, thereby more efficiently exploring and optimizing the control strategy in the subsequent learning process, improving the overall performance and economic benefits of the power system preventive control method.

[0049] As a preferred embodiment of the second aspect, the transient stability margin is calculated according to the transient energy function, and the transient stability margin is used as the second embedded physical knowledge of the initial deep learning agent loss function, specifically:

[0050] According to the transient energy function method, the transient stability kinetic energy function and the transient stability potential energy function are calculated to form a transient energy function;

[0051] According to the instability limit, calculating the critical transient energy of the transient energy function when a preset fault occurs, and calculating the transient stability margin according to the critical transient energy;

[0052] The transient stability margin is the second embedded physical knowledge of the loss function of the initial deep learning agent.

[0053] In this preferred embodiment, the present application first constructs a transient energy function by calculating a transient stability kinetic energy function and a transient stability potential energy function, and then uses the instability limit of the system to determine the critical transient energy under preset fault conditions, and then calculates the transient stability margin of the system. The embedding of this transient stability margin enables the intelligent agent to not only consider economic benefits during the learning process, but also pay more attention to the dynamic stability of the system. The intelligent agent can make more accurate control decisions based on the stability margin of the system, so that when the power system faces transient disturbances, it can respond quickly and take preventive measures to effectively avoid system instability and ensure the safe operation and reliability of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 : A flowchart of an embodiment of a reinforcement learning prevention control method with loss function embedded knowledge provided in this application;

[0055] Figure 2 : Schematic diagram of the process flow generated for the preventive control strategy provided in this application;

[0056] Figure 3 : A structural diagram of an embodiment of a reinforcement learning prevention control device with loss function embedded knowledge provided in this application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] Embodiment 1

[0059] Please refer to Figure 1 , which is a reinforcement learning preventive control method with loss function embedded knowledge provided by an embodiment of the present invention.

[0060] In this embodiment, the process of the reinforcement learning preventive control method of embedding loss function knowledge in this application is described in detail through steps S01-S03.

[0061] S01: Obtain real-time operation status information of the power system;

[0062] S02: inputting the real-time operation status information into a deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control; wherein the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent with an environment constructed by a power system simulation model; and the loss function of the initial deep learning agent is constructed based on embedded physical knowledge;

[0063] The real-time operation status information is input into the deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control. The generation method of the entire preventive strategy is as follows: Figure 2 As shown;

[0064] More specifically, the initial deep learning agent is constructed according to the Markov process, and the construction process of the initial deep learning agent is:

[0065] The actual power system or simulation software is used as the environment for the initial deep reinforcement learning agent to interact. The initial deep reinforcement learning agent optimizes and adjusts the value function according to the state and reward feedback from the environment and takes actions on the environment according to the value function. The environment forms the corresponding state and reward according to the input action and feeds back to the agent. The initial deep reinforcement learning agent stores the current state, the action taken, the reward feedback from the environment, and the previous state of the environment in the experience playback module. Among them:

[0066] The reward function of the environment:

[0067]

[0068] ΔF(S t )=F(S t )-F(S t+1 ) (2)

[0069]

[0070] G PF (S t )=-R P (S t )-R V (S t ) (4)

[0071]

[0072] Where, the power flow constraint is the equation (2-12); F(S t ) is the S in the environment t The trend economic function under the state of is the change in the trend economic function after the initial deep reinforcement learning agent action; p, r, α are positive weight coefficients; G PF (S t ) is the power flow constraint penalty function, R P (S t ) is the active constraint penalty function, R V (S t ) is the voltage constraint penalty function, P G,i,t is the active output of generator set i at time t, P G,max,i and P G,min,i They are the upper and lower limits of the generator output, V i,t is the voltage amplitude of node i at time t, V i,max and V i,min is the upper and lower limits of the amplitude of node i; G TS (S t ) is the transient stability penalty function, Ω instability It is the unstable region of the system.

[0073] The value function is built by a deep neural network, which is divided into a value network and a target value network. The input of the value network is the current state S t , output all actions A made in this state i , the output is the value Q predicted by the neural network under all actions s,i ; The target value network input is the environment state S t The next state S t+1 , predict S t+1 All actions in state A i And the corresponding value Q t+1,i , and select the one with the largest value.

[0074] For the value network, the loss function is:

[0075] L=[Q(S t ,A t )-(R+γmaxQ(S t+1 ,A t+1 )] 2 (9)

[0076] In the formula, Q(S t ,A t ) is the value network in S t Status, A t The predicted value of the action, R is the reward of the environment after the action, and γ is the decay coefficient of the reward. Based on the Bayesian principle, the value network updates itself after calculating the loss based on the target value network and the reward fed back by the environment:

[0077] Q'(S t ,A t )=Q(S t ,A t )+α[(R+γmaxQ(S t+1 ,A t+1 )-Q(S t ,A t )] (10)

[0078] In the formula, Q'(S t ,A t ) is the updated value network, and α is the learning rate of the agent. After T updates, the agent copies the value network to the target value network; the initial deep reinforcement learning agent calculates the value Q provided by the value function s,i Act on the environment, that is, control each controllable state of the power system's generators, loads, etc.

[0079] In summary, an initial deep reinforcement learning agent framework suitable for safe and stable preventive control of power systems can be formed.

[0080] More specifically, the deep reinforcement learning agent is obtained by interactive training of the initial deep reinforcement learning agent and the environment constructed by the power system simulation model, specifically:

[0081] The initial deep reinforcement learning agent probabilistically outputs control signals based on the output of the value function for the controllable variables in the environment constructed by the power system simulation model, namely, the active output power and reactive output power of the generator set and the load removal amount;

[0082] After receiving the control signal, the state of the environment constructed by the power system simulation model, i.e., the voltage of each node, the power transmitted by each line, etc., changes accordingly, and accordingly generates corresponding rewards and state signals which are transmitted to the initial deep reinforcement learning agent; the initial deep reinforcement learning agent receives the rewards and state signals and stores them as a set of data in the experience playback component;

[0083] The experience replay component uses experience to continuously train the agent until the value network and the target value network converge to unity. The experience replay component continuously trains the agent including: 1) the experience replay component samples a data and inputs them into the value network and the target value network respectively in turn; 2) the value network updates its own parameters according to the loss function after each data input; 3) after repeating steps 1) and 2) for 10 times, the value network parameters are copied to the target value network; 4) steps 1) to 3) are repeated until the value network and the target value network converge, at which time the two network parameters are exactly the same and form a unity, the strategy network achieves the optimal decision-making result, and a deep reinforcement learning agent is obtained.

[0084] In summary, through interactive learning with the environment, the deep reinforcement learning agent generates the optimal value function. The value function takes various states of the power system as input, outputs actions and their values ​​under various states, and selects the action with the best value to form two corresponding state-action pairs, that is, a deep reinforcement learning transient stability preventive control strategy with loss function embedded in the knowledge.

[0085] This application forms experience replay component information by collecting reward and state signal data generated by the interaction between the agent and the environment. Using this information, the loss function of the agent is trained and optimized, in which key physical knowledge is embedded to guide the learning process. As the training progresses, the loss function continues to converge until the preset performance target value is reached, and the training stops. The resulting deep learning enhanced agent can efficiently output control signals to ensure that the power system can maintain safe and stable operation under various operating conditions, while also improving the intelligence level of preventive control of the power system.

[0086] More specifically, the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge, specifically:

[0087] Calculate the optimization result according to the optimal power flow calculation model, and use the optimization result as the first embedded physical knowledge of the loss function of the initial deep learning agent;

[0088] Calculating a transient stability margin according to a transient energy function, wherein the transient stability margin is used as second embedded physical knowledge of a loss function of an initial deep learning agent;

[0089] The first embedded physical knowledge and the second embedded physical knowledge are embedded into the loss function of the initial deep learning agent.

[0090] This application embeds two sets of key physical knowledge, the optimization results obtained by the optimal power flow calculation model and the transient stability margin calculated by the transient energy function, into the loss function of the initial deep learning intelligent agent. This method ensures that the intelligent agent not only pursues the maximization of economic benefits during the learning process, but also fully considers the safe and stable operation of the power system. This method of embedding physical knowledge improves the accuracy and reliability of the intelligent agent's decision-making, allowing the intelligent agent to generate more scientific and effective control strategies. Therefore, the intelligent agent can comprehensively consider the various needs of the power system and optimize the preventive control strategy, thereby improving the stability and economic operation efficiency of the power system when facing various operating conditions and potential faults.

[0091] More specifically, the optimization result is calculated according to the optimal power flow calculation model, and the optimization result is used as the first embedded physical knowledge of the loss function of the initial deep learning agent, specifically:

[0092] According to the basic constraints of power system safety and stability and the safety and stability constraints under N-1 faults, the objective function with economic benefits as the optimization goal is solved to obtain the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source; where N is a positive integer greater than 1;

[0093] The active power and reactive power of the generator under the optimal power flow and the reactive power of the reactive power source are the first embedded physical knowledge of the loss function of the initial deep learning agent.

[0094] Specifically, the first method for generating embedded physical knowledge is:

[0095] Based on the N-1 rule, the optimal power flow model considering safety and stability preventive control constraints is established:

[0096] The optimization model constructs the objective function with the best economic benefit:

[0097]

[0098] Where F is the objective function; NT is the total number of time periods in the dispatch cycle, NG is the number of generator sets in the system, and P is G,i,t is the power generation of generator set i in time period t; a i is the coefficient of the quadratic term of fuel cost, b i Fuel cost primary coefficient, c i It is the fixed cost of fuel; i,t is the operating status logic parameter of the generator set, I i,t =1 means that at time t, generator set i is in operation state, I i,t =0 means that generator set i is in shutdown state at time t; Y i,t is the number of startups of generator set i up to time t, c ST,t The fixed cost of starting the unit; Z i,t is the number of shutdowns of generator set i up to time t, c SD,t Fixed cost for shutting down the unit; VOLL d P is the unit penalty fee for load shedding at the electricity user end, LS,d,t It is the amount of active load that is forced to be removed at the power consumption terminal d at time t.

[0099] The basic constraints on the operation of the power system include power balance constraints, upper and lower limit constraints on unit output, unit climbing constraints, load shedding upper limit constraints, and power flow safety constraints.

[0100] The power balance constraint is:

[0101]

[0102] Where ND is the total number of power consumption terminals, P D,d,t is the original active load at the power consumption terminal d at time t, S T A collection of time periods.

[0103] Unit output upper and lower limit constraints:

[0104]

[0105] Where P G,min,i is the lower limit of active power output of generator set i in operation, P G,max,i It is the active power output online of generator set i in running state.

[0106] Unit climbing constraints:

[0107]

[0108] In the formula, R U,i is the maximum climbing value of generator set i per hour, R D,i is the maximum downhill slope of generator set i per hour, P ST,iis the minimum active power output of generator set i when it is turned on, P SD,i It is the minimum active power output when generator set i is shut down.

[0109] Load shedding upper limit constraint:

[0110]

[0111] Power flow node equality constraints:

[0112]

[0113] In the formula, S B is the set of all nodes in the power network; KL is a NB*NL order matrix, and the element KL b,l Represents the connection relationship between node b and branch l, KL b,l =1 means that there is a connection between node b and branch l, KL b,l = 0, otherwise; KG is a NB*NG order matrix, and the element KG b,i Represents the connection relationship between node b and generator set i, KG b,i =1 means that node b is connected to generator set i, KG b,i On the contrary; KD is a NB*ND order matrix, and the element KD b,d Characterizes the connection relationship between node b and load d, KD b,d =1 indicates that there is a connection between node b and load d, KD b,d =0, otherwise; P L,l,t is the active power flow of line l at time t.

[0114] Power flow safety inequality constraints:

[0115] V i,min,t ≤V i,t ≤V i,max,t ,i∈S B ,t∈S T (18)

[0116] P L,l,min,t ≤P L,l,t ≤P L,l,max,t ,l∈S L ,t∈S T (19)

[0117] In the formula, S L is the collection of all branches of the power system; V i is the voltage amplitude of node i, V i,max 、V i,min They correspond to the maximum and minimum values ​​of the voltage amplitude at node i during safe and stable operation; P L,l,max , P L,l,minThey correspond to the maximum and minimum values ​​of the useful power of the power flow in branch l respectively.

[0118] Power flow node constraints under N-1 expected failure:

[0119]

[0120] In the formula, S c is the fault set, is the matrix that represents the connection relationship between lines and nodes in the power network under the expected fault state c, is the power flow of line l under the expected fault state c at time t.

[0121] Power flow safety inequality constraints under N-1 expected faults:

[0122]

[0123] In the formula, is the voltage amplitude of node i at time t under the expected fault state c, is the power flow active power of branch l in fault state c at time t.

[0124] In this way, an optimization model for the optimal power flow problem can be formed which takes into account safety, stability, preventive control constraints and takes optimal economic benefits as the optimization goal. The corresponding results obtained by calculating the power system in provinces based on this model are used to extract the optimization results as the physical knowledge of the intelligent agent. The extracted physical knowledge includes the active power and reactive power of the generator under the optimal power flow, and the reactive power of the reactive power source.

[0125] This application solves the objective function with the goal of maximizing economic benefits while satisfying the basic constraints of power system safety and stability and N-1 fault conditions, thereby obtaining the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source. These optimization results, as the first set of embedded physical knowledge, provide an economic benefit-oriented reference benchmark for the intelligent agent, ensuring that the intelligent agent can generate control strategies that both meet the safety and stability requirements of the power system and optimize economic benefits during the learning process. This embedding method enables the intelligent agent to internalize the economic operation characteristics of the power system in the early stages of training, so that it can more efficiently explore and optimize the control strategy in the subsequent learning process, thereby improving the overall performance and economic benefits of the preventive control method of the power system.

[0126] The transient stability margin is calculated according to the transient energy function, and the transient stability margin is used as the second embedded physical knowledge of the initial deep learning agent loss function, specifically:

[0127] According to the transient energy function method, the transient stability kinetic energy function and the transient stability potential energy function are calculated to form a transient energy function;

[0128] According to the instability limit, calculating the critical transient energy of the transient energy function when a preset fault occurs, and calculating the transient stability margin according to the critical transient energy;

[0129] The transient stability margin is the second embedded physical knowledge of the loss function of the initial deep learning agent.

[0130] Specifically, the second method for generating embedded physical knowledge is:

[0131] Power system vulnerability assessment system based on energy function method:

[0132] Synchronous generator model rotor equation:

[0133]

[0134] Where M i is the inertia time constant of generator set i; ω i is the angular velocity relative to the system's stationary reference axis, ω R is the angular velocity of the system synchronization reference axis; δ i is the power angle of generating set i relative to the system synchronous reference axis; P mi is the output power of the prime mover of generator set i, P ei is the electromagnetic output power of generator set i; f i (δ) is the unbalanced power of the generator rotor.

[0135] The transient energy kinetic energy function of the system:

[0136]

[0137]

[0138] In the formula, is the kinetic energy of transient stability of the system; A is the set of leading generators, B is the set of lagging generators; M eq is the equivalent inertia constant of the system.

[0139] The transient energy potential function of the system:

[0140]

[0141] In the formula, is the transient energy potential; α A is the weighted power angle of the leading generator group, α B is the price power angle of the lagging generator group, α AB is the power angle difference between the two units; P AB is the acceleration power of the system.

[0142] The transient energy function of the system is comprehensively formed:

[0143]

[0144] Among them, V cr is the transient energy of the system.

[0145] Determine the critical energy V of the system through the instability limit cr And design the transient stability margin index:

[0146]

[0147] In the formula, is the transient stability margin of the system when the expected fault i occurs, is the critical transient energy of the system under the anticipated fault i.

[0148] In summary, a power system vulnerability assessment system based on the energy function method can be comprehensively formed. Through the understanding of this system, the transient critical energy estimation method calculates the transient critical energy of the power system in provinces as the physical knowledge of the intelligent agent.

[0149] This application first constructs a transient energy function by calculating the transient stability kinetic energy function and the transient stability potential energy function, and then uses the system's instability limit to determine the critical transient energy under preset fault conditions, and then calculates the system's transient stability margin. The embedding of this transient stability margin allows the intelligent agent to not only consider economic benefits during the learning process, but also pay more attention to the dynamic stability of the system. The intelligent agent can make more accurate control decisions based on the system's stability margin, so that when the power system faces transient disturbances, it can respond quickly and take preventive measures to effectively avoid system instability and ensure the safe operation and reliability of the power system.

[0150] More specifically, the embedding of the first embedded physical knowledge and the second embedded physical knowledge into the loss function of the initial deep learning agent is specifically as follows:

[0151] The loss function after embedding the first embedded physical knowledge and the second embedded physical knowledge is:

[0152]

[0153] Where λ1 and λ2 are the weight coefficients of the optimal power flow and the transient stability margin, respectively; n is the total number of controllable variables, x is i is the controllable variable i of the system after the action, x' i is the optimization result of extracting the corresponding variable from the first embedded physical knowledge. In summary, a complete deep reinforcement learning agent with loss function embedded knowledge can be formed.

[0154] S03: According to the control signal, the output power and switching of the generator and reactive power source in the power system are adjusted to make the power system operate in a safe and stable working state.

[0155] This application obtains and analyzes the operating status information of the power system in real time, and uses the intelligent agent trained through interaction with the simulation environment to make rapid decisions. The intelligent agent is also trained using a loss function embedded with physical knowledge, and outputs control signals to dynamically adjust the output of generators and reactive power sources, effectively improving the reliability of preventive control strategies, thereby ensuring that the power system can maintain a safe and stable operating state under various operating conditions.

[0156] Embodiment 2

[0157] Please refer to Figure 3 , which is a reinforcement learning prevention and control device with loss function embedded knowledge provided in an embodiment of the present application.

[0158] In this embodiment, the reinforcement learning prevention and control device with loss function embedded in knowledge includes an acquisition module 10, an input and output module 20 and an adjustment module 30.

[0159] The acquisition module 10 is used to obtain real-time operation status information of the power system;

[0160] The input-output module 20 is used to input the real-time operation status information into the deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control; wherein the deep reinforcement learning agent is obtained by interactive training of the initial deep reinforcement learning agent and the environment constructed by the power system simulation model; the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge;

[0161] The real-time operation status information is input into the deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control. The generation method of the entire preventive strategy is as follows: Figure 2 As shown;

[0162] More specifically, the initial deep learning agent is constructed according to the Markov process, and the construction process of the initial deep learning agent is:

[0163] The actual power system or simulation software is used as the environment for the initial deep reinforcement learning agent to interact. The initial deep reinforcement learning agent optimizes and adjusts the value function according to the state and reward feedback from the environment and takes actions on the environment according to the value function. The environment forms the corresponding state and reward according to the input action and feeds back to the agent. The initial deep reinforcement learning agent stores the current state, the action taken, the reward feedback from the environment, and the previous state of the environment in the experience playback module. Among them:

[0164] The reward function of the environment:

[0165]

[0166] ΔF(S t )=F(S t )-F(S t+1 ) (33)

[0167]

[0168] G PF (S t )=-R P (S t )-R V (S t ) (35)

[0169]

[0170]

[0171] Where, the power flow constraint is the equation (2-12); F(S t ) is the S in the environment t The trend economic function under the state of is the change in the trend economic function after the initial deep reinforcement learning agent action; p, r, α are positive weight coefficients; G PF (S t ) is the power flow constraint penalty function, R P (St) is the active constraint penalty function, R V (S t ) is the voltage constraint penalty function, P G,i,t is the active output of generator set i at time t, P G,max,i and P G,min,i They are the upper and lower limits of the generator output, V i,t is the voltage amplitude of node i at time t, V i,max and V i,min is the upper and lower limits of the amplitude of node i; G TS (S t ) is the transient stability penalty function, Ω instability It is the unstable region of the system.

[0172] The value function is built by a deep neural network, which is divided into a value network and a target value network. The input of the value network is the current state S t , output all actions A made in this state i , the output is the value Q predicted by the neural network under all actions s,i; The target value network input is the environment state S t The next state S t+1 , predict S t+1 All actions in state A i And the corresponding value Q t+1,i , and select the one with the largest value.

[0173] For the value network, the loss function is:

[0174] L=[Q(S t ,A t )-(R+γmaxQ(S t+1 ,A t+1 )] 2 (40)

[0175] In the formula, Q(S t ,A t ) is the value network in S t Status, A t The predicted value of the action, R is the reward of the environment after the action, and γ is the decay coefficient of the reward. Based on the Bayesian principle, the value network updates itself after calculating the loss based on the target value network and the reward fed back by the environment:

[0176] Q'(S t ,A t )=Q(S t ,A t )+α[(R+γmaxQ(S t+1 ,A t+1 )-Q(S t ,A t )] (41)

[0177] In the formula, Q'(S t ,A t ) is the updated value network, and α is the learning rate of the agent. After T updates, the agent copies the value network to the target value network; the initial deep reinforcement learning agent calculates the value Q provided by the value function s,i Act on the environment, that is, control each controllable state of the power system's generators, loads, etc.

[0178] In summary, an initial deep reinforcement learning agent framework suitable for safe and stable preventive control of power systems can be formed.

[0179] More specifically, the deep reinforcement learning agent is obtained by interactive training of the initial deep reinforcement learning agent and the environment constructed by the power system simulation model, specifically:

[0180] The initial deep reinforcement learning agent probabilistically outputs control signals based on the output of the value function for the controllable variables in the environment constructed by the power system simulation model, namely, the active output power and reactive output power of the generator set and the load removal amount;

[0181] After receiving the control signal, the state of the environment constructed by the power system simulation model, i.e., the voltage of each node, the power transmitted by each line, etc., changes accordingly, and accordingly generates corresponding rewards and state signals which are transmitted to the initial deep reinforcement learning agent; the initial deep reinforcement learning agent receives the rewards and state signals and stores them as a set of data in the experience playback component;

[0182] The experience replay component uses experience to continuously train the agent until the value network and the target value network converge to unity. The experience replay component continuously trains the agent including: 1) the experience replay component samples a data and inputs them into the value network and the target value network respectively in turn; 2) the value network updates its own parameters according to the loss function after each data input; 3) after repeating steps 1) and 2) for 10 times, the value network parameters are copied to the target value network; 4) steps 1) to 3) are repeated until the value network and the target value network converge, at which time the two network parameters are exactly the same and form a unity, the strategy network achieves the optimal decision-making result, and a deep reinforcement learning agent is obtained.

[0183] In summary, through interactive learning with the environment, the deep reinforcement learning agent generates the optimal value function. The value function takes various states of the power system as input, outputs actions and their values ​​under various states, and selects the action with the best value to form two corresponding state-action pairs, that is, a deep reinforcement learning transient stability preventive control strategy with loss function embedded in the knowledge.

[0184] This application forms experience replay component information by collecting reward and state signal data generated by the interaction between the agent and the environment. Using this information, the loss function of the agent is trained and optimized, in which key physical knowledge is embedded to guide the learning process. As the training progresses, the loss function continues to converge until the preset performance target value is reached, and the training stops. The resulting deep learning enhanced agent can efficiently output control signals to ensure that the power system can maintain safe and stable operation under various operating conditions, while also improving the intelligence level of preventive control of the power system.

[0185] More specifically, the loss function of the initial deep learning agent is constructed based on the embedded physical knowledge, specifically:

[0186] Calculate the optimization result according to the optimal power flow calculation model, and use the optimization result as the first embedded physical knowledge of the loss function of the initial deep learning agent;

[0187] Calculating a transient stability margin according to a transient energy function, wherein the transient stability margin is used as second embedded physical knowledge of a loss function of an initial deep learning agent;

[0188] The first embedded physical knowledge and the second embedded physical knowledge are embedded into the loss function of the initial deep learning agent.

[0189] This application embeds two sets of key physical knowledge, the optimization results obtained by the optimal power flow calculation model and the transient stability margin calculated by the transient energy function, into the loss function of the initial deep learning intelligent agent. This method ensures that the intelligent agent not only pursues the maximization of economic benefits during the learning process, but also fully considers the safe and stable operation of the power system. This method of embedding physical knowledge improves the accuracy and reliability of the intelligent agent's decision-making, allowing the intelligent agent to generate more scientific and effective control strategies. Therefore, the intelligent agent can comprehensively consider the various needs of the power system and optimize the preventive control strategy, thereby improving the stability and economic operation efficiency of the power system when facing various operating conditions and potential faults.

[0190] More specifically, the optimization result is calculated according to the optimal power flow calculation model, and the optimization result is used as the first embedded physical knowledge of the loss function of the initial deep learning agent, specifically:

[0191] According to the basic constraints of power system safety and stability and the safety and stability constraints under N-1 faults, the objective function with economic benefits as the optimization goal is solved to obtain the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source; where N is a positive integer greater than 1;

[0192] The active power and reactive power of the generator under the optimal power flow and the reactive power of the reactive power source are the first embedded physical knowledge of the loss function of the initial deep learning agent.

[0193] Specifically, the first method for generating embedded physical knowledge is:

[0194] Based on the N-1 rule, the optimal power flow model considering safety and stability preventive control constraints is established:

[0195] The optimization model constructs the objective function with the best economic benefit:

[0196]

[0197] Where F is the objective function; NT is the total number of time periods in the dispatch cycle, NG is the number of system generators, and P is G,i,t is the power generation of generator set i in time period t; a i is the coefficient of the quadratic term of fuel cost, b i Fuel cost primary coefficient, ci It is the fixed cost of fuel; i,t is the operating status logic parameter of the generator set, I i,t =1 means that at time t, generator set i is in operation state, I i,t =0 means that generator set i is in shutdown state at time t; Y i,t is the number of startups of generator set i up to time t, c ST,t The fixed cost of starting the unit; Z i,t is the number of shutdowns of generator set i up to time t, c SD,t Fixed cost for shutting down the unit; VOLL d P is the unit penalty fee for load shedding at the electricity user end, LS,d,t It is the amount of active load that is forced to be removed at the power consumption terminal d at time t.

[0198] The basic constraints on the operation of the power system include power balance constraints, upper and lower limit constraints on unit output, unit climbing constraints, load shedding upper limit constraints, and power flow safety constraints.

[0199] The power balance constraint is:

[0200]

[0201] Where ND is the total number of power consumption terminals, P D,d,t is the original active load at the power consumption terminal d at time t, S T A collection of time periods.

[0202] Unit output upper and lower limit constraints:

[0203]

[0204] Where P G,min,i is the lower limit of active power output of generator set i in operation, P G,max,i It is the active power output online of generator set i in running state.

[0205] Unit climbing constraints:

[0206]

[0207] In the formula, R U,i is the maximum climbing value of generator set i per hour, R D,i is the maximum downhill slope of generator set i per hour, P ST,i is the minimum active power output of generator set i when it is turned on, P SD,i It is the minimum active power output when generator set i is shut down.

[0208] Load shedding upper limit constraint:

[0209]

[0210] Power flow node equality constraints:

[0211]

[0212] In the formula, S B is the set of all nodes in the power network; KL is a NB*NL order matrix, and the element KL b,l Represents the connection relationship between node b and branch l, KL b,l =1 means that there is a connection between node b and branch l, KL b,l = 0, otherwise; KG is a NB*NG order matrix, and the element KG b,i Represents the connection relationship between node b and generator set i, KG b,i =1 indicates that node b is connected to generator set i, KG b,i On the contrary; KD is a NB*ND order matrix, and the element KD b,d Characterizes the connection relationship between node b and load d, KD b,d =1 indicates that there is a connection between node b and load d, KD b,d =0, otherwise; P L,l,t is the active power flow of line l at time t.

[0213] Power flow safety inequality constraints:

[0214] V i,min,t ≤V i,t ≤V i,max,t ,i∈S B ,t∈S T (49)

[0215] P L,l,min,t ≤P L,l,t ≤P L,l,max,t ,l∈S L ,t∈S T (50)

[0216] In the formula, S L is the collection of all branches of the power system; V i is the voltage amplitude of node i, V i,max 、V i,min They correspond to the maximum and minimum values ​​of the voltage amplitude of node i during safe and stable operation; P L,l,max , P L,l,min They correspond to the maximum and minimum values ​​of the useful power of the power flow in branch l respectively.

[0217] Power flow node constraints under N-1 expected failure:

[0218]

[0219] In the formula, S c is the fault set, is the matrix that represents the connection relationship between lines and nodes in the power network under the expected fault state c, is the power flow of line l under the expected fault state c at time t.

[0220] Power flow safety inequality constraints under N-1 expected faults:

[0221]

[0222] In the formula, is the voltage amplitude of node i at time t under the expected fault state c, is the power flow active power of branch l in fault state c at time t.

[0223] In this way, an optimization model for the optimal power flow problem can be formed which takes into account safety, stability, preventive control constraints and takes optimal economic benefits as the optimization goal. The corresponding results obtained by calculating the power system in provinces based on this model are used to extract the optimization results as the physical knowledge of the intelligent agent. The extracted physical knowledge includes the active power and reactive power of the generator under the optimal power flow, and the reactive power of the reactive power source.

[0224] This application solves the objective function with the goal of maximizing economic benefits while satisfying the basic constraints of power system safety and stability and N-1 fault conditions, thereby obtaining the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source. These optimization results, as the first set of embedded physical knowledge, provide an economic benefit-oriented reference benchmark for the intelligent agent, ensuring that the intelligent agent can generate control strategies that both meet the safety and stability requirements of the power system and optimize economic benefits during the learning process. This embedding method enables the intelligent agent to internalize the economic operation characteristics of the power system in the early stages of training, so that it can more efficiently explore and optimize the control strategy in the subsequent learning process, thereby improving the overall performance and economic benefits of the preventive control method of the power system.

[0225] The transient stability margin is calculated according to the transient energy function, and the transient stability margin is used as the second embedded physical knowledge of the initial deep learning agent loss function, specifically:

[0226] According to the transient energy function method, the transient stability kinetic energy function and the transient stability potential energy function are calculated to form a transient energy function;

[0227] According to the instability limit, calculating the critical transient energy of the transient energy function when a preset fault occurs, and calculating the transient stability margin according to the critical transient energy;

[0228] The transient stability margin is the second embedded physical knowledge of the loss function of the initial deep learning agent.

[0229] Specifically, the second method for generating embedded physical knowledge is:

[0230] Power system vulnerability assessment system based on energy function method:

[0231] Synchronous generator model rotor equation:

[0232]

[0233] Where M i is the inertia time constant of generator set i; ω i is the angular velocity relative to the system's stationary reference axis, ω R is the angular velocity of the system synchronization reference axis; δ i is the power angle of generating set i relative to the system synchronous reference axis; P mi is the output power of the prime mover of generator set i, P ei is the electromagnetic output power of generator set i; f i (δ) is the unbalanced power of the generator rotor.

[0234] The transient energy kinetic energy function of the system:

[0235]

[0236] In the formula, is the kinetic energy of transient stability of the system; A is the set of leading generators, B is the set of lagging generators; M eq is the equivalent inertia constant of the system.

[0237] The transient energy potential function of the system:

[0238]

[0239] In the formula, is the transient energy potential; α A is the weighted power angle of the leading generator group, α B is the price power angle of the lagging generator group, α AB is the power angle difference between the two units; P AB is the acceleration power of the system.

[0240] The transient energy function of the system is comprehensively formed:

[0241]

[0242] Among them, V cr is the transient energy of the system.

[0243] Determine the critical energy V of the system through the instability limit cr And design the transient stability margin index:

[0244]

[0245] In the formula, is the transient stability margin of the system when the expected fault i occurs, is the critical transient energy of the system under the anticipated fault i.

[0246] In summary, a power system vulnerability assessment system based on the energy function method can be comprehensively formed. Through the understanding of this system, the transient critical energy estimation method calculates the transient critical energy of the power system in provinces as the physical knowledge of the intelligent agent.

[0247] This application first constructs a transient energy function by calculating the transient stability kinetic energy function and the transient stability potential energy function, and then uses the system's instability limit to determine the critical transient energy under preset fault conditions, and then calculates the system's transient stability margin. The embedding of this transient stability margin allows the intelligent agent to not only consider economic benefits during the learning process, but also pay more attention to the dynamic stability of the system. The intelligent agent can make more accurate control decisions based on the system's stability margin, so that when the power system faces transient disturbances, it can respond quickly and take preventive measures to effectively avoid system instability and ensure the safe operation and reliability of the power system.

[0248] More specifically, the embedding of the first embedded physical knowledge and the second embedded physical knowledge into the loss function of the initial deep learning agent is specifically as follows:

[0249] The loss function after embedding the first embedded physical knowledge and the second embedded physical knowledge is:

[0250]

[0251] Where λ1 and λ2 are the weight coefficients of the optimal power flow and the transient stability margin, respectively; n is the total number of controllable variables, x is i is the controllable variable i of the system after the action, x' i is the optimization result of extracting the corresponding variable from the first embedded physical knowledge. In summary, a complete deep reinforcement learning agent with loss function embedded knowledge can be formed.

[0252] The adjustment module 30 is used to adjust the output power and switching of the generator and reactive power source in the power system according to the control signal, so that the power system operates in a safe and stable working state.

[0253] This device uses three modules to divide the work and coordinate work to better output preventive control strategies according to the power system status. This application obtains and analyzes the operating status information of the power system in real time, and uses the intelligent agent obtained by interactive training with the simulation environment to make quick decisions. The intelligent agent also uses the loss function embedded with physical knowledge for training, and outputs control signals to dynamically adjust the output of generators and reactive power sources, effectively improving the reliability of preventive control strategies, thereby ensuring that the power system can maintain a safe and stable operating state under various working conditions.

[0254] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A reinforcement learning preventive control method with loss function embedded knowledge, characterized in that: include: Obtain real-time operating status information of the power system; The real-time operation status information is input into a deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control; wherein the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent and an environment constructed by a power system simulation model; and the loss function of the initial deep learning agent is constructed based on embedded physical knowledge; According to the control signal, the output power and switching of the generator and reactive power source in the power system are adjusted to make the power system operate in a safe and stable working state.

2. The reinforcement learning prevention and control method of embedding loss function knowledge according to claim 1 is characterized in that: The deep reinforcement learning agent is obtained by interactive training between the initial deep reinforcement learning agent and the environment constructed by the power system simulation model, specifically: Acquire experience replay component information; wherein the experience replay component information is obtained by collecting rewards and state signal data generated during the interaction between the initial deep reinforcement learning agent and the environment constructed by the power system simulation model; Training the initial deep learning agent according to the experience playback component information, and optimizing the loss function of the initial deep learning agent; When the loss function of the initial deep learning agent converges to the preset target value, the training is stopped and a deep learning reinforcement agent is obtained.

3. The reinforcement learning prevention and control method of embedding loss function knowledge according to claim 1 is characterized in that: The loss function of the initial deep learning agent is constructed based on the embedded physical knowledge, specifically: Calculate the optimization result according to the optimal power flow calculation model, and use the optimization result as the first embedded physical knowledge of the loss function of the initial deep learning agent; Calculating a transient stability margin according to a transient energy function, wherein the transient stability margin is used as second embedded physical knowledge of a loss function of an initial deep learning agent; The first embedded physical knowledge and the second embedded physical knowledge are embedded into the loss function of the initial deep learning agent.

4. The reinforcement learning prevention and control method of embedding loss function knowledge according to claim 3 is characterized in that: The optimization result is calculated according to the optimal power flow calculation model, and the optimization result is used as the first embedded physical knowledge of the loss function of the initial deep learning agent, specifically: According to the basic constraints of power system safety and stability and the safety and stability constraints under N-1 faults, the objective function with economic benefits as the optimization goal is solved to obtain the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source; where N is a positive integer greater than 1; The active power and reactive power of the generator under the optimal power flow and the reactive power of the reactive power source are the first embedded physical knowledge of the loss function of the initial deep learning agent.

5. The reinforcement learning prevention and control method of embedding loss function knowledge according to claim 3 is characterized in that: The transient stability margin is calculated according to the transient energy function, and the transient stability margin is used as the second embedded physical knowledge of the initial deep learning agent loss function, specifically: According to the transient energy function method, the transient stability kinetic energy function and the transient stability potential energy function are calculated to form a transient energy function; According to the instability limit, calculating the critical transient energy of the transient energy function when a preset fault occurs, and calculating the transient stability margin according to the critical transient energy; The transient stability margin is the second embedded physical knowledge of the loss function of the initial deep learning agent.

6. A reinforcement learning prevention and control device with loss function embedded knowledge, characterized in that: It includes an acquisition module, an input and output module, and an adjustment module; The acquisition module is used to obtain the real-time operation status information of the power system; The input-output module is used to input the real-time operation status information into the deep reinforcement learning agent so that the deep reinforcement learning agent outputs a control signal for preventive control; wherein the deep reinforcement learning agent is obtained by interactive training of an initial deep reinforcement learning agent and an environment constructed by a power system simulation model; and the loss function of the initial deep learning agent is constructed based on embedded physical knowledge; The adjustment module is used to adjust the output power and switching of the generator and reactive power supply in the power system according to the control signal, so that the power system operates in a safe and stable working state.

7. The reinforcement learning prevention and control device with loss function embedded knowledge according to claim 6 is characterized in that: The deep reinforcement learning agent is obtained by interactive training between the initial deep reinforcement learning agent and the environment constructed by the power system simulation model, specifically: Acquire experience replay component information; wherein the experience replay component information is obtained by collecting rewards and state signal data generated during the interaction between the initial deep reinforcement learning agent and the environment constructed by the power system simulation model; Training the initial deep learning agent according to the experience playback component information, and optimizing the loss function of the initial deep learning agent; When the loss function of the initial deep learning agent converges to the preset target value, the training is stopped and a deep learning reinforcement agent is obtained.

8. The reinforcement learning prevention and control device with loss function embedded knowledge according to claim 6 is characterized in that: The loss function of the initial deep learning agent is constructed based on the embedded physical knowledge, specifically: Calculate the optimization result according to the optimal power flow calculation model, and use the optimization result as the first embedded physical knowledge of the loss function of the initial deep learning agent; Calculating a transient stability margin according to a transient energy function, wherein the transient stability margin is used as second embedded physical knowledge of a loss function of an initial deep learning agent; The first embedded physical knowledge and the second embedded physical knowledge are embedded into the loss function of the initial deep learning agent.

9. The reinforcement learning prevention and control device with loss function embedded knowledge according to claim 7, characterized in that: The optimization result is calculated according to the optimal power flow calculation model, and the optimization result is used as the first embedded physical knowledge of the loss function of the initial deep learning agent, specifically: According to the basic constraints of power system safety and stability and the safety and stability constraints under N-1 faults, the objective function with economic benefits as the optimization goal is solved to obtain the active power and reactive power of the generator under the optimal power flow, as well as the reactive power of the reactive power source; where N is a positive integer greater than 1; The active power and reactive power of the generator under the optimal power flow and the reactive power of the reactive power source are the first embedded physical knowledge of the loss function of the initial deep learning agent.

10. The reinforcement learning prevention and control device with loss function embedded knowledge according to claim 7, characterized in that: The transient stability margin is calculated according to the transient energy function, and the transient stability margin is used as the second embedded physical knowledge of the initial deep learning agent loss function, specifically: According to the transient energy function method, the transient stability kinetic energy function and the transient stability potential energy function are calculated to form a transient energy function; According to the instability limit, calculating the critical transient energy of the transient energy function when a preset fault occurs, and calculating the transient stability margin according to the critical transient energy; The transient stability margin is the second embedded physical knowledge of the loss function of the initial deep learning agent.

Images