Virtual PSS2A parameter setting method based on reverse reinforcement learning
By constructing a virtual PSS2A structure based on inverse reinforcement learning and optimizing parameter tuning, the "reverse adjustment" problem of the virtual power system stabilizer is solved, thereby improving the stability and enhancing the anti-disturbance capability of the new power system.
Patent Information
- Application Number
- CN202411320885.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-23
- Publication Date
- 2025-01-10
AI Technical Summary
During the transition from traditional power systems to new power systems, virtual power system stabilizers exhibit a "reverse adjustment" phenomenon. Furthermore, the reward function design of reinforcement learning algorithms is highly subjective during parameter tuning, leading to prominent stability issues and weak anti-disturbance capabilities in the new power systems.
A virtual PSS2A structure is constructed using an inverse reinforcement learning approach. The virtual PSS2A parameters are optimized by combining a deep deterministic gradient policy algorithm and maximum entropy inverse reinforcement learning. The network parameters are further optimized through Markov decision process and expert experience loss, eliminating the subjectivity of the reward function and providing an adaptive positive damping configuration.
It improves the stability of the new power system, effectively suppresses low-frequency oscillations, enhances the system's anti-disturbance capability, and adapts to the steady-state operating point under various operating conditions.
Smart Images
Figure CN119298095A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of power system stability control, and in particular relates to a virtual power system stability (VirtualPSS2A) parameter setting method, and more specifically to a virtual PSS2A parameter setting method based on inverse reinforcement learning. Background Art
[0002] In the context of the transformation from traditional power systems to new power systems, more and more thermal power units are participating in deep peak load regulation. Due to the low inertia and weak damping characteristics of new energy power generation units, the new power system has weak anti-disturbance ability and greater operating risks. However, new energy units cannot participate in system frequency regulation under general control methods, making the power system present a "low inertia" feature, resulting in prominent stability issues in the new power system.
[0003] The virtual power system stabilizer in the existing technology has the phenomenon of "reverse adjustment", which is not conducive to improving the anti-disturbance capability of the new power system. Moreover, in the process of using reinforcement learning algorithm for real-time parameter tuning optimization, most of the artificially designed reward functions are highly subjective, and the rewards obtained only through interaction with the environment are usually sparse, which makes it extremely difficult for the algorithm to converge during training and the parameter tuning effect is poor. Summary of the invention
[0004] In view of the shortcomings of the above-mentioned prior art, the present invention provides a virtual PSS2A parameter setting method based on inverse reinforcement learning. Its purpose is to solve the "reverse adjustment" problem of the virtual PSS and eliminate the subjectivity of the artificially designed reward function in the parameter setting process, so as to achieve the invention purpose of improving the stability of the new power system.
[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:
[0006] A virtual PSS2A parameter setting method based on inverse reinforcement learning comprises the following steps:
[0007] Construct a virtual PSS2A structure;
[0008] The virtual PSS2A structure is introduced into the reactive power control link of VSG, and a linear model of VSG grid-connected system considering the access of virtual PSS2A is constructed.
[0009] The virtual PSS2A parameter tuning problem is modeled as a Markov decision process, and the network parameters are optimized using a deep deterministic gradient strategy algorithm based on fusion expert experience loss.
[0010] The reward function implicit in the expert demonstration trajectory is solved by the maximum entropy inverse reinforcement learning algorithm as the reward function of the Markov decision process;
[0011] The parameters of virtual PSS2A are optimized by combining the strategy update algorithm integrating expert experience loss and the reward function solution algorithm based on maximum entropy inverse reinforcement learning.
[0012] Furthermore, the virtual PSS2A structure is constructed, wherein:
[0013] The virtual PSS2A structure expression is:
[0014]
[0015] Where s represents the complex variable used for Laplace transform, G VPSS2A (s) is the transfer function of the virtual PSS2A, K VPSS2A is the proportional coefficient of virtual PSS2A, T1-T6 are the time constants of virtual PSS2A;
[0016] Taking the fitted power deviation of the VSG virtual angular frequency deviation and the output active power deviation as input, and the reactive power deviation of the virtual PSS2A as output, the relationship between the input and output of the virtual PSS2A is:
[0017] ΔQ VPSS2A (s) = G VPSS2A (s)Δω1
[0018] Where s represents the complex variable used for Laplace transform, ΔQ VPSS2A is the output reactive power deviation of VPSS2A, G VPSS2A is the transfer function of VPSS2A added to the VSG reactive loop, and Δω1 is the fitting signal of VSG virtual angular frequency deviation Δω and output active power deviation ΔP.
[0019] Furthermore, the virtual PSS2A parameter setting problem is modeled as a Markov decision process, and a deep deterministic gradient strategy algorithm based on fusion expert experience loss is used to optimize network parameters; including: an online strategy function network μ(sθ μ ), online value function network Q(s,aθ Q ), target policy network μ′(sθ μ′ ), objective value function network parameters Q′(s,aθ Q′ );
[0020] The algorithm’s experience pool is composed of an expert demonstration trajectory dataset and a self-exploration trajectory dataset. A hybrid sampling mechanism is used to sample from the two datasets to form the final training data. The loss function is minimized by the gradient descent method, and the network parameter θ of the online value function is QThe optimization update is performed. The optimization of the online policy network parameters is divided into expert demonstration trajectory samples and self-exploration samples. While introducing the expert experience loss gradient to optimize the online policy iteration process, the self-exploration trajectory dataset is retained and the online policy network parameters θ are updated according to the online policy gradient value of the original DDPG algorithm. μ .
[0021] Furthermore, the process of solving the virtual PSS2A parameters by combining the strategy update algorithm integrating the expert experience loss and the maximum entropy-based inverse reinforcement learning reward function solution algorithm includes:
[0022] Step (1) Randomly initialize the online policy network μ(sθ μ ) and the online value function network Q(s,aθ Q )’s online policy network parameters θ μ and the online value function network parameter θ Q , initialize the target policy network parameters θ μ′ and the target value function network parameter θ Q′ and its weight, where: s is the state; a is the action;
[0023] Step (2) According to the formula r(ζ)=θ T F(ζ) constructs the reward function, where ζ is the expert trajectory, r(ζ) is the reward value of the expert trajectory, F(ζ) is the sum of the characteristic components of each state, and the reward function weight θ is initialized i ;
[0024] Step (3) Initialize the experience pool and take the expert demonstration trajectory dataset T expert Deposit into experience pool;
[0025] Step (4) performs network training with M iterations.
[0026] Furthermore, the network training with the number of iterations M includes:
[0027] ① The online policy network obtains action a=πθ(s)+η based on the current state s t , where η t is random noise; θ(s) is the online policy network under the current state s; the action selection strategy π depends on the design of the reward function;
[0028] ②Interact with the environment to perform action a, obtain a new state s′ and an immediate reward value r;
[0029] ③ The self-exploration sample data (s, a, r, s′) generated by interaction with the environment, namely T discover Deposit into the experience pool;
[0030] ④The current state s is equal to the new state s′;
[0031] ⑤ Randomly sample N sample data from the experience pool for training, and predict the distribution function based on the expert trajectory under the current strategy Estimate the partition function Z(θ), minimizing The target value shown, the weight of the reward function θ i Optimize and obtain the optimal reward function;
[0032] ⑥According to Update the online value function network parameters θ Q , if the training data T∈T expert , then according to Update the policy network parameters θ a , if the training data T∈T discover , then according to the online policy gradient value of the original deep deterministic gradient policy algorithm Update the online policy network parameters θ μ ;
[0033] ⑦ The update of target network parameters is based on the online network parameters using soft update method, the update method is as follows:
[0034]
[0035] Where τ is the soft update coefficient, θ μ′ ,θ Q′ They are target strategy network parameters and target value function network parameters respectively;
[0036] ⑧When s′ is in the terminal state, the current iteration ends, otherwise go to step ①.
[0037] A virtual PSS2A parameter setting device based on inverse reinforcement learning, comprising:
[0038] A virtual PSS2A structure building module, used to build a virtual PSS2A structure;
[0039] The VSG grid-connected system linearization model construction module is used to introduce the virtual PSS2A structure into the reactive power control link of the VSG and construct the VSG grid-connected system linearization model taking into account the access of the virtual PSS2A;
[0040] The optimization module is used to model the virtual PSS2A parameter tuning problem as a Markov decision process and optimize the network parameters using a deep deterministic gradient strategy algorithm based on fusion expert experience loss;
[0041] The reward function solving module is used to solve the implicit reward function in the expert demonstration trajectory through the maximum entropy inverse reinforcement learning algorithm as the reward function of the Markov decision process;
[0042] The reward function algorithm optimizes the virtual PSS2A parameter solution module, which is used to combine the strategy update algorithm integrating expert experience loss and the maximum entropy-based inverse reinforcement learning to solve the reward function algorithm to optimize the virtual PSS2A parameters.
[0043] Furthermore, the virtual PSS2A parameter setting device based on inverse reinforcement learning further includes:
[0044] Virtual power system stabilizer VPSS2A, used to suppress low-frequency oscillations caused by negative damping when VSG is connected to the power system;
[0045] The inverse reinforcement learning algorithm control module uses the maximum entropy inverse reinforcement learning algorithm to train the intelligent agent, and uses the intelligent agent to control the virtual PSS2A model, so that the virtual PSS2A can effectively suppress low-frequency oscillations when the steady-state operating point of the power system changes randomly.
[0046] Furthermore, the virtual PSS2A parameter setting device based on inverse reinforcement learning is used to implement any step of the virtual PSS2A parameter setting method based on inverse reinforcement learning.
[0047] A computer device comprises a storage medium, a processor and a computer program stored on the storage medium and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the methods for virtual PSS2A parameter setting based on inverse reinforcement learning are implemented.
[0048] A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any one of the methods for virtual PSS2A parameter setting based on inverse reinforcement learning.
[0049] The present invention has the following beneficial effects and advantages:
[0050] The present invention draws on the traditional power system stabilizer to construct a dual-input virtual PSS2A attached to the VSG virtual excitation regulation part, which solves the "reverse regulation" problem existing in the single-input virtual PSS, improves the problem of the VSG reactive loop providing negative damping torque, and enhances the anti-disturbance capability of the new power system.
[0051] The present invention takes into account the randomness of steady-state operating points in the actual operation of the power system, models the virtual PSS2A parameter setting problem into a Markov decision process, and adopts a deep deterministic gradient strategy algorithm based on fusion expert experience loss to optimize network parameters, so that the virtual PSS2A can adapt to various operating conditions of the VSG and achieve the goal of "self-adaptation" of various parameters of the virtual PSS2A.
[0052] The present invention uses the maximum entropy inverse reinforcement learning algorithm to self-learn the optimal reward function through expert trajectory, thereby eliminating the subjectivity of the artificially designed reward function in the process of parameter setting optimization of the virtual PSS2A reinforcement learning algorithm, ensuring that the virtual PSS2A provides the system with the optimal damping configuration in real time, achieving effective suppression of electromechanical oscillations under various working conditions, and improving the stability of the new power system. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0054] Figure 1 is a diagram of a virtual PSS2A structure of the present invention;
[0055] Figure 2 The Phillips-Heffron model of the VSG grid-connected system taking into account the access of the virtual PSS2A of the present invention;
[0056] Figure 3 It is a training framework diagram of the reverse reinforcement learning algorithm of the present invention. DETAILED DESCRIPTION
[0057] In order to more clearly understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0058] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0059] Refer to the following Figure 1-Figure 3 The technical solutions of some embodiments of the present invention are described.
[0060] Example 1
[0061] The present invention provides an embodiment, which is a virtual PSS2A parameter setting method based on inverse reinforcement learning. Figure 1 As shown, Figure 1 It is a diagram of the virtual PSS2A structure of the present invention.
[0062] In the present invention, VPSS2A is a virtual power system stabilizer, VPSS2A is a dual-input type virtual power system stabilizer, and VPSS is a single-input type virtual power system stabilizer.
[0063] The method of the present invention comprises the following steps:
[0064] Step 1. Construct a virtual PSS2A structure, taking the VSG virtual angular frequency deviation and the fitted power deviation of the output active power deviation as input, and taking the reactive power deviation of the virtual PSS2A as output.
[0065] Drawing on the traditional accelerating power type power system stabilizer, a virtual PSS2A structure is constructed. The fitting signals of VSG virtual angular frequency deviation and output active power deviation are taken as input, and the reactive power deviation of virtual PSS2A is taken as output, and the general structure of virtual PSS2A is obtained.
[0066] VSG is usually the mainstream control method for new energy units to provide virtual inertia and improve system frequency regulation. However, the VSG reactive loop will provide a certain amount of negative damping to the system, making the overall VSG-containing system present a weak damping characteristic, which is not conducive to the stable operation of the new power system. Therefore, the virtual PSS2A is introduced into the VSG reactive loop to provide positive damping to the system to offset the influence of negative damping. The general expression of the virtual PSS2A structure is:
[0067]
[0068] Where s represents the complex variable used for Laplace transform, G VPSS2A (s) is the transfer function of the virtual PSS2A, K VPSS2A is the proportional coefficient of the virtual PSS2A, and T1-T6 are the time constants of the virtual PSS2A.
[0069] The virtual PSS2A structure of the present invention takes the fitting signal Δω1 of the VSG virtual angular frequency deviation Δω and the output active power deviation ΔP as input, and takes the reactive power deviation ΔQ of the virtual PSS2A as input. VPSS2A For output, the relationship between virtual PSS2A input and output is:
[0070] ΔQ VPSS2A (s) = G VPSS2A (s)Δω1
[0071] Where s represents the complex variable used for Laplace transform, ΔQ VPSS2A is the output reactive power deviation of VPSS2A, G VPSS2A is the transfer function of VPSS2A added to the VSG reactive loop, Δω1 is the fitting signal of the VSG virtual angular frequency deviation Δω and the output active power deviation ΔP, and the fitting signal is the fitting power deviation.
[0072] Step 2. Introduce the virtual PSS2A structure into the reactive power control link of VSG, and construct a Phillips-Heffron linearized model of the VSG grid-connected system taking into account the access of virtual PSS2A.
[0073] like Figure 2As shown, Figure 2 The virtual PSS2A structure of the present invention introduces the reactive power control link of the VSG, and constructs a Phillips-Heffron model of the VSG grid-connected system taking into account the access of the virtual PSS2A.
[0074] In order to take into account the Phillips-Heffron model of the VSG grid-connected system with virtual PSS2A access, the virtual PSS2A is introduced into the VSG reactive loop to provide positive damping to the system to offset the influence of negative damping.
[0075] Step 3. Provide positive damping torque to the electromechanical oscillation circuit by adjusting the relevant control parameters of the virtual PSS2A. The virtual PSS2A parameter adjustment problem is modeled as a Markov decision process, and the online policy function network μ(sθ) in the deep neural network of the virtual PSS2A parameters is optimized using the deep deterministic gradient strategy algorithm (DDPG) based on the fusion of expert experience loss. μ ), online value function network Q(s,aθ Q ), target policy network μ′(sθ μ′ ), objective value function network parameters Q′(s,aθ Q′ ). The algorithm’s experience pool consists of an expert demonstration trajectory dataset and a self-exploration trajectory dataset, and a mixed sampling mechanism is used to sample from the two datasets to form the final training data. The loss function is minimized by the gradient descent method, and the network parameter θ of the online value function is Q The optimization update is performed, and the optimization of the online policy network parameters is divided into two parts: expert demonstration trajectory samples and self-exploration samples. While introducing the expert experience loss gradient to optimize the online policy iteration process, the self-exploration trajectory dataset is also retained, and the online policy network parameters θ are updated according to the online policy gradient value of the original DDPG algorithm. μ .
[0076] Step 4. Solve the reward function implicit in the expert demonstration trajectory through the maximum entropy inverse reinforcement learning algorithm as the reward function of the Markov decision process.
[0077] Considering the randomness of the steady-state operating point in the actual operation of the power system and eliminating the subjectivity of the artificially designed reward function in the process of parameter setting optimization of the virtual PSS2A, the maximum entropy inverse reinforcement learning algorithm is used to optimize the parameter setting of the virtual PSS2A.
[0078] At a series of consecutive decision moments, the policy network makes a decision based on the current state s of the intelligent agent; after the decision is implemented, the policy network obtains an immediate reward value based on the reward function designed by inverse reinforcement learning. The reward corresponds to the network decision and the environment state. After that, the network will enter the state corresponding to the next moment of the decision and forward update the network parameters through the deep deterministic gradient strategy algorithm that integrates expert supervision loss; at the new training time step, the network will execute a new decision and obtain a new reward value based on the current new state, and repeat this cycle until the network completes parameter tuning with the optimal strategy under the optimal reward function implied by the expert trajectory.
[0079] Among them, expert demonstration refers to the data and trajectories of the optimal operating strategies and behavior patterns under various operating conditions of the actual power system.
[0080] Step 5. Combining the strategy update algorithm integrating expert experience loss and the maximum entropy-based inverse reinforcement learning reward function algorithm to optimize the virtual PSS2A parameter solution process, including:
[0081] Step (1) Randomly initialize the online policy network μ(sθ μ ) and the online value function network Q(s,aθ Q )’s online policy network parameters θ μ and the online value function network parameter θ Q , initialize the target policy network parameters θ μ′ and the target value function network parameter θ Q′ and its weight. Among them: s is the state; a is the action.
[0082] Step (2) According to the formula r(ζ)=θ T F(ζ) constructs the reward function, where ζ is the expert trajectory, r(ζ) is the reward value of the expert trajectory, and F(ζ) is the sum of the characteristic components of each state. Initialize the reward function weight θ i .
[0083] Step (3) Initialize the experience pool and take the expert demonstration trajectory dataset T expert Deposit into experience pool.
[0084] Step (4) performs network training with an iteration number of M, specifically comprising the following steps:
[0085] ① The online policy network obtains action a=πθ(s)+η based on the current state s t , where η t is random noise; θ(s) is the online policy network under the current state s; the action selection strategy π depends on the design of the reward function;
[0086] ②Interact with the environment to perform action a, obtain a new state s′ and an immediate reward value r;
[0087] ③ The self-exploration sample data (s, a, r, s′) generated by interaction with the environment, namely T discover Deposit into the experience pool;
[0088] ④The current state s is equal to the new state s′;
[0089] ⑤ Randomly sample N sample data from the experience pool for training, and predict the distribution function based on the expert trajectory under the current strategy Estimate the partition function Z(θ), minimizing The target value shown, the weight of the reward function θ i Optimize and obtain the optimal reward function;
[0090] ⑥According to Update the online value function network parameters θ Q , if the training data T∈T expert , then according to Update the policy network parameters θ a , if the training data T∈T discover , then according to the online policy gradient value of the original deep deterministic gradient policy algorithm Update the online policy network parameters θ μ ;
[0091] ⑦ The update of target network parameters is based on the online network parameters using soft update method, and the update method is:
[0092]
[0093] Where τ is the soft update coefficient, θ μ′ ,θ Q′ They are the target strategy network parameters and the target value function network parameters respectively.
[0094] ⑧When s′ is in the terminal state, the current iteration ends, otherwise go to step ①.
[0095] Example 2
[0096] The present invention further provides an embodiment, which is a virtual power system stabilizer parameter setting device, comprising:
[0097] A virtual PSS2A structure building module, used to build a virtual PSS2A structure;
[0098] The VSG grid-connected system linearization model building module is used to introduce the virtual PSS2A structure into the reactive power control link of the VSG and build the VSG grid-connected system linearization model taking into account the virtual PSS2A access;
[0099] The optimization module is used to model the virtual PSS2A parameter tuning problem as a Markov decision process and optimize the network parameters using a deep deterministic gradient strategy algorithm based on fusion expert experience loss;
[0100] The reward function solving module is used to solve the implicit reward function in the expert demonstration trajectory through the maximum entropy inverse reinforcement learning algorithm as the reward function of the Markov decision process;
[0101] The reward function algorithm optimizes the virtual PSS2A parameter solution module, which is used to combine the strategy update algorithm integrating expert experience loss and the maximum entropy-based inverse reinforcement learning to solve the reward function algorithm to optimize the virtual PSS2A parameters.
[0102] Also includes:
[0103] The virtual power system stabilizer VPSS2A is used to suppress the low-frequency oscillation caused by negative damping when VSG is connected to the power system.
[0104] The inverse reinforcement learning algorithm control module uses the maximum entropy inverse reinforcement learning algorithm to train the intelligent agent, and uses the intelligent agent to control the virtual PSS2A model, so that the virtual PSS2A can effectively suppress low-frequency oscillations when the steady-state operating point of the power system changes randomly.
[0105] Example 3
[0106] Based on the same inventive concept, an embodiment of the present invention further provides a computer device, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor. When the processor executes the computer program, the steps of any one of the virtual PSS2A parameter tuning methods based on inverse reinforcement learning described in Embodiment 1 or 2 are implemented.
[0107] Example 4
[0108] Based on the same inventive concept, an embodiment of the present invention also provides a computer storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the virtual PSS2A parameter adjustment methods based on inverse reinforcement learning described in Example 1 or 2 are implemented.
[0109] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0110] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0111] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0112] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A virtual PSS2A parameter setting method based on inverse reinforcement learning, characterized by: The following steps are involved: Construct a virtual PSS2A structure; The virtual PSS2A structure is introduced into the reactive power control link of VSG, and a linear model of VSG grid-connected system considering the access of virtual PSS2A is constructed. The virtual PSS2A parameter tuning problem is modeled as a Markov decision process, and the network parameters are optimized using a deep deterministic gradient strategy algorithm based on fusion expert experience loss. The reward function implicit in the expert demonstration trajectory is solved by the maximum entropy inverse reinforcement learning algorithm as the reward function of the Markov decision process; The parameters of virtual PSS2A are optimized by combining the strategy update algorithm integrating expert experience loss and the reward function solution algorithm based on maximum entropy inverse reinforcement learning.
2. A virtual PSS2A parameter setting method based on inverse reinforcement learning according to claim 1, characterized in that: The virtual PSS2A structure is constructed, wherein: The virtual PSS2A structure expression is: Where s represents the complex variable used for Laplace transform, G VPSS2A (s) is the transfer function of the virtual PSS2A, K VPSS2A is the proportional coefficient of virtual PSS2A, T1-T6 are the time constants of virtual PSS2A; Taking the fitted power deviation of the VSG virtual angular frequency deviation and the output active power deviation as input, and the reactive power deviation of the virtual PSS2A as output, the relationship between the input and output of the virtual PSS2A is: ΔQ VPSS2A (s)=G VPSS2A (s)Give me 1 Where s represents the complex variable used for Laplace transform, ΔQ VPSS2A is the output reactive power deviation of VPSS2A, G VPSS2A is the transfer function of VPSS2A added to the VSG reactive loop, and Δω1 is the fitting signal of VSG virtual angular frequency deviation Δω and output active power deviation ΔP.
3. A virtual PSS2A parameter setting method based on inverse reinforcement learning according to claim 1, characterized in that: The virtual PSS2A parameter setting problem is modeled as a Markov decision process, and a deep deterministic gradient strategy algorithm based on fusion expert experience loss is used to optimize network parameters; including: an online strategy function network μ(sθ μ ), online value function network Q(s,aθ Q ), target policy network μ′(sθ μ′ ), objective value function network parameters Q′(s,aθ Q′ ); The algorithm’s experience pool is composed of an expert demonstration trajectory dataset and a self-exploration trajectory dataset. A mixed sampling mechanism is used to sample from the two datasets to form the final training data. The gradient descent method is used to minimize the loss function and the network parameter θ of the online value function is used. Q The optimization update is performed. The optimization of the online policy network parameters is divided into expert demonstration trajectory samples and self-exploration samples. While introducing the expert experience loss gradient to optimize the online policy iteration process, the self-exploration trajectory dataset is retained and the online policy network parameters θ are updated according to the online policy gradient value of the original DDPG algorithm. μ .
4. A virtual PSS2A parameter setting method based on inverse reinforcement learning according to claim 1, characterized in that: The process of solving the virtual PSS2A parameters by combining the strategy update algorithm integrating the expert experience loss and the maximum entropy-based inverse reinforcement learning reward function solution algorithm includes: Step (1) Randomly initialize the online policy network μ(sθ μ ) and the online value function network Q(s,aθ Q )’s online policy network parameters θ μ and the online value function network parameter θ Q , initialize the target policy network parameters θ μ′ and the target value function network parameter θ Q′ and its weight, where: s is the state; a is the action; Step (2) According to the formula r(ζ)=θ T F(ζ) constructs the reward function, where ζ is the expert trajectory, r(ζ) is the reward value of the expert trajectory, F(ζ) is the sum of the characteristic components of each state, and the reward function weight θ is initialized i ; Step (3) Initialize the experience pool and take the expert demonstration trajectory dataset T expert Deposit into experience pool; Step (4) performs network training with M iterations.
5. A virtual PSS2A parameter setting method based on inverse reinforcement learning according to claim 4, characterized in that: The network training with the number of iterations M includes: ① The online policy network obtains action a=πθ(s)+η based on the current state s t , where η t is random noise; θ(s) is the online policy network under the current state s; the action selection strategy π depends on the design of the reward function; ②Interact with the environment to perform action a, obtain a new state s′ and an immediate reward value r; ③ The self-exploration sample data (s, a, r, s′) generated by interaction with the environment, namely T discover Deposit into the experience pool; ④The current state s is equal to the new state s′; ⑤ Randomly sample N sample data from the experience pool for training, and predict the distribution function based on the expert trajectory under the current strategy Estimate the partition function Z(θ), minimizing The target value shown, the weight of the reward function θ i Optimize and obtain the optimal reward function; ⑥According to Update the online value function network parameters θ Q , if the training data T∈T expert , then according to Update the policy network parameters θ a , if the training data T∈T discover , then according to the online policy gradient value of the original deep deterministic gradient policy algorithm Update the online policy network parameters θ μ ; ⑦ The target network parameters are updated based on the online network parameters using a soft update method. The update method is as follows: Where τ is the soft update coefficient, θ μ′ ,θ Q′ They are target strategy network parameters and target value function network parameters respectively; ⑧When s′ is in the terminal state, the current iteration ends, otherwise go to step ①.
6. A virtual PSS2A parameter setting device based on inverse reinforcement learning, characterized in that it includes: A virtual PSS2A structure building module, used to build a virtual PSS2A structure; The VSG grid-connected system linearization model building module is used to introduce the virtual PSS2A structure into the reactive power control link of the VSG and build the VSG grid-connected system linearization model taking into account the virtual PSS2A access; The optimization module is used to model the virtual PSS2A parameter tuning problem as a Markov decision process and optimize the network parameters using a deep deterministic gradient strategy algorithm based on fusion expert experience loss; The reward function solving module is used to solve the implicit reward function in the expert demonstration trajectory through the maximum entropy inverse reinforcement learning algorithm as the reward function of the Markov decision process; The reward function algorithm optimizes the virtual PSS2A parameter solution module, which is used to combine the strategy update algorithm integrating expert experience loss and the maximum entropy-based inverse reinforcement learning to solve the reward function algorithm to optimize the virtual PSS2A parameters.
7. A virtual PSS2A parameter setting device based on inverse reinforcement learning according to claim 6, characterized in that: include: Virtual power system stabilizer VPSS2A, used to suppress low-frequency oscillations caused by negative damping when VSG is connected to the power system; The inverse reinforcement learning algorithm control module uses the maximum entropy inverse reinforcement learning algorithm to train the intelligent agent, and uses the intelligent agent to control the virtual PSS2A model, so that the virtual PSS2A can effectively suppress low-frequency oscillations when the steady-state operating point of the power system changes randomly.
8. A virtual PSS2A parameter setting device based on inverse reinforcement learning according to claim 6, characterized in that: The virtual PSS2A parameter setting device based on inverse reinforcement learning is used to implement the steps of a virtual PSS2A parameter setting method based on inverse reinforcement learning described in any one of claims 1-5.
9. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a virtual PSS2A parameter adjustment method based on inverse reinforcement learning as described in any one of claims 1 to 5 are implemented.
10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, which, when executed by a processor, implements the steps of a virtual PSS2A parameter adjustment method based on inverse reinforcement learning as described in any one of claims 1 to 5.