Physical guidance primary frequency modulation adaptive control method and system

Abstract

The invention relates to the technical field of power system control, in particular to a physical guidance primary frequency modulation adaptive control method and system, and the method comprises the steps: constructing a linearized offshore wind power system transfer function model; determining a dominant pole and carrying out characteristic root analysis on the dominant pole; introducing a modal response regulation and control coefficient, and quantifying the regulation and control capability of the control parameters on the dominant modal of the system so as to determine the differentiated adjustment step length proportion of the control parameters; defining an energy function describing the system frequency deviation, and analyzing the derivative of the energy function to represent the evolution direction of the system state; constructing a reinforcement learning agent based on a double-delay depth deterministic strategy gradient TD3 algorithm; applying the modal response regulation and control equilibrium principle to the action space design of the intelligent agent, and fusing the derivative information of the energy function into the reward function design; and the trained reinforcement learning agent is utilized to adaptively adjust the control parameters of the network construction type energy storage according to the real-time state of the system so as to complete the adaptive optimization of the primary frequency modulation.

Description

Technical Field

[0001] This invention relates to the field of power system control technology, and in particular to a physically guided primary frequency regulation adaptive control method and system. Background Technology

[0002] Offshore wind power, as an important component of clean energy, has seen its installed capacity and penetration rate continuously increase. However, most wind turbines currently connected to the grid adopt a grid-following control strategy and lack the ability to independently construct grid voltage and frequency. With the high proportion of offshore wind power integrated into the system, the system inertia support and primary frequency regulation capability continue to decline, and frequency stability issues are becoming increasingly prominent, posing a severe challenge to the safe and stable operation of the power system.

[0003] To address these challenges, grid-based control technology has been proposed and applied to energy storage systems. This technology uses a virtual synchronous generator control strategy to simulate the operating mechanism of a synchronous generator, actively providing the system with the necessary inertia and damping support, and participating in primary frequency regulation. The core of virtual synchronous generator control lies in its two key parameters: the virtual inertial time constant and the virtual damping coefficient. Their values ​​and their coordination directly determine the dynamic response characteristics and stability of the system's frequency.

[0004] To address this, various parameter control strategies have been proposed in existing technologies. Traditional fixed-parameter control methods struggle to maintain optimal performance under changing operating conditions. While adaptive control methods based on fixed rules can adjust parameters according to single variables such as frequency deviation, their linear and piecewise adjustment rules are insufficient to accurately characterize and adapt to the complex dynamic processes of high-order nonlinear systems. Consequently, their coordination capabilities and control accuracy are limited when dealing with complex and ever-changing offshore wind power operation scenarios. Furthermore, some control methods based on fuzzy logic, although reducing reliance on precise models, often depend on expert experience in their rule design and fail to fundamentally solve the coupling and coordination problems among multiple parameters.

[0005] In recent years, data-driven intelligent control methods, especially reinforcement learning, have been introduced into this field in hopes of achieving better adaptive control. However, purely data-driven reinforcement learning methods have significant limitations: their training process is blind and lacks physical guidance, resulting in low learning efficiency; moreover, the control strategies obtained through training have poor physical interpretability, making it difficult to analyze and verify their rationality, which is not conducive to their application and trust in practical power systems; when dealing with high-order, strongly nonlinear systems such as offshore wind power, without physical guidance, the agent will find it difficult to learn truly coordinated and robust control strategies.

[0006] Therefore, existing technologies have failed to effectively address the balance between efficient collaborative optimization of control parameters and physical interpretability of control strategies when grid-based energy storage participates in primary frequency regulation in complex offshore wind power systems.

[0007] The information disclosed in this background section is intended only to enhance the understanding of the general background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0008] This invention provides a physically guided primary frequency modulation adaptive control method and system, thereby effectively solving the problems in the background art.

[0009] To achieve the above objectives, the technical solution adopted by this invention is: a physically guided primary frequency modulation adaptive control method, comprising the following steps:

[0010] Construct a linearized transfer function model for an offshore wind power system that includes various synchronous machines, grid-connected control of direct-drive wind turbines, and grid-connected control of energy storage.

[0011] Based on the linear offshore wind power system transfer function model, the dominant poles are determined and eigenvalue analysis of the dominant poles is performed to determine the influence mechanism of the main control parameters in grid-type energy storage on system stability.

[0012] A modal response modulation coefficient is introduced to quantify the ability of the control parameters to regulate the dominant mode of the system, and a modal response modulation equilibrium principle is proposed to determine the differential adjustment step size ratio of the control parameters.

[0013] Lyapunov stability theory is introduced to define an energy function that describes the frequency deviation of the system, and its derivative is analyzed to characterize the direction of the evolution of the system state.

[0014] A reinforcement learning agent is constructed based on the dual-delay deep deterministic policy gradient TD3 algorithm; the modal response regulation and equilibrium principle is applied to the action space design of the agent, and the derivative information of the energy function is incorporated into the reward function design;

[0015] The trained reinforcement learning agent adaptively adjusts the control parameters of the grid-type energy storage according to the real-time state of the system to complete the first frequency regulation adaptive optimization.

[0016] Furthermore, the linearized offshore wind power system transfer function model is as follows:

[0017] ;

[0018] In the formula, and These are the system's equivalent inertial constant and equivalent damping coefficient, respectively. For system frequency deviation, The disturbance power is expressed in pu, and is positive when the load decreases or the generator power increases. , and These are the transfer functions for reheat steam turbines, non-reheat steam turbines, and water turbines, respectively. The transfer function of the wind turbine; The transfer function of grid-type energy storage is expressed as follows:

[0019] ;

[0020] In the formula, This represents the change in power on the energy storage side. , and These are the terminal voltage of the energy storage system and the grid voltage, respectively, both measured in pu. The virtual reactance unit is pu. The rated angular frequency; This is the virtual inertial time constant; This is the virtual damping coefficient.

[0021] Furthermore, the determination of the dominant pole and the eigenvalue analysis of the dominant pole include:

[0022] Fixed virtual damping coefficient Analyzing the virtual inertial time constant When the value increases, the dominant eigenvalue of the system shifts towards the imaginary axis, resulting in a slower decay rate of the system; when fixed... ,analyze When the value increases, the dominant eigenvalue shifts to the left half-plane, and the transient process of the system decays faster; however, excessively large values... This will also cause the poles to start approaching the imaginary axis again;

[0023] Considering The physical meaning of this value is that an excessively large value means that for the same frequency deviation, the energy storage system needs to provide greater power support, while energy storage devices all have capacity limitations. It must not exceed the threshold.

[0024] Furthermore, the proposed modal response control equilibrium principle, to determine the differential adjustment step size ratio of the control parameters, includes:

[0025] Define the modal response modulation coefficients for the parameter, for the parameter virtual inertia time constant. and virtual damping coefficient Its modal response modulation coefficient The calculation formula is:

[0026] ;

[0027] In the formula, The number of dominant poles For the first One dominant extreme point No. The real part of the dominant pole. Represents control parameters or ;

[0028] If a parameter has a high control coefficient, it means that even a small adjustment will have a significant impact on the system; therefore, during parameter optimization, a finer adjustment step size should be given to this parameter to prevent overshoot. Conversely, a parameter with a low control coefficient requires a larger adjustment step size to drive it; thus, the unit control efficiency of the parameter... It should be proportional to its modal response modulation coefficient, that is:

[0029] ;

[0030] In order to make and The adjustments should have a comparable impact on the system, ensuring the coordination of collaborative optimization, and should satisfy the following:

[0031] ;

[0032] This leads to the conclusion and The adjustment step size should satisfy the following relationship, namely the modal response control equilibrium principle:

[0033] ;

[0034] in, and Parameters and Adjustment step size, and Parameters and Modal response modulation coefficient.

[0035] Furthermore, by introducing Lyapunov stability theory, an energy function describing the frequency deviation of the system is defined, including:

[0036] Lyapunov energy function The definition is as follows:

[0037] ;

[0038] Wherein, derivative ,when If the derivative is positive definite, then the system is gaining "energy" and deviating from a steady state. and Same number; when If the derivative is negative and definite, then the system is consuming "energy" and converging towards a steady state. and Different sign.

[0039] Furthermore, incorporating the derivative information of the energy function into the reward function design includes:

[0040] Reward function of reinforcement learning The design is as follows:

[0041] ;

[0042] In the formula, , , It is a scaling factor introduced to unify the dimensions. , , Let be the weight coefficient, and satisfy... ;

[0043] In the reward function, the first term This is the frequency deviation term, and it is the main penalty term. The squaring operation ensures that both positive and negative deviations are penalized.

[0044] Second item As a smoothing term, it suppresses rapid frequency changes when a disturbance first occurs and encourages a smooth frequency recovery process;

[0045] Third item For Lyapunov's stable term, when and When the signs are the same, the system state deteriorates, and a penalty is imposed; when... and When the sign is different, the system state is restored and a reward is given; this item essentially guides the agent's control strategy in the direction of the negative definite derivative of the system energy function, providing physical guidance for the agent's learning.

[0046] Furthermore, the state space of the reinforcement learning agent is a five-dimensional vector:

[0047] ;

[0048] In the formula, The system frequency deviation reflects the current degree of imbalance in the system. The rate of change of frequency is used to characterize the trend of frequency change. and It provides historical information about the action value output by the agent in the previous moment, so that the agent can determine to what extent the current state is affected by the action in the previous moment; The output change of grid-type energy storage reflects the power support provided by the energy storage side to the system.

[0049] Furthermore, the action space of the reinforcement learning agent is a two-dimensional vector:

[0050] ;

[0051] In the formula, and It is the direct output of the agent network. and The unit adjustment amount; since the output layer of the Actor network uses the Tanh activation function, each action component is constrained within the interval [-1, 1]. Therefore, the action needs to be converted into network control parameters with actual physical meaning through a parameter mapping function:

[0052] ;

[0053] In the formula, and The center value of the parameter. and The scaling factor, set according to the modal response control and equalization principle, physically corresponds to the adjustment step size. and .

[0054] Furthermore, in the reinforcement learning agent that has been trained, the reinforcement learning agent uses a transfer function model as the environment during the offline training phase and transfers the trained strategy to the actual detailed model for application.

[0055] The present invention also includes a physically guided primary frequency modulation adaptive control system using the method described above, the system comprising:

[0056] The transfer function modeling unit is used to construct a linearized transfer function model of an offshore wind power system that includes various synchronous machines, grid-connected control of direct-drive wind turbines, and grid-connected control of energy storage.

[0057] The dominant pole analysis unit is used to determine the dominant poles and perform eigenvalue analysis of the dominant poles based on the linearized offshore wind power system transfer function model, and to determine the influence mechanism of the main control parameters in grid-type energy storage on system stability.

[0058] The step size adaptive unit is used to introduce modal response control coefficients, quantify the control parameters' ability to control the dominant mode of the system, and propose modal response control balance principles to determine the differential adjustment step size ratio of the control parameters.

[0059] Energy function building blocks are used to introduce Lyapunov stability theory, define the energy function describing the frequency deviation of the system, and analyze its derivative to characterize the direction of the evolution of the system state;

[0060] The agent unit is used to construct a reinforcement learning agent based on the dual-delay deep deterministic policy gradient TD3 algorithm; the modal response regulation and balance principle is applied to the action space design of the agent, and the derivative information of the energy function is incorporated into the reward function design;

[0061] An adaptive optimization unit is used to adaptively adjust the control parameters of the grid-type energy storage according to the real-time state of the system using a trained reinforcement learning agent, so as to complete a primary frequency regulation adaptive optimization.

[0062] The present invention also includes a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method as described above.

[0063] The present invention also includes a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described above.

[0064] The beneficial effects of this invention are as follows: By guiding reinforcement learning through physical mechanisms such as eigenvalue loci and modal response regulation coefficient analysis, a network-based energy storage control framework with both physical interpretability and data-driven adaptability is constructed. This method utilizes the modal response regulation equilibrium principle to guide the action space design of reinforcement learning, ensuring efficient collaborative exploration and optimization of virtual inertia and damping coefficients. By introducing Lyapunov stability theory to construct a reward function, the system energy change trend is integrated into the agent's learning objective, significantly improving learning efficiency and control performance. This effectively increases the frequency minimum point, reduces steady-state deviation, and achieves rapid and smooth frequency recovery. Furthermore, the use of the dual-delay deep deterministic policy gradient TD3 algorithm effectively improves the stability of training and the reliability of the policy. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 The primary frequency regulation adaptive control process for grid-type energy storage connected to offshore wind power systems;

[0067] Figure 2Frequency response model for grid-connected energy storage systems integrated into offshore wind power systems;

[0068] Figure 3 Pole-zero distribution diagram of the system;

[0069] Figure 4 For J and D p Root locus curve during change;

[0070] Figure 5 For comparison of modal response modulation coefficients;

[0071] Figure 6 This is a schematic diagram of frequency variation;

[0072] Figure 7 Here is a diagram of the TD3 algorithm structure;

[0073] Figure 8 Comparison of frequency changes under different methods;

[0074] Figure 9 Diagram of a grid-connected energy storage system integrated into an offshore wind power system;

[0075] Figure 10 For the frequency variation of different methods in the detailed model;

[0076] Figure 11 The changes in energy storage and wind turbine power under different methods;

[0077] Figure 12 This is a schematic diagram of the structure of a computer device. Detailed Implementation

[0078] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0079] like Figure 1 As shown: A physical-guided primary frequency regulation adaptive control method for grid-connected energy storage systems integrated into offshore wind power systems includes the following steps:

[0080] Step 1: Establish a linearized transfer function model of the offshore wind power system, including various synchronous machines (reheat turbines, non-reheat turbines, and hydro turbines), grid-connected direct-drive turbines, and grid-connected energy storage. To enable the turbines to have frequency regulation capabilities, droop control is added to the turbine-side control; the grid-connected control strategy adopts virtual synchronous generator (VSG) control.

[0081] Step 2: Based on the established linearized offshore wind power system transfer function model, determine the dominant poles. Furthermore, eigenvalue locus analysis of the dominant poles was conducted to reveal the main control parameter in grid-type energy storage—the virtual inertial time constant. and virtual damping coefficient Mechanism of influence on system stability;

[0082] Step 3: Introduce modal response modulation coefficients and quantify. and The system's ability to regulate the dominant mode is assessed, and a modal response regulation equilibrium principle is proposed to determine subsequent... and Differentiated adjustment step size ratio;

[0083] Step 4: Introduce Lyapunov stability theory, define the energy function describing the frequency deviation energy of the system, and analyze its derivative to characterize the direction of the evolution of the system state;

[0084] Step 5: Construct a reinforcement learning agent based on the dual-delay deep deterministic policy gradient TD3 algorithm; apply the modal response regulation and equilibrium principle to the action space design of the agent, and incorporate the derivative information of the energy function into the reward function design;

[0085] Step 6: Using the trained reinforcement learning agent, adaptively adjust the virtual inertial time constant of the grid-type energy storage according to the real-time state of the system. and virtual damping coefficient This is to complete a frequency modulation adaptive optimization.

[0086] By guiding reinforcement learning through physical mechanisms such as eigenvalue trajectories and modal response regulation coefficient analysis, a network-based energy storage control framework with both physical interpretability and data-driven adaptability is constructed. This method utilizes the modal response regulation equilibrium principle to guide the action space design of reinforcement learning, ensuring efficient collaborative exploration and optimization of virtual inertia and damping coefficients. By introducing Lyapunov stability theory to construct a reward function, the system energy change trend is integrated into the agent's learning objective, significantly improving learning efficiency and control performance. This effectively improves the frequency minimum point, reduces steady-state deviation, and achieves rapid and smooth frequency recovery. Furthermore, the dual-delay deep deterministic policy gradient (TD3) algorithm effectively enhances training stability and policy reliability.

[0087] like Figure 2 As shown, the linearized transfer function model of the offshore wind power system is:

[0088] ;

[0089] in, and These are the system's equivalent inertial constant and equivalent damping coefficient, respectively. For system frequency deviation, The disturbance power (pu) is positive when the load decreases or the generator power increases. , and These are the transfer functions for reheat steam turbines, non-reheat steam turbines, and water turbines, respectively. The transfer function of the wind turbine; The transfer function of grid-type energy storage is expressed as follows:

[0090] ;

[0091] in, This represents the change in power on the energy storage side. , and These are the energy storage system terminal voltage and the grid voltage (pu), respectively. For virtual reactance (pu). The rated angular frequency; The virtual inertial time constant. This is the virtual damping coefficient.

[0092] The parameters in the transfer function model of the offshore wind power system are shown in Table 1.

[0093] Table 1 System Frequency Response Model Parameters

[0094] Substituting the data from Table 1 into the transfer function model, the dominant poles of the system can be determined, and the root locus of the dominant poles can be plotted as follows: Figure 3 , Figure 4 As shown. The eigenvalue locus analysis of the dominant pole specifically includes: fixing ,analyze When the value increases, the dominant eigenvalue of the system shifts towards the imaginary axis, resulting in a slower decay rate of the system; when fixed... ,analyze As the value increases, the dominant eigenvalue shifts to the left half-plane, and the transient process of the system decays more rapidly. However, excessively large values... This will also cause the poles to start approaching the imaginary axis again; and considering The physical meaning of this value is that an excessively large value means that for the same frequency deviation, the energy storage system needs to provide greater power support, while energy storage devices all have capacity limitations. It shouldn't be too big either.

[0095] The process of determining the modal response modulation equilibrium principle includes:

[0096] Define the modal response modulation coefficients for the parameters. and Its modal response modulation coefficient The calculation formula is:

[0097] ;

[0098] in, The number of dominant poles For the first One dominant extreme point No. The real part of the dominant pole. Represents control parameters or .

[0099] based on Figure 4 The root locus can be used to calculate the four pole pairs. or The distribution of modal response modulation coefficients, such as Figure 5 As shown.

[0100] If a parameter has a high control coefficient, it means that even a small adjustment will have a significant impact on the system. Therefore, during parameter optimization, a finer adjustment step size should be given to this parameter to prevent overshoot. Conversely, parameters with low control coefficients require a larger adjustment step size to drive them. This leads to the question of how to improve the unit control efficiency of the parameter. It should be proportional to its modal response modulation coefficient, that is:

[0101] ;

[0102] In order to make and The adjustments should have a comparable impact on the system, ensuring the coordination of collaborative optimization, and should satisfy the following:

[0103] ;

[0104] This leads to the conclusion and The adjustment step size should satisfy the following relationship, namely the modal response control equilibrium principle:

[0105] ;

[0106] in, and Parameters and Adjustment step size, and Parameters and The modal response control coefficients. The adjustment step size of each parameter should be inversely proportional to its modal response control coefficient. Therefore, under the parameters given in this example, ,but The step size of the change should be This is five times greater than the previous one, which makes the impact of the two actions on system performance tend to be balanced.

[0107] Lyapunov energy function The definition is as follows:

[0108] ;

[0109] Wherein, derivative ,when If the derivative is positive definite, then the system is gaining "energy" and deviating from a steady state. and Same number; when If the derivative is negative and definite, then the system is consuming "energy" and converging towards a steady state. and Different sign.

[0110] Reward function of reinforcement learning The design is as follows:

[0111] ;

[0112] in, , , It is a scaling factor introduced to unify the dimensions. , , Let be the weight coefficient, and satisfy... .

[0113] In the reward function, the first term is the frequency deviation term, and this term should be the main penalty term. The squaring operation ensures that both positive and negative deviations are penalized.

[0114] The second term is the smoothness term, which suppresses rapid frequency changes when a disturbance first occurs and encourages a smooth frequency recovery process.

[0115] The third term is the Lyapunov stability term, such as Figure 6 As shown, when and When the signs are the same, the system state deteriorates, and a penalty is imposed; when... and When the sign changes, the system state is restored, and a reward is given. Essentially, this guides the agent's control strategy towards a direction where the derivative of the system's energy function is negatively definite, providing physical guidance for the agent's learning.

[0116] The state space of a reinforcement learning agent is a five-dimensional vector:

[0117] ;

[0118] in, The system frequency deviation reflects the current degree of imbalance in the system. The rate of change of frequency is used to characterize the trend of frequency change. and It provides historical information about the action value output by the agent in the previous moment, so that the agent can determine to what extent the current state is affected by the action in the previous moment; The output change of grid-type energy storage reflects the power support provided by the energy storage side to the system.

[0119] The action space of a reinforcement learning agent is a two-dimensional vector:

[0120] ;

[0121] in, and It is the direct output of the agent network. and The unit adjustment amount. Since the output layer of the Actor network uses the Tanh activation function, each action component is constrained within the interval [-1, 1]. Therefore, the action needs to be converted into network control parameters with actual physical meaning through a parameter mapping function:

[0122] ;

[0123] in, and The center value of the parameter. and The scaling factor, set according to the modal response control and equalization principle, physically corresponds to the "adjustment step size". and .

[0124] The structure of the dual-delay deep deterministic policy gradient TD3 algorithm is as follows: Figure 7 As shown, the reinforcement learning agent uses a transfer function model as its environment during the offline training phase. To verify the superiority of the proposed method, it is compared with traditional adaptive control strategies and constant-parameter control strategies. Under a power disturbance of -0.05, the frequency dynamic response curves of the three methods are compared as follows: Figure 8 As shown.

[0125] Because the parameters are fixed, the control strategy of the fixed-parameter method is a compromise for specific operating conditions. When faced with the current disturbance, it cannot quickly provide significant inertia support, resulting in a lower minimum frequency and an inability to fully increase inertia in the later stages of frequency modulation. The mobilization of stored energy leads to a lower steady-state frequency value. Traditional adaptive methods outperform fixed-parameter methods, demonstrating the necessity of adaptive control. However, their parameter adjustments based on fixed rules are inherently linear and piecewise, making it difficult to accurately characterize the complex dynamics of high-order nonlinear systems. Therefore, when faced with disturbances, their parameter adjustments exhibit a certain lag and coarseness, resulting in performance inferior to reinforcement learning methods in metrics such as minimum frequency and steady-state frequency value.

[0126] The trained strategy is then transferred to a detailed real-world model. A detailed model is built on the Matlab / Simulink simulation platform, and its structure diagram is shown below. Figure 9 As shown, the frequency modulation control parameters in the detailed model should be consistent with the parameters of the transfer function in Table 1. The proposed method is compared with the traditional adaptive method under a power perturbation of -0.05.

[0127] like Figure 10 As shown, the proposed method outperforms traditional adaptive methods in terms of system frequency minimum point and steady-state frequency value. Figure 11 (a) shows the changes in energy storage power. In the early stage of the disturbance, the energy storage quickly provides or absorbs power to suppress frequency changes and raise the lowest frequency point. In the recovery period, the energy storage can be fully mobilized to improve the steady-state frequency value. Figure 11 (b) illustrates the changes in wind turbine power. When using the proposed method, the frequency regulation requirement is effectively shared because the energy storage is fully mobilized, thus reducing the frequency regulation pressure on the wind turbine. This means that in scenarios where wind turbines are equipped with active power reserves, the proposed method can be used to appropriately reduce the reserved reserve capacity of the wind turbine while ensuring frequency stability, which helps to improve wind energy utilization efficiency.

[0128] This embodiment also includes a physically guided primary frequency modulation adaptive control system, using the method described above. The system includes:

[0129] The transfer function modeling unit is used to construct a linearized transfer function model of an offshore wind power system that includes various synchronous machines, grid-connected control of direct-drive wind turbines, and grid-connected control of energy storage.

[0130] The dominant pole analysis unit is used to determine the dominant poles and perform eigenvalue analysis of the dominant poles based on the linearized offshore wind power system transfer function model, and to determine the influence mechanism of the main control parameters on the system stability in grid-type energy storage.

[0131] The step size adaptive unit is used to introduce modal response control coefficients, quantify the control parameters’ ability to control the dominant mode of the system, and propose modal response control balance principle to determine the differential adjustment step size ratio of control parameters.

[0132] Energy function building blocks are used to introduce Lyapunov stability theory, define the energy function describing the frequency deviation of the system, and analyze its derivative to characterize the direction of the evolution of the system state;

[0133] The agent unit is used to construct reinforcement learning agents based on the dual-delay deep deterministic policy gradient TD3 algorithm; the modal response regulation and balance principle is applied to the action space design of the agent, and the derivative information of the energy function is incorporated into the reward function design;

[0134] The adaptive optimization unit is used to adaptively adjust the control parameters of the grid-type energy storage based on the real-time state of the system using a trained reinforcement learning agent, so as to complete the first frequency regulation adaptive optimization.

[0135] Please see Figure 12 The diagram shows a structural schematic of a computer device provided in an embodiment of this application. An embodiment of this application provides a computer device 400, including a processor 410 and a memory 420. The memory 420 stores a computer program executable by the processor 410. When the computer program is executed by the processor 410, it performs the method described above.

[0136] This application embodiment also provides a storage medium 430, on which a computer program is stored, and the computer program is executed by a processor 410 to perform the above method.

[0137] The storage medium 430 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0138] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. "A plurality of" means two or more, unless otherwise explicitly specified.

[0139] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0140] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0141] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0142] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0143] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0144] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0145] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A physically guided primary frequency modulation adaptive control method, characterized in that, Includes the following steps: Construct a linearized transfer function model for an offshore wind power system that includes various synchronous machines, grid-connected control of direct-drive wind turbines, and grid-connected control of energy storage. Based on the linear offshore wind power system transfer function model, the dominant poles are determined and eigenvalue analysis of the dominant poles is performed to determine the influence mechanism of the main control parameters in grid-type energy storage on system stability. A modal response modulation coefficient is introduced to quantify the ability of the control parameters to regulate the dominant mode of the system, and a modal response modulation equilibrium principle is proposed to determine the differential adjustment step size ratio of the control parameters. Lyapunov stability theory is introduced to define an energy function that describes the frequency deviation of the system, and its derivative is analyzed to characterize the direction of the evolution of the system state. A reinforcement learning agent is constructed based on the dual-delay deep deterministic policy gradient TD3 algorithm. The modal response regulation and equilibrium principle is applied to the action space design of the agent, and the derivative information of the energy function is incorporated into the reward function design. The trained reinforcement learning agent adaptively adjusts the control parameters of the grid-type energy storage according to the real-time state of the system to complete the first frequency regulation adaptive optimization.

2. The physical-guided primary frequency-modulated adaptive control method according to claim 1, characterized in that, The linear offshore wind power system transfer function model is as follows: ； In the formula, and These are the system's equivalent inertial constant and equivalent damping coefficient, respectively. For system frequency deviation, The disturbance power is expressed in pu, and is positive when the load decreases or the generator power increases. , and These are the transfer functions for reheat steam turbines, non-reheat steam turbines, and water turbines, respectively. The transfer function of the wind turbine; The transfer function of grid-type energy storage is expressed as follows: ； In the formula, This represents the change in power on the energy storage side. , and These are the terminal voltage of the energy storage system and the grid voltage, respectively, both measured in pu. The virtual reactance unit is pu. This is the rated angular frequency; This is the virtual inertial time constant; This is the virtual damping coefficient.

3. The physical-guided primary frequency-modulated adaptive control method according to claim 1, characterized in that, The process of determining the dominant pole and performing eigenvalue analysis on the dominant pole includes: Fixed virtual damping coefficient Analyzing the virtual inertial time constant When the value increases, the dominant eigenvalue of the system shifts towards the imaginary axis, resulting in a slower decay rate of the system; when fixed... ,analyze When the value increases, the dominant eigenvalue shifts to the left half-plane, and the transient process of the system decays faster; however, excessively large values... This will also cause the poles to start approaching the imaginary axis again; Considering The physical meaning of this value is that an excessively large value means that for the same frequency deviation, the energy storage system needs to provide greater power support, while energy storage devices all have capacity limitations. It must not exceed the threshold.

4. The physical-guided primary frequency modulation adaptive control method according to claim 1, characterized in that, The proposed modal response control equilibrium principle, used to determine the differential adjustment step size ratio of the control parameters, includes: Define the modal response modulation coefficients for the parameter, for the parameter virtual inertia time constant. and virtual damping coefficient Its modal response modulation coefficient The calculation formula is: ； In the formula, The number of dominant poles, For the first One dominant extreme point No. The real part of the dominant pole. Represents control parameters or ; If a parameter has a high control coefficient, it means that even a small adjustment will have a significant impact on the system; therefore, during parameter optimization, a finer adjustment step size should be given to this parameter to prevent overshoot. Conversely, a parameter with a low control coefficient requires a larger adjustment step size to drive it; thus, the unit control efficiency of the parameter... It should be proportional to its modal response modulation coefficient, that is: ； In order to make and The adjustments should have a comparable impact on the system, ensuring the coordination of collaborative optimization, and should satisfy the following: ； This leads to the conclusion and The adjustment step size should satisfy the following relationship, namely the modal response control equilibrium principle: ； in, and Parameters and Adjustment step size, and Parameters and Modal response modulation coefficient.

5. The physical-guided primary frequency modulation adaptive control method according to claim 1, characterized in that, The introduction of Lyapunov stability theory defines an energy function describing the frequency deviation of the system, including: Lyapunov energy function The definition is as follows: ； Wherein, derivative ,when If the derivative is positive definite, then the system is gaining "energy" and deviating from a steady state. and Same number; when If the derivative is negative and definite, then the system is consuming "energy" and converging towards a steady state. and Different sign.

6. The physical-guided primary frequency modulation adaptive control method according to claim 5, characterized in that, The step of incorporating the derivative information of the energy function into the design of the reward function includes: Reward function of reinforcement learning The design is as follows: ； In the formula, , , It is a scaling factor introduced to unify the dimensions. , , Let be the weight coefficient, and satisfy... ; In the reward function, the first term This is the frequency deviation term, and it is the main penalty term. The squaring operation ensures that both positive and negative deviations are penalized. Second item As a smoothing term, it suppresses rapid frequency changes when a disturbance first occurs and encourages a smooth frequency recovery process; Third item For Lyapunov's stable term, when and When the signs are the same, the system state deteriorates, and a penalty is imposed; when... and When the sign is different, the system state is restored and a reward is given; this item essentially guides the agent's control strategy in the direction of the negative definite derivative of the system energy function, providing physical guidance for the agent's learning.

7. The physical-guided primary frequency modulation adaptive control method according to claim 6, characterized in that, The state space of the reinforcement learning agent is a five-dimensional vector: ； In the formula, The system frequency deviation reflects the current degree of imbalance in the system. The rate of change of frequency is used to characterize the trend of frequency change. and It provides historical information about the action value output by the agent in the previous moment, so that the agent can determine to what extent the current state is affected by the action in the previous moment; The output change of grid-type energy storage reflects the power support provided by the energy storage side to the system.

8. The physical-guided primary frequency-modulated adaptive control method according to claim 5, characterized in that, The action space of the reinforcement learning agent is a two-dimensional vector: ； In the formula, and It is the direct output of the agent network. and The unit adjustment amount; since the output layer of the Actor network uses the Tanh activation function, each action component is constrained within the interval [-1, 1]. Therefore, the action needs to be converted into network control parameters with actual physical meaning through a parameter mapping function: ； In the formula, and The center value of the parameter. and The scaling factor, set according to the modal response control and equalization principle, physically corresponds to the adjustment step size. and .

9. The physical-guided primary frequency modulation adaptive control method according to claim 1, characterized in that, In the reinforcement learning agent that has been trained, the reinforcement learning agent uses a transfer function model as the environment during the offline training phase and transfers the trained policy to the actual detailed model for application.

10. A physically guided primary frequency modulation adaptive control system, characterized in that, Using the method of any one of claims 1 to 9, the system comprises: The transfer function modeling unit is used to construct a linearized transfer function model of an offshore wind power system that includes various synchronous machines, grid-connected control of direct-drive wind turbines, and grid-connected control of energy storage. The dominant pole analysis unit is used to determine the dominant poles and perform eigenvalue analysis of the dominant poles based on the linearized offshore wind power system transfer function model, and to determine the influence mechanism of the main control parameters in grid-type energy storage on system stability. The step size adaptive unit is used to introduce modal response control coefficients, quantify the control parameters' ability to control the dominant mode of the system, and propose modal response control balance principles to determine the differential adjustment step size ratio of the control parameters. Energy function building blocks are used to introduce Lyapunov stability theory, define the energy function describing the frequency deviation of the system, and analyze its derivative to characterize the direction of the evolution of the system state; The agent unit is used to construct a reinforcement learning agent based on the dual-delay deep deterministic policy gradient TD3 algorithm; the modal response regulation and balance principle is applied to the action space design of the agent, and the derivative information of the energy function is incorporated into the reward function design; An adaptive optimization unit is used to adaptively adjust the control parameters of the grid-type energy storage according to the real-time state of the system using a trained reinforcement learning agent, so as to complete a primary frequency regulation adaptive optimization.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-9.

12. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-9.

Images