A method for adaptive adjustment of parameters of a rotating power flow controller based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's rotary power flow controller parameter self-adapting adjustment method, comprising the following steps: step S1, constructs RPFC double-loop electromechanical coupling dynamic model including power loop and angle loop;Step S2, based on DDGP algorithm, construct Actor-Critic model, train Actor-Critic model, obtain trained Actor-Critic model;Step S3, the PID adjustment parameter of RPFC double-loop electromechanical coupling dynamic model is obtained using trained Actor-Critic model output;Step S4, in power loop, according to PID adjustment parameter, calculate active reference current command value and reactive reference current command value, and conversion obtains corresponding target angle reference value, in angle loop, according to target angle reference value, drive RPST action in RPFC;According to the method, the wear and tear of mechanical structure can be avoided, the quick tracking of target power angle reference value is realized, and the quick response of system is guaranteed.

Description

An adaptive parameter adjustment method for rotating power flow controllers based on reinforcement learning Technical Field

[0001] This invention relates to the field of power system control technology, and in particular to an adaptive adjustment method for the parameters of a rotating power flow controller based on reinforcement learning. Background Technology

[0002] Currently, urban power distribution networks are evolving towards a closed-loop "petal-shaped" architecture. This architecture improves power supply reliability through a multi-ring network structure, but it is prone to problems such as uneven power flow distribution and voltage exceeding limits due to the natural distribution characteristics of line impedance. The Rotary Power Flow Controller (RPFC), as an economical flexible AC transmission device capable of simultaneously regulating line voltage amplitude and phase, can achieve decoupled control of active and reactive power, becoming a key device for solving power flow control in "petal-shaped" distribution networks.

[0003] However, existing technologies have significant drawbacks in the application of "petal-shaped" distribution networks. First, existing PID control parameters are typically fixed based on the system's rated operating point. The "petal-shaped" architecture, due to its closed-loop operation characteristics, frequent line switching, fault isolation, and distributed power source integration, causes significant fluctuations in the system's equivalent impedance. Fixed-parameter PID controllers cannot adapt to this time-varying characteristic, resulting in slow regulation or overshoot oscillations. Second, RPFCs inherently possess complex nonlinear coupling between power, voltage, and angle, requiring extremely high parameter robustness for control accuracy. When faced with drift in internal coupling parameters, traditional fixed-parameter PID controllers struggle to provide stable and effective control parameters.

[0004] Furthermore, existing adaptive control algorithms (such as fuzzy control and neural network PID) often focus solely on minimizing electrical errors, neglecting the physical constraints of the RPFC as a mechanical rotating device. For example, in pursuit of millisecond-level electrical response, control parameters may be adjusted drastically at high frequencies, leading to frequent acceleration and deceleration of the servo motor. This can cause stress concentration in the rotor bearings, increased cogging torque fluctuations, and severe mechanical wear and fatigue life loss. Therefore, current technologies lack a parameter adaptive adjustment scheme that can achieve both rapid and error-free tracking of electrical quantities and ensure long-term stable operation of the mechanical actuator. Summary of the Invention

[0005] In view of the above-mentioned prior art, the present invention provides a method for adaptive adjustment of parameters of a rotating power flow controller based on reinforcement learning, which mainly solves the technical problems existing in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] An adaptive parameter adjustment method for a rotating power flow controller based on reinforcement learning includes the following steps:

[0008] Step S1: Construct a dual closed-loop electromechanical coupling dynamic model of RPFC including a power loop and an angle loop. The power loop includes an active power control loop, a reactive power control loop, and a power decoupling control model.

[0009] Step S2: Construct an Actor-Critic model based on the DDGP algorithm, train the Actor-Critic model, and obtain a trained Actor-Critic model.

[0010] Step S3: Use the trained Actor-Critic model output to obtain the PID control parameters of the RPFC dual closed-loop electromechanical coupling dynamic model. The PID control parameters include the adjustment components of the proportional gain, integral gain and derivative gain in the active power control loop and the reactive power control loop.

[0011] Step S4: In the power loop, calculate the active reference current command value and reactive reference current command value according to the PID adjustment parameters, and convert them into the corresponding target angle reference value. In the angle loop, drive the RPST action in RPFC according to the target angle reference value.

[0012] Optionally, the formula for calculating the active reference current command value is:

[0013]

[0014]

[0015] in, for The active power reference current command value on the d-axis at any given time; for The proportional gain of the active power control loop at all times. for The integral gain of the active power control loop at all times. for The differential gain of the active power control loop at all times; This refers to the active power tracking error. For discrete sampling sequence numbers; To control the sampling period; This represents the measured active power. This is a reference value for active power.

[0016] Optionally, the formula for calculating the reactive reference current command value is:

[0017]

[0018]

[0019] in, for The q-axis reactive current command value at any given time; for The proportional gain of the reactive power control loop at all times. for The integral gain of the reactive power control loop at all times. for The differential gain of the reactive power control loop at any given time; This refers to reactive power tracking error; For discrete sampling sequence numbers; To control the sampling period; This is the actual measured reactive power; This is a reference value for reactive power.

[0020] Optionally, the construction of the Actor-Critic model based on the DDGP algorithm in step S2 includes:

[0021] Step S21: Construct the state space;

[0022] Step S22: Define the action space and construct the parameter mapping mechanism;

[0023] Step S23: Design a composite reward function that combines graded punishment and transient inhibition.

[0024] Optionally, the expression for the state space in step S21 is:

[0025]

[0026] in, for The state vector at any given time; The normalized error of the active power control loop. This is the normalized error of the reactive power control loop; This is the cumulative error term for the active power control loop. This is the cumulative error term for the reactive power control loop; The rate of change of active power in the active power control loop. The reactive power change rate of the reactive power control loop; This is the effective value of the voltage at the point of common coupling; This represents the standard deviation of voltage fluctuations. This refers to the mechanical state characteristics of the servo motor; This is the effective value of the load current.

[0027] Optionally, step S22 includes:

[0028] Define a reference parameter, the expression of which is:

[0029]

[0030] in, Reference parameter vector; The reference value for the proportional gain. This is the reference value for the integral gain. This is the reference value for the differential gain;

[0031] Define the action space as follows:

[0032]

[0033] in, for The action vector output by the agent at any given time. The proportional gain in the active power control loop Adjustment component, Integral gain in the active power control loop Adjustment component, The differential gain in the active power control loop Adjustment component, The proportional gain in the reactive power control loop Adjustment component, Integral gain in the reactive power control loop Adjustment component, The differential gain in the reactive power control loop Adjustment component;

[0034] Design an asymmetric scaling function, and calculate the adjustment components of the proportional gain, integral gain, and derivative gain of the active power control loop and the reactive power control loop based on the asymmetric scaling function. The expression of the asymmetric scaling function is:

[0035]

[0036] in, This is the vector of PID control parameters that actually take effect; To adjust the gain coefficient; It is an asymmetric scaling mapping function; The action vector output by the agent.

[0037] Optionally, the adjustment components of the proportional gain, integral gain, and derivative gain of the active power control loop are calculated based on the asymmetric scaling function, using the following formula:

[0038]

[0039]

[0040]

[0041] in, The proportional gain of the active power control loop; Action vector The corresponding proportional gain adjustment component; The integral gain of the active power control loop; Action vector The corresponding adjustment component of the integral gain; The differential gain of the active power control loop; Action vector The corresponding adjustment component of the differential gain;

[0042] The adjustment components of the proportional gain, integral gain, and derivative gain of the reactive power control loop are calculated based on the asymmetric scaling function, using the following formula:

[0043]

[0044]

[0045]

[0046] in, The proportional gain of the reactive power control loop; Action vector The corresponding proportional gain adjustment component; The integral gain of the reactive power control loop; Action vector The corresponding adjustment component of the integral gain; The differential gain of the reactive power control loop; Action vector The corresponding adjustment component of the differential gain.

[0047] Optionally, the composite reward function in step S23 includes:

[0048]

[0049] in, for Momentary compound rewards; This is a reward item for tracking error; This is a peak penalty item; This is a steady-state reward term; For motion smoothing constraints;

[0050]

[0051] in, The weighting coefficient for the tracking error reward; This represents the absolute value of the active power error. This represents the absolute value of the reactive power error.

[0052]

[0053] in, Base penalty value; The set power safety threshold; Sensitivity coefficient; This represents the measured active power.

[0054]

[0055] in, This is the additional reward value after entering the steady-state dead zone; The preset steady-state error dead zone threshold;

[0056]

[0057] in, The weighting coefficient for the penalty of action change; The action vector from the previous moment; This represents the square of the L2 norm.

[0058] Optionally, step S4 includes:

[0059] In the active power control loop, the active power tracking error is obtained based on the measured active power and the active power reference value.

[0060] In the reactive power control loop, the reactive power tracking error is obtained based on the measured reactive power and the reactive power reference value;

[0061] The active power tracking error and the reactive power tracking error are respectively fed into the PID controller to obtain the active power reference current command value and the reactive power reference current command value.

[0062] The active power reference current command value and the reactive power reference current command value are input into the power decoupling control model to obtain the target rotation angle reference value.

[0063] The target angle reference value is input into the angle loop, and the RPST action in RPFC is driven according to the target angle reference value.

[0064] The beneficial effects of this invention are as follows: This invention provides a method for adaptive adjustment of parameters of a rotating power flow controller based on reinforcement learning. It outputs PID adjustment parameters through a trained Actor-Critic model, and adaptively adjusts the real-time active power and reactive power of the active power control loop and reactive power control loop in the dual-closed-loop electromechanical coupling dynamic model of the RPFC according to the PID adjustment parameters. The PID adjustment parameters include adjustment components of proportional gain, integral gain, and derivative gain. Due to the presence of derivative gain, severe power spikes or oscillations at the output can be avoided during adaptive adjustment, thereby preventing wear on the mechanical structure. Simultaneously, through the setting of the Actor-Critic model, power loop, and angle loop, closed-loop control can be formed, enabling rapid tracking of the target power angle reference value and ensuring a fast system response.

[0065] This invention utilizes a deep reinforcement learning mechanism to break free from the dependence of traditional control on precise system models. When a topology reconfiguration or line impedance parameter drift occurs in a "petal-shaped" distribution network, this invention does not require manual parameter retuning. The intelligent agent can automatically sense environmental changes based on the error differential characteristics in the state space and adjust the PID control parameters in real time, exhibiting extremely strong robustness.

[0066] In terms of control performance, this invention overcomes the limitation of traditional PI control in simultaneously achieving fast response and suppressing transient shocks. By introducing a derivative control loop and establishing a six-parameter adjustment mechanism, the system can utilize the derivative gain to provide crucial "virtual damping" in the early stages of disturbances, effectively resisting oscillations caused by voltage or load changes. Combined with the physical constraint of the proportional gain, the transient power spikes in the RPFC response process are reduced from a mechanistic perspective. This method can significantly reduce transient peak values ​​and solve the problem that RPFC is prone to triggering overcurrent protection at the moment of loop closure.

[0067] Regarding system safety, this invention innovatively introduces a tiered penalty mechanism. Unlike traditional algorithms that only pursue error minimization, this invention sets a strict power safety threshold in the reward function. Once the system predicts a potential risk of exceeding the threshold, it provides significant negative feedback. This mechanism forces the agent to develop a keen awareness of power spikes during the learning process, prioritizing the system's operation within absolutely safe physical boundaries while ensuring control accuracy.

[0068] Regarding equipment protection, this invention continues and optimizes the protection logic for mechanical components. By balancing electrical response and mechanical action amplitude in the optimization objectives, and combining the suppression effect of the differential term on high-frequency oscillations, it effectively reduces invalid jitter and torque pulsation of the servo motor during steady-state operation. This not only achieves smooth regulation of grid power, but also significantly reduces wear on the internal mechanical structure of the RPFC, and significantly extends the service life of the device. Attached Figure Description

[0069] Figure 1 is a flowchart illustrating the adaptive adjustment method for rotating power flow controller parameters based on reinforcement learning provided in an embodiment of the present invention.

[0070] Figure 2 is a schematic diagram of the RPFC provided in an embodiment of the present invention;

[0071] Figure 3 is a circuit diagram of the RPFC access line provided in an embodiment of the present invention;

[0072] Figure 4 is a schematic diagram of the processing procedure of the adaptive adjustment method for rotating power flow controller parameters based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0073] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. In the following description, the expression "some embodiments" refers to a subset of all possible embodiments; however, it should be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0074] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0075] It should be understood that the present invention can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Furthermore, the terminology used herein is intended only to describe particular embodiments and is not intended to limit the invention. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “compose” and / or “comprising,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0076] To fully understand this invention, a detailed structure will be presented in the following description to illustrate the technical solution proposed by this invention. Optional embodiments of the invention are described in detail below; however, in addition to these detailed descriptions, the invention may have other embodiments.

[0077] Example

[0078] Referring to Figures 1 and 4, this application provides a reinforcement learning-based adaptive adjustment method for the parameters of a rotating power flow controller, comprising the following steps:

[0079] Step S1: Construct a dual closed-loop electromechanical coupling dynamic model of RPFC including a power loop and an angle loop. The power loop includes an active power control loop, a reactive power control loop, and a power decoupling control model.

[0080] Step S2: Construct an Actor-Critic model based on the DDGP algorithm, train the Actor-Critic model, and obtain a trained Actor-Critic model.

[0081] Step S3: Use the trained Actor-Critic model output to obtain the PID control parameters of the RPFC dual closed-loop electromechanical coupling dynamic model. The PID control parameters include the adjustment components of the proportional gain, integral gain and derivative gain in the active power control loop and the reactive power control loop.

[0082] Step S4: In the power loop, calculate the active reference current command value and reactive reference current command value according to the PID adjustment parameters, and convert them into the corresponding target angle reference value. In the angle loop, drive the RPST action in RPFC according to the target angle reference value.

[0083] Specifically, the rotating power flow controller (RPFC) differs from fully electric electronic devices. It includes a phase-shifting transformer driven by a servo motor, and there is a coupling between the electrical response at the millisecond level and the mechanical response at the hundred-millisecond level. In the dq coordinate system, the regulation effect of the RPFC on the line current needs to take into account the execution delay of the servo motor. Therefore, it is necessary to first establish a power response equation containing a mechanical time constant, i.e., a state equation.

[0084] In adaptive parameter adjustment methods for rotating power flow controllers, traditional PI control is prone to overshoot due to response lag during sudden error changes. Specifically, traditional PI control only includes proportional (P) and integral (I) terms. The proportional term provides immediate response, while the integral term eliminates steady-state error. However, in systems with mechanical time constants, when an error changes abruptly, the adjustment command issued by the PI controller cannot be immediately translated into the physical action of the phase-shifting transformer, resulting in response lag. During the lag period, because the error persists, the integral term in the PI controller continues to accumulate until the mechanical structure begins to operate after the delay. By then, the accumulated integral term has become very large, causing the actuator to overshoot, resulting in severe power spikes or oscillations at the output, which can cause significant damage to the mechanical structure. In other words, in existing adaptive parameter adjustment methods for rotating power flow controllers, traditional PI control cannot simultaneously achieve both rapid adjustment response and suppression of transient impacts on the mechanical structure, thus exhibiting certain limitations.

[0085] Construct state equations to describe the dynamic characteristics of RPFC. The expressions for the state equations are as follows:

[0086]

[0087] in, The equivalent inductance of the line; This represents the current component of the d-axis active power control loop. This refers to the current component of the q-axis reactive power control loop. For time; The equivalent resistance of the line; The angular frequency of the power grid; This represents the d-axis component of the grid voltage. This represents the q-axis component of the grid voltage. This represents the d-axis component of the load terminal voltage. This represents the q-axis component of the load terminal voltage. The d-axis component of the injected voltage for RPFC. The q-axis component of the voltage injected into the RPFC; It is the equivalent mechanical time constant.

[0088] The state equation describes the physical response process of a rotating power flow controller (RPFC) after it is connected to the grid, and it determines the injected voltage. How does it affect electric current? This, in turn, affects power; that is, RPFC changes the voltage distribution of the power grid by adjusting the amplitude and phase of the injected voltage. The change in the injected voltage will change the total voltage drop of the line, thereby changing the current in the line. In the dq rotating coordinate system, the active and reactive components of the current correspond to active power P and reactive power Q, respectively. Due to the existence of active power control loop and reactive power control loop, the active and reactive components of the current will be adjusted accordingly. That is, the change in current directly affects active power and reactive power. Therefore, by adjusting the amplitude and phase of the voltage, the power output can be precisely controlled.

[0089] This technical solution constructs a state equation containing a mechanical time constant, which describes the physical response process of the rotating power flow controller after it is connected to the power grid, providing a physical basis for subsequent adaptive parameter adjustment. Then, a PID controller (containing proportional P, integral I, and derivative D) is used to introduce a derivative control loop into the active power control loop and the reactive power control loop. This allows the derivative control loop to directly act on the measured active power / reactive power when the measured active power / reactive power fluctuates drastically due to grid disturbances, rather than on the error. This eliminates the instantaneous spikes in the measured active power / reactive power and avoids impact damage to the mechanical structure during subsequent feedback-control adjustment.

[0090] An Actor-Critic model is constructed using the DDGP algorithm, a specific deep learning algorithm suitable for the "continuous action space" scenario in this invention. The constructed Actor-Critic model is trained to output PID control parameters. These PID control parameters include six parameters: the proportional gain, integral gain, and derivative gain adjustment components in the active power control loop, and the proportional gain, integral gain, and derivative gain adjustment components in the reactive power control loop. In other words, the trained Actor-Critic model... The itic model determines how the six parameters of the active power control loop and reactive power control loop are dynamically and optimally adjusted according to operating conditions. After obtaining the optimal PID control parameters through a trained Actor-Critic model, the PID control parameters are applied to the RPFC dual-closed-loop electromechanical coupling dynamic model to calculate the active power reference current command value and the reactive power reference current command value. These calculated values ​​are then converted to drive the RPFC, thereby adjusting the measured active power / reactive power. By continuously outputting the PID control parameters and adjusting accordingly, this process is repeated to achieve closed-loop power control. Specifically, the active power control loop / reactive power control loop calculates the deviation between the measured active / reactive power and the target power reference value, and uses an adaptive PID algorithm to convert it into the corresponding active / reactive power reference current command value. This command is then further converted into an injected voltage vector and decomposed into executable mechanical angle commands to drive the RPST in the RPFC for adjustment. Meanwhile, the system collects physical quantities such as line voltage, current, and power (P / Q) in real time through sensors as feedback signals, compares these feedback quantities with the state variables predicted by the state equation and updates them in real time, thereby triggering the control loop to correct the command value, realizing dynamic closed-loop control of 'physical quantity acquisition - state update - control command adjustment - execution feedback'.

[0091] It should be noted that an RPST is an electromagnetic sensor used to measure the angular displacement and angular velocity of a rotating object. It consists of a stator and a rotor, and its output voltage has a specific functional relationship with the rotor angle (such as sine, cosine, or linear). In power systems, RPSTs are commonly used for voltage regulation and phase control. By adjusting the rotor angle, the phase of the output voltage is changed, thereby controlling the voltage and power in the line.

[0092] The RPFC consists of two RPSTs, with their rotor windings connected in parallel and stator windings connected in series. Each RPST generates a voltage vector with a constant amplitude but an adjustable phase angle. Therefore, by adjusting the rotation angle of the two RPSTs... and This generates two stator voltage vectors with constant amplitude and adjustable phase, and their vector sum forms the series voltage of the injection line. It is used for voltage and phase regulation in the line. The RPFC combines the outputs of two RPSTs, expanding the voltage regulation range and enabling more flexible power and voltage control.

[0093] As an optional implementation, the formula for calculating the active power reference current command value is:

[0094]

[0095]

[0096] in, for The active power reference current command value on the d-axis at any given time; for The proportional gain of the active power control loop at all times. for The integral gain of the active power control loop at all times. for The differential gain of the active power control loop at all times; This refers to the active power tracking error. For discrete sampling sequence numbers; To control the sampling period; This represents the measured active power. This is a reference value for active power.

[0097] The formula for calculating the reactive reference current command value is as follows:

[0098]

[0099]

[0100] in, for The q-axis reactive current command value at any given time; for The proportional gain of the reactive power control loop at all times. for The integral gain of the reactive power control loop at all times. for The differential gain of the reactive power control loop at any given time; This refers to reactive power tracking error; For discrete sampling sequence numbers; To control the sampling period; This is the actual measured reactive power; This is a reference value for reactive power.

[0101] Specifically, in the active power control loop and reactive power control loop of the power loop, the power tracking error (active power tracking error and reactive power tracking error) is obtained by inputting the reference power (active power reference value and reactive power reference value) and the measured power (measured active power and measured reactive power). This power tracking error is processed by a PID controller with adaptive adjustment parameters obtained through reinforcement learning, and the output is used to calculate the reference current command value (active reference current command value and reactive reference current command value). Subsequently, the reference current command value is used as an intermediate variable and input into the power decoupling control model, and is transformed into the required injected voltage vector according to the voltage dq calculation formula.

[0102] The PID controllers in both the active power control loop and the reactive power control loop include proportional control, integral control, and derivative control elements. The control objective of the active power control loop is active power P, and the corresponding execution variable is the d-axis active power reference current command value. The structure of the reactive power control loop is symmetrical to that of the active power control loop, but its control objective is reactive power. The corresponding execution variable is Shaft reactive reference current command value.

[0103] The proportional control stage provides real-time feedback, rapidly generating an adjustment based on the current error magnitude. It is the main control mechanism, but excessive adjustment can cause system instability. The integral control stage eliminates steady-state error, continuously increasing as long as the error exists, ensuring that the measured active power in steady state is exactly equal to the reference active power, thus eliminating residual error. The derivative control stage predicts trends, generating a reverse suppression force when the power rises too rapidly by sensing the trend of power changes. The sum of these three stages constitutes a complete control law, determining the dynamic trajectory of the active power reference current command value, thereby achieving precise control of complex electromechanical coupling systems.

[0104] In the differential control stage, this invention applies the measured active power value rather than the error value to avoid "differential shocks" caused by sudden changes in the setpoint. Specifically, when a disturbance in the power grid causes a sharp fluctuation in the measured active power value, regardless of how the active power reference value changes (the active power reference value is the power target reference value set by the user or the upper-level power grid dispatch center, indicating how much active power the RPFC expects to output), the differential control stage will generate a reverse "virtual damping force" to directly offset the instantaneous power spike. This differs from conventional PID control and is specifically designed to address the mechanical oscillations of the RPFC.

[0105] The derivative control element of a conventional PID controller typically differentiates the error, i.e. If the power reference value changes abruptly, it will instantly generate an infinitely large spike, causing a huge impact on the mechanical structure.

[0106] As an optional implementation, the Actor-Critic model includes:

[0107] Step S21: Construct the state space;

[0108] Step S22: Define the action space and construct the parameter mapping mechanism;

[0109] Step S23: Design a composite reward function that combines graded punishment and transient inhibition.

[0110] As an optional implementation, the expression for the state space in step S21 is:

[0111]

[0112] in, for The state vector at any given time; The normalized error of the active power control loop. This is the normalized error of the reactive power control loop; This is the cumulative error term for the active power control loop. This is the cumulative error term for the reactive power control loop; The rate of change of active power in the active power control loop. The reactive power change rate of the reactive power control loop; This is the effective value of the voltage at the point of common coupling; This represents the standard deviation of voltage fluctuations. This refers to the mechanical state characteristics of the servo motor; This is the effective value of the load current.

[0113] Specifically, in the defined state vector, the rate of change of power includes the rate of change of active power and the rate of change of reactive power. : , representing the increment of active power / reactive power during the sampling period, is equivalent to the first derivative of active power with respect to time in discrete control systems; the rate of change of power is used by the agent to determine whether a large output is needed. The core basis;

[0114] The mechanical state of the servo motor is obtained by collecting feedback data from the servo driver, including the current phase shift angle, so that the algorithm can perceive the boundary of the physical actuator. The boundary of the physical actuator is mainly reflected in the extreme position of the rotation angle and the maximum speed of the motor.

[0115] The power errors and cumulative errors of the active power control loop / reactive power control loop in the state vector need to be normalized by subtracting their rated operating mean and dividing by the standard deviation, thus mapping to... The distribution is obtained by normalizing its value.

[0116] As an optional implementation, step S22 includes:

[0117] Define a reference parameter, the expression of which is:

[0118]

[0119] in, Reference parameter vector; The reference value for the proportional gain. This is the reference value for the integral gain. This is the reference value for the differential gain;

[0120] Define the action space as follows:

[0121]

[0122] in, for The action vector output by the agent at any given time. The proportional gain in the active power control loop Adjustment component, Integral gain in the active power control loop Adjustment component, The differential gain in the active power control loop Adjustment component, The proportional gain in the reactive power control loop Adjustment component, Integral gain in the reactive power control loop Adjustment component, The differential gain in the reactive power control loop The adjustment component, each component in the action vector is between [-1, 1];

[0123] Design an asymmetric scaling function, and calculate the adjustment components of the proportional gain, integral gain, and derivative gain of the active power control loop and the reactive power control loop based on the asymmetric scaling function. The expression of the asymmetric scaling function is:

[0124]

[0125] in, This is the vector of PID control parameters that actually take effect; To adjust the gain coefficient; It is an asymmetric scaling mapping function; The action vector output by the agent.

[0126] Specifically, a set of baseline parameters is defined and pre-tuned using the offline Ziegler-Nichols method or frequency domain analysis. These parameters are stable under rated operating conditions and ensure the system does not go out of control. An action space is defined, and an asymmetric scaling function is designed. This asymmetric scaling function is set to address the "peak suppression" requirement of RPFC. It establishes a mapping framework, stipulating that parameter adjustments are percentage increases or decreases based on baseline values, rather than aimless random searches. The proportional gain and derivative gain are calculated based on the asymmetric scaling function. This limits the proportional gain from being too large while allowing the derivative gain to be significantly increased, ensuring that the agent's output actions optimize performance without compromising the stability of the physical system.

[0127] The mapping calculation of proportional gain is to physically limit overshoot. Applying a negative bias to the proportional gain allows the agent to reduce the proportional gain to 20% of the baseline value, but can only increase it to a maximum of 110%. This forces the agent to mainly operate in the "low gain region" during exploration, reducing energy impact from the source from a physical mechanism.

[0128] The mapping calculation of the differential gain is to provide strong damping in transients. The differential gain can be increased significantly (e.g., by a factor of 10), and the basic damping is still retained even when the action output is -1. When the action output is +1, it provides extremely strong transient suppression capability. Therefore, the stability of the system can be guaranteed by using proportional gain and differential gain.

[0129] By calculating the proportional gain, integral gain, and derivative gain of the active power control loop and the reactive power control loop, it can be ensured that the proportional gain is limited to a low range, while the derivative gain has a large enhancement space, thereby achieving the technical effect of "suppressing transient spikes" at the physical level.

[0130] As an optional implementation, the adjustment components of the proportional gain, integral gain, and derivative gain of the active power control loop are calculated based on the asymmetric scaling function, using the following formula:

[0131]

[0132]

[0133]

[0134] in, The proportional gain of the active power control loop; Action vector The corresponding proportional gain adjustment component; The integral gain of the active power control loop; Action vector The corresponding adjustment component of the integral gain; The differential gain of the active power control loop; Action vector The corresponding adjustment component of the differential gain;

[0135] The adjustment components of the proportional gain, integral gain, and derivative gain of the reactive power control loop are calculated based on the asymmetric scaling function, using the following formula:

[0136]

[0137]

[0138]

[0139] in, The proportional gain of the reactive power control loop; Action vector The corresponding proportional gain adjustment component; The integral gain of the reactive power control loop; Action vector The corresponding adjustment component of the integral gain; The differential gain of the reactive power control loop; Action vector The corresponding adjustment component of the differential gain.

[0140] As an optional implementation, the composite reward function in step S23 includes:

[0141]

[0142] in, for Momentary compound rewards; Rewards for tracking errors; Peak penalty; For steady-state reward; For motion smoothing constraints;

[0143]

[0144] in, The weighting coefficient for the tracking error reward; This represents the absolute value of the active power error. This represents the absolute value of the reactive power error.

[0145]

[0146] in, Base penalty value; The set power safety threshold; Sensitivity coefficient; This represents the measured active power.

[0147]

[0148] in, This is the additional reward value after entering the steady-state dead zone; The preset steady-state error dead zone threshold;

[0149]

[0150] in, The weighting coefficient for the penalty of action change; The action vector from the previous moment; This represents the square of the L2 norm.

[0151] Specifically, to address the "peak" problem in RPFC power regulation, a composite reward function incorporating graded penalties and transient suppression was designed. This composite reward function includes tracking error reward, peak penalty, steady-state reward, and action smoothing constraint. The tracking error reward forces the agent to learn how to adjust parameters to make the error approach zero, thereby guiding the agent to reduce the absolute error and achieve power tracking. The tracking error reward uses an exponential form to improve convergence accuracy. This term is negative, and the larger the deviation of the measured power from the target value, the smaller the reward.

[0152] Peak penalty is key to suppressing transient spikes. By setting a power safety threshold (e.g., 2.5 MW), a large, non-linear penalty is applied when the absolute value of the measured power exceeds this threshold. This design forces the agent to automatically search for parameter combinations during training that can control the power peak below the power safety threshold (i.e., low peak power). high (Combination). As long as the measured power is within the power safety threshold, the penalty is 0; once it exceeds the threshold, the penalty increases exponentially with the amount of excess. By imposing huge penalties, the agent can avoid overshooting in subsequent decisions, and this also provides mathematical constraints for achieving the goal of "suppressing transient spikes".

[0153] Steady-state reward, when the error enters a very small dead zone (e.g. When the error in both the active power control loop and the reactive power control loop falls within a very small preset range, a positive reward is given to encourage the agent to increase the integral gain to eliminate steady-state error. Within this process, an additional positive reward is given to the agent, thus compensating for the insufficient driving force of the tracking error reward when the error is very small. It guides the agent to finely adjust the parameters (such as increasing the integral term) after entering a steady state in order to pursue high-precision control with zero steady-state error.

[0154] Motion smoothing constraints are used to protect the servo motor and mechanical transmission structure of the RPFC and prevent drastic parameter changes. The square penalty is applied to the change in motion. Motion smoothing constraints can protect the hardware and prevent PID parameters from changing drastically between adjacent sampling times, thereby avoiding high-frequency vibration or wear of the servo motor and ensuring that the actuator's motion is smooth and continuous.

[0155] As an optional implementation, step S4 includes:

[0156] In the active power control loop, the active power tracking error is obtained based on the measured active power and the active power reference value.

[0157] In the reactive power control loop, the reactive power tracking error is obtained based on the measured reactive power and the reactive power reference value;

[0158] The active power tracking error and the reactive power tracking error are respectively fed into the PID controller to obtain the active power reference current command value and the reactive power reference current command value.

[0159] The active power reference current command value and the reactive power reference current command value are input into the power decoupling control model to obtain the target rotation angle reference value.

[0160] The target angle reference value is input into the angle loop, and the RPST action in RPFC is driven according to the target angle reference value.

[0161] Specifically, please refer to Figure 3, which illustrates a dual-power supply loop-operated power distribution configuration. This is a schematic diagram of the circuit in which the RPFC device is applied in this embodiment of the invention. The RPFC device is installed within the red dashed box, where R1+jX1 is the line impedance of bus 1, R2+jX2 is the impedance of bus 2; P1+jQ1 is the load power of bus 1, P2+jQ2 is the load power of bus 2; and P3+jQ3 is the power flowing through the loop-closing point. and These represent the voltages on both sides of switch K1; I represents the line current l3 under closed-loop operation.

[0162] Referring to Figure 2, the Actor-Critic model includes a training phase and an application phase. During the training phase, the agent interacts with the controller in a simulation environment or a protected hardware environment, obtaining a five-tuple of information in the form of (state, action, reward, state transition probability, discount factor). The state transition probability is automatically inferred by the model and does not require explicit modeling. The discount factor is a hyperparameter, set to 0.99 in this embodiment, representing the model's focus on future rewards. The Actor-Critic model is further trained using this five-tuple information. This training process is a mature algorithm in deep reinforcement learning and is not within the scope of innovation of this invention, so it will not be elaborated upon here. The training result is an excellent Actor network, which can obtain the optimized control parameters of the two PID controllers (corresponding to the six adjustment components of the action vector in the action space, i.e., the adjustment components of the proportional gain, integral gain, and derivative gain corresponding to the active power control loop and the reactive power control loop) based on the state vector information in the state space. The Actor network is then applied.

[0163] It should be noted that in the Actor-Critic model, the state vector... The input is passed through two fully connected layers in the range [400, 300] to obtain the action. The action is the parameter adjustment coefficient of the PID (the adjustment components of the proportional gain, integral gain, and derivative gain corresponding to the active power control loop and the reactive power control loop). The union of the state and the action is input into the Critic network, and the output is the Q value. The Critic network is used to evaluate whether the Actor's "action" is performed well. Based on the state input to the Actor network and the selected action, the Critic network outputs a Q value. The Actor network updates its parameters based on the Q value gradient provided by the Critic network, so that the Actor network's goal is to make the output action obtain a higher Q value. In this process, the Critic network also predicts the sum of future cumulative rewards by observing the composite reward obtained at each time step. It continuously adjusts its criteria for judging good and bad (i.e., updates the Q-value function).

[0164] In the application phase, state vector information in the state space is acquired / calculated and input into the Actor network to obtain the current optimal PID control parameters (proportional gain, integral gain, and derivative gain corresponding to the active power control loop and reactive power control loop). The error between the measured power (measured active power and measured reactive power) and the reference power (active power reference value and reactive power reference value) is calculated through the power loop, and the active / reactive power reference current command value is output using the adaptive PID controller. Then, using the voltage dq calculation formula, the current command value is converted into the voltage vector required to eliminate the power deviation. Through geometric calculations, the target voltage vector is decomposed into two RPST target rotation angle reference values. and .

[0165] The inner loop angle control process then begins: two PI controllers receive the target angle reference value and output commands to control the servo motor to drive the rotor to rotate. The commands drive the servo motor to generate a rotational speed (…). The actual rotor angle is obtained by integrating the rotational speed. ), and continuously The feedback is compared with the target rotation angle reference value to form an angle closed loop. By changing the rotor angle, the phase and amplitude of the injected voltage are ultimately adjusted, enabling the measured power to track the reference power signal quickly and smoothly.

[0166] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for adaptive adjustment of parameters of a rotating power flow controller based on reinforcement learning, characterized in that, Includes the following steps: Step S1: Construct a dual-loop electromechanical coupling dynamic model of RPFC including a power loop and an angle loop. The power loop includes an active power control loop, a reactive power control loop, and a power decoupling control model. Step S2: Construct an Actor-Critic model based on the DDGP algorithm and train the Actor-Critic model to obtain a trained Actor-Critic model. Step S3: Use the output of the trained Actor-Critic model to obtain the PID control parameters of the RPFC dual-loop electromechanical coupling dynamic model. The PID control parameters include the adjustment components of the proportional gain, integral gain, and derivative gain in the active power control loop and the reactive power control loop. Step S4: In the power loop, calculate the active power reference current command value and the reactive power reference current command value according to the PID adjustment parameters, and convert them into the corresponding target angle reference value. In the angle loop, drive the RPST action in the RPFC according to the target angle reference value. The Actor-Critic model constructed based on the DDGP algorithm in step S2 includes: Step S21: Constructing the state space; Step S22: Defining the action space and constructing a parameter mapping mechanism, specifically including: defining the reference parameter, the expression of which is: in, Reference parameter vector; The reference value for the proportional gain. This is the reference value for the integral gain. Let be the reference value for the differential gain; define the action space as follows: in, for The action vector output by the agent at each moment. The proportional gain in the active power control loop Adjustment component, Integral gain in the active power control loop Adjustment component, The differential gain in the active power control loop Adjustment component, The proportional gain in the reactive power control loop Adjustment component, Integral gain in the reactive power control loop Adjustment component, The differential gain in the reactive power control loop The adjustment components are determined; an asymmetric scaling function is designed, and the adjustment components of the proportional gain, integral gain, and derivative gain of the active power control loop and the reactive power control loop are calculated based on the asymmetric scaling function. The expression of the asymmetric scaling function is as follows: in, This is the vector of PID control parameters that actually take effect; To adjust the gain coefficient; It is an asymmetric scaling mapping function; The action vector output by the agent; Step S23: Design a composite reward function of hierarchical penalty and transient inhibition, wherein the composite reward function includes: in, for Momentary compound rewards; This is a reward item for tracking error; This is a peak penalty item; This is a steady-state reward term; This is a motion smoothing constraint.

2. The adaptive parameter adjustment method for a rotating power flow controller based on reinforcement learning according to claim 1, characterized in that, The formula for calculating the active reference current command value is as follows: in, for The active power reference current command value on the d-axis at any given time; for The proportional gain of the active power control loop at all times. for The integral gain of the active power control loop at all times. for The differential gain of the active power control loop at all times; This refers to the active power tracking error. For discrete sampling sequence numbers; To control the sampling period; This represents the measured active power. This is a reference value for active power.

3. The adaptive parameter adjustment method for a rotating power flow controller based on reinforcement learning according to claim 1, characterized in that, The formula for calculating the reactive reference current command value is as follows: in, for The q-axis reactive current command value at any given time; for The proportional gain of the reactive power control loop at all times. for The integral gain of the reactive power control loop at all times. for The differential gain of the reactive power control loop at any given time; This refers to reactive power tracking error; For discrete sampling sequence numbers; To control the sampling period; This is the actual measured reactive power; This is a reference value for reactive power.

4. The adaptive adjustment method for rotating power flow controller parameters based on reinforcement learning according to claim 1, characterized in that, The expression for the state space in step S21 is: in, for The state vector at any given time; The normalized error of the active power control loop. This is the normalized error of the reactive power control loop; This is the cumulative error term for the active power control loop. This is the cumulative error term for the reactive power control loop; The rate of change of active power in the active power control loop. The reactive power change rate of the reactive power control loop; This is the effective value of the voltage at the point of common coupling; This represents the standard deviation of voltage fluctuations. This refers to the mechanical state characteristics of the servo motor; This is the effective value of the load current.

5. The adaptive parameter adjustment method for a rotating power flow controller based on reinforcement learning according to claim 1, characterized in that, The adjustment components of the proportional gain, integral gain, and derivative gain of the active power control loop are calculated based on the asymmetric scaling function, using the following formula: in, The proportional gain of the active power control loop; Action vector The corresponding proportional gain adjustment component; The integral gain of the active power control loop; Action vector The corresponding adjustment component of the integral gain; The differential gain of the active power control loop; Action vector The adjustment components corresponding to the derivative gain; the adjustment components of the proportional gain, integral gain, and derivative gain of the reactive power control loop are calculated according to the asymmetric scaling function, and the calculation formula is as follows: in, The proportional gain of the reactive power control loop; Action vector The corresponding proportional gain adjustment component; The integral gain of the reactive power control loop; Action vector The corresponding adjustment component of the integral gain; The differential gain of the reactive power control loop; Action vector The corresponding adjustment component of the differential gain.

6. The adaptive parameter adjustment method for a rotating power flow controller based on reinforcement learning according to claim 1, characterized in that, The composite reward function in step S23 is specifically as follows: in, The weighting coefficient for the tracking error reward; This represents the absolute value of the active power error. This represents the absolute value of the reactive power error. in, Base penalty value; The set power safety threshold; Sensitivity coefficient; This represents the measured active power. in, This is the additional reward value after entering the steady-state dead zone; The preset steady-state error dead zone threshold; in, The weighting coefficient for the penalty of action change; The action vector from the previous moment; This represents the square of the L2 norm.

7. The adaptive parameter adjustment method for a rotating power flow controller based on reinforcement learning according to claim 1, characterized in that, Step S4 includes: obtaining the active power tracking error based on the measured active power and the active power reference value in the active power control loop; obtaining the reactive power tracking error based on the measured reactive power and the reactive power reference value in the reactive power control loop; sending the active power tracking error and the reactive power tracking error into the PID controller to obtain the active power reference current command value and the reactive power reference current command value, respectively; inputting the active power reference current command value and the reactive power reference current command value into the power decoupling control model to obtain the target rotation angle reference value; inputting the target rotation angle reference value into the angle loop, and driving the RPST action in the RPFC according to the target rotation angle reference value.

Images