Inertia flywheel control method and system based on reinforcement learning

Abstract

The application discloses a kind of inertia flywheel control method and system based on reinforcement learning, the method includes: the conversion experience multiple tuple obtained by interactive simulation environment and neural network is stored in experience pool, and the experience multiple tuple of the set number of random sampling from current time experience pool is used to update neural network;According to the reward in current experience pool, judge whether to meet neural network update end condition, if meet, then the updated neural network is used as the neural network of training completion, otherwise continue to update neural network until meeting end condition;The state quantity of inertia flywheel output end needing control quantity calculation is input to the trained neural network to obtain action, and the control quantity to be input to inertia flywheel is obtained based on action and torque setting value.Utilize the scheme of the application, neural network can adjust its strategy according to actual performance, more accurately predict and calculate the action corresponding to the state quantity of inertia flywheel output end.

Description

TECHNICAL FIELD

[0001] The present application generally relates to the technical field of energy storage. More specifically, the present application relates to a reinforcement learning-based inertia flywheel control method and system. BACKGROUND

[0002] In the prior art, inertia flywheel technology with large inertia, strong reactive power support and high overload capacity has become one of the effective ways to solve the frequency and voltage regulation problems of new energy power systems. Inertia flywheel can provide efficient inertia and voltage support, effectively alleviating system frequency and voltage fluctuations. Its high overload capacity enables it to quickly respond to system frequency and voltage fluctuations and provide a large amount of mechanical inertia and reactive power in a short time, helping to maintain the stability of system frequency and voltage.

[0003] However, when used as a frequency and voltage regulation device for a power system, inertia flywheel faces the problem of its own low-frequency oscillation and the impact of responding to grid low-frequency oscillation, which can cause serious harm to the power system, such as reducing system stability and damaging equipment life.

[0004] Therefore, there is an urgent need to provide a reinforcement learning-based inertia flywheel control scheme to solve the problem of inertia flywheel's own low-frequency oscillation and the impact of responding to grid low-frequency oscillation. SUMMARY

[0005] To at least solve one or more of the above-mentioned technical problems, the present application provides a reinforcement learning-based inertia flywheel control scheme in multiple aspects.

[0006] In a first aspect, the present application provides a reinforcement learning-based inertia flywheel control method, comprising: storing a converted experience tuple (s t ,a t ,r t ,s t+1 ) obtained through an interactive simulation environment and a neural network into an experience pool, and updating the neural network with a set number of experience tuples randomly sampled from the experience pool at the current time to update the neural network, wherein s t is the state quantity of the output end of the inertia flywheel at the current time t, a t is the action corresponding to the state quantity at the current time t, r t is the reward at the current time t, and s t+1is an action for the next time; determining whether a neural network update end condition is satisfied according to rewards in all experience tuples in the experience pool at the current time; in response to the neural network update end condition being satisfied, taking the updated neural network as a trained neural network; and in response to the neural network update end condition not being satisfied, returning to the step of updating the neural network until the neural network update end condition is satisfied, taking the neural network updated in the last execution of the step of updating the neural network as the trained neural network; inputting a state quantity of an output end of the inertia flywheel for which a control quantity needs to be calculated to the trained neural network to obtain an action, and obtaining a control quantity to be input to the inertia flywheel based on the action and a torque setting value.

[0007] In some embodiments, the state quantity includes active power, a differential of the active power, reactive power, a differential of the reactive power, terminal voltage, a differential of the terminal voltage, rotational speed, and a differential of the rotational speed.

[0008] In some embodiments, the action includes a damping voltage component and a damping torque component.

[0009] In some embodiments, the neural network update end condition includes that a sum of the rewards in all experience tuples in the current experience pool is greater than a preset value of the sum of the rewards.

[0010] In some embodiments, r t is obtained by a reward function, wherein the reward function is: t step is a set time step, t is a current time, ω e is an actual rotational speed of a rotor of the output end of the inertia flywheel, ω ref is a rated rotational speed of the rotor of the output end of the inertia flywheel.

[0011] In some embodiments, the torque setting value is a torque setting value input to the output end of the inertia flywheel, and the torque setting value is obtained by a torque setting value formula, wherein the torque setting value formula is: T is the torque setting value, P ref is an active power output by the inertia flywheel, ω m is an angular speed of the rotor of the output end of the inertia flywheel, K flag is an inertia flywheel charging and discharging flag, K flag is -1 when the inertia flywheel is charging, and K flag is +1 when the inertia flywheel is discharging, and ΔP is a power compensation term. R a is a phase resistance of the output end of the inertia flywheel.

[0012] In some embodiments, the control quantity includes a driving signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel.

[0013] In some embodiments, obtaining the control amount to be input to the inertia flywheel based on the action and the torque set value comprises: adding the damping torque component and the torque set value to obtain a torque base value; obtaining a torque measured value corresponding to the inertia flywheel for which the control amount is to be calculated, and subtracting the torque base value from the torque measured value to obtain an error value; inputting the obtained error value to a PID controller for proportional, integral and differential operations, and outputting the error value to a gain amplifier for processing to obtain a first signal; subtracting the first signal from a q-axis current of an output end of the inertia flywheel for which the control amount is to be calculated, and inputting the result to the PID controller for proportional, integral and differential operations to obtain a second signal; subtracting the second signal from ω e i d L d and adding ω e Flux signals to obtain a third signal, inputting the third signal to an integrator for integral processing, and inputting the signal after the integral processing to a pulse width modulator to obtain a drive signal β-axis component input to the inertia flywheel, wherein ω e is an actual rotational speed of a rotor at an output end of the inertia flywheel, i d is a d-axis current of the output end of the inertia flywheel for which the control amount is to be calculated, L d is a d-axis inductance of the output end of the inertia flywheel for which the control amount is to be calculated, and Flux is a magnetic flux of the output end of the inertia flywheel for which the control amount is to be calculated.

[0014] In some embodiments, obtaining the control amount to be input to the inertia flywheel based on the action and the torque set value further comprises: inputting a damping voltage component, a voltage measured value of the output end of the inertia flywheel for which the control amount is to be calculated, and a voltage given value of the output end of the inertia flywheel for which the control amount is to be calculated to an excitation controller to obtain an excitation voltage.

[0015] In a second aspect, the application provides a reinforcement learning-based inertia flywheel control system, which controls an inertia flywheel by using the reinforcement learning-based inertia flywheel control method according to any one of the embodiments of the first aspect. The system comprises: a neural network updating module, which is configured to store a conversion experience tuple obtained by interacting with a simulation environment and a neural network into an experience pool, and randomly sample a set number of experience tuples from the experience pool at a current time to update the neural network; a data processing module, which is configured to determine whether a neural network updating end condition is met according to a reward in all experience tuples in the experience pool at the current time, respond to the neural network updating end condition being met by taking the updated neural network as a trained neural network, and respond to the neural network updating end condition not being met by using the neural network updating module to update the neural network again until the neural network updating end condition is met, and take the neural network updated in the last time of updating the neural network as the trained neural network; and a control quantity obtaining module, which is configured to input a state quantity of an output end of the inertia flywheel that needs to be controlled into the trained neural network to obtain an action, and obtain a control quantity to be input into the inertia flywheel based on the action and a torque set value.

[0016] By using the reinforcement learning-based inertia flywheel control method provided above, the embodiments of the application determine whether the neural network needs to be updated again according to a reward in all experience tuples in the experience pool at the current time, and take the finally updated neural network as the trained neural network to obtain an action corresponding to a state quantity of an output end of the inertia flywheel that needs to be controlled, and obtain a control quantity to be input into the inertia flywheel based on the action and a torque set value, so that the neural network can adjust its strategy according to actual performance, more accurately predict and calculate the action corresponding to the state quantity of the output end of the inertia flywheel, and more accurately control the state of the inertia flywheel, thereby improving the control precision of the inertia flywheel.

[0017] Further, in some embodiments, by setting the action to include a damping voltage component and a damping torque component, the problem of the rotor of the inertia flywheel oscillating greatly at the output end of the inertia flywheel can be solved when the control quantity to be input into the inertia flywheel is obtained based on the action and the torque set value, and the anti-grid impact capability and self-stability of the inertia flywheel are improved. Further, in some embodiments, by taking the sum of the rewards in all experience tuples in the current experience pool being greater than a reward sum preset value as the neural network updating end condition, it can be ensured that the neural network reaches a certain performance level after training, and overfitting of the neural network on the training data and invalid training of the neural network are avoided. BRIEF DESCRIPTION OF DRAWINGS

[0018] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description read in conjunction with the accompanying drawings, in which:

[0019] Figure 1 An exemplary flow chart of the inertia flywheel control method based on reinforcement learning of the embodiments of the present application is shown;

[0020] Figure 2 A schematic diagram of the composition of the inertia flywheel of the embodiments of the present application is shown;

[0021] Figure 3 A schematic diagram of the interaction of the agent model with the simulation environment of the embodiments of the present application is shown;

[0022] Figure 4 A schematic diagram of the obtaining of the inertia flywheel control quantity based on the action and torque set value of the embodiments of the present application is shown;

[0023] Figure 5 A schematic diagram of the obtaining of the inertia flywheel control quantity based on the action and torque set value of some embodiments of the present application is shown;

[0024] Figure 6 A schematic diagram of the obtaining of the driving signal of the embodiments of the present application is shown;

[0025] Figure 7 An exemplary structural block diagram of the inertia flywheel control system of the embodiments of the present application is shown. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0027] It should be understood that the terms “include” and “contain” used in the specification and claims of the present application indicate the presence of the described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0028] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this specification and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this specification and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0029] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] The specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0031] In existing technologies, the traditional solution to the low-frequency oscillation problem of inertial flywheels is to add a power system stabilizer (PSS). This mainly involves the PSS adjusting the excitation system of the generator within the flywheel to improve the excitation system and thus suppress low-frequency oscillations. However, the PSS modifies the generator's reactive power output by adjusting the excitation current. Since changes in the excitation current require time to affect the generator's electromagnetic characteristics, this adjustment method is relatively slow. Furthermore, due to the dynamic characteristics of the excitation system, the control accuracy of the PSS may be limited, especially under rapidly changing system conditions, where the excitation system's response may be imprecise, resulting in poor control performance. In addition, the PSS needs to be integrated with the existing excitation system, increasing system complexity.

[0032] In view of this, embodiments of this application provide an inertial flywheel control scheme based on reinforcement learning, which can solve the problem of low-frequency oscillation of the inertial flywheel itself and its response to the impact of low-frequency oscillation of the power grid.

[0033] Figure 1 An exemplary flowchart of an inertial flywheel control method 100 based on reinforcement learning according to an embodiment of this application is shown.

[0034] like Figure 1 As shown, in step S110, the transformation experience multivariate (s) obtained through the interactive simulation environment and neural network is... t ,a t ,r t,s t+1 ) Stored in the experience pool, and a set number of experience tuples are randomly sampled from the experience pool at the current time to update the neural network, where s t Let a be the state quantity at the current time t of the inertial flywheel output. t For the action corresponding to the state variable at the current time t, r t s is the reward at the current time t. t+1 For the action in the next moment.

[0035] In the embodiments of this application, the inertia flywheel includes: a flywheel, an electromagnetic coupler, a synchronous condenser, and a converter controller.

[0036] Figure 2 A schematic diagram of the composition of the inertia flywheel 200 according to an embodiment of this application is shown.

[0037] like Figure 2 As shown, the inertia flywheel 200 includes a flywheel 210, an electromagnetic coupler 220, a synchronous condenser 230, and a converter controller 240. Specifically, the electromagnetic coupler 220 includes an inner rotor and an outer rotor. The rotor of the flywheel 210 is connected to the inner rotor via a first drive shaft 221, and the rotor of the synchronous condenser 230 is connected to the outer rotor via a first drive shaft 222. The inner rotor uses permanent magnets, and the outer rotor has an armature winding. The electromagnetic coupler 220 is equivalent to a permanent magnet synchronous motor in which the stator and rotor rotate simultaneously, enabling the electromagnetic coupler 220 to transmit torque and inertia between the flywheel 210 and the synchronous condenser 230. The stator of the synchronous condenser 230 is directly connected to the power grid 250. The converter controller 240 can control the electromagnetic coupler 220, and the power grid 250 can supply power to the electromagnetic coupler 220 through the converter controller 240.

[0038] In the embodiments of this application, the synchronous condenser 230 is used as the output terminal of the inertia flywheel 200.

[0039] Specifically, the converter controller 240 employs a back-to-back converter. When the external frequency rises or falls, the converter controller 240 controls the output torque of the electromagnetic coupler 220. Since the electromagnetic coupler 220 is directly connected to the rotor of the synchronous condenser 230 via a mechanical shaft, the output torque of the electromagnetic coupler 220 is the input torque of the synchronous condenser 230. Because the electromagnetic coupler 220 is equivalent to a permanent magnet synchronous motor with both stator and rotor rotating simultaneously, its output torque can be controlled by a vector controller.

[0040] In the embodiments of this application, the aforementioned simulation environment is constructed based on the parameters of the real-time running data of the inertial flywheel. It is mainly constructed by giving the system state space (i.e., output quantity) and action space (i.e., input quantity) during the real-time running process of the inertial flywheel in the training process of deep reinforcement learning.

[0041] In the embodiments of this application, a neural network including an Actor network and a Critic network is built based on graph convolutional neural networks and deep reinforcement learning algorithms, and an intelligent agent model is constructed based on the neural network and the experience pool.

[0042] In the embodiments of this application, after the simulation environment and the intelligent agent model are constructed, the constructed simulation environment and the intelligent agent model are initialized. Then, the initialized intelligent agent model interacts with the initialized simulation environment to obtain the initial state s0 of the inertia flywheel output. Then, the interaction between the intelligent agent model and the simulation environment is repeated to obtain multiple conversion experience tuples.

[0043] Figure 3 This diagram illustrates the interaction between the intelligent agent model and the simulation environment according to an embodiment of this application.

[0044] like Figure 3 As shown, the Actor network includes a first optimizer, an online policy network, and a target policy network, while the Critic network includes a second optimizer, an online Q network, and a target Q network. The first optimizer and the second optimizer may be the same or different.

[0045] During the interaction between the intelligent agent model and the simulation environment, the Actor network collects the state quantity s at the current time t of the inertial flywheel output terminal from the simulation environment. t Based on the state quantity s at the current time t of the inertia flywheel output terminal t Random noise is introduced to obtain the output state variable s. t The corresponding action a t And apply it to the simulation environment to obtain the action a at the current time t. t The corresponding reward r t and the state quantity s at the output of the inertial flywheel in the simulation environment at the next moment. t+1 And the state quantity s at the current time t will be generated by the output terminal of the inertial flywheel. t Output state variable s t The corresponding action a t At the current time t, action a t The corresponding reward r t and the state quantity s at the output of the inertial flywheel in the simulation environment at the next moment. t+1 The transformed experience plural (s) constitute t ,a t ,r t ,s t+1 The data is stored in the experience pool. Random noise is introduced to obtain the output state variable s. t The corresponding action a t It can enhance the ability of intelligent agent models to explore the environment.

[0046] Next, the Critic network randomly samples a predetermined number of empirical plurals (s) from the current time-instance empirical pool. i ,a i ,r i ,s i+1 ), where s i Let a be the state quantity at time i of the inertial flywheel output terminal. i Let r be the action corresponding to the state variable at time i. i For the reward at time i, s i+1 This refers to the action at time i+1. Specifically, during the random sampling process, the specific value of the sampled empirical tuple can be set according to the total number of empirical tuples in the current empirical pool, which is not limited in this application.

[0047] Then, by sampling a predetermined number of empirical plural groups (s) i ,a i ,r i ,s i+1 Calculate the objective function y corresponding to the target Q-network in the Critic network. i The target Q-network will target the function y i Send to an online Q-network, which is based on the objective function y. i The loss function L of the online Q-network is obtained. The Q-gradient is obtained by minimizing the loss function L and sent to the second optimizer. The second optimizer updates the parameters of the online Q-network based on the received Q-gradient. The online policy network in the Actor network obtains the policy gradient based on the gradients of the online Q-network with respect to the online policy network and the gradients of the online Q-network with respect to the parameters of the online Q-network. This policy gradient is sent to the first optimizer, which updates the parameters of the online policy network based on the received policy gradient. Furthermore, the online policy network performs soft updates to the target policy network using a smoothing factor, and the online Q-network also performs soft updates to the target Q-network using a smoothing factor.

[0048] In the embodiments of this application, the state quantities at the output terminal of the inertia flywheel include active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed. Since the output terminal of the inertia flywheel uses a synchronous condenser, the state quantities collected in the simulation environment are the active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed at the stator terminal of the synchronous condenser.

[0049] In the embodiments of the present application, the action includes a damping voltage component and a damping torque component. Specifically, the damping voltage component acts on an excitation controller that controls the excitation current of the synchronous compensator in the inertia flywheel, and the damping torque component is an input signal of a vector controller of the electromagnetic coupler in the inertia flywheel. The excitation controller changes the magnetic field strength generated by the synchronous compensator by adjusting the direct current supplied to the rotor of the synchronous compensator, thereby affecting the active and reactive power output by the synchronous compensator.

[0050] Specifically, the damping voltage component changes the magnetic field strength of the synchronous compensator by adjusting the excitation current, thereby suppressing the oscillation of the rotor of the synchronous compensator. When the power grid is subjected to an impact or disturbance, the damping voltage component can quickly adjust the excitation current, helping the synchronous compensator to quickly recover to a stable state and reducing the impact on the system. The damping torque component directly acts on the rotor of the synchronous compensator by adjusting the output torque of the electromagnetic coupler, thereby reducing the oscillation of the rotor of the synchronous compensator. When the power grid is subjected to an impact or disturbance, the damping torque component can quickly adjust the output of the electromagnetic coupler, helping the inertia flywheel to quickly recover to a stable state and reducing the impact on the system.

[0051] Therefore, by making the action include a damping voltage component and a damping torque component, the problem of large rotor oscillation at the output end of the inertia flywheel when outputting can be solved when obtaining the control amount to be input to the inertia flywheel based on the action and the torque set value, and the anti-impact ability of the power grid and the self-stability of the inertia flywheel can be improved.

[0052] In the embodiments of the present application, the reward r t is obtained by a reward function, wherein the reward function is:

[0053] t step is a set time step, t is the current time, ω e is the actual speed of the rotor at the output end of the inertia flywheel, ω ref is the rated speed of the rotor at the output end of the inertia flywheel.

[0054] As can be seen from the reward function, the speed deviation is used as the evaluation standard of the action. Since the speed deviation directly reflects the oscillation of the rotor of the synchronous compensator, if the speed deviation is small, it means that the oscillation of the rotor is effectively suppressed and the system stability is good. The goal of the present application is to minimize the oscillation of the synchronous compensator when dealing with sudden changes in output power and power grid impacts, and the speed deviation directly reflects the degree of realization of this goal. Therefore, by setting the reward function that uses the speed deviation as the evaluation standard of the action, the effectiveness of suppressing the oscillation can be directly reflected.

[0055] After step S110 is performed, in step S120, it is determined whether a neural network update end condition is met according to the rewards in all experience tuples in the experience pool at the current time.

[0056] In some embodiments of the present application, the neural network update end condition can be that the sum of the rewards in all experience tuples in the current experience pool is greater than a preset reward sum value. In other embodiments of the present application, the neural network update end condition can also be that the average of the rewards in all experience tuples in the current experience pool is greater than a preset reward average value, which is not limited in the present application.

[0057] By taking the sum of the rewards in all experience tuples in the current experience pool being greater than the preset reward sum value as the neural network update end condition, the training can only be stopped when the accumulated reward sum exceeds the preset value, which means that the performance of the neural network on the training data has reached the expected standard, and the training can be stopped after the neural network reaches the expected standard level, avoiding over-fitting of the model to the training data caused by continued training, which can prevent the neural network from over-learning specific noise and details on the training data. At the same time, by taking the sum of the rewards in all experience tuples in the current experience pool being greater than the preset reward sum value as the neural network update end condition, it can avoid wasting computing resources and time in the case where the performance of the neural network improves very limitedly or even no longer improves.

[0058] In response to the neural network update end condition being met, in step S130, the current updated neural network is taken as the trained neural network.

[0059] In response to the neural network update end condition not being met, return to step S110 to re-perform the step of updating the neural network until the neural network update end condition is met. When step S110 is performed multiple times, the neural network updated by the last time step S110 is performed is taken as the trained neural network.

[0060] After obtaining the trained neural network, in step S140, the state quantity of the output end of the inertia flywheel for which the control quantity needs to be calculated is input to the trained neural network to obtain an action, and the control quantity to be input to the inertia flywheel is obtained based on the action and the torque set value.

[0061] Specifically, since the neural network can process the input state quantity and output the action in a very short time, by inputting the state quantity of the inertia flywheel output end which needs to be calculated to the control quantity into the trained neural network to obtain the action, the inertia flywheel can be controlled efficiently. At the same time, since the trained neural network can find the optimal control strategy, by inputting the state quantity of the inertia flywheel output end which needs to be calculated to the control quantity into the trained neural network to obtain the action, the action corresponding to the state quantity of the inertia flywheel output end can be more accurately predicted and calculated, and the state of the inertia flywheel can be more accurately controlled, thereby improving the control precision of the inertia flywheel.

[0062] In the embodiment of the present application, the torque set value is a torque set value input to the output end of the inertia flywheel, and the torque set value is obtained through a torque set value formula, wherein the torque set value formula is:

[0063] T is the torque set value, P ref is the active power output by the inertia flywheel, ω m is the angular velocity of the rotor at the output end of the inertia flywheel, K flag is the charge-discharge flag bit of the inertia flywheel, K flag is-1 when the inertia flywheel is charging, and K flag is+1 when the inertia flywheel is discharging, and ΔP is a power compensation term. R a is the phase resistance at the output end of the inertia flywheel.

[0064] The power compensation term is used to compensate for the difference between the output power of the synchronous condenser and the input mechanical power. By introducing the power compensation term into the torque set value, the case where the output power of the synchronous condenser is lower than the input mechanical power due to the loss of the synchronous condenser winding in practice is considered, so that the torque set value obtained is more suitable for actual needs, which makes the control quantity obtained based on the action and the torque set value more accurate.

[0065] In the embodiment of the present application, the control quantity includes a drive signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel. After the control quantity is obtained, the control quantity is input to the inertia flywheel to control it.

[0066] Figure 4 A schematic diagram of the present application for obtaining the control quantity of the inertia flywheel based on the action and the torque set value is shown.

[0067] As Figure 4As shown, the agent model collects the state quantity of the output end of the inertia flywheel from the simulation environment, updates the adopted strategy based on the collected state quantity and the reward function, and outputs the action including the damping torque component and the damping voltage component when the optimal strategy is obtained after multiple updates. Then, the damping torque component and the aforementioned torque set value are added to obtain a torque base value, and the torque base value is taken as the input of the vector controller to obtain a driving signal. At the same time, the damping voltage component is input into the excitation controller to obtain an excitation voltage after processing. Then, the control quantity including the driving signal and the excitation voltage is input into the inertia flywheel to control the same.

[0068] In some embodiments of the present application, the process of obtaining the inertia flywheel control quantity based on the action and the torque set value can refer to Figure 5 .

[0069] As shown in Figure 5 , the agent model collects the state quantity of the stator end of the synchronous condenser from the simulation environment, and outputs the action including the damping torque component and the damping voltage component using the trained agent model. Then, the damping torque component and the aforementioned torque set value are added to obtain a torque base value, and the torque base value is taken as the input of the vector controller to obtain a driving signal. The output torque of the electromagnetic coupler is controlled based on the driving signal by the vector controller, so that the output torque directly acts on the rotor of the synchronous condenser, reducing the oscillation of the rotor of the synchronous condenser. At the same time, the damping voltage component, the voltage measured value of the stator end of the synchronous condenser which needs to be calculated for control quantity, and the voltage given value of the output end of the inertia flywheel which needs to be calculated for control quantity are input into the excitation controller to obtain an excitation voltage, and the excitation current flowing into the rotor winding of the synchronous condenser is controlled by the excitation voltage. Then, by adjusting the excitation current, the magnetic field strength of the synchronous condenser is changed to suppress the oscillation of the rotor of the synchronous condenser.

[0070] Specifically, the excitation controller can calculate the excitation voltage using the control algorithm set therein. The control algorithm used can be an adaptive control algorithm, a model predictive control algorithm, etc., which is not limited in the present application.

[0071] Specifically, taking the torque base value as the input of the vector controller to obtain the driving signal can refer to Figure 6 .

[0072] As shown in Figure 6 , the torque measured value T e of the inertia flywheel which needs to be calculated for control quantity is obtained. eref The torque base value T eThe error value is obtained by subtraction, and the error value is input into a PID controller to output a first signal after proportional, integral and differential operations, and the first signal is input into a gain amplifier to obtain a d-axis component u of the driving signal of the inertia flywheel q The second signal is obtained by inputting into a PID controller after proportional, integral and differential operations, and the second signal is input into a gain amplifier to obtain a q-axis component u of the driving signal of the inertia flywheel e i d L d The third signal is obtained by adding the flux signal and the ω e The third signal is input into an integrator to perform integral processing, and the signal after integral processing is input into a pulse width modulator to obtain the driving signal of the inertia flywheel β wherein ω e is the actual rotating speed of the rotor of the inertia flywheel output end, i d is the d-axis current of the inertia flywheel output end, L d is the d-axis inductance of the inertia flywheel output end, and Flux is the magnetic flux of the inertia flywheel output end.

[0073] At the same time of obtaining the driving signal of the β-axis component, the d-axis current of the inertia flywheel output end is given as i dref The error value is obtained by subtraction, and the error value is input into a PID controller to output a fourth signal after proportional, integral and differential operations, and the fourth signal is input into a gain amplifier to obtain a d-axis component u of the driving signal of the inertia flywheel e i q L q The third signal is obtained by adding the flux signal and the ω α The third signal is input into an integrator to perform integral processing, and the signal after integral processing is input into a pulse width modulator to obtain the driving signal of the inertia flywheel e wherein ω q is the actual rotating speed of the rotor of the inertia flywheel output end, i q is the q-axis current of the inertia flywheel output end, L is the q-axis inductance of the inertia flywheel output end.

[0074] In the embodiment of the present application, the aforementioned pulse width modulator can be an SVPWM pulse width modulator.

[0075] In summary, by means of the inertia flywheel control scheme based on reinforcement learning provided above, the embodiments of the present application determine whether the neural network needs to be updated again according to the rewards in all experience tuples in the experience pool at the current time, and take the finally updated neural network as the trained neural network to obtain the action corresponding to the state quantity of the output end of the inertia flywheel that needs to be controlled, and obtain the control quantity to be input to the inertia flywheel based on the action and the torque set value, so that the neural network can adjust its strategy according to the actual performance, more accurately predict and calculate the action corresponding to the state quantity of the output end of the inertia flywheel, and more accurately control the state of the inertia flywheel, thereby improving the control precision of the inertia flywheel.

[0076] Further, in some embodiments, by setting the action to include the damping voltage component and the damping torque component, the problem of large rotor oscillation of the output end of the inertia flywheel when outputting can be solved when obtaining the control quantity to be input to the inertia flywheel based on the action and the torque set value, and the anti-grid impact capability and self-stability of the inertia flywheel are improved. Further, in some embodiments, by taking the sum of the rewards in all experience tuples in the current experience pool being greater than a reward sum preset value as the neural network update end condition, it can be ensured that the neural network reaches a certain performance level after training, while avoiding overfitting of the neural network on the training data and invalid training of the neural network.

[0077] The embodiments of the present application also provide an inertia flywheel control system based on reinforcement learning, which can control the inertia flywheel by means of the inertia flywheel control method 100 based on reinforcement learning described above.

[0078] Figure 7 An exemplary structural block diagram of the inertia flywheel control system 700 of the embodiments of the present application is shown.

[0079] As shown in Figure 7 The system 700 includes a neural network updating module 710, a data processing module 720 and a control quantity obtaining module 730. The data processing module 720 and the neural network updating module 710 are connected to each other, and the control quantity obtaining module 730 is connected to the data processing module 720. In the embodiments of the present application, the neural network updating module 710, the data processing module 720 and the control quantity obtaining module 730 can be separate units or can be integrated in the same controller.

[0080] Specifically, the neural network updating module 710 is configured to store the converted experience tuples obtained through the interaction simulation environment and the neural network into the experience pool, and update the neural network by randomly sampling a set number of experience tuples from the experience pool at the current time.

[0081] Specifically, the data processing module 720 is configured to determine whether a neural network update end condition is met according to the rewards in all experience tuples in the experience pool at the current time, to take the updated neural network as a trained neural network in response to the neural network update end condition being met, and to update the neural network again by the neural network update module in response to the neural network update end condition not being met until the neural network update end condition is met, and to take the neural network updated in the last time of updating the neural network as the trained neural network.

[0082] Specifically, the control quantity acquisition module 730 is configured to input the state quantity of the inertia flywheel output end requiring control quantity calculation to the trained neural network to obtain an action, and to obtain the control quantity to be input to the inertia flywheel based on the action and the torque setting value.

[0083] When the system 700 controls the inertia flywheel by using the aforementioned inertia flywheel control method 100 based on reinforcement learning, the neural network update module 710 performs the aforementioned step S110, the data processing module 720 performs the aforementioned steps S110 and S120, and the control quantity acquisition module 730 performs the aforementioned step S140. The specific execution process can be referred to the foregoing, and will not be described here again.

[0084] Although the embodiments of the present application have been shown and described herein, it should be apparent to those skilled in the art that such embodiments are merely illustrative of the present application. Many changes, modifications and alternative ways of implementing the present application can be suggested to one skilled in the art and it is intended to encompass such changes, modifications and alternative ways within the scope of the appended claims.

Claims

1. A method for controlling an inertial flywheel based on reinforcement learning, characterized in that, include: The transformation experience multivariate (s) obtained through interactive simulation environment and neural network t ,a t ,r t ,s t+1 ) Stored in the experience pool, and a set number of experience tuples are randomly sampled from the experience pool at the current time to update the neural network, where s t Let a be the state quantity at the output of the inertial flywheel at the current time t. t For the action corresponding to the state variable at the current time t, r t s is the reward at the current time t. t+1 For the action in the next moment; Determine whether the neural network update termination condition is met based on the rewards in all experience tuples in the experience pool at the current moment. In response to the satisfaction of the neural network update termination condition, the updated neural network is designated as the trained neural network; and In response to the failure to meet the neural network update termination condition, return to the step of updating the neural network until the neural network update termination condition is met, and take the neural network updated by the last step of updating the neural network as the neural network that has been trained. The state quantity of the inertial flywheel output terminal that needs to be calculated for control is input to the trained neural network to obtain the action, and the control quantity to be input to the inertial flywheel is obtained based on the action and torque setpoint.

2. The inertial flywheel control method based on reinforcement learning according to claim 1, characterized in that, The state quantities include active power, the derivative of active power, reactive power, the derivative of reactive power, terminal voltage, the derivative of terminal voltage, rotational speed, and the derivative of rotational speed.

3. The inertial flywheel control method based on reinforcement learning according to claim 1, characterized in that, The action includes a damping voltage component and a damping torque component.

4. The inertial flywheel control method based on reinforcement learning according to claim 1, characterized in that, The conditions for ending the neural network update include: the sum of rewards in all experience tuples in the current experience pool is greater than a preset value for the sum of rewards.

5. The reinforcement learning-based inertial flywheel control method according to claim 1 or 4, characterized in that, r t It is obtained through a reward function, wherein the reward function is: t step The set time step is t, where t is the current time, and ω is the current time step. e ω represents the actual rotational speed of the rotor at the output end of the inertia flywheel. ref It is the rated speed of the rotor at the output end of the inertia flywheel.

6. The inertial flywheel control method based on reinforcement learning according to claim 1, characterized in that, The torque setpoint is the torque setpoint input to the output terminal of the inertia flywheel. The torque setpoint is obtained using a torque setpoint formula, which is: T is the torque setpoint, P ref ω is the active power output of the inertial flywheel. m K is the angular velocity of the rotor at the output end of the inertia flywheel. flag This is the charge / discharge flag for the inertia flywheel. K is used when the inertia flywheel is charging. flag The value of K is -1, and the inertia flywheel discharges. flag The value is +1, and ΔP is the power compensation term; R a This is the phase resistance at the output terminal of the inertia flywheel.

7. The inertial flywheel control method based on reinforcement learning according to claim 3, characterized in that, The control quantities include the drive signal input to the inertia flywheel and the excitation voltage input to the inertia flywheel.

8. The inertial flywheel control method based on reinforcement learning according to claim 7, characterized in that, The control quantities to be input to the inertial flywheel are obtained based on the motion and torque setpoints, including: The damping torque component and the torque set value are added together to obtain the torque base value; Obtain the measured torque value corresponding to the inertia flywheel for which control quantity calculation is required, and subtract the measured torque value from the torque base value to obtain the error value; The obtained error value is input into the PID controller, and after proportional, integral and derivative operations, it is output to the gain amplifier for processing to obtain the first signal; The first signal is subtracted from the q-axis current at the output of the inertia flywheel, which is to be calculated as the control quantity, and then input into the PID controller to obtain the second signal after proportional, integral and derivative operations. The second signal and ω e i d L d Signal and ω e The Flux signals are summed to obtain a third signal. This third signal is then input to an integrator for integration. The integrated signal is then input to a pulse width modulator to obtain the β-axis component of the drive signal input to the inertial flywheel, where ω e i represents the actual rotational speed of the rotor at the output end of the inertia flywheel. d For the d-axis current at the output of the inertia flywheel that requires control calculation, L d d-axis inductance of the inertia flywheel output terminal that requires control quantity calculation, and Flux is the magnetic flux of the inertia flywheel output terminal that requires control quantity calculation.

9. The inertial flywheel control method based on reinforcement learning according to claim 5, characterized in that, Obtaining the control quantity to be input to the inertia flywheel based on the action and torque setpoints also includes: inputting the damping voltage component, the measured voltage value of the inertia flywheel output terminal that needs to be calculated for control quantity calculation, and the voltage setpoint value of the inertia flywheel output terminal that needs to be calculated for control quantity calculation to the excitation controller to obtain the excitation voltage.

10. An inertial flywheel control system based on reinforcement learning, characterized in that, The system controls an inertial flywheel using the reinforcement learning-based inertial flywheel control method as described in any one of claims 1-9, wherein the system comprises: The neural network update module is used to store the transformation experience tuples obtained through the interactive simulation environment and the neural network into the experience pool, and to randomly sample a set number of experience tuples from the experience pool at the current moment to update the neural network. The data processing module is used to determine whether the neural network update termination condition is met based on the rewards in all experience tuples in the experience pool at the current moment. In response to meeting the neural network update termination condition, the updated neural network is used as the trained neural network. In response to not meeting the neural network update termination condition, the neural network update module is used to update the neural network again until the neural network update termination condition is met. The neural network updated during the last execution of the neural network update is used as the trained neural network. The control quantity acquisition module inputs the state quantity of the inertial flywheel output terminal that needs to be calculated into the trained neural network to acquire the action, and obtains the control quantity to be input to the inertial flywheel based on the action and torque setpoint.

Images