Inertia flywheel control method and system based on reinforcement learning

Abstract

The invention discloses an inertia flywheel control method and system based on reinforcement learning, and the method comprises the steps: storing a conversion experience multi-tuple obtained through an interactive simulation environment and a neural network into an experience pool, and randomly sampling a set number of experience multi-tuples from the experience pool at the current moment to update the neural network; according to the reward in the current experience pool, judging whether a neural network updating ending condition is satisfied, if so, taking the updated neural network as a trained neural network, otherwise, continuing to update the neural network until the ending condition is satisfied; and inputting the state quantity of the output end of the inertia flywheel, which needs to be subjected to control quantity calculation, into the trained neural network to obtain an action, and obtaining the control quantity to be input into the inertia flywheel based on the action and the torque set value. According to the scheme of the invention, the neural network can adjust the strategy according to the actual performance, and the motion corresponding to the state quantity of the output end of the inertia flywheel can be predicted and calculated more accurately.

Description

Technical Field

[0001] The present application generally relates to the field of energy storage technology. More specifically, the present application relates to an inertia flywheel control method and system based on reinforcement learning. Background Art

[0002] In the existing technology, inertia flywheel technology with large inertia, strong reactive power support and high overload capacity has become one of the effective ways to solve the frequency and voltage regulation problems of new energy power systems. Inertia flywheel can provide efficient inertia and voltage support, effectively alleviating system frequency and voltage fluctuations. Its high overload capacity enables it to quickly respond to system frequency and voltage fluctuations, and provide a large amount of mechanical inertia and reactive power in a short time, which helps to maintain the stability of system frequency and voltage.

[0003] However, when the inertial flywheel is used as a frequency and voltage regulating device in the power system, it faces the problem of its own low-frequency oscillation and the impact of low-frequency oscillation in the power grid, which can cause serious harm to the power system, such as reducing system stability and damaging equipment life.

[0004] In view of this, there is an urgent need to provide an inertia flywheel control solution based on reinforcement learning in order to solve the problem of low-frequency oscillation of the inertia flywheel itself and its impact on the low-frequency oscillation of the power grid. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present application proposes an inertia flywheel control solution based on reinforcement learning in multiple aspects.

[0006] In a first aspect, the present application provides an inertia flywheel control method based on reinforcement learning, comprising: converting a conversion experience multi-tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience pool, and a set number of experience tuples are randomly sampled from the current experience pool to update the neural network, where s t is the state quantity of the inertia flywheel output end at the current time t, a t is the action corresponding to the state quantity at the current time t, r t is the reward at the current moment t, s t+1for the action at the next moment; judging whether the neural network update end condition is met according to the rewards in all the experience tuples in the experience pool at the current moment; in response to the neural network update end condition being met, taking the updated neural network as a trained neural network; and in response to the neural network update end condition not being met, returning to the step of updating the neural network until the neural network update end condition is met, taking the neural network updated by the last execution of the step of updating the neural network as a trained neural network; inputting the state quantity of the output end of the inertia flywheel for which the control quantity calculation is required to be performed into the trained neural network to obtain the action, and obtaining the control quantity to be input to the inertia flywheel based on the action and the torque set value.

[0007] In some embodiments, the state quantities include active power, differential of active power, reactive power, differential of reactive power, terminal voltage, differential of terminal voltage, rotation speed, and differential of rotation speed.

[0008] In some embodiments, the action includes a damping voltage component and a damping torque component.

[0009] In some embodiments, the neural network update termination condition includes: the sum of rewards in all experience tuples in the current experience pool is greater than a preset value of the sum of rewards.

[0010] In some embodiments, r t Obtained through a reward function, wherein the reward function is: t step is the set time step, t is the current moment, ω e is the actual speed of the rotor at the output end of the inertia flywheel, ω ref It is the rated speed of the rotor at the output end of the inertia flywheel.

[0011] In some embodiments, the torque setting value is a torque setting value input to the output end of the inertia flywheel, and the torque setting value is obtained by a torque setting value formula, wherein the torque setting value formula is: T is the torque setting value, P ref is the active power output by the inertia flywheel, ω m is the angular velocity of the rotor at the output end of the inertia flywheel, K flag It is the charge and discharge flag of the inertia flywheel. When the inertia flywheel is charged, K flag The value is -1, when the inertia flywheel discharges, K flag The value of is +1, ΔP is the power compensation term; R a is the phase resistance at the output end of the inertia flywheel.

[0012] In some embodiments, the control variable includes a drive signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel.

[0013] In some embodiments, obtaining the control amount to be input to the inertia flywheel based on the action and the torque setting value includes: adding the damping torque component and the torque setting value to obtain a torque base value; obtaining the torque measured value corresponding to the inertia flywheel for which the control amount calculation is required, and subtracting the torque base value from the torque measured value to obtain an error value; inputting the obtained error value into the PID controller, and then outputting it to the gain amplifier for processing to obtain a first signal after undergoing proportional, integral and differential operations; subtracting the first signal from the q-axis current at the output end of the inertia flywheel for which the control amount calculation is required, and then inputting it into the PID controller for undergoing proportional, integral and differential operations to obtain a second signal; and comparing the second signal with ω e i d L d Signal and ω e Flux signals are added to obtain a third signal, the third signal is input to an integrator for integration processing, and the signal after integration processing is input to a pulse width modulator to obtain a β-axis component of a driving signal input to the inertia flywheel, where ω e is the actual speed of the rotor at the output end of the inertia flywheel, i d is the d-axis current at the output end of the inertia flywheel for which the control quantity needs to be calculated, L d is the d-axis inductance of the inertia flywheel output end for which the control quantity calculation is required, and Flux is the magnetic flux of the inertia flywheel output end for which the control quantity calculation is required.

[0014] In some embodiments, obtaining the control quantity to be input to the inertia flywheel based on the action and torque set values ​​also includes: inputting the damping voltage component, the measured value of the voltage at the output end of the inertia flywheel for which the control quantity calculation is required, and the voltage given value at the output end of the inertia flywheel for which the control quantity calculation is required into the excitation controller to obtain the excitation voltage.

[0015] In a second aspect, the present application provides an inertia flywheel control system based on reinforcement learning, which controls the inertia flywheel using the inertia flywheel control method based on reinforcement learning as described in any embodiment of the first aspect, and the system includes: a neural network update module, which is used to store the conversion experience tuples obtained through the interactive simulation environment and the neural network into an experience pool, and randomly sample a set number of experience tuples from the experience pool at the current moment to update the neural network; a data processing module, which is used to determine whether the neural network update end condition is met according to the rewards in all experience tuples in the experience pool at the current moment, and in response to the neural network update end condition being met, the updated neural network is used as a trained neural network; and in response to the neural network update end condition not being met, the neural network is updated again using the neural network update module until the neural network update end condition is met, and the neural network updated when the neural network is updated for the last time is used as the trained neural network; a control quantity acquisition module, which inputs the state quantity of the inertia flywheel output end that needs to be calculated for the control quantity into the trained neural network acquisition action, and acquires the control quantity to be input to the inertia flywheel based on the action and the torque set value.

[0016] Through the inertia flywheel control method based on reinforcement learning provided above, the embodiment of the present application determines whether the neural network needs to be updated again according to the rewards in all experience multi-groups in the experience pool at the current moment, and uses the neural network that has finally been updated as the trained neural network to obtain the action corresponding to the state quantity of the inertia flywheel output end that needs to be calculated for the control quantity, and obtains the control quantity to be input to the inertia flywheel based on the action and the torque set value, so that the neural network can adjust its strategy according to the actual performance, more accurately predict and calculate the action corresponding to the state quantity of the inertia flywheel output end, and can more accurately control the state of the inertia flywheel, thereby improving the control accuracy of the inertia flywheel.

[0017] Furthermore, in some embodiments, by setting the action to include a damping voltage component and a damping torque component, when the control quantity to be input to the inertia flywheel is obtained based on the action and the torque setting value, the problem of large rotor oscillation at the output end of the inertia flywheel during output can be solved, thereby improving its ability to resist power grid impact and self-stability. Furthermore, in some embodiments, by taking the sum of rewards in all experience multi-tuples in the current experience pool greater than the preset value of the sum of rewards as the end condition of the neural network update, it can be ensured that the neural network reaches a certain performance level after the training is completed, while avoiding overfitting of the neural network on the training data and invalid training of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easy to understand. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0019] Figure 1 An exemplary flow chart of an inertia flywheel control method based on reinforcement learning according to an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram showing the composition of an inertia flywheel according to an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram showing the interaction between the intelligent agent model and the simulation environment according to an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram showing an embodiment of the present application of obtaining an inertia flywheel control amount based on an action and a torque setting value;

[0023] Figure 5 A schematic diagram showing some embodiments of the present application of obtaining the inertia flywheel control amount based on the action and torque setting value;

[0024] Figure 6 A schematic diagram of obtaining a driving signal according to an embodiment of the present application is shown;

[0025] Figure 7 An exemplary structural block diagram of an inertia flywheel control system according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0027] It should be understood that the terms "include" and "comprising" used in the specification and claims of the present application indicate the presence of described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in this application specification and claims, unless the context clearly indicates otherwise, the singular forms of "a", "an" and "the" are intended to include plural forms. It should also be further understood that the term "and / or" used in this application specification and claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0029] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0030] The specific implementation of the present application is described in detail below with reference to the accompanying drawings.

[0031] In the prior art, the traditional solution to the problem of low-frequency oscillation of the inertial flywheel is to install a power system stabilizer (PSS), which mainly adjusts the excitation system of the generator in the inertial flywheel through the power system stabilizer (PSS) to improve the excitation system, thereby suppressing the low-frequency oscillation. However, PSS changes the reactive power output of the generator by adjusting the excitation current. Since the change of the excitation current takes a certain amount of time to affect the electromagnetic characteristics of the generator, this adjustment method is relatively slow; at the same time, due to the dynamic characteristics of the excitation system, the control accuracy of PSS may be limited, especially under rapidly changing system conditions, the response of the excitation system may not be accurate enough, resulting in poor control effect of PSS; in addition, PSS needs to be integrated with the existing excitation system, which increases the complexity of the system.

[0032] In view of this, an embodiment of the present application provides an inertia flywheel control solution based on reinforcement learning, which can solve the problem of low-frequency oscillation of the inertia flywheel itself and its response to the impact of low-frequency oscillation of the power grid.

[0033] Figure 1 An exemplary flow chart of an inertia flywheel control method 100 based on reinforcement learning according to an embodiment of the present application is shown.

[0034] like Figure 1 As shown, in step S110, the transformation experience multi-tuple (s t ,a t ,r t,s t+1 ) is stored in the experience pool, and a set number of experience tuples are randomly sampled from the current experience pool to update the neural network, where s t is the state quantity of the inertia flywheel output end at the current time t, a t is the action corresponding to the state quantity at the current time t, r t is the reward at the current moment t, s t+1 For the next moment's action.

[0035] In an embodiment of the present application, the inertia flywheel includes: a flywheel, an electromagnetic coupler, a synchronous phase condenser and a current converter controller.

[0036] Figure 2 A schematic diagram of the composition of the inertia flywheel 200 according to an embodiment of the present application is shown.

[0037] like Figure 2 As shown, the inertia flywheel 200 includes a flywheel 210, an electromagnetic coupler 220, a synchronous condenser 230 and a current converter 240. Specifically, the electromagnetic coupler 220 includes an inner rotor and an outer rotor. The rotor of the flywheel 210 is connected to the inner rotor through a first transmission shaft 221, and the rotor of the synchronous condenser 230 is connected to the outer rotor through a first transmission shaft 222. The inner rotor uses a permanent magnet, and the outer rotor is provided with an armature winding. The electromagnetic coupler 220 is equivalent to a permanent magnet synchronous motor with a stator and a rotor rotating simultaneously, which enables the electromagnetic coupler 220 to realize the transmission of torque and inertia between the flywheel 210 and the synchronous condenser 230. The stator end of the synchronous condenser 230 is directly connected to the power grid 250, and the current converter 240 can control the electromagnetic coupler 220, and the power grid 250 can supply power to the electromagnetic coupler 220 through the current converter 240.

[0038] In the embodiment of the present application, the synchronous condenser 230 is used as the output end of the inertia flywheel 200 .

[0039] Specifically, the current converter controller 240 uses a back-to-back converter. When the external frequency rises or falls, the current converter controller 240 controls the output torque of the electromagnetic coupler 220. Since the electromagnetic coupler 220 is directly connected to the rotor of the synchronous condenser 230 through a mechanical shaft, the output torque of the electromagnetic coupler 220 is the input torque of the synchronous condenser 230. Since the electromagnetic coupler 220 is equivalent to a permanent magnet synchronous motor with a stator and a rotor rotating simultaneously, its output torque can be controlled by a vector controller.

[0040] In an embodiment of the present application, the aforementioned simulation environment is constructed based on the parameters of the real-time operation data of the inertia flywheel, which is mainly constructed based on the system state space (i.e., output) and action space (i.e., input) of the real-time operation process of the given inertia flywheel during the training process of deep reinforcement learning.

[0041] In an embodiment of the present application, a neural network including an Actor network and a Critic network is built based on a graph convolutional neural network and a deep reinforcement learning algorithm, and an agent model is constructed based on the neural network and the experience pool.

[0042] In an embodiment of the present application, after constructing the simulation environment and the intelligent agent model, the constructed simulation environment and the intelligent agent model are initialized, then, the initialized intelligent agent model is used to interact with the initialized simulation environment to obtain the initial state s0 of the output end of the inertia flywheel, and then, the interaction between the intelligent agent model and the simulation environment is repeated to obtain multiple transformation experience tuples.

[0043] Figure 3 A schematic diagram showing the interaction between the intelligent agent model and the simulation environment according to an embodiment of the present application.

[0044] like Figure 3 As shown, the Actor network includes a first optimizer, an online policy network and a target policy network, and the Critic network includes a second optimizer, an online Q network and a target Q network, wherein the first optimizer and the second optimizer may be the same or different.

[0045] In the process of interaction between the intelligent model and the simulation environment, the Actor network collects the state quantity s of the inertia flywheel output end at the current time t from the simulation environment. t , based on the state quantity s at the output end of the inertia flywheel at the current time t t And introduce random noise to get the output state quantity s t The corresponding action a t And act on the simulation environment to get the current time t action a t The corresponding reward r t and the state quantity s at the output end of the inertia flywheel in the simulation environment at the next moment t+1 The state quantity s at the current time t of the inertia flywheel output end is t , output state quantity s t The corresponding action a t , current time t action a t The corresponding reward r t and the state quantity s at the output end of the inertia flywheel in the simulation environment at the next moment t+1 The transformation experience tuple (s t ,a t ,r t ,s t+1 ) is stored in the experience pool. The output state quantity s is obtained by introducing random noise t The corresponding action a t It can increase the intelligent model's ability to explore the environment.

[0046] Next, the Critic network randomly samples a set number of experience tuples (s i ,a i ,r i ,s i+1 ), where s i is the state quantity of the inertia flywheel output end at time i, a i is the action corresponding to the state quantity at time i, r i is the reward at time i, s i+1 is the action at time i+1. Specifically, in the random sampling process, the specific value of the sampled experience multi-group can be set according to the total number of experience multi-groups in the experience pool at the current moment, and this application does not limit this.

[0047] Then, by sampling a set number of empirical multivariate sets (s i ,a i ,r i ,s i+1 ) Calculate the objective function y corresponding to the target Q network in the Critic network i , the target Q network converts the target function y i Sent to the online Q network, the online Q network is based on the objective function y i The loss function L of the online Q network is obtained, the Q gradient is obtained by minimizing the loss function L, and the Q gradient is sent to the second optimizer, which updates the parameters of the online Q network based on the received Q gradient. The online policy network in the Actor network obtains the policy gradient based on the gradient of the online Q network to the online policy network and the gradient of the online Q network to the parameters of the online Q network, and sends the policy gradient to the first optimizer, which updates the parameters of the online policy network based on the received policy gradient. In addition, the online policy network soft-updates the target policy network through the smoothing factor, and the online Q network soft-updates the target Q network through the smoothing factor.

[0048] In the embodiment of the present application, the state quantity of the inertia flywheel output terminal includes active power, differential of active power, reactive power, differential of reactive power, terminal voltage, differential of terminal voltage, speed and differential of speed. Since the inertia flywheel output terminal adopts a synchronous condenser, the state quantity collected in the simulation environment is the active power, differential of active power, reactive power, differential of reactive power, terminal voltage, differential of terminal voltage, speed and differential of speed at the stator terminal of the synchronous condenser.

[0049] In the embodiment of the present application, the action includes a damping voltage component and a damping torque component. Specifically, the damping voltage component acts on an excitation controller that controls the excitation current of the synchronous condenser in the inertia flywheel, and the damping torque component serves as an input signal of a vector controller of an electromagnetic coupler in the inertia flywheel. The excitation controller changes the magnetic field strength generated by the synchronous condenser by adjusting the DC current supplied to the synchronous condenser rotor, thereby affecting the active and reactive power output by the synchronous condenser.

[0050] Specifically, the damping voltage component changes the magnetic field strength of the synchronous condenser by adjusting the excitation current, thereby suppressing the oscillation of its rotor. When the power grid is impacted or disturbed, the damping voltage component can quickly adjust the excitation current to help the synchronous condenser quickly recover to a stable state and reduce the impact of the impact on the system. The damping torque component directly acts on the rotor of the synchronous condenser by adjusting the output torque of the electromagnetic coupler to reduce the oscillation of the synchronous condenser rotor. When the power grid is impacted or disturbed, the damping torque component can quickly adjust the output of the electromagnetic coupler to help the inertia flywheel quickly recover to a stable state and reduce the impact of the impact on the system.

[0051] Thus, by making the action include a damping voltage component and a damping torque component, when the control quantity to be input to the inertia flywheel is obtained based on the action and the torque setting value, the problem of large rotor oscillation at the output end of the inertia flywheel during output can be solved, thereby improving its ability to resist grid impact and self-stability.

[0052] In the embodiment of the present application, the reward r at the current time t is t Obtained through a reward function, wherein the reward function is:

[0053] t step is the set time step, t is the current moment, ω e is the actual speed of the rotor at the output end of the inertia flywheel, ω ref It is the rated speed of the rotor at the output end of the inertia flywheel.

[0054] It can be seen from the reward function that it uses the speed deviation as the evaluation standard for the action. Since the speed deviation directly reflects the oscillation of the synchronous condenser rotor, if the speed deviation is small, it means that the oscillation of the rotor is effectively suppressed and the system stability is good. The goal of this application is to minimize the oscillation of the synchronous condenser when responding to sudden changes in output power and power grid shocks, and the speed deviation directly reflects the degree of achievement of this goal. Therefore, by setting a reward function that uses the speed deviation as the evaluation standard for the action, the effectiveness of suppressing oscillations can be directly reflected.

[0055] After executing step S110, in step S120, it is determined whether the neural network update termination condition is met based on the rewards in all experience multi-tuples in the experience pool at the current moment.

[0056] In some embodiments of the present application, the neural network update termination condition may be that the sum of rewards in all experience multi-groups in the current experience pool is greater than a preset value of the sum of rewards. In other embodiments of the present application, the neural network update termination condition may also be that the mean of rewards in all experience multi-groups in the current experience pool is greater than a preset value of the mean of rewards, which is not limited in the present application.

[0057] By using the sum of rewards in all experience tuples in the current experience pool greater than the preset value of the sum of rewards as the end condition for the neural network update, the training can be stopped only when the accumulated sum of rewards exceeds the preset value, which means that the performance of the neural network on the training data has reached the expected standard, and the training can be stopped after the neural network reaches the expected standard level, avoiding continued training that causes the model to overfit the training data, and preventing the neural network from over-learning specific noise and details on the training data. At the same time, by using the sum of rewards in all experience tuples in the current experience pool greater than the preset value of the sum of rewards as the end condition for the neural network update, it is possible to avoid wasting computing resources and time when the performance improvement of the neural network is very limited or even no longer improves.

[0058] In response to the neural network update end condition being met, in step S130, the currently updated neural network is used as a trained neural network.

[0059] In response to the neural network update termination condition not being met, the process returns to step S110 and the step of updating the neural network is re-executed until the neural network update termination condition is met. When step S110 is executed multiple times, the neural network updated by the last execution of step S110 is regarded as the neural network that has completed training.

[0060] After obtaining the trained neural network, in step S140, the state quantity of the output end of the inertia flywheel that needs to be calculated for the control quantity is input into the trained neural network to obtain the action, and the control quantity to be input to the inertia flywheel is obtained based on the action and the torque setting value.

[0061] Specifically, since the neural network can process the input state quantity and output the action in a very short time, by inputting the state quantity of the output end of the inertia flywheel that needs to be calculated for the control quantity into the trained neural network to obtain the action, the inertia flywheel can be efficiently controlled. At the same time, since the trained neural network can find the optimal control strategy, by inputting the state quantity of the output end of the inertia flywheel that needs to be calculated for the control quantity into the trained neural network to obtain the action, the action corresponding to the state quantity of the output end of the inertia flywheel can be more accurately predicted and calculated, and the state of the inertia flywheel can be more accurately controlled, thereby improving the control accuracy of the inertia flywheel.

[0062] In an embodiment of the present application, the torque setting value is a torque setting value input to the output end of the inertia flywheel, and the torque setting value is obtained by a torque setting value formula, wherein the torque setting value formula is:

[0063] T is the torque setting value, P ref is the active power output by the inertia flywheel, ω m is the angular velocity of the rotor at the output end of the inertia flywheel, K flag It is the charge and discharge flag of the inertia flywheel. When the inertia flywheel is charged, K flag The value is -1, when the inertia flywheel discharges, K flag The value of is +1, ΔP is the power compensation term; R a is the phase resistance at the output end of the inertia flywheel.

[0064] The power compensation term is used to compensate for the difference between the output power of the synchronous condenser and the input mechanical power. By introducing the power compensation term in the torque setting value, the situation in practice that the output power of the synchronous condenser is lower than the input mechanical power due to the loss of the synchronous condenser winding is taken into account. Therefore, the obtained torque setting value better meets the actual needs. This makes the control amount to be input to the inertia flywheel based on the action and torque setting value The obtained control amount is more accurate.

[0065] In an embodiment of the present application, the control amount includes a drive signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel. After the control amount is obtained, the control amount is input to the inertia flywheel to control it.

[0066] Figure 4 A schematic diagram of obtaining the inertia flywheel control quantity based on the action and torque setting value in an embodiment of the present application is shown.

[0067] like Figure 4As shown, the intelligent agent model collects the state quantity of the inertia flywheel output terminal from the simulation environment, and based on the collected state quantity and reward function, uses the reinforcement learning algorithm to update the adopted strategy. When the optimal strategy is obtained after multiple updates, the output includes the damping torque component and the damping voltage component. Then, the damping torque component is added to the aforementioned torque setting value to obtain the torque base value, and the torque base value is used as the input of the vector controller to obtain the drive signal. At the same time, the damping voltage component is input to the excitation controller for processing to obtain the excitation voltage. Then, the control quantity including the drive signal and the excitation voltage is input to the inertia flywheel to control it.

[0068] In some embodiments of the present application, the process of obtaining the inertia flywheel control value based on the action and torque setting value can be referred to in Figure 5 .

[0069] like Figure 5 As shown, the intelligent agent model collects the state quantity of the synchronous condenser stator end from the simulation environment, and uses the trained intelligent agent model to output the action including the damping torque component and the damping voltage component. Then, the damping torque component and the aforementioned torque setting value are added to obtain the torque base value, and the torque base value is used as the input of the vector controller to obtain the driving signal. The vector controller controls the output torque of the electromagnetic coupler based on the driving signal, so that the output torque directly acts on the rotor of the synchronous condenser to reduce the oscillation of the synchronous condenser rotor. At the same time, the damping voltage component, the measured voltage value of the synchronous condenser stator end that needs to be calculated for the control quantity, and the voltage set value of the inertia flywheel output end that needs to be calculated for the control quantity are input into the excitation controller to obtain the excitation voltage, and the excitation voltage is used to control the excitation current flowing into the synchronous condenser rotor winding, and then the magnetic field strength of the synchronous condenser is changed by adjusting the excitation current to suppress the oscillation of the synchronous condenser rotor.

[0070] Specifically, the excitation controller may calculate the excitation voltage using a control algorithm set inside the controller, which may be an adaptive control algorithm, a model predictive control algorithm, etc., which is not limited in this application.

[0071] Specifically, the torque base value is used as the input of the vector controller to obtain the drive signal. For details, please refer to Figure 6 .

[0072] like Figure 6 As shown, the torque measured value T corresponding to the inertia flywheel that needs to be calculated for the control quantity is obtained. e , the torque base value T eref and the torque measured value T eSubtract the error value to obtain the error value; input the obtained error value into the PID controller, and then output it to the gain amplifier for processing to obtain the first signal after the proportional, integral and differential operations; compare the first signal with the q-axis current i at the output end of the inertia flywheel for which the control quantity calculation is required q After subtraction, it is input into the PID controller and subjected to proportional, integral and differential operations to obtain the second signal; the second signal is compared with ω e i d L d Signal and ω e Flux signals are added to obtain a third signal, the third signal is input to an integrator for integration processing, and the signal after integration processing is input to a pulse width modulator to obtain a drive signal β-axis component u input to the inertia flywheel β , where ω e is the actual speed of the rotor at the output end of the inertia flywheel, i d is the d-axis current at the output end of the inertia flywheel for which the control quantity needs to be calculated, L d is the d-axis inductance of the inertia flywheel output end for which the control quantity calculation is required, and Flux is the magnetic flux of the inertia flywheel output end for which the control quantity calculation is required.

[0073] While obtaining the β-axis component of the drive signal, the d-axis current at the output end of the inertia flywheel is given by the current i dref The d-axis current i at the output end of the inertia flywheel that needs to be calculated d The error value is obtained by subtracting the error value, and the error value is input into the PID controller, and after the proportional, integral and differential operations, the fourth signal is output, and the fourth signal is compared with ω e i q L q After the signal is subtracted, it is input to the integrator for integration processing, and the signal after integration processing is input to the pulse width modulator to obtain the α-axis component u of the driving signal input to the inertia flywheel α , where ω e is the actual speed of the rotor at the output end of the inertia flywheel, i q is the q-axis current at the output end of the inertia flywheel for which the control quantity needs to be calculated, L q It is the q-axis inductance at the output end of the inertia flywheel for which the control quantity calculation is required.

[0074] In the embodiment of the present application, the aforementioned pulse width modulator may adopt a SVPWM pulse width modulator.

[0075] In summary, through the reinforcement learning-based inertia flywheel control scheme provided above, the embodiment of the present application determines whether the neural network needs to be updated again according to the rewards in all experience multi-tuples in the experience pool at the current moment, and uses the finally updated neural network as the trained neural network to obtain the action corresponding to the state quantity of the inertia flywheel output end that needs to be calculated for the control quantity, and obtains the control quantity to be input to the inertia flywheel based on the action and the torque set value, so that the neural network can adjust its strategy according to the actual performance, more accurately predict and calculate the action corresponding to the state quantity of the inertia flywheel output end, and can more accurately control the state of the inertia flywheel, thereby improving the control accuracy of the inertia flywheel.

[0076] Furthermore, in some embodiments, by setting the action to include a damping voltage component and a damping torque component, when the control quantity to be input to the inertia flywheel is obtained based on the action and the torque setting value, the problem of large rotor oscillation at the output end of the inertia flywheel during output can be solved, thereby improving its ability to resist power grid impact and self-stability. Furthermore, in some embodiments, by taking the sum of rewards in all experience multi-tuples in the current experience pool greater than the preset value of the sum of rewards as the end condition of the neural network update, it can be ensured that the neural network reaches a certain performance level after the training is completed, while avoiding overfitting of the neural network on the training data and invalid training of the neural network.

[0077] The embodiment of the present application further provides an inertia flywheel control system based on reinforcement learning, which can use the aforementioned inertia flywheel control method 100 based on reinforcement learning to control the inertia flywheel.

[0078] Figure 7 An exemplary structural block diagram of an inertia flywheel control system 700 according to an embodiment of the present application is shown.

[0079] like Figure 7 As shown, the system 700 includes a neural network update module 710, a data processing module 720, and a control amount acquisition module 730. The data processing module 720 is interconnected with the neural network update module 710, and the control amount acquisition module 730 is connected with the data processing module 720. In the embodiment of the present application, the neural network update module 710, the data processing module 720, and the control amount acquisition module 730 can be separate units or integrated in the same controller.

[0080] Specifically, the neural network update module 710 is used to store the converted experience tuples obtained through the interactive simulation environment and the neural network into the experience pool, and randomly sample a set number of experience tuples from the experience pool at the current moment to update the neural network.

[0081] Specifically, the data processing module 720 is used to determine whether the neural network update end condition is met based on the rewards in all experience multi-tuples in the experience pool at the current moment, and in response to the neural network update end condition being met, use the updated neural network as a trained neural network, and in response to the neural network update end condition not being met, use the neural network update module to update the neural network again until the neural network update end condition is met, and use the neural network updated when the neural network is updated for the last time as a trained neural network.

[0082] Specifically, the control quantity acquisition module 730 is used to input the state quantity of the inertia flywheel output end for which the control quantity calculation is required into the trained neural network to obtain the action, and obtain the control quantity to be input into the inertia flywheel based on the action and the torque setting value.

[0083] When the system 700 uses the aforementioned reinforcement learning-based inertia flywheel control method 100 to control the inertia flywheel, the neural network update module 710 executes the aforementioned step S110, the data processing module 720 executes the aforementioned steps S110 and S120, and the control amount acquisition module 730 executes the aforementioned step S140. The specific execution process can be referred to above, and will not be repeated here.

[0084] Although multiple embodiments of the present application have been shown and described herein, it is obvious to those skilled in the art that such embodiments are provided only by way of example. Those skilled in the art can think of many changes, modifications and alternatives without departing from the thought and spirit of the present application. It should be understood that in the process of practicing the present application, various alternatives to the embodiments of the present application described herein can be adopted. The attached claims are intended to limit the scope of protection of the present application, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for controlling an inertia flywheel based on reinforcement learning, characterized in that: include: The transformed experience multi-tuple (s) obtained through the interactive simulation environment and the neural network t ,a t ,r t ,s t+1 ) is stored in the experience pool, and a set number of experience tuples are randomly sampled from the current experience pool to update the neural network, where s t is the state quantity of the inertia flywheel output end at the current time t, a t is the action corresponding to the state quantity at the current time t, r t is the reward at the current time t, s t+1 For the next moment’s action; Determine whether the neural network update end condition is met based on the rewards in all experience multi-groups in the current experience pool; In response to the neural network update end condition being met, using the updated neural network as a trained neural network; and In response to the neural network update termination condition not being met, returning to the step of updating the neural network until the neural network update termination condition is met, and taking the neural network updated by the last execution of the step of updating the neural network as the neural network that has completed training; The state quantity of the output end of the inertia flywheel for which control quantity calculation is required is input into the trained neural network to obtain the action, and the control quantity to be input into the inertia flywheel is obtained based on the action and the torque setting value.

2. The inertia flywheel control method based on reinforcement learning according to claim 1, characterized in that: The state quantities include active power, differential of active power, reactive power, differential of reactive power, terminal voltage, differential of terminal voltage, rotation speed and differential of rotation speed.

3. The inertia flywheel control method based on reinforcement learning according to claim 1, characterized in that: The action includes a damping voltage component and a damping torque component.

4. The inertia flywheel control method based on reinforcement learning according to claim 1, characterized in that: The neural network update termination condition includes: the sum of rewards in all experience multi-tuples in the current experience pool is greater than a preset value of the sum of rewards.

5. The inertia flywheel control method based on reinforcement learning according to claim 1 or 4, characterized in that: r t Obtained through a reward function, wherein the reward function is: t step is the set time step, t is the current moment, ω e is the actual speed of the rotor at the output end of the inertia flywheel, ω ref It is the rated speed of the rotor at the output end of the inertia flywheel.

6. The inertia flywheel control method based on reinforcement learning according to claim 1, characterized in that: The torque setting value is a torque setting value input to the output end of the inertia flywheel, and the torque setting value is obtained by a torque setting value formula, wherein the torque setting value formula is: T is the torque setting value, P ref is the active power output by the inertia flywheel, ω m is the angular velocity of the rotor at the output end of the inertia flywheel, K flag It is the charge and discharge flag of the inertia flywheel. When the inertia flywheel is charged, K flag The value is -1, when the inertia flywheel discharges, K flag The value of is +1, ΔP is the power compensation term; R a is the phase resistance at the output end of the inertia flywheel.

7. The inertia flywheel control method based on reinforcement learning according to claim 3, characterized in that: The control amount includes a drive signal input to the inertia flywheel and an excitation voltage input to the inertia flywheel.

8. The inertia flywheel control method based on reinforcement learning according to claim 7, characterized in that: The control quantity to be input to the inertia flywheel based on the action and torque setting value includes: Adding the damping torque component and the torque setting value to obtain a torque base value; Obtain the torque measured value corresponding to the inertia flywheel for which the control quantity calculation is required, and subtract the torque base value from the torque measured value to obtain the error value; The obtained error value is input into the PID controller, and then output to the gain amplifier for processing to obtain a first signal after being subjected to proportional, integral and differential operations; The first signal is subtracted from the q-axis current at the output end of the inertia flywheel for which the control amount calculation is required, and then input into the PID controller for proportional, integral and differential operations to obtain a second signal; The second signal is compared with ω e i d L d Signal and ω e Flux signals are added to obtain a third signal, the third signal is input to an integrator for integration processing, and the signal after integration processing is input to a pulse width modulator to obtain a β-axis component of a driving signal input to the inertia flywheel, where ω e is the actual speed of the rotor at the output end of the inertia flywheel, i d is the d-axis current at the output end of the inertia flywheel for which the control quantity needs to be calculated, L d is the d-axis inductance of the inertia flywheel output end for which the control quantity calculation is required, and Flux is the magnetic flux of the inertia flywheel output end for which the control quantity calculation is required.

9. The inertia flywheel control method based on reinforcement learning according to claim 5, characterized in that: Acquiring the control quantity to be input to the inertia flywheel based on the action and torque set values ​​also includes: inputting the damping voltage component, the measured value of the voltage at the output end of the inertia flywheel for which the control quantity calculation is required, and the voltage given value at the output end of the inertia flywheel for which the control quantity calculation is required into the excitation controller to obtain the excitation voltage.

10. An inertial flywheel control system based on reinforcement learning, characterized in that: The inertia flywheel control method based on reinforcement learning as described in any one of claims 1 to 9 is used to control the inertia flywheel, and the system comprises: A neural network update module, which is used to store the converted experience tuples obtained through the interactive simulation environment and the neural network into the experience pool, and randomly sample a set number of experience tuples from the experience pool at the current moment to update the neural network; A data processing module, which is used to determine whether a neural network update end condition is satisfied according to rewards in all experience multi-groups in the experience pool at the current moment, and in response to satisfying the neural network update end condition, use the updated neural network as a trained neural network, and in response to not satisfying the neural network update end condition, use the neural network update module to update the neural network again until the neural network update end condition is satisfied, and use the neural network updated when the neural network is updated for the last time as the trained neural network; The control quantity acquisition module inputs the state quantity of the output end of the inertia flywheel for which the control quantity calculation is required into the trained neural network to obtain the action, and obtains the control quantity to be input into the inertia flywheel based on the action and the torque setting value.

Images