Active power control method, system and device based on reinforcement learning and storage medium

Abstract

The embodiment of the invention provides an active power control method, system and device based on reinforcement learning and a storage medium, and is applied to an inertia flywheel, and the method comprises the steps: collecting the power and rotating speed of a synchronous phase modifier included in the inertia flywheel; inputting the power and the rotating speed into a reinforcement learning neural network to obtain a proportion, an integral and a differential; determining power deviation according to the power and preset target power; based on a PID controller, determining a torque base value according to the proportion, the integral, the differential and the power deviation; determining an optimal control signal based on vector control by taking the torque base value as input; and controlling a variable flow controller according to the optimal control signal so as to control an inertia flywheel. The problems that the system is low in response speed and large in output oscillation are solved, and the dynamic response performance and the control precision of the system are improved.

Description

Technical Field

[0001] This document relates to the field of motor control technology, and in particular to an active power control method, system, device and storage medium based on reinforcement learning. Background Art

[0002] Grid inertia is the ability to provide inertial support through the kinetic energy of its rotor when the grid frequency changes, helping to stabilize the grid frequency.

[0003] In the prior art, when renewable energy is affected by power outage, the energy stored in the inertia flywheel is mainly supplemented to the power grid through the synchronous motor through active control to maintain the stability of the power grid frequency.

[0004] However, due to the limitation of the transient process of the synchronous motor during active control, the existing technology will cause rotor oscillation when torque is suddenly input to the synchronous motor, causing large fluctuations in the system output power, affecting the response speed of the motor and the normal operation and power safety of electrical equipment. Summary of the invention

[0005] In view of the above scheme, the present application aims to propose an active power control method, system, device and storage medium based on reinforcement learning to solve at least one of the above technical problems.

[0006] In a first aspect, one or more embodiments of this specification provide an active power control method based on reinforcement learning, which is applied to an inertia flywheel, including:

[0007] Collect the power and speed of the synchronous condenser included in the inertia flywheel;

[0008] Inputting the power and speed into a reinforcement learning neural network to obtain proportional, integral and differential;

[0009] Determining a power deviation according to the power and a preset target power;

[0010] Determine a torque base value based on the PID controller according to the proportion, the integral, the differential and the power deviation;

[0011] Taking the torque base value as input, determining an optimal control signal based on vector control; and

[0012] According to the optimal control signal, the inverter controller is controlled to realize the control of the inertia flywheel.

[0013] Furthermore, the training process of the reinforcement learning neural network includes:

[0014] According to the data of the inertia flywheel, a simulation model is constructed; the simulation model includes parameters of the flywheel, the electromagnetic coupler, the synchronous condenser, the current converter controller and the PID controller;

[0015] Based on the DDPG algorithm, the simulation model is called to determine the reward function, power, speed, proportion, integration and differentiation;

[0016] Determining whether the reward function is greater than a preset threshold;

[0017] If so, updating the reinforcement learning neural network according to the reward function, the power, the speed, the ratio, the integral and the differential; and

[0018] If not, the simulation model is called to determine the reward function, power, speed, proportion, integration and differentiation.

[0019] Furthermore, the reward function is calculated as follows:

[0020]

[0021] Among them, R t represents the reward function;

[0022] t represents the simulation start time;

[0023] t step represents the simulation step length;

[0024] k represents the weight coefficient;

[0025] ω represents the actual speed of the synchronous condenser;

[0026] ω ref Indicates the rated speed of the synchronous condenser;

[0027] P represents the actual output power of the inertia flywheel system;

[0028] P ref Indicates the target output power of the inertia flywheel.

[0029] In a second aspect, an embodiment of the present application provides an active control system based on reinforcement learning, including a flywheel rotor, an electromagnetic coupler, a synchronous condenser, a power grid and a current converter controller;

[0030] One end of the electromagnetic coupler is connected to the flywheel rotor, and the other end is connected to the synchronous condenser;

[0031] The synchronous condenser is connected to the power grid;

[0032] Controlling the variable current controller by the method of claim 1; and

[0033] The current converter controller controls the electromagnetic coupler.

[0034] Further, the electromagnetic coupler is connected to the flywheel rotor at one end and the synchronous condenser at the other end, comprising:

[0035] The electromagnetic coupler comprises an inner rotor and an outer rotor;

[0036] The inner rotor is connected to the flywheel rotor via a mechanical shaft; and

[0037] The outer rotor is connected to the rotor of the synchronous condenser.

[0038] Furthermore, the synchronous condenser is connected to the power grid including:

[0039] The synchronous condenser comprises a stator and a rotor; and

[0040] The stator of the synchronous condenser is connected to the power grid.

[0041] In a third aspect, an embodiment of the present application provides an active power control device based on reinforcement learning, the device comprising a collection module, a first calculation module, a second calculation module, a third calculation module, a fourth calculation module and a fifth calculation module;

[0042] The acquisition module is used to acquire the power and speed of the synchronous condenser included in the inertia flywheel;

[0043] The first calculation module is used to input the power and speed into a reinforcement learning neural network to obtain a proportion, an integral and a differential;

[0044] The second calculation module is used to determine the power deviation according to the power and the preset target power;

[0045] The third calculation module is used to determine the torque base value based on the PID controller according to the proportion, the integral, the differential and the power deviation;

[0046] The fourth calculation module is used to determine the optimal control signal based on vector control by taking the torque base value as input; and

[0047] The fifth calculation module is used to control the variable current controller according to the optimal control signal, so as to realize the control of the inertia flywheel.

[0048] Furthermore, the device also includes:

[0049] The first determination module is used to construct a simulation model according to the flywheel, the electromagnetic coupler, the synchronous phase condenser, the current converter controller and the PID controller;

[0050] The second determination module is used to call the simulation model based on the DDPG algorithm to determine the reward function, power, speed, proportion, integration and differentiation;

[0051] The first determination module is used to determine whether the reward function is greater than a preset threshold;

[0052] The second discrimination module is configured to, if yes, update the reinforcement learning neural network according to the reward function, the power, the rotation speed, the ratio, the integral and the differential; and

[0053] The third judgment module is used to call the simulation model to determine the reward function, power, speed, proportion, integration and differentiation.

[0054] Furthermore, the device also includes:

[0055] The reward function is specifically:

[0056]

[0057] Among them, R t represents the reward function;

[0058] t represents the simulation start time;

[0059] t step represents the simulation step length;

[0060] k represents the weight coefficient;

[0061] ω represents the actual speed of the synchronous condenser;

[0062] ω ref Indicates the rated speed of the synchronous condenser;

[0063] P represents the actual output power of the inertia flywheel system;

[0064] P ref Indicates the target output power of the inertia flywheel.

[0065] In a fourth aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the steps of the active power control method based on reinforcement learning described in any one of the first aspects are implemented.

[0066] Compared with the prior art, this application can at least achieve the following technical effects:

[0067] This application can use the reinforcement learning neural network to dynamically adjust the P, I, and D parameters of the PID controller, thereby achieving refined control of the synchronous condenser output power and speed, so as to effectively cope with the grid frequency fluctuations caused by the intermittent nature of renewable energy in the power system; and then by determining the speed and power angle of the synchronous condenser in the inertia flywheel, more precise control can be achieved. This method can ensure that the synchronous condenser always works in the best state, further improving the operating efficiency and reliability of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0069] Figure 1 A flow chart of an active power control method based on reinforcement learning provided for one or more embodiments of this specification;

[0070] Figure 2 Active power waveform diagram of experimental tests of the control strategy of the present application and the traditional control strategy provided for one or more embodiments of this specification;

[0071] Figure 3 A schematic diagram of the structure of an inertia flywheel system provided for one or more embodiments of this specification;

[0072] Figure 4 A block diagram of an inertia flywheel active power control strategy of a variable parameter PID based on reinforcement learning provided in one or more embodiments of this specification;

[0073] Figure 5 A schematic diagram of the structure of an active power control device based on reinforcement learning provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0075] The power generation of renewable energy is affected by weather and time, resulting in unstable output. Therefore, inertia flywheel energy storage is used to absorb excess electricity and store it when there is an excess of electricity. When the demand for electricity increases or decreases, the stored energy is released to supplement the power grid to maintain the stability of the grid frequency. However, the grid inertia response phase occurs within a few seconds of the frequency drop. In the existing technology, it is difficult for inertia flywheels to respond quickly and provide inertia support to the grid in a timely manner, resulting in large fluctuations in output power, affecting the normal operation of electrical equipment and power safety.

[0076] In view of the above technical problems, this application proposes an active power control method based on reinforcement learning, and the specific steps are as follows:

[0077] Step S1, collecting the power and speed of the synchronous condenser included in the inertia flywheel.

[0078] In the embodiment of the present application, the synchronous condenser is a rotating device that can provide or absorb reactive power to support the voltage stability of the power system. When using renewable energy to generate electricity, since the output power of renewable energy depends on natural conditions, the voltage of the power grid will be unstable, so the synchronous condenser is used to transfer excess electric energy to the inertia flywheel for storage; if the renewable energy stops generating electricity, the electric energy stored in the inertia flywheel is accurately output to the power grid using the synchronous condenser. Therefore, by collecting the power and speed of the synchronous condenser, the output power of the synchronous condenser can be dynamically adjusted to ensure voltage stability.

[0079] Step S2, inputting the power and speed into a reinforcement learning neural network to obtain proportion, integration and differentiation.

[0080] In an embodiment of the present application, reinforcement learning is used to automatically adjust the proportional (P), integral (I) and differential (D) coefficients of the PID controller to optimize the control performance of the inertia flywheel.

[0081] A simulation model is constructed using visual simulation tools such as MATLAB / simulink to simulate the working status of the flywheel, electromagnetic coupler and synchronous condenser in the inertia flywheel when the power grid stops supplying power. In this simulation environment, the output of the PID controller is used as input and input into the inertia flywheel to control the torque in the synchronous condenser.

[0082] During training, a test power command is randomly given to the simulation model. During the test time, the sensor measures the state of the synchronous condenser in the inertia flywheel at each moment, including active power, differential of active power, speed and differential of speed. Then the state is input into the neural network, and the action value of the neural network is output. The action value includes the P, I and D parameters of the PID controller. Then the action value is input into the PID controller, and the reward value is calculated according to the power and power command, and the neural network is updated using the reward value. During training, the training is cyclic, and when the reward value is within the preset threshold range, the neural network training is completed and the reinforced neural network is obtained. When in use, the reinforcement learning neural network is used to calculate the P, I and D parameters.

[0083] Specifically, the neural network is updated using the DDPG algorithm, and the algorithm flow is as follows:

[0084] initialization:

[0085] Randomly initialize the parameters of the Actor network and the Critic network;

[0086] Initialize the target network;

[0087] Initialize the experience replay pool R.

[0088] Repeat the number of episodes:

[0089] Randomly initialize the process for motion exploration.

[0090] Get the initial state value s0.

[0091] Repeat t to T(step):

[0092] According to the current status t Calculate the action a at the current time step t .

[0093] Execute action a t , and record the reward r t and the new state s t+1 .

[0094] Storage conversion experience(s) t , a t , r t ,s t+1 ) in the experience pool R.

[0095] Randomly sample a small batch of transformed experience samples (s) from the experience pool R i , a i , r i ,s i+1 ).

[0096] Minimize the Loss function and update the Critic network.

[0097] Update the Actor network using a gradient algorithm.

[0098] Update the target network.

[0099] Among them, the state s t It includes four state quantities: active power, differential of active power, speed, differential of speed, action a t It contains three variables, the P, I, and D parameters in the PID controller. The reward function is used to evaluate the quality of the action. The purpose of the variable parameter PID controller is to reduce the oscillation of the motor while making the motor output a specified power. Therefore, the evaluation criteria are power deviation and speed deviation. The calculation method of the reward function is as follows:

[0100]

[0101] Among them, r t represents the reward function;

[0102] t represents the simulation start time;

[0103] t step represents the simulation step length;

[0104] k represents the weight coefficient;

[0105] ω represents the actual speed of the synchronous condenser;

[0106] ω ref Indicates the rated speed of the synchronous condenser;

[0107] P represents the actual output power of the inertia flywheel system;

[0108] P ref Indicates the target output power of the inertia flywheel.

[0109] Step S3, determining a power deviation according to the power and a preset target power.

[0110] In the embodiment of the present application, the preset target power value is the power level that the system expects to achieve, and this value is set based on the demand of the power grid. The power deviation can be calculated using the actual output power of the synchronous condenser and the preset target power.

[0111] Specifically, power deviation=preset target power-actual power.

[0112] For example, the target power of a synchronous condenser is 100 kW, while the current actual output power is 95 kW, ΔP = 100 kW - 95 kW = 5 kW, which means that the system needs to increase the power output by 5 kW to reach the target value.

[0113] The power deviation can be used to adjust the control strategy, and the control system can adjust the output of the synchronous condenser in real time to ensure the stability and reliability of the power system. For example, the excitation current of the synchronous condenser can be adjusted to make the actual power close to the target power.

[0114] Step S4, based on the PID controller, determine the torque base value according to the proportion, the integral, the differential and the power deviation.

[0115] Determine the base torque using the following formula:

[0116]

[0117] Where, T eref is the torque base value, K p is the P parameter of the PID controller, K i is the I parameter of the PID controller, K d is the D parameter of the PID controller, s is the Laplace operator, and ΔP is the power deviation.

[0118] Step S5, taking the torque base value as input and determining the optimal control signal based on vector control.

[0119] The vector control process is as follows:

[0120] The speed, winding resistance, d-axis current, d-axis inductance, q-axis inductance, and q-axis current of the flywheel rotor and the synchronous motor rotor are collected; the slip of the flywheel rotor and the synchronous motor rotor is determined according to the speed of the flywheel rotor and the synchronous motor rotor; based on the voltage equation of the electromagnetic coupler, the d-axis voltage and q-axis voltage are determined according to the slip, winding resistance, d-axis current, d-axis inductance, q-axis inductance, and q-axis current of the flywheel rotor and the synchronous motor rotor.

[0121] The voltage equation of the electromagnetic coupler is:

[0122]

[0123] Where Δω is the slip between the flywheel rotor and the synchronous motor rotor, u d is the d-axis voltage, R s is the winding resistance, i d is the d-axis current, L d is the d-axis inductance, L q is the q-axis inductance, i q is the q-axis current, u q is the q-axis voltage, ψ r is the magnetic linkage coefficient.

[0124] Based on the electromagnetic torque equation of the electromagnetic coupler, the electromagnetic torque is determined according to the d-axis inductance, q-axis inductance, d-axis current, and q-axis current.

[0125] The electromagnetic torque equation of the electromagnetic coupler is:

[0126] T e =1.5p n [ψ r +(L d -L q )i d ]i q ;

[0127] Among them, T e is the electromagnetic torque, p n is the pole pair number.

[0128] Adopt i d =0 vector control strategy, i d The reference value of is 0. According to the expression of electromagnetic torque of electromagnetic coupler, it can be known that the electromagnetic torque T is e Proportional to the q-axis current, the coupler output torque is approximately equal to the electromagnetic torque; the feedback signal of the electromagnetic coupler output torque is measured by the sensor; the feedback value is determined according to the feedback signal; the difference is calculated according to the feedback value and the torque base value. The difference is input into the pi loop, and the q-axis current is controlled by the pi loop to make the electromagnetic coupler output the target electromagnetic torque. The d-axis current loop, the q-axis current loop output, and the coupling term calculated by the dq-axis current and the slip Δω are substituted into the voltage equation to obtain the voltage reference value of the voltage signal. The output of the converter is controlled by space vector modulation SVPWM to control the current of the electromagnetic coupler based on the voltage reference value.

[0129] Step S6, controlling the inverter controller according to the optimal control signal to achieve control of the inertia flywheel.

[0130] In the embodiment of the present application, by comparing the overshoot, cumulative error and time required to reach steady state in the control strategy of the present application with that in the traditional control strategy, it can be seen that the active power outputted by the present application has smaller fluctuations, smaller overshoot and shorter time required to reach steady state than the traditional inertia flywheel system, as shown in Table 1, thereby effectively suppressing the power oscillation problem of the inertia flywheel system when outputting active power. Figure 2 It can be seen that the traditional control strategy will produce huge power fluctuations at the moment when the inertia flywheel outputs active power or controls the power size, thus affecting the safety of power equipment and the stable operation of the power system. Figure 2 It can be clearly seen that huge fluctuations occur between 0.5 seconds and 1 second in the figure; the control strategy of the present application can reduce power fluctuations at the moment when the inertia flywheel outputs active power or controls the power size, thereby ensuring the safety and stability of power equipment and power systems, such as Figure 2It can be clearly seen between 0.5 seconds and 1 second that the power fluctuation is relatively stable compared with the traditional control strategy.

[0131] Table 1 compares the active power indicators of the control strategy of this application and the traditional control strategy experimental test

[0132]

[0133] Furthermore, in an embodiment of the present application, the training process of the reinforcement learning neural network includes: constructing a simulation model based on the data of the inertia flywheel; the simulation model includes parameters of the flywheel, electromagnetic coupler, synchronous phase regulator, current converter and PID controller; based on the DDPG algorithm, calling the simulation model to determine the reward function, power, speed, proportion, integration and differentiation; determining whether the reward function is greater than a preset threshold; if so, updating the reinforcement learning neural network based on the reward function, the power, the speed, the proportion, the integration and the differentiation; and if not, calling the simulation model to determine the reward function, power, speed, proportion, integration and differentiation.

[0134] Specifically, the parameters of the flywheel, electromagnetic coupler, synchronous phase condenser, current converter and PID controller in the actual equipment are obtained, wherein the flywheel is used to store and release energy; the electromagnetic coupler is used to control the transmission of energy; the synchronous phase condenser is used to provide reactive power support and control the output power; the current converter is used to control the conversion of energy; and the PID controller is used to adjust the control signal.

[0135] These parameters are then used to construct a simulation model to simulate the operating state of the inertia flywheel when the grid is powered and when the grid is not powered. When the grid is powered, the synchronous condenser does not generate active power, the flywheel speed is constant, and the electromagnetic coupler does not output torque; when the grid is not powered, the synchronous condenser outputs active power, the flywheel speed decreases to release energy, and the electromagnetic coupler begins to output torque to transfer energy from the flywheel side to the synchronous condenser side.

[0136] Reinitialize the reinforcement learning model, including the Actor network: used to generate control actions; the Critic network: used to evaluate the quality of control actions. Get the initial state from the simulation model, including power and speed, etc.; based on the current state, generate a control action (such as the parameters of the PID controller) through the Actor network. Use the generated action to update the state of the simulation model and calculate the new power, speed, etc. Calculate the reward value based on the new state, and store the current state, action, reward and new state in the experience replay buffer. Randomly extract a batch of samples from the experience replay buffer to update the Actor and Critic networks. Check whether the current reward value is greater than the preset threshold. If so, continue to update the reinforcement learning model; otherwise, re-obtain the power and speed for calculation. Once the model training is completed through the above steps, it is applied to the actual system to dynamically adjust the parameters of the PID controller to optimize the performance of the synchronous condenser in the inertia flywheel.

[0137] Furthermore, the reward function is calculated as:

[0138]

[0139] Among them, R t represents the reward function;

[0140] t represents the simulation start time;

[0141] t step represents the simulation step length;

[0142] k represents the weight coefficient;

[0143] ω represents the actual speed of the synchronous condenser;

[0144] ω ref Indicates the rated speed of the synchronous condenser;

[0145] P represents the actual output power of the inertia flywheel system;

[0146] P ref Indicates the target output power of the inertia flywheel.

[0147] The embodiment of the present application provides an active control system based on reinforcement learning, such as Figure 3 The method comprises a flywheel rotor 10, an electromagnetic coupler 30, a synchronous condenser 40, a power grid 60 and a current converter 50; one end of the electromagnetic coupler 30 is connected to the flywheel rotor 10, and the other end is connected to the synchronous condenser 40; the synchronous condenser 40 is connected to the power grid 60; the current converter 50 is controlled by the method in the above embodiment; and the current converter 50 controls the electromagnetic coupler 30.

[0148] In the embodiment of the present application, the flywheel rotor 10 is connected to the synchronous condenser 40 through the electromagnetic coupler 30. The electromagnetic coupler 30 includes two parts, an inner rotor and an outer rotor. The inner rotor is directly connected to the flywheel rotor 10 through the transmission shaft 1 21, and the outer rotor is directly connected to the rotor of the synchronous condenser 40 through the transmission shaft 2 22. The inner rotor of the electromagnetic coupler 30 adopts a permanent magnet, and the outer rotor is placed with an armature winding, which is equivalent to a permanent magnet synchronous motor with a stator and a rotor rotating at the same time. The electromagnetic coupler 30 can realize the transmission of torque and inertia between the flywheel 10 and the synchronous condenser 40. The electromagnetic coupler 30 is controlled by the inverter controller 50. The rotor of the synchronous condenser 40 is maintained at the industrial frequency speed, and the stator end is directly connected to the power grid 60. The power grid 60 can supply power to the electromagnetic coupler 30 through the coupler inverter controller 50.

[0149] When the external frequency rises or falls, the inverter controller 50 controls the output torque of the electromagnetic coupler 30. Since the electromagnetic coupler 30 is directly connected to the rotor of the synchronous condenser 40 through a mechanical shaft, the output torque of the electromagnetic coupler 30 is the input torque of the synchronous condenser 40. Since the electromagnetic coupler 30 is equivalent to a permanent magnet synchronous motor with a stator and a rotor rotating simultaneously, its output torque can be controlled by vector control.

[0150] The inertia flywheel active power control strategy of variable parameter PID based on reinforcement learning, such as Figure 4 As shown in the figure: The grid side of the converter controller adopts the control strategy of DC bus voltage outer loop and current inner loop to maintain the DC bus voltage constant. The machine side of the converter controller adopts the control strategy of torque outer loop and current inner loop. The reference value of d-axis current in the current inner loop is 0, and the reference value of q-axis current is the output value of the torque outer loop.

[0151] The set value of the electromagnetic coupler output torque is obtained by the PID controller. The input of the PID controller is the deviation of the output power. The PID parameters are obtained by the neural network trained by the reinforcement learning algorithm. The reinforcement learning algorithm without model reference is adopted. The algorithm does not need to know the state equation of the control model. The PID controller outputs the torque base value, and then vector controls the torque base value to determine the optimal drive signal; it is input into the current converter controller, which can control the current and torque input to the electromagnetic coupler winding, and control the torque in the synchronous condenser by controlling the current. The torque is used to adjust the speed and power angle of the synchronous condenser, thereby improving the dynamic response performance and control accuracy of the system.

[0152] Specifically, the actual power P and speed ω are obtained from the synchronous condenser, and the preset target output power P is determined. ref, the derivative of power dP / dt and the derivative of speed dω / dt, and input the actual power P and speed ω into the reinforcement learning neural network to obtain the P, I, D parameter values; then according to the actual power P and the target output power P ref , determine the power deviation; then input the P, I, D parameter values ​​and the power deviation into the PID controller to determine the torque base value T eref , according to the torque base value T efef , the optimal drive signal is obtained through the vector control strategy of the electromagnetic coupler; the optimal drive signal is then transmitted to the inverter controller, and the inverter controller is used to control the current of the winding in the electromagnetic coupler and then control the output torque of the electromagnetic coupler, that is, the input torque of the synchronous motor, and then accurately control the speed and power angle of the rotor in the synchronous phase regulator to suppress the oscillation of the rotor.

[0153] The embodiment of the present application provides an active power control device based on reinforcement learning, such as Figure 5 As shown, the device includes a collection module 101, a first calculation module 102, a second calculation module 103, a third calculation module 104, a fourth calculation module 105 and a fifth calculation module 106;

[0154] The acquisition module 101 is used to acquire the power and speed of the synchronous condenser included in the inertia flywheel;

[0155] The first calculation module 102 is used to input the power and speed into a reinforcement learning neural network to obtain a proportion, an integral and a differential;

[0156] The second calculation module 103 is used to determine the power deviation according to the power and the preset target power;

[0157] The third calculation module 104 is used to determine the torque base value based on the PID controller according to the proportion, the integral, the differential and the power deviation;

[0158] The fourth calculation module 105 is used to determine the optimal control signal based on vector control by taking the torque base value as input; and

[0159] The fifth calculation module 106 is used to control the inverter controller according to the optimal control signal, so as to realize the control of the inertia flywheel.

[0160] Furthermore, the device also includes:

[0161] The first determination module is used to construct a simulation model according to the flywheel, the electromagnetic coupler, the synchronous phase condenser, the current converter controller and the PID controller;

[0162] The second determination module is used to call the simulation model based on the DDPG algorithm to determine the reward function, power, speed, proportion, integration and differentiation;

[0163] The first determination module is used to determine whether the reward function is greater than a preset threshold;

[0164] The second discrimination module is configured to, if yes, update the reinforcement learning neural network according to the reward function, the power, the rotation speed, the ratio, the integral and the differential; and

[0165] The third judgment module is used to call the simulation model to determine the reward function, power, speed, proportion, integration and differentiation.

[0166] Furthermore, the device also includes:

[0167] The reward function is specifically:

[0168]

[0169] Among them, r t represents the reward function;

[0170] t represents the simulation start time;

[0171] t step represents the simulation step length;

[0172] k represents the weight coefficient;

[0173] ω represents the actual speed of the synchronous condenser;

[0174] ω ref Indicates the rated speed of the synchronous condenser;

[0175] P represents the actual output power of the inertia flywheel system;

[0176] P ref Indicates the target output power of the inertia flywheel.

[0177] In an embodiment of the present application, a storage medium for storing computer-executable instructions is provided, wherein the computer-executable instructions, when executed, implement the steps of the active power control method based on reinforcement learning described in any one of the above embodiments.

[0178] It should be noted that the embodiments of the storage medium in this specification and the embodiments of the blockchain-based service provision method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the aforementioned corresponding implementation of the blockchain-based service provision method, and the repeated parts will not be repeated.

[0179] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0180] In the 1930s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0181] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0182] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0183] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0184] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0189] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0190] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0191] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0192] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0193] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0194] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A reinforcement learning based active power control method, applied to inertia flywheel, characterized in that include: Collect the power and speed of the synchronous condenser included in the inertia flywheel; Inputting the power and speed into a reinforcement learning neural network to obtain proportional, integral and differential; Determining a power deviation according to the power and a preset target power; Determine a torque base value based on the PID controller according to the proportion, the integral, the differential and the power deviation; Taking the torque base value as input, determining an optimal control signal based on vector control; as well as According to the optimal control signal, the inverter controller is controlled to realize the control of the inertia flywheel.

2. The method according to claim 1, characterized in that The training process of the reinforcement learning neural network includes: According to the data of the inertia flywheel, a simulation model is constructed; the simulation model includes parameters of the flywheel, the electromagnetic coupler, the synchronous condenser, the current converter controller and the PID controller; Based on the DDPG algorithm, the simulation model is called to determine the reward function, power, speed, proportion, integration and differentiation; Determining whether the reward function is greater than a preset threshold; If so, updating the reinforcement learning neural network according to the reward function, the power, the speed, the ratio, the integral and the differential; and If not, the simulation model is called to determine the reward function, power, speed, proportion, integration and differentiation.

3. The method according to claim 2, characterized in that The reward function is calculated as follows: Among them, R t represents the reward function; t represents the simulation start time; t step represents the simulation step length; k represents the weight coefficient; ω represents the actual speed of the synchronous condenser; ω ref Indicates the rated speed of the synchronous condenser; P represents the actual output power of the inertia flywheel system; P ref Indicates the target output power of the inertia flywheel.

4. An active power control system based on reinforcement learning, characterized in that: Including, flywheel rotor, electromagnetic coupler, synchronous condenser, power grid and converter controller; One end of the electromagnetic coupler is connected to the flywheel rotor, and the other end is connected to the synchronous condenser; The synchronous condenser is connected to the power grid; Controlling the variable current controller by the method described in claim 1; and The current converter controller controls the electromagnetic coupler.

5. The system according to claim 4, characterized in that: The electromagnetic coupler has one end connected to the flywheel rotor and the other end connected to the synchronous condenser, comprising: The electromagnetic coupler comprises an inner rotor and an outer rotor; The inner rotor is connected to the flywheel rotor via a mechanical shaft; and The outer rotor is connected to the rotor of the synchronous condenser.

6. The system according to claim 4, characterized in that The synchronous condenser is connected with the power grid including: The synchronous condenser comprises a stator and a rotor; and The stator of the synchronous condenser is connected to the power grid.

7. An active power control device based on reinforcement learning, characterized in that: The device includes a collection module, a first calculation module, a second calculation module, a third calculation module, a fourth calculation module and a fifth calculation module; The acquisition module is used to acquire the power and speed of the synchronous condenser included in the inertia flywheel; The first calculation module is used to input the power and speed into a reinforcement learning neural network to obtain a proportion, an integral and a differential; The second calculation module is used to determine the power deviation according to the power and the preset target power; The third calculation module is used to determine the torque base value based on the PID controller according to the proportion, the integral, the differential and the power deviation; The fourth calculation module is used to determine the optimal control signal based on vector control by taking the torque base value as input; as well as The fifth calculation module is used to control the variable current controller according to the optimal control signal, so as to realize the control of the inertia flywheel.

8. The device according to claim 7, characterized in that: The device also includes: A first determination module is used to construct a simulation model according to a flywheel, an electromagnetic coupler, a synchronous condenser, a current converter controller and a PID controller; A second determination module is used to call the simulation model based on the DDPG algorithm to determine the reward function, power, speed, proportion, integration and differentiation; A first determination module, used to determine whether the reward function is greater than a preset threshold; A second discrimination module is configured to, if yes, update the reinforcement learning neural network according to the reward function, the power, the rotation speed, the ratio, the integral and the differential; and The third judgment module is used to call the simulation model to determine the reward function, power, speed, proportion, integration and differentiation.

9. The device according to claim 8, characterized in that The device also includes: The reward function is: Among them, R t represents the reward function; t represents the simulation start time; t step represents the simulation step length; k represents the weight coefficient; ω represents the actual speed of the synchronous condenser; ω ref Indicates the rated speed of the synchronous condenser; P represents the actual output power of the inertia flywheel system; P ref Indicates the target output power of the inertia flywheel.

10. A storage medium for storing computer executable instructions, characterized in that: When the computer executable instructions are executed, the steps of the active power control method based on reinforcement learning described in any one of claims 1 to 3 are implemented.

Images