Active power control method, system, device and storage medium based on reinforcement learning

Abstract

The embodiments of this specification provide a reinforcement learning-based active power control method, system, device, and storage medium for use with an inertial flywheel. The method includes: collecting the power and speed of a synchronous condenser included in the inertial flywheel; inputting the power and speed into a reinforcement learning neural network to obtain a proportional, integral, and differential; determining a power deviation based on the power and a preset target power; determining a torque base value based on the proportional, integral, differential, and power deviation using a PID controller; determining an optimal control signal based on vector control using the torque base value as input; and controlling a current converter controller based on the optimal control signal to achieve control of the inertial flywheel. This method solves the problems of slow system response and large output oscillation, thereby improving the system's dynamic response performance and control accuracy.

Description

Technical Field

[0001] This document relates to the field of motor control technology, and in particular to an active power control method, system, device and storage medium based on reinforcement learning. Background Art

[0002] Grid inertia is the ability to provide inertial support through the kinetic energy of its rotor when the grid frequency changes, helping to stabilize the grid frequency.

[0003] In the prior art, when renewable energy is affected by power outages, the energy stored in the inertia flywheel is mainly replenished into the power grid through a synchronous motor through active control to maintain the stability of the power grid frequency.

[0004] However, due to the limitations of the transient process of the synchronous motor during active control, the existing technology will cause rotor oscillation when torque is suddenly input to the synchronous motor, causing large fluctuations in the system output power, affecting the response speed of the motor, the normal operation of electrical equipment, and power safety. Summary of the Invention

[0005] In view of the above solutions, the present application aims to propose an active power control method, system, device and storage medium based on reinforcement learning to solve at least one of the above technical problems.

[0006] In a first aspect, one or more embodiments of this specification provide a reinforcement learning-based active power control method applied to an inertia flywheel, including:

[0007] Collect the power and speed of the synchronous condenser included in the inertia flywheel;

[0008] Inputting the power and speed into a reinforcement learning neural network to obtain proportional, integral and differential;

[0009] determining a power deviation based on the power and a preset target power;

[0010] Determine a torque base value based on the proportion, the integral, the differential, and the power deviation based on a PID controller;

[0011] Taking the torque base value as input, determining an optimal control signal based on vector control; and

[0012] According to the optimal control signal, the current converter is controlled to realize the control of the inertia flywheel.

[0013] Furthermore, the training process of the reinforcement learning neural network includes:

[0014] Constructing a simulation model based on the inertia flywheel data; the simulation model includes parameters of the flywheel, electromagnetic coupler, synchronous condenser, current converter controller, and PID controller;

[0015] Based on the DDPG algorithm, the simulation model is called to determine the reward function, power, speed, proportion, integral and differential;

[0016] Determining whether the reward function is greater than a preset threshold;

[0017] If so, updating a reinforcement learning neural network based on the reward function, the power, the speed, the ratio, the integral, and the differential; and

[0018] If not, the simulation model is called to determine the reward function, power, speed, proportion, integration and differentiation.

[0019] Furthermore, the reward function is calculated as follows:

[0020]

[0021] Among them, R t represents the reward function;

[0022] t represents the simulation start time;

[0023] t step represents the simulation step size;

[0024] k represents the weight coefficient;

[0025] ω represents the actual speed of the synchronous condenser;

[0026] ω ref Indicates the rated speed of the synchronous condenser;

[0027] P represents the actual output power of the inertia flywheel system;

[0028] P ref Indicates the target output power of the inertia flywheel.

[0029] In a second aspect, an embodiment of the present application provides an active power control system based on reinforcement learning, comprising a flywheel rotor, an electromagnetic coupler, a synchronous condenser, a power grid, and a current converter controller;

[0030] One end of the electromagnetic coupler is connected to the flywheel rotor, and the other end is connected to the synchronous condenser;

[0031] The synchronous condenser is connected to the power grid;

[0032] Controlling the variable current controller by the method of claim 1; and

[0033] The current converter controller controls the electromagnetic coupler.

[0034] Furthermore, the electromagnetic coupler is connected to the flywheel rotor at one end and the synchronous condenser at the other end, comprising:

[0035] The electromagnetic coupler includes an inner rotor and an outer rotor;

[0036] The inner rotor is connected to the flywheel rotor via a mechanical shaft; and

[0037] The outer rotor is connected to the rotor of the synchronous condenser.

[0038] Furthermore, the connecting of the synchronous condenser to the power grid comprises:

[0039] The synchronous condenser comprises a stator and a rotor; and

[0040] The stator of the synchronous condenser is connected to the power grid.

[0041] In a third aspect, an embodiment of the present application provides an active power control device based on reinforcement learning, the device comprising an acquisition module, a first calculation module, a second calculation module, a third calculation module, a fourth calculation module, and a fifth calculation module;

[0042] The acquisition module is used to acquire the power and speed of the synchronous condenser included in the inertia flywheel;

[0043] The first calculation module is used to input the power and speed into a reinforcement learning neural network to obtain proportional, integral and differential;

[0044] The second calculation module is used to determine the power deviation according to the power and the preset target power;

[0045] The third calculation module is configured to determine a torque base value based on the proportion, the integral, the differential and the power deviation based on a PID controller;

[0046] The fourth calculation module is configured to determine an optimal control signal based on vector control using the torque base value as input; and

[0047] The fifth calculation module is used to control the current converter according to the optimal control signal, so as to realize the control of the inertia flywheel.

[0048] Furthermore, the device further comprises:

[0049] The first determining module is used to construct a simulation model based on the flywheel, the electromagnetic coupler, the synchronous condenser, the current converter controller and the PID controller;

[0050] The second determination module is configured to call the simulation model based on the DDPG algorithm to determine the reward function, power, speed, proportion, integration, and differentiation;

[0051] The first determination module is used to determine whether the reward function is greater than a preset threshold;

[0052] The second discrimination module is configured to, if yes, update the reinforcement learning neural network according to the reward function, the power, the speed, the ratio, the integral and the differential; and

[0053] The third judgment module is used to call the simulation model to determine the reward function, power, speed, proportion, integration and differentiation if no.

[0054] Furthermore, the device further comprises:

[0055] The reward function is specifically:

[0056]

[0057] Among them, R t represents the reward function;

[0058] t represents the simulation start time;

[0059] t step represents the simulation step size;

[0060] k represents the weight coefficient;

[0061] ω represents the actual speed of the synchronous condenser;

[0062] ω ref Indicates the rated speed of the synchronous condenser;

[0063] P represents the actual output power of the inertia flywheel system;

[0064] P ref Indicates the target output power of the inertia flywheel.

[0065] In a fourth aspect, an embodiment of the present application provides a storage medium for storing computer-executable instructions, characterized in that when the computer-executable instructions are executed, the steps of the active power control method based on reinforcement learning described in any one of the first aspects are implemented.

[0066] Compared with the existing technology, this application can at least achieve the following technical effects:

[0067] This application utilizes a reinforcement learning neural network to dynamically adjust the P, I, and D parameters of the PID controller, thereby achieving refined control of the synchronous condenser's output power and speed, effectively addressing grid frequency fluctuations caused by the intermittent nature of renewable energy in the power system. Furthermore, by determining the speed and power angle of the synchronous condenser in the inertia flywheel, even more precise control can be achieved. This approach ensures that the synchronous condenser always operates at its optimal state, further improving the operational efficiency and reliability of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0069] Figure 1 A flow chart of an active power control method based on reinforcement learning provided in one or more embodiments of this specification;

[0070] Figure 2 Active power waveform diagram of experimental tests of the control strategy of the present application and the traditional control strategy provided in one or more embodiments of this specification;

[0071] Figure 3 A schematic diagram of the structure of an inertia flywheel system provided for one or more embodiments of this specification;

[0072] Figure 4 A block diagram of an inertia flywheel active power control strategy based on a variable parameter PID of reinforcement learning provided in one or more embodiments of this specification;

[0073] Figure 5 A schematic structural diagram of an active power control device based on reinforcement learning provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this document.

[0075] Renewable energy generation is affected by weather and time, resulting in unstable output. Therefore, flywheel energy storage is used to absorb and store excess power when there is an oversupply of electricity. When power demand increases or decreases, the stored energy is released to replenish the grid to maintain a stable frequency. However, the grid's inertia response phase occurs within a few seconds of a frequency drop. Existing flywheels struggle to respond quickly and provide timely inertia support to the grid, resulting in significant fluctuations in output power, impacting the proper operation of electrical equipment and power safety.

[0076] To address the above technical issues, this application proposes an active power control method based on reinforcement learning. The specific steps are as follows:

[0077] Step S1 : collecting the power and speed of the synchronous condenser included in the inertia flywheel.

[0078] In an embodiment of the present application, a synchronous condenser is a rotating device that can provide or absorb reactive power to support the voltage stability of the power system. When generating electricity from renewable energy, the grid voltage can be unstable because the output power of renewable energy depends on natural conditions. Therefore, a synchronous condenser is used to transfer excess electrical energy to an inertia flywheel for storage. If the renewable energy stops generating electricity, the electrical energy stored in the inertia flywheel is accurately output to the grid using the synchronous condenser. Therefore, by collecting the power and speed of the synchronous condenser, the output power of the synchronous condenser can be dynamically adjusted to ensure voltage stability.

[0079] Step S2: input the power and speed into a reinforcement learning neural network to obtain proportion, integration and differentiation.

[0080] In an embodiment of the present application, reinforcement learning is used to automatically adjust the proportional (P), integral (I) and differential (D) coefficients of the PID controller to optimize the control performance of the inertia flywheel.

[0081] A simulation model was constructed using visual simulation tools such as MATLAB / Simulink to simulate the operating conditions of the flywheel, electromagnetic coupler, and synchronous condenser in the inertia flywheel when the power grid is disconnected. In this simulation environment, the output of a PID controller was fed into the inertia flywheel to control the torque in the synchronous condenser.

[0082] During training, the simulation model is randomly given a test power command. At each moment during the test, sensors measure the state variables of the synchronous condenser in the inertia flywheel. These variables include active power, its derivative, speed, and its derivative. These state variables are then fed into a neural network, which outputs action values ​​for the neural network, including the P, I, and D parameters of the PID controller. This action value is then fed into the PID controller, where a reward value is calculated based on the power and power command, and the neural network is updated using this reward value. Training is repeated repeatedly, and when the reward value is within a preset threshold, the neural network is trained, resulting in a reinforced neural network. During use, the reinforcement learning neural network calculates the P, I, and D parameters.

[0083] Specifically, the neural network is updated using the DDPG algorithm, and the algorithm flow is as follows:

[0084] initialization:

[0085] Randomly initialize the parameters of the Actor network and Critic network;

[0086] Initialize the target network;

[0087] Initialize the experience replay pool R.

[0088] Repeat the number of iterations (episode):

[0089] Randomly initialize the process for action exploration.

[0090] Get the initial state value s0.

[0091] Repeat t to T(step):

[0092] According to the current state t Calculate the action a at the current time step t .

[0093] Execute action a t , and record the reward r t and the new state s t+1 .

[0094] Storage conversion experience(s t , a t , r t , s t+1 ) in the experience pool R.

[0095] Randomly sample a small batch of conversion experience samples (s) from the experience pool R i , a i , r i , s i+1 ).

[0096] Minimize the Loss function and update the Critic network.

[0097] Update the Actor network using a gradient algorithm.

[0098] Update the target network.

[0099] Among them, the state s t Including active power, differential of active power, speed, differential of speed four state quantities, action a t It contains three variables: the P, I, and D parameters of the PID controller. The reward function is used to evaluate the quality of the action. The purpose of the variable parameter PID controller is to make the motor output a specified power while reducing the motor oscillation. Therefore, the evaluation criteria are power deviation and speed deviation. The reward function is calculated as follows:

[0100]

[0101] Among them, r t represents the reward function;

[0102] t represents the simulation start time;

[0103] t step represents the simulation step size;

[0104] k represents the weight coefficient;

[0105] ω represents the actual speed of the synchronous condenser;

[0106] ω ref Indicates the rated speed of the synchronous condenser;

[0107] P represents the actual output power of the inertia flywheel system;

[0108] P ref Indicates the target output power of the inertia flywheel.

[0109] Step S3: determining a power deviation according to the power and a preset target power.

[0110] In the embodiment of the present application, the preset target power value is the power level that the system expects to achieve, which is set based on the demand of the power grid. The power deviation can be calculated using the actual output power of the synchronous condenser and the preset target power.

[0111] Specifically, power deviation=preset target power-actual power.

[0112] For example, if the target power of a synchronous condenser is 100 kW, and the actual output power is 95 kW, ΔP = 100 kW - 95 kW = 5 kW, which means the system needs to increase its power output by 5 kW to reach the target value.

[0113] The power deviation can be used to adjust the control strategy. The control system can adjust the synchronous condenser output in real time to ensure the stability and reliability of the power system. For example, the synchronous condenser's excitation current can be adjusted to keep the actual power close to the target power.

[0114] Step S4: determining a torque base value based on the PID controller according to the proportion, the integral, the differential and the power deviation.

[0115] Use the following formula to determine the base torque:

[0116]

[0117] Where, T eref is the torque base value, K p is the P parameter of the PID controller, K i is the I parameter of the PID controller, K d is the D parameter of the PID controller, s is the Laplace operator, and ΔP is the power deviation.

[0118] Step S5: Taking the torque base value as input, determine the optimal control signal based on vector control.

[0119] The vector control process is as follows:

[0120] The speed, winding resistance, d-axis current, d-axis inductance, q-axis inductance, and q-axis current of the flywheel rotor and the synchronous motor rotor are collected; the slip of the flywheel rotor and the synchronous motor rotor is determined based on the speed; based on the voltage equation of the electromagnetic coupler, the d-axis voltage and q-axis voltage are determined based on the slip, winding resistance, d-axis current, d-axis inductance, q-axis inductance, and q-axis current of the flywheel rotor and the synchronous motor rotor.

[0121] The voltage equation of the electromagnetic coupler is:

[0122]

[0123] Where Δω is the slip between the flywheel rotor and the synchronous motor rotor, u d is the d-axis voltage, R s is the winding resistance, i d is the d-axis current, L d is the d-axis inductance, L q is the q-axis inductance, i q is the q-axis current, u q is the q-axis voltage, ψ r is the magnetic linkage coefficient.

[0124] Then, based on the electromagnetic torque equation of the electromagnetic coupler, the electromagnetic torque is determined according to the d-axis inductance, q-axis inductance, d-axis current, and q-axis current.

[0125] Electromagnetic torque equation of electromagnetic coupler:

[0126] T e =1.5p n [ψ r +(L d -L q )i d ]i q ;

[0127] Among them, T e is the electromagnetic torque, p n is the pole pair number.

[0128] Adopt i d =0 vector control strategy, i d The reference value is 0. According to the expression of electromagnetic torque of electromagnetic coupler, it can be known that the electromagnetic torque T is e The coupling output torque is proportional to the q-axis current, approximately assuming the coupler's output torque is equal to the electromagnetic torque. A sensor measures the electromagnetic coupler's output torque feedback signal. A feedback value is determined based on the feedback signal. The difference between the feedback value and the torque base value is calculated. This difference is input into the PI loop, which controls the q-axis current to achieve the target electromagnetic torque output from the electromagnetic coupler. The outputs of the d-axis and q-axis current loops, as well as the coupling term calculated from the d / q-axis currents and slip Δω, are substituted into the voltage equation to obtain a voltage reference value. Space vector modulation (SVPWM) is used to control the converter's output, controlling the electromagnetic coupler's current based on the voltage reference value.

[0129] Step S6: controlling the current converter according to the optimal control signal to achieve control of the inertia flywheel.

[0130] In the embodiment of the present application, by comparing the overshoot, cumulative error and time required to reach steady state in the control strategy of the present application with that of the traditional control strategy, it can be seen that the active power output by the present application is less volatile, the overshoot is smaller, and the time required to reach steady state is shorter than that of the traditional inertia flywheel system, as shown in Table 1, thereby effectively suppressing the power oscillation problem of the inertia flywheel system when outputting active power. Figure 2 It can be seen that the traditional control strategy will produce huge power fluctuations at the moment when the inertia flywheel outputs active power or controls the power size, thereby affecting the safety of power equipment and the stable operation of the power system. Figure 2 It can be clearly seen that huge fluctuations occur between 0.5 seconds and 1 second; the control strategy of this application can reduce power fluctuations at the moment when the inertia flywheel outputs active power or controls the power size, ensuring the safety and stability of power equipment and power systems, such as Figure 2It can be clearly seen from the 0.5 second to 1 second that the power fluctuation is relatively stable compared with the traditional control strategy.

[0131] Table 1 compares the active power indicators of the control strategy of this application and the traditional control strategy experimental test

[0132]

[0133] Furthermore, in an embodiment of the present application, the training process of the reinforcement learning neural network includes: constructing a simulation model based on the data of the inertia flywheel; the simulation model includes parameters of the flywheel, electromagnetic coupler, synchronous phase regulator, current converter and PID controller; based on the DDPG algorithm, calling the simulation model to determine the reward function, power, speed, proportion, integral and differential; determining whether the reward function is greater than a preset threshold; if so, updating the reinforcement learning neural network according to the reward function, the power, the speed, the proportion, the integral and the differential; and if not, calling the simulation model to determine the reward function, power, speed, proportion, integral and differential.

[0134] Specifically, the parameters of the flywheel, electromagnetic coupler, synchronous condenser, current converter and PID controller in the actual equipment are obtained, among which the flywheel is used to store and release energy; the electromagnetic coupler is used to control the transmission of energy; the synchronous condenser is used to provide reactive power support and control the output power; the current converter is used to control the conversion of energy; and the PID controller is used to adjust the control signal.

[0135] These parameters were then used to construct a simulation model to simulate the operating states of the inertia flywheel when the grid was powered and when it was not. When the grid was powered, the synchronous condenser did not generate any active power, the flywheel speed remained constant, and the electromagnetic coupler did not output any torque. When the grid was not powered, the synchronous condenser began to generate active power, the flywheel speed decreased, and energy was released. The electromagnetic coupler began to output torque, transferring energy from the flywheel to the synchronous condenser.

[0136] Reinitialize the reinforcement learning model, including the Actor network for generating control actions and the Critic network for evaluating the quality of control actions. Obtain the initial state from the simulation model, including power and speed. Based on the current state, generate a control action (such as the parameters of the PID controller) through the Actor network. Use the generated action to update the state of the simulation model and calculate the new power, speed, etc. Calculate the reward value based on the new state, and store the current state, action, reward, and new state in the experience replay buffer. Randomly extract a batch of samples from the experience replay buffer to update the Actor and Critic networks. Check whether the current reward value is greater than the preset threshold. If so, continue updating the reinforcement learning model; otherwise, re-obtain the power and speed for calculation. Once the model training is completed through the above steps, it is applied to the actual system to dynamically adjust the parameters of the PID controller to optimize the performance of the synchronous condenser in the inertia flywheel.

[0137] Furthermore, the reward function is calculated as:

[0138]

[0139] Among them, R t represents the reward function;

[0140] t represents the simulation start time;

[0141] t step represents the simulation step size;

[0142] k represents the weight coefficient;

[0143] ω represents the actual speed of the synchronous condenser;

[0144] ω ref Indicates the rated speed of the synchronous condenser;

[0145] P represents the actual output power of the inertia flywheel system;

[0146] P ref Indicates the target output power of the inertia flywheel.

[0147] The embodiment of the present application provides an active power control system based on reinforcement learning, such as Figure 3 The present invention includes a flywheel rotor 10, an electromagnetic coupler 30, a synchronous condenser 40, a power grid 60 and a current converter 50; one end of the electromagnetic coupler 30 is connected to the flywheel rotor 10, and the other end is connected to the synchronous condenser 40; the synchronous condenser 40 is connected to the power grid 60; the current converter 50 is controlled by the method in the above embodiment; and the current converter 50 controls the electromagnetic coupler 30.

[0148] In the embodiment of the present application, the flywheel rotor 10 is connected to the synchronous condenser 40 via an electromagnetic coupler 30. The electromagnetic coupler 30 comprises an inner rotor and an outer rotor. The inner rotor is directly connected to the flywheel rotor 10 via drive shaft 1 21, while the outer rotor is directly connected to the rotor of the synchronous condenser 40 via drive shaft 2 22. The inner rotor of the electromagnetic coupler 30 utilizes permanent magnets, while the outer rotor is equipped with armature windings, effectively acting as a permanent magnet synchronous motor with simultaneous stator and rotor rotation. The electromagnetic coupler 30 enables the transfer of torque and inertia between the flywheel 10 and the synchronous condenser 40. The electromagnetic coupler 30 is controlled by a current converter 50. The synchronous condenser 40 rotor is maintained at the power frequency, and the stator end is directly connected to the power grid 60, which can supply power to the electromagnetic coupler 30 via the coupler current converter 50.

[0149] When the external frequency rises or falls, the inverter controller 50 controls the output torque of the electromagnetic coupler 30. Because the electromagnetic coupler 30 is directly connected to the synchronous condenser 40 rotor via a mechanical shaft, the output torque of the electromagnetic coupler 30 is equal to the input torque of the synchronous condenser 40. Since the electromagnetic coupler 30 is equivalent to a permanent magnet synchronous motor with simultaneous rotation of the stator and rotor, its output torque can be controlled using vector control.

[0150] The inertia flywheel active power control strategy of variable parameter PID based on reinforcement learning, such as Figure 4 As shown in the figure, the grid-side controller uses a DC bus voltage outer loop and a current inner loop control strategy to maintain a constant DC bus voltage. The generator-side controller uses a torque outer loop and a current inner loop control strategy. The d-axis current reference value in the inner current loop is 0, and the q-axis current reference value is the output value of the torque outer loop.

[0151] The setpoint for the electromagnetic coupler's output torque is determined by a PID controller, whose input is the output power deviation. The PID parameters are derived from a neural network trained using a reinforcement learning algorithm. This algorithm utilizes a model-free reinforcement learning algorithm that does not require knowledge of the control model's state equations. The PID controller outputs a base torque value, which is then vector-controlled to determine the optimal drive signal. This input is then fed into a current converter controller, which controls the current and torque input to the electromagnetic coupler windings. This current control, in turn, controls the torque in the synchronous condenser. The torque is then used to adjust the synchronous condenser's speed and power angle, thereby improving the system's dynamic response and control accuracy.

[0152] Specifically, the actual power P and speed ω are obtained from the synchronous condenser, and the preset target output power P is determined. ref, the power derivative dP / dt and the speed derivative dω / dt, the actual power P and speed ω are input into the reinforcement learning neural network to obtain the P, I, D parameter values; then according to the actual power P and the target output power P ref , determine the power deviation; then input the P, I, D parameter values ​​and power deviation into the PID controller to determine the torque base value T eref , according to the torque base value T efef , the optimal drive signal is obtained through the vector control strategy of the electromagnetic coupler; the optimal drive signal is then transmitted to the current converter controller, which controls the current of the winding in the electromagnetic coupler and thus controls the output torque of the electromagnetic coupler, that is, the input torque of the synchronous motor, thereby accurately controlling the speed and power angle of the rotor in the synchronous condenser and suppressing the oscillation of the rotor.

[0153] The embodiment of the present application provides an active power control device based on reinforcement learning, such as Figure 5 As shown, the apparatus includes a collection module 101, a first calculation module 102, a second calculation module 103, a third calculation module 104, a fourth calculation module 105 and a fifth calculation module 106;

[0154] The acquisition module 101 is used to acquire the power and speed of the synchronous condenser included in the inertia flywheel;

[0155] The first calculation module 102 is used to input the power and speed into the reinforcement learning neural network to obtain the proportion, integral and differential;

[0156] The second calculation module 103 is used to determine the power deviation according to the power and the preset target power;

[0157] The third calculation module 104 is configured to determine a torque base value based on the proportion, the integral, the differential, and the power deviation based on a PID controller;

[0158] The fourth calculation module 105 is configured to determine an optimal control signal based on vector control using the torque base value as input; and

[0159] The fifth calculation module 106 is configured to control the variable current controller according to the optimal control signal, so as to realize control of the inertia flywheel.

[0160] Furthermore, the device further comprises:

[0161] The first determining module is used to construct a simulation model based on the flywheel, the electromagnetic coupler, the synchronous condenser, the current converter controller and the PID controller;

[0162] The second determination module is configured to call the simulation model based on the DDPG algorithm to determine the reward function, power, speed, proportion, integration, and differentiation;

[0163] The first determination module is used to determine whether the reward function is greater than a preset threshold;

[0164] The second discrimination module is configured to, if yes, update the reinforcement learning neural network according to the reward function, the power, the speed, the ratio, the integral and the differential; and

[0165] The third judgment module is used to call the simulation model to determine the reward function, power, speed, proportion, integration and differentiation if no.

[0166] Furthermore, the device further comprises:

[0167] The reward function is specifically:

[0168]

[0169] Among them, r t represents the reward function;

[0170] t represents the simulation start time;

[0171] t step represents the simulation step size;

[0172] k represents the weight coefficient;

[0173] ω represents the actual speed of the synchronous condenser;

[0174] ω ref Indicates the rated speed of the synchronous condenser;

[0175] P represents the actual output power of the inertia flywheel system;

[0176] P ref Indicates the target output power of the inertia flywheel.

[0177] In an embodiment of the present application, a storage medium for storing computer-executable instructions is provided, wherein the computer-executable instructions, when executed, implement the steps of the active power control method based on reinforcement learning described in any one of the above embodiments.

[0178] It should be noted that the embodiment of the storage medium in this specification and the embodiment of the blockchain-based service provision method in this specification are based on the same inventive concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the corresponding blockchain-based service provision method mentioned above, and the repeated parts will not be repeated.

[0179] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0180] In the 1930s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0181] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0182] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0183] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0184] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0189] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0190] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0191] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0192] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0193] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0194] The foregoing description is merely an example of the present invention and is not intended to limit the present invention. Persons skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be included within the scope of the claims herein.

Claims

1. A method for active power control based on reinforcement learning, applied to inertia flywheel, characterized in that include: Collect the power and speed of the synchronous condenser included in the inertia flywheel; Inputting the power and speed into a reinforcement learning neural network to obtain proportional, integral and differential; determining a power deviation based on the power and a preset target power; Determine a torque base value based on the proportion, the integral, the differential, and the power deviation based on a PID controller; Taking the torque base value as input, determining an optimal control signal based on vector control; as well as Controlling the current converter controller according to the optimal control signal so as to realize control of the inertia flywheel; The training process of the reinforcement learning neural network includes: Constructing a simulation model based on the inertia flywheel data; the simulation model includes parameters of the flywheel, electromagnetic coupler, synchronous condenser, current converter controller, and PID controller; Based on the DDPG algorithm, the simulation model is called to determine the reward function, power, speed, proportion, integral and differential; Determining whether the reward function is greater than a preset threshold; If so, updating a reinforcement learning neural network based on the reward function, the power, the speed, the ratio, the integral, and the differential; and If not, the simulation model is called to determine the reward function, power, speed, proportion, integration and differentiation.

2. The method according to claim 1, characterized in that The reward function is calculated as follows: in, represents the reward function; t represents the simulation start time; represents the simulation step size; represents the weight coefficient; Indicates the actual speed of the synchronous condenser; Indicates the rated speed of the synchronous condenser; Indicates the actual output power of the inertia flywheel system; Indicates the target output power of the inertia flywheel.

3. An active power control system based on reinforcement learning, characterized in that: Including flywheel rotor, electromagnetic coupler, synchronous condenser, power grid and converter controller; One end of the electromagnetic coupler is connected to the flywheel rotor, and the other end is connected to the synchronous condenser; The synchronous condenser is connected to the power grid; Controlling the variable current controller by the method described in claim 1; and The current converter controller controls the electromagnetic coupler.

4. The system according to claim 3, characterized in that The electromagnetic coupler has one end connected to the flywheel rotor and the other end connected to the synchronous condenser, comprising: The electromagnetic coupler includes an inner rotor and an outer rotor; The inner rotor is connected to the flywheel rotor via a mechanical shaft; and The outer rotor is connected to the rotor of the synchronous condenser.

5. The system according to claim 3, wherein: The synchronous condenser is connected to the power grid including: The synchronous condenser comprises a stator and a rotor; and The stator of the synchronous condenser is connected to the power grid.

6. An active power control device based on reinforcement learning, characterized in that: The device includes an acquisition module, a first calculation module, a second calculation module, a third calculation module, a fourth calculation module and a fifth calculation module; The acquisition module is used to acquire the power and speed of the synchronous condenser included in the inertia flywheel; The first calculation module is used to input the power and speed into a reinforcement learning neural network to obtain proportional, integral and differential; The second calculation module is used to determine the power deviation according to the power and the preset target power; The third calculation module is configured to determine a torque base value based on the proportion, the integral, the differential and the power deviation based on a PID controller; The fourth calculation module is configured to determine an optimal control signal based on vector control using the torque base value as input; as well as The fifth calculation module is used to control the variable current controller according to the optimal control signal, so as to realize the control of the inertia flywheel; The device further comprises: A first determination module is used to construct a simulation model based on a flywheel, an electromagnetic coupler, a synchronous condenser, a current converter controller and a PID controller; A second determination module is configured to call the simulation model based on the DDPG algorithm to determine the reward function, power, speed, proportion, integration, and differentiation; A first discrimination module, configured to determine whether the reward function is greater than a preset threshold; A second discrimination module is configured to, if yes, update a reinforcement learning neural network according to the reward function, the power, the speed, the ratio, the integral, and the differential; and The third judgment module is used to call the simulation model to determine the reward function, power, speed, proportion, integration and differentiation if the answer is no.

7. The device according to claim 6, characterized in that The device further comprises: The reward function is: in, represents the reward function; t represents the simulation start time; represents the simulation step size; represents the weight coefficient; Indicates the actual speed of the synchronous condenser; Indicates the rated speed of the synchronous condenser; Indicates the actual output power of the inertia flywheel system; Indicates the target output power of the inertia flywheel.

8. A storage medium for storing computer-executable instructions, characterized in that: When the computer executable instructions are executed, the steps of the active power control method based on reinforcement learning according to claim 1 or 2 are implemented.

Images