Multi-energy power system active power dynamic distribution method based on GRU and DRL

Abstract

The invention relates to a multi-energy power system active power dynamic distribution method based on a GRU and a DRL, and belongs to the technical field of active power dynamic distribution of various power sources in a power system. And scheduling strategies for units with different operation characteristics during dynamic power distribution are optimized accurately and quickly. Meanwhile, the deep reinforcement learning performs optimization of a dynamic power distribution algorithm based on historical operation data actually acquired by the multi-energy power system, and considers the nonlinear and uncertain problems, which are difficult to describe in a traditional scheduling mode, existing in the multi-energy power system in an off-line training part of a model by using a soft action-Critic, SAC algorithm. And the power distribution real-time calculation efficiency of the multi-energy power system is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of active power dynamic allocation of multiple power sources in a power system, and particularly relates to a method for active power dynamic allocation of a multi-energy power system based on GRU and DRL. BACKGROUND

[0002] With the proposal of "carbon peak" and "carbon neutrality", the development and utilization of new energy has become the key to improving the utilization efficiency of traditional energy, developing smart grids and new power systems with new energy as the main body. The increasing penetration rate of new energy has occupied the space of traditional hydro and thermal generating units, leading to structural difficulties in grid frequency control, and new energy units are urgently needed to participate in grid fast frequency regulation to improve the safety level of large grid frequency. However, the output capacity of new energy sources is difficult to predict, and the calculation is large and has strong uncertainty, so it is difficult to fully utilize. SUMMARY

[0003] The purpose of the present application is to overcome the shortcomings of the prior art, and provide a method for active power dynamic allocation of a multi-energy power system based on GRU and DRL, which fully utilizes the rapidity of the inverter frequency response technology of new energy units, dynamically optimizes the generation power instruction distribution of the multi-energy power system according to the power generation capacity of wind and light stations under different weather conditions and the fast frequency regulation demand of the grid, and further comprehensively improves the rapidity and economy of the fast frequency regulation of the multi-energy power system.

[0004] The technical scheme adopted by the present application is: a method for active power dynamic allocation of a multi-energy power system based on GRU and DRL, comprising the following steps:

[0005] S1: collecting historical operation data of the multi-energy power system; and taking the future available power of the wind farm and photovoltaic power station within a certain time period as the true output of the model, and setting the prediction step number as the input of the GRU prediction model;

[0006] S2: preprocessing the collected data, correcting abnormal data, and converting time series data into supervised sequence data;

[0007] S3: inputting the input quantity into the GRU neural network, and initializing the GRU neural network weight training model;

[0008] S4: performing precision testing on the GRU neural network prediction model using the test set;

[0009] S5: defining the state space of the reinforcement learning algorithm, fully considering the observed environmental information and the prediction results of the GRU neural network, to ensure that the model can accurately perceive the state of the system and make corresponding decisions during the SAC parameter learning process;

[0010] S6: Determine the exploration strategy of the action, define the action space

[0011] S7: Define the reward function of the deep reinforcement model;

[0012] S8: Use Record all experience data of active power distribution of multi-energy power system, and express as

[0013]

[0014] S9: Update the SAC network parameters;

[0015] S10: Conduct a comprehensive evaluation of the system, including the prediction accuracy of the model, the accuracy and speed of power distribution decision, etc.

[0016] Specifically, in step S1, the historical operation data of the multi-energy power system includes the wind speed V of the wind farm, the available power S wg of the wind farm at the corresponding time, the light intensity In of the photovoltaic power station, the outdoor temperature T out , and the available power S pv of the photovoltaic power station at the corresponding time, the system frequency deviation Δf at the current time, and the available power S th of the thermal power unit at the corresponding time, wherein the available power of each power supply unit is the difference between the theoretical full-power output and the actual grid-connected power of the unit.

[0017] In step S2, the data is divided into training set and test set.

[0018] In step S3, the structure of the GRU model is determined, including the number of layers of the neural network model, the number of neurons in each layer, the activation function and the loss function, and the model parameters are initialized. The wind speed V, light intensity In, outdoor temperature T out , system frequency deviation Δf, and available power S wg of the wind farm and available power S pv of the photovoltaic power station at the corresponding time are imported into the GRU neural network for training to obtain the model parameters.

[0019] In step S4, the test set is used to correct the parameters of the GRU model, and the prediction model of the available power of the wind farm and the photovoltaic power station in the multi-energy power system is obtained.

[0020] In step S5, the state space s tUsing a GRU neural network prediction model, historical data for each multi-energy power system is used to predict the available power values ​​of wind farms and photovoltaic power plants at n future time points. Simultaneously, the power system frequency deviation Δf and the available power of the generating units corresponding to the prediction time are combined as environmental information for reinforcement learning. Based on the prediction results of the GRU neural network, the power system frequency deviation Δf corresponding to the prediction time of the available power of wind farms and photovoltaic power plants is collected. t and the available power S of thermal power units t th Together, they serve as state and environment information for reinforcement learning, and h t , Δf t S t th As environmental information for reinforcement learning, it is represented as: s t ={h t ,Δf t ,S t th}

[0021] In step S6, the current state information is input into the Actor network of the deep reinforcement learning algorithm SAC. A multinomial distribution is constructed based on the parameters output by the Actor to generate actions. This multinomial distribution is constructed based on the disturbance information in the historical data collected by the power system and is associated with the time series.

[0022] In step S7, taking into account both the accuracy and speed of multi-energy power systems participating in grid frequency regulation, a reward function for a deep reinforcement learning model is constructed. This function guides the multi-energy power system in executing power allocation decisions. t Simultaneously, based on its participation in the system's frequency response, the instant reward r is calculated. t And record the execution of a t The updated environmental state calculation value s t+1 .

[0023] In step S8, s t Indicates the current state, a t This represents the power allocation decision of a multi-energy power system under the current state, r. t To enable each power source unit within a multi-energy power system to perform a t The reward value obtained afterwards, s t+1 Performing power allocation decisions for multi-energy power systems t The subsequent environmental state.

[0024] In the step S9, the SAC algorithm is an off-policy algorithm developed for maximum entropy reinforcement learning, and the experience replay area is used to store historical data in the form of tuples to update the parameters of the Vcritic network, the Qcritic network and the policy network to improve the training stability. A copy of the value network, i.e., the target value network, is developed as a supervised network to estimate the target value of training. The target value changes slowly due to the soft update of network parameters, and has the advantages of high sample efficiency and strong robustness.

[0025] In the step S10, a comprehensive evaluation is performed on the system, including the prediction accuracy of the model, the accuracy and speed of the power allocation decision, etc. The trained result μ(sθ μ ) network is used as a multi-energy power system active power allocation controller to realize dynamic allocation of multi-source power generation power of the multi-energy power system. The real-time operation environment data of the multi-energy power system and the trained multi-energy power system active power dynamic allocation model are loaded, the wind and light unit available power prediction accuracy is compared, the consistency of the algorithm variable space and time is determined, and the necessity of re-adjusting the parameters is determined. Then, the multi-energy power system running according to the power allocation decision of the model is evaluated, and the accuracy and speed when participating in the overall grid frequency control are evaluated.

[0026] The beneficial effects of the present application are: through the synergistic optimization mechanism of the GRU neural network and the SAC algorithm, the dynamic allocation efficiency and response speed of the multi-energy power system are significantly improved, and the synergistic effect is realized, and the overall performance is significantly improved beyond the effect of simply superimposing the two technologies. The GRU neural network provides high-precision renewable energy power generation prediction, and the SAC algorithm performs adaptive power allocation optimization on this basis, thereby ensuring fast response of the system while improving overall economic benefit and operation stability. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 It is the internal structure diagram of the gate recurrent unit of the present application;

[0028] Figure 2 It is the SAC algorithm framework diagram of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application, which will be specifically described below in combination with the embodiments.

[0030] As Figures 1-2 shown, the present application includes the following steps:

[0031] S1: Collect historical operating data of multi-energy power systems; the specific process is as follows:

[0032] The internal structure of the GRU neural network is as follows: Figure 1 As shown, the steps for obtaining training data are as follows:

[0033] Step S1.1: Collect historical operating data of the multi-energy power system, including wind speed V of the wind farm and the available power S of the wind farm at the corresponding time. wg The solar irradiance In of the photovoltaic power station and the outdoor temperature T out and the available power S of the photovoltaic power station at the corresponding time. pv The current system frequency deviation Δf and the corresponding available power S of the thermal power unit. th The sampling interval is every 15 minutes. The available power of wind farms and photovoltaic power stations is obtained from the difference between the full-capacity power and the grid-connected power under the current weather conditions.

[0034] Step S1.2: Collect historical operating data of the multi-energy power system and corresponding meteorological information. Input the GRU neural network model. Where T... out Let V be the outdoor temperature, V be the wind speed, In be the irradiance, and Δf be the system frequency deviation; its output is the available power of the wind farm and photovoltaic power station at n future moments, and the variables are represented as follows: in, These are the available power outputs of wind farms and photovoltaic power plants, respectively.

[0035] S2: Preprocess the collected data, correct outliers, and convert the time series data into supervised sequence data; the specific process is as follows:

[0036] In step S2, the collected data is preprocessed to remove invalid data, fill in missing data, and perform data smoothing to divide the data into 5000 training sets and 2000 test sets.

[0037] S3: Input the input values ​​into the GRU neural network and initialize the GRU neural network weights to train the model; the specific process is as follows:

[0038] The GRU neural network is configured with 100 neurons, 1000 training iterations, a learning rate of 0.001, and a batch size of 32 for minimum batch gradient descent.

[0039] Step S3.1: Initialize the model parameters by inputting the input values ​​into the first block of the GRU neural network. The internal structure diagram of the GRU model is shown below. Figure 1 As shown, the GRU neural network update process is as follows:

[0040] The update formula for the updated door is:

[0041] z t =σ(ω) z ·[h t-1 ,x t ])

[0042] The formula for resetting the door is:

[0043] r t =σ(ω) r ·[h t-1 ,x t ])

[0044] The expression for the candidate hiding function is as follows:

[0045]

[0046] The final update formula for the output on the time series is:

[0047]

[0048] Let the number of iterations s = 1, where the expression for the loss function L of the GRU model is as follows:

[0049]

[0050] In the formula, Y t h represents the true value corresponding to the subsequence or residual component. t Here, N represents the model's predicted value, and N is the number of training samples used in the GRU model training. The Sigmoid activation function expression is as follows:

[0051]

[0052] The hyperbolic tangent activation function expression is as follows:

[0053]

[0054] The tanh function maps input values ​​to a range between -1 and 1. Here, e is the base of the natural logarithm. The function e x Let e ​​represent an exponential function, where e increases as x increases. x Rapid increase; e -x This indicates exponential decay; as x increases, e... -x Rapidly decrease.

[0055] Step S3.2: Forward compute the output value of each neuron, i.e., calculate z. t r t , h t The values ​​of the four vectors;

[0056] Step S3.3: Backpropagation of the error term for each neuron. The backpropagation of the GRU error term has two directions: one is backpropagation along the time direction, that is, starting from the current time t, calculating the error term at each time step; the other is propagating the error term to the next layer.

[0057] Step S3.4: Determine whether the loss function has converged and whether the maximum number of iterations has been reached. If either of these conditions is met, stop updating; otherwise, set the number of iterations s = s + 1 and return to step S3.2.

[0058] Step S3.5: The final output is h. t That is, the prediction results of the GRU neural network on historical data;

[0059] Step S3.6: Calculate the true value Y based on the loss function L t Compared with the predicted value h t The error between the true and predicted values ​​is minimized through multiple iterations of training and updating of the neural network parameters using the minimum batch gradient descent method. This ultimately yields a predictive model for the available power of wind farms and photovoltaic power plants within a multi-energy power system.

[0060] In the above expressions, ω z It is the weight coefficient matrix of the update gate; ω r This is the weight coefficient matrix of the reset gate; ω is the weight coefficient matrix during state update; σ and tanh are the sigmoid and hyperbolic tangent activation functions, respectively, and h t-1 This represents the output of the previous one, x. t z represents the current input. t r t h t These are the results of the update gate, the reset gate, and the time series output, respectively.

[0061] S4: Use the test set to test the accuracy of the GRU neural network prediction model; the specific process is as follows:

[0062] The accuracy of the GRU prediction model is evaluated using a test set. Evaluation metrics include root mean square error (RMSE) and , which are defined as follows:

[0063]

[0064] In the formula Y i h i For each time step, the corresponding true value and predicted value are... is the mean of the real samples at each step, and n is the number of samples in the test set.

[0065] S5: Define the state space of the reinforcement learning algorithm, fully considering the observed environmental information and the prediction results of the GRU neural network, to ensure that the model can accurately perceive the system's state and make corresponding decisions during the SAC parameter learning process; the specific process is as follows:

[0066] Based on the prediction results of the GRU neural network, the power system frequency deviation Δf corresponding to the prediction time of available power of wind farms and photovoltaic power plants is collected. t and the available power S of thermal power units th Together, they serve as state and environment information for reinforcement learning. The data is divided into a training set of 5000 and a test set of 2000. t , Δf t S th As environmental information for reinforcement learning, it is represented as:

[0067] Furthermore, five deep neural networks are configured for the deep reinforcement learning SAC algorithm: one Actor network, two V Critic networks (one V Critic network and one Target V Critic network), and two Q Critic networks. The policy network parameters θ, the double Q function, and the target network parameters ω are then evaluated. (1) and ω (2) V Ψ Value network parameters ψ and Target value network parameters By definition, the V Critic network and Q Critic network have the same structure, using a three-layer network with neurons in each layer connected in pairs, and the activation function is the linear rectified function ReLU; the Actor network also uses a three-layer network with neurons in each layer connected in pairs, and the activation function is the hyperbolic tangent function tanh.

[0068] S6: Determine the action exploration strategy and define the action space. The specific process is as follows:

[0069] The current state information S in the training set t The current neural network input to the Actor is further used to construct a probability distribution based on the parameters output by the Actor to generate the controller action 'a'. t a t ∈[P min ,P max ]. θ μ P represents the current neural network parameters of the Actor. min and P max These represent the lower limit of power for load shedding operation of power supply units within a multi-energy power system and the upper limit of power that can be achieved through increased generation;

[0070] S7: Define the reward function for the deep reinforcement model; the specific process is as follows:

[0071] By combining the high-precision prediction results provided by the GRU neural network, the SAC algorithm can perform decision optimization more effectively. This ensemble approach leverages the predictive power of GRU to enhance the action selection and policy adjustment of the SAC algorithm, ensuring a more accurate and adaptive control strategy. A reward function for the deep reinforcement model is defined to guide the parameter optimization of the active power allocation model in a multi-energy power system. The power allocation command of the multi-energy power system is executed, and then an immediate reward r is obtained. t , and with a t The corresponding calculated value s for updating the environmental state t+1 reward function r t Setting and adjusting the power generation ΔP of each controlled unit in a multi-energy power system i out Response to AGC power generation command ΔP order-Σ The accuracy and speed are related. The coordinated control objective of a multi-source power system is expressed as follows:

[0072]

[0073] Where χ is a penalty term, χ = 0 when the preset system safety operation constraints are met; otherwise, χ = -0.5.

[0074] S8: Use Record all empirical data on active power distribution in a multi-energy power system and represent it as... The specific process is as follows:

[0075] In the formula, s t Indicates the current state, a t This represents the power allocation decision of a multi-energy power system under the current state, r. t To enable each power source unit within a multi-energy power system to perform a t The reward value obtained afterwards, s t+1 Performing power allocation decisions for multi-energy power systems t The subsequent environmental state;

[0076] Step S8.1: (s t ,a t ,r t ,s t+1 Store in an experience pool middle;

[0077] Step S8.2: By integrating the predicted output of the GRU neural network with the SAC policy learning process, the decision quality of the SAC algorithm is enhanced using an experience replay mechanism. In this process, the prediction results provided by the GRU are used to optimize state evaluation and action selection in the SAC algorithm, thereby improving the overall policy performance. Through the experience replay mechanism, historical interaction data is stored in tuples for subsequent training batches. The parameters of the V Critic network, Q Critic network, and policy network are updated using mini-batch samples drawn from the experience pool. In the early stages of training, due to the scarcity and high temporal correlation of experience data, direct use may lead to parameter instability. Therefore, an experience pool is established. After the internal data volume is greater than the initial accumulated experience volume CM, it is taken from the experience pool. M samples are randomly selected from the sample, (s i ,a i ,s i+1 ), i = 1, 2, ..., M;

[0078] S9: Update SAC network parameters; the specific process is as follows:

[0079] Neural networks are used to parameterize the V-Critic network, Q-Critic network, and policy function, where the policy function outputs the probability distribution of actions. During training, the parameters of the neural network are updated based on rewards from environmental feedback. Furthermore, the prediction data generated by the GRU neural network is used to guide the decision-making process of the SAC algorithm, influencing its action selection and policy optimization, thereby ensuring consistency between prediction-based decisions and the actual environmental state.

[0080] Step S9.1: For the Q-network, the loss function is defined as:

[0081] J Q(ω) =E (s,a)～D [(Q ω (s,a)-y q ) 2 ]

[0082] in Here, Q is the target Q-value, and D is the empirical replay pool. The gradient update formula for the Q-network is:

[0083] Step S9.2: The objective function of the value network is optimized by minimizing the sum of squared residuals.

[0084]

[0085] in, Two Q-networks are used to reduce estimation bias; the gradient update formula is as follows:

[0086]

[0087] The actions To obtain samples from the current strategy.

[0088] Step S9.3: The policy network updates its parameters by minimizing the following objective function:

[0089]

[0090] Where, f θ (ξ;s) represents the actions sampled from the policy using a reparameterization technique, where ξ is random noise from a standard normal distribution. This form of objective function helps balance exploration and exploitation while controlling the randomness of the policy by adjusting the entropy regularization coefficient α.

[0091] Step S9.4: Target Value Network The parameters are updated through a soft update mechanism to maintain training stability.

[0092]

[0093] Where τ is the soft update coefficient.

[0094] Step S9.6: Treat the state of the next time step as the state of the current time step: i.e., s t ←s t+1 The training loop iterates, using the reward value obtained in each iteration and the error value L at each step as the evaluation metric for network convergence. After training on the training set for M epochs (m = M), μ(s) t θ μ The network is the training result. The output neural network parameters θ are... μ The power allocation model of the multi-energy power system is obtained. Proceed to step S10.

[0095] S10: Conduct a comprehensive evaluation of the system, including aspects such as the model's prediction accuracy, the accuracy and speed of power allocation decisions; the specific process is as follows:

[0096] Training complete. The trained Actor target neural network is used as the active power distribution controller for the multi-energy power system, and an 'a' is selected based on the optimal strategy obtained during training. t Next, each power supply unit within the multi-energy power system will, according to a t Adjusting power generation capacity allows for the evaluation of the accuracy and speed of the multi-energy power system as a whole in the grid frequency response process.

[0097] This invention significantly improves the dynamic power allocation efficiency and response speed of multi-energy power systems through a collaborative optimization mechanism of GRU neural network and SAC algorithm, achieving a synergistic effect. Its overall performance improvement significantly exceeds the effect of simply combining the two technologies. The GRU neural network provides high-precision renewable energy generation prediction, and the SAC algorithm performs adaptive power allocation optimization based on this, thereby ensuring rapid system response while improving overall economic efficiency and operational stability.

[0098] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the equivalent elements of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for dynamic active power allocation in a multi-energy power system based on GRU and DRL, characterized in that, Includes the following steps: S1: Collect historical operating data of multi-energy power systems; S2: Preprocess the collected data, correct abnormal data, and convert the time series data into supervised sequence data; S3: Input the input quantities into the GRU neural network and initialize the GRU neural network weights to train the model; S4: Test the accuracy of the GRU neural network prediction model using the test set; S5: Define the state space of the reinforcement learning algorithm, fully considering the observed environmental information and the prediction results of the GRU neural network, to ensure that the model can accurately perceive the state of the system and make corresponding decisions during the SAC parameter learning process. S6: Determine the action exploration strategy and define the action space. S7: Define the reward function for the deep reinforcement model; S8: Use Record all empirical data on active power distribution in a multi-energy power system and represent it as... S9: Update SAC network parameters; S10: Conduct a comprehensive evaluation of the system.

2. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S1, the historical operating data of the multi-energy power system includes the wind speed V of the wind farm and the available power S of the wind farm at the corresponding time. wg The solar irradiance In of the photovoltaic power station and the outdoor temperature T out and the available power S of the photovoltaic power station at the corresponding time. pv The current system frequency deviation Δf and the corresponding available power S of the thermal power unit. th The available power of each power unit is the difference between the theoretical full-capacity power of the unit and the actual grid-connected power.

3. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S2, the data is divided into a training set and a test set.

4. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S3, the structure of the GRU model is determined, including the number of layers and neurons in each layer of the neural network model, the activation function and the loss function, and the model parameters are initialized, including wind speed V, light intensity In, and outdoor temperature T. out System frequency deviation Δf and the corresponding available power S of the wind farm at that time. wg and the available power S of photovoltaic power plants pv Import the GRU neural network for training to obtain model parameters.

5. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S4, the GRU model is calibrated using a test set to obtain a prediction model for the available power of wind farms and photovoltaic power plants in a multi-energy power system.

6. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S5, based on the prediction results of the GRU neural network, the power system frequency deviation Δf corresponding to the available power prediction time of wind farms and photovoltaic power plants is collected. t and the available power of thermal power units Together, they serve as state and environment information for reinforcement learning, and h t , Δf t , As environmental information for reinforcement learning, it is represented as:

7. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S6, the current state information is input into the Actor network of the deep reinforcement learning algorithm SAC. A multinomial distribution is constructed based on the parameters output by the Actor to generate actions. This multinomial distribution is constructed based on the disturbance information in the historical data collected by the power system and is associated with the time series.

8. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S7, the reward function of the deep reinforcement learning model is constructed to take into account both the accuracy and speed of multi-energy power systems participating in grid frequency regulation.

9. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S8, s t Indicates the current state, a t This represents the power allocation decision of a multi-energy power system under the current state, r. t To enable each power source unit within a multi-energy power system to perform a t The reward value obtained afterwards, s t+1 Performing power allocation decisions for multi-energy power systems a t The subsequent environmental state.

10. The method for dynamic active power allocation in a multi-energy power system based on GRU and DRL according to claim 1, characterized in that: In step S9, the SAC algorithm is an off-policy algorithm developed for maximum entropy reinforcement learning. It utilizes the experience replay area to store historical data in the form of tuples to update the parameters of the Vcritic network, Qcritic network, and policy network to improve training stability.

Images