A method for load frequency control of power system using deep reinforcement learning

Abstract

The present invention provides a method for controlling the load frequency of an electric power system by using deep reinforcement learning, and relates to the technical field of electric power system control. The method comprises: constructing an electric power system model, wherein the electric power system model comprises a power generation unit, a power transmission network, a load unit, and a frequency control device; obtaining historical operation data of the electric power system, including load demand records, power generation output logs, and frequency fluctuations, and preprocessing the historical operation data of the electric power system to obtain processed historical operation data of the system; training a deep neural network model according to the processed historical operation data of the system, extracting key features of the historical operation data of the system, and predicting future load demand and frequency change trends to obtain prediction results. The present invention utilizes deep reinforcement learning to fuse the electric power system model to achieve accurate prediction and automatic real-time control of load frequency, and improve the stability and economy of the electric power system.

Description

Technical Field

[0001] The present invention relates to the technical field of power system control, and in particular to a method for controlling power system load frequency by using deep reinforcement learning. Background Art

[0002] Traditional methods, such as PI control, are based on fixed mathematical models and parameters. These models and parameters are derived under specific conditions. When the power system structure (such as new generators, line modifications, etc.) or load characteristics (such as load fluctuations, changes in user behavior, etc.) undergo significant changes, the original models and parameters may no longer be applicable, resulting in a decrease in control effect. Due to the lack of real-time update and adaptive adjustment capabilities, it may seem powerless in the face of dynamic changes in the power system, and manual intervention is required to readjust parameters or models, which increases the complexity and workload of operation and maintenance.

[0003] In the face of various disturbances (such as sudden load changes, line failures, etc.) and uncertain factors (such as the randomness of renewable energy generation, changes in meteorological conditions, etc.), it may lack sufficient robustness. These disturbances and uncertain factors may cause the power system frequency to deviate from the set value, and may not be able to respond quickly and accurately, making it difficult to maintain the stability and accuracy of frequency control.

[0004] Due to the lack of systematic optimization methods and tool support, the optimization process may be time-consuming and labor-intensive, with poor results, and cannot meet the increasingly high control performance requirements of the power system. In addition, the optimization results are limited by human experience and technical level, making it difficult to achieve true intelligent and automated control. The lack of self-learning and intelligent decision-making capabilities makes it impossible to dynamically adjust and optimize control strategies based on historical data and real-time status of power system operation. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a method for controlling the load frequency of an electric power system using deep reinforcement learning, which can more accurately perform frequency control according to the actual operating status of the electric power system.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] In a first aspect, a method for controlling load frequency of a power system using deep reinforcement learning is provided, the method comprising:

[0008] Constructing a power system model, wherein the power system model includes a power generation unit, a transmission network, a load unit, and a frequency control device;

[0009] Obtaining historical operation data of the power system, including load demand records, power generation output logs, and frequency fluctuations, and preprocessing the historical operation data of the power system to obtain processed historical operation data of the system;

[0010] Based on the processed system historical operation data, a deep neural network model is trained to extract the key features of the system historical operation data, and the future load demand and frequency change trends are predicted to obtain the prediction results;

[0011] Based on the prediction results, the load frequency control scheme is determined through continuous iterative optimization through reinforcement learning;

[0012] The trained deep neural network model is integrated with reinforcement learning to build a deep reinforcement learning controller to achieve automatic and real-time control of load frequency in the power system.

[0013] Furthermore, based on the processed system historical operation data, a deep neural network model is trained to extract the key features of the system historical operation data, and to predict the future load demand and frequency change trend, and obtain the prediction results, including:

[0014] The processed system historical operation data is divided into a training set, a validation set, and a test set, and the gated recurrent unit time series prediction model is used as a deep neural network model;

[0015] Define the architecture of the deep neural network model, including input layer, hidden layer and output layer, and use mean absolute error as the loss function;

[0016] The training set is input into the deep neural network model to train the deep neural network model. During the training process, the deep neural network model is verified using the verification set data to obtain the verification result.

[0017] Adjust the deep neural network model architecture and parameters based on the verification results;

[0018] The verified deep neural network model is used to predict the test set and obtain the future load demand and frequency change trend prediction results.

[0019] Furthermore, the calculation formula of the mean absolute error is:

[0020] ;

[0021] in, represents the mean absolute error; represents the total number of time steps for the forecast; Indicates The actual load demand for each time step; Indicates The predicted value for each time step; represents the input feature vector; represents the model parameter set; Represents environment variables; represents the weight coefficient; represents the global adjustment coefficient; The index value representing the time step.

[0022] Furthermore, the training set is input into the deep neural network model to train the deep neural network model. During the training process, the deep neural network model is verified using the verification set data to obtain verification results, including:

[0023] The training set is input into the deep neural network model for training, and in each training cycle, the deep neural network model traverses the training set;

[0024] Through forward propagation, the deep neural network model calculates the predicted value, and through back propagation, the deep neural network model parameters are updated;

[0025] After each training cycle, the deep neural network model is verified using the validation set, and the verification results are obtained by comparing the prediction results of the validation set with the actual data.

[0026] Furthermore, based on the prediction results, through reinforcement learning and continuous iterative optimization, the load frequency control scheme is determined, including:

[0027] Define the state space, action space and reward function of reinforcement learning, and initialize the value network parameters;

[0028] Obtain the current state of the power system from the state space, determine the control action based on the current state, and initialize the value network;

[0029] The deep neural network model is used to predict the impact of the determined control action on future load demand and frequency change trends, and the reward value after the action is executed is calculated;

[0030] Use the calculated reward value to update the network parameters of reinforcement learning, and repeat state acquisition, action execution and future prediction, reward calculation and parameter update until the preset number of iterations is reached, extract the control scheme, and use the control scheme as the load frequency control scheme.

[0031] Furthermore, the calculation formula of the reward function is:

[0032] ;

[0033] in, Indicates in status Take action After transfer to state Rewards received when represents the weight of frequency deviation; Indicates the frequency of the power system after the action is performed; Indicates the set frequency of the power system; The weight representing the matching degree between power generation and load; Indicates the power generation output after the action is executed; Indicates the load requirement after performing the action; represents the weight of the stability index; Indicates the weight of voltage stability in the stability index; Indicates the voltage value after the action is performed; represents the weight of current stability in stability calculation; Indicates the current value after the action is performed; represents the weight of the total cost; represents the power generation cost coefficient; Represents the equipment maintenance cost.

[0034] Furthermore, the trained deep neural network model is integrated with reinforcement learning to construct a deep reinforcement learning controller to automatically and in real time control the load frequency in the power system, including:

[0035] The power system model is used to build a reinforcement learning environment, and the deep reinforcement learning controller is used for trial-and-error learning in the reinforcement learning environment. During the trial-and-error process, the deep reinforcement learning controller determines the action based on the current state, and accepts the reward and new state information fed back by the environment to obtain the trial-and-error learning result.

[0036] According to the trial-and-error learning results, the deep reinforcement learning controller continuously updates the control scheme until the deep reinforcement learning controller learns how to determine the final action based on the current state and future predictions, thus obtaining a trained deep reinforcement learning controller;

[0037] The trained deep reinforcement learning controller is integrated into the power system to automatically and in real time control the load frequency in the power system.

[0038] In a second aspect, a system for controlling load frequency of a power system using deep reinforcement learning includes:

[0039] A construction module, used to construct a power system model, wherein the power system model includes a power generation unit, a power transmission network, a load unit and a frequency control device;

[0040] An acquisition module is used to acquire the historical operation data of the power system, including load demand records, power generation output logs, and frequency fluctuations, and to pre-process the historical operation data of the power system to obtain the processed historical operation data of the system;

[0041] The prediction module is used to train a deep neural network model based on the processed system historical operation data, extract key features of the system historical operation data, and predict future load demand and frequency change trends to obtain prediction results;

[0042] The processing module is used to determine the load frequency control plan based on the prediction results through reinforcement learning and continuous iterative optimization; the trained deep neural network model is integrated with reinforcement learning to build a deep reinforcement learning controller to achieve automatic and real-time control of the load frequency in the power system.

[0043] According to a third aspect, a computing device includes:

[0044] one or more processors;

[0045] The storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0046] In a fourth aspect, a computer-readable storage medium stores a program, and when the program is executed by a processor, the method described is implemented.

[0047] The above solution of the present invention includes at least the following beneficial effects:

[0048] By building a power system model and training it with historical operation data, the deep neural network model can accurately extract the key features of the system's historical operation, thereby more accurately predicting future load demand and frequency change trends. This high-precision prediction provides a reliable foundation for load frequency control, enabling the control scheme to more accurately respond to the actual needs of the power system and improve control accuracy and stability. The deep reinforcement learning controller is not only trained based on historical data, but also continuously iterates and optimizes the control strategy through reinforcement learning. This self-learning and optimization capability enables the controller to adaptively adjust parameters to cope with changes in the power system structure or load characteristics. The deep reinforcement learning controller combines the advantages of deep neural networks and reinforcement learning and has the ability to make intelligent decisions. It can automatically adjust the control strategy according to the real-time status and historical data of the power system to achieve efficient optimization. Compared with traditional methods, the deep reinforcement learning controller does not need to rely on manual experience and trial and error for adjustment and optimization, which reduces the complexity and workload of operation and maintenance, while improving the control effect.

[0049] By accurately predicting load demand and frequency change trends, combined with the optimization control strategy of deep reinforcement learning, power resources can be allocated more effectively and waste can be reduced. This efficient resource allocation and optimized control strategy helps to improve the overall operating efficiency of the power system and reduce operating costs, while also helping to improve the reliability and quality of power supply. The application of deep reinforcement learning controllers is an important part of the intelligent upgrade of power systems. It not only improves the performance of load frequency control, but also provides new ideas and methods for other control tasks of power systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a method for controlling load frequency of a power system using deep reinforcement learning provided by an embodiment of the present invention.

[0051] Figure 2 It is a system schematic diagram of a power system load frequency control using deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] like Figure 1 As shown, an embodiment of the present invention proposes a method for controlling load frequency of a power system using deep reinforcement learning, the method comprising the following steps:

[0054] Step 11, constructing a power system model, wherein the power system model includes a power generation unit, a transmission network, a load unit, and a frequency control device;

[0055] Step 12, obtaining historical operation data of the power system, including load demand records, power generation output logs, and frequency fluctuations, and preprocessing the historical operation data of the power system to obtain processed system historical operation data;

[0056] Step 13: train a deep neural network model based on the processed system historical operation data, extract key features of the system historical operation data, and predict future load demand and frequency change trends to obtain prediction results;

[0057] Step 14, based on the prediction results, the load frequency control scheme is determined through continuous iterative optimization through reinforcement learning;

[0058] Step 15, integrate the trained deep neural network model with reinforcement learning to build a deep reinforcement learning controller to automatically and in real time control the load frequency in the power system.

[0059] In an embodiment of the present invention, by constructing a power system model including a power generation unit, a transmission network, a load unit, and a frequency control device, the actual operation status of the power system can be simulated in all aspects. This comprehensive model provides a solid foundation for subsequent deep learning and reinforcement learning, making the control scheme closer to reality, thereby improving the control effect. Acquiring and preprocessing historical operation data of the power system, including load demand records, power generation output logs, frequency fluctuations, etc., can mine the potential laws and key features in the data. These features and laws provide valuable information for the training of deep neural network models, thereby improving the accuracy of predicting future load demand and frequency change trends.

[0060] The deep neural network model is integrated with reinforcement learning to construct a deep reinforcement learning controller. This integration not only takes advantage of the deep neural network in feature extraction and prediction, but also makes full use of the reinforcement learning in strategy optimization and decision-making. The two complement each other to achieve intelligent control of the load frequency of the power system. Through the continuous iterative optimization of reinforcement learning, the load frequency control scheme can be gradually adjusted and improved, and stable and accurate frequency control can be achieved in the face of various complex power system operating conditions. The deep reinforcement learning controller can automatically and in real time control the load frequency in the power system. Through accurate prediction and intelligent control, power resources can be allocated more effectively to avoid waste. This efficient use of resources helps to reduce the operating cost of the power system and improve economic benefits. At the same time, stable frequency control can also extend the service life of power equipment and further reduce maintenance costs.

[0061] In a preferred embodiment of the present invention, the above step 11, constructing a power system model, the power system model includes a power generation unit, a power transmission network, a load unit and a frequency control device; the above step 12, obtaining the power system historical operation data, including load demand records, power generation output logs, frequency fluctuations, and pre-processing the power system historical operation data to obtain the processed system historical operation data, may include:

[0062] In an embodiment of the present invention, various types of power generation units are identified and defined, such as thermal power generation, hydropower generation, wind power generation, solar power generation, etc. Parameters such as power generation capacity, power generation efficiency, response time, fuel consumption rate, etc. are set for each power generation unit. A mathematical model of the power generation unit is established to describe the relationship between its power generation power and input (such as fuel quantity, water flow rate, wind speed, light intensity). A transmission network diagram of the power system is drawn, including transmission lines, substations, buses, etc. Parameters of the transmission line are set, such as resistance, reactance, susceptance, transmission capacity, etc. A power flow model is established to describe the flow and distribution of power in the transmission network. Different types of load units are classified and defined, such as industrial loads, commercial loads, residential loads, etc. A load demand model is established, taking into account the daily cycle, weekly cycle, seasonal changes of the load, and the impact of emergencies on the load. The types and characteristics of frequency control devices are defined, such as automatic generation control (AGC), load frequency control (LFC), etc. A mathematical model of the frequency control device is established to describe how it adjusts the output of the power generation unit according to the system frequency deviation.

[0063] Integrate the generation units, transmission network, load units and frequency control devices into a unified power system model. Set the initial and boundary conditions of the model, such as initial load demand, initial generation power, initial state of the transmission network, etc. Establish a simulation environment for the model to simulate the actual operation and response of the power system. Obtain historical operation data from the power system operation management system, including load demand records, generation output logs, frequency fluctuations, etc. Remove duplicates, invalid items and erroneous items in the data.

[0064] Interpolate or estimate missing data to complete the data set. Convert data into a format suitable for model processing, such as time series data, tabular data, etc. Normalize or standardize data to eliminate the impact of different dimensions and data ranges on model training. Extract key features from raw data, such as daily cycles, weekly cycles, seasonal changes in load demand, etc. Extract features such as stability and response speed of power generation output. Extract features such as frequency and amplitude of frequency fluctuations.

[0065] In a preferred embodiment of the present invention, the above step 13, based on the processed system historical operation data, trains a deep neural network model, extracts key features of the system historical operation data, and predicts future load demand and frequency change trends to obtain prediction results, which may include:

[0066] Step 133, dividing the processed system historical operation data into a training set, a validation set, and a test set, and using the gated recurrent unit time series prediction model as a deep neural network model;

[0067] Step 134, defining the architecture of the deep neural network model, including an input layer, a hidden layer, and an output layer, and using the mean absolute error as a loss function;

[0068] Step 135, inputting the training set into the deep neural network model, training the deep neural network model, and during the training process, using the validation set data to validate the deep neural network model to obtain a validation result;

[0069] Step 136, adjusting the deep neural network model architecture and parameters according to the verification results;

[0070] Step 137, use the verified deep neural network model to predict the test set to obtain the future load demand and frequency change trend prediction results.

[0071] In an embodiment of the present invention, the processed system historical operation data is divided into a training set, a validation set and a test set. The training set is used for model training, the validation set is used for model verification and parameter adjustment, and the test set is used to evaluate model performance. The division ratio can be flexibly adjusted according to the amount of data, and is generally recommended to be 70% training set, 15% validation set, and 15% test set. The gated recurrent unit (GRU) time series prediction model is selected as the deep neural network model. GRU is a variant of the recurrent neural network (RNN) that can effectively handle long-term dependency problems in time series data. The input layer receives the processed system historical operation data, such as load demand, power generation output, frequency fluctuation and other time series data. The hidden layer is composed of multiple layers of GRU units, which are used to extract key features and time dependencies in the data. The output layer outputs the future load demand and frequency change trend prediction results. Set other parameters of the model, such as learning rate, batch size, number of iterations, etc. The training set is input into the deep neural network model, and the model is trained using an optimization algorithm (such as Adam optimizer). During the training process, the validation set data is used to validate the model, the loss function value (such as mean absolute error) on the validation set is calculated, and the model architecture and parameters are adjusted according to the validation results.

[0072] According to the loss function value and prediction accuracy on the validation set, adjust the model architecture (such as increasing or decreasing the number of GRU layers, adjusting the number of hidden layer neurons) and parameters (such as learning rate, batch size). Repeat the training and validation process until the model achieves satisfactory performance on the validation set. Use the verified deep neural network model to predict the test set and obtain the future load demand and frequency change trend prediction results. Evaluate the prediction performance of the model on the test set, such as calculating indicators such as prediction accuracy and mean square error.

[0073] Suppose there is a power system whose historical operating data includes load demand, power generation output, and frequency fluctuations. We hope to use a deep neural network model to predict the load demand and frequency change trends for the next day.

[0074] The data of the past year are divided into a training set (70%), a validation set (15%), and a test set (15%). The GRU time series forecasting model is selected as the deep neural network model. The input layer receives the load demand, power generation output, and frequency fluctuation data of the past 24 hours. The hidden layer consists of two layers of GRU units, each containing 128 neurons. The output layer outputs the forecast results of the load demand and frequency change trend in the next 24 hours. The model is trained using the Adam optimizer, with the learning rate set to 0.001, the batch size set to 64, and the number of iterations set to 100. During the training process, the model is validated using the validation set data, and the mean absolute error is calculated as the loss function value. According to the loss function value and prediction accuracy on the validation set, the architecture and parameters of the model are adjusted. For example, increase the number of GRU layers to three, adjust the learning rate to 0.0005, etc.

[0075] Repeat the training and validation process until the mean absolute error of the model on the validation set drops to a low level. Use the validated deep neural network model to predict the test set and obtain the load demand and frequency change trend forecast results for the next day. Evaluate the prediction performance of the model on the test set, such as calculating the prediction accuracy of 95% and the mean square error of 0.01.

[0076] By dividing the data set, we can ensure that the model will not overfit to specific data during training, thereby improving the model's generalization ability to new data. The validation set is used to evaluate the model performance during the training process, and the test set is used to finally evaluate the model to ensure the accuracy and objectivity of the model evaluation. A clear model architecture helps to understand the working principle and optimization direction of the model. Choosing MAE as the loss function can intuitively reflect the absolute gap between the model's predicted value and the actual value, which is convenient for optimization and adjustment. Through real-time feedback from the validation set data, the model training process can be monitored, and overfitting or underfitting phenomena can be discovered and corrected in a timely manner. The validation results provide a basis for model parameter adjustment and help find the optimal model parameter combination. By continuously adjusting and optimizing the model architecture and parameters, the model's prediction performance and stability can be improved. A fully trained and validated model can provide more accurate and reliable prediction results.

[0077] In a preferred embodiment of the present invention, the calculation formula of the mean absolute error is:

[0078] ;

[0079] in, represents the mean absolute error; represents the total number of time steps for the forecast; Indicates The actual load demand for each time step; Indicates The predicted value for each time step; represents the input feature vector; represents the model parameter set; Represents environment variables; represents the weight coefficient; represents the global adjustment coefficient; The index value representing the time step.

[0080] In the embodiment of the present invention, historical operation data of the power system is collected and sorted, including load demand, input feature vectors, environmental variables, etc. The integrity and accuracy of the data are ensured, and missing or abnormal data are processed. The total number of time steps for prediction is set. , determined according to actual needs. Initialize the model parameter set , which includes weights and biases in the Gated Recurrent Unit (GRU). Determine the weight coefficients and the global adjustment factor , these coefficients can be adjusted according to actual conditions to reflect the importance of different time steps or different load demands. The deep neural network model is trained using the training set data, and the model parameter set is continuously adjusted during the training process. , in order to minimize the mean absolute error . After each training cycle, use the validation set data to validate the model and evaluate the model performance. According to the validation results, adjust the model architecture (such as increasing or decreasing the number of GRU layers, adjusting the number of hidden layer neurons) and parameters (such as learning rate, batch size). Repeat the training and validation process until the model achieves satisfactory performance on the validation set. Use the validated deep neural network model to predict the test set and obtain the future load demand and frequency change trend prediction results. Calculate the mean absolute error on the test set , to evaluate the model’s predictive performance. For each time step , calculate the actual load demand With the predicted value The absolute error between each time step is multiplied by the corresponding weight coefficient. , and sum them. Divide the sum by the total number of time steps , then multiplied by the global adjustment factor , and get the final mean absolute error .

[0081] Suppose there is a power system whose historical operation data includes load demand, power output, weather conditions, etc. We want to use a deep neural network model to predict the load demand for the next day. Collect the load demand data, power output data, weather condition data, etc. for the past year. Preprocess the data, such as removing outliers, filling missing values, etc. Set the total number of time steps for the prediction =24 (forecast load demand for the next day, one time step per hour). Initialize the model parameter set , including weights and biases in GRU. Set weight coefficients = 1 (assuming the same weight at each time step) and a global adjustment factor =1 (no additional adjustment). Use the training set data (such as data from the past year) to train the deep neural network model. During the training process, the model parameter set is continuously adjusted. , in order to minimize the mean absolute error . Use validation set data (such as data from the past month) to validate the model.

[0082] According to the verification results, adjust the model architecture and parameters, such as increasing the number of GRU layers, adjusting the learning rate, etc. Use the verified deep neural network model to predict the test set (such as the data of the last week). Obtain the load demand forecast result for the next day. For each time step (from 1 to 24), calculate the actual load demand With the predicted value The absolute error between the two time steps. Sum the absolute errors of each time step. Divide the sum by 24 (the total number of time steps) to get the final mean absolute error .

[0083] For example, if the actual load demand at a time step is 100MW and the predicted value is 95MW, the absolute error at that time step is 5MW. The mean absolute error is obtained by summing the absolute errors of all time steps and dividing by the total number of time steps.

[0084] M directly calculates the absolute difference between the predicted value and the actual value and takes the average value, so it can intuitively reflect the prediction accuracy of the model. Since M gives the same weight to positive and negative errors, it will not be biased due to high or low predicted values. By introducing the weight coefficient , the error can be weighted according to the importance of different time steps, so that the model pays more attention to the prediction accuracy of the key time period. Environmental variables This allows the model to take into account the impact of external factors on load demand, such as weather, holidays, etc., thereby improving the accuracy and robustness of the forecast. Global adjustment coefficient The introduction of provides additional flexibility for model optimization. In practical applications, it can be adjusted according to specific needs. The value of to balance prediction accuracy and other performance indicators (such as computational efficiency, model complexity, etc.). By continuously adjusting the model parameter set , and combined with the M value on the validation set for feedback, the model performance can be gradually optimized to better adapt to changes in actual load demand. Models trained and validated based on historical data can better capture the changing patterns of load demand, thereby improving the generalization ability of new data. By continuously optimizing the model architecture and parameters, the model can better adapt to the changing trends of load demand and frequency in different scenarios, and improve the accuracy and stability of predictions.

[0085] In a preferred embodiment of the present invention, the above step 135, inputting the training set into the deep neural network model, training the deep neural network model, and during the training process, using the validation set data to validate the deep neural network model to obtain the validation result, may include:

[0086] Step 1351, inputting the training set into the deep neural network model for training, and in each training cycle, the deep neural network model traverses the training set;

[0087] Step 1352, the deep neural network model calculates the prediction value through forward propagation, and updates the deep neural network model parameters through back propagation;

[0088] Step 1353, after each training cycle is completed, the deep neural network model is verified using the verification set, and the verification result is obtained by comparing the prediction result of the verification set with the actual data.

[0089] In an embodiment of the present invention, the training set and validation set data are collected and organized to ensure the quality and consistency of the data. The data is preprocessed, such as normalization, standardization or feature extraction, to meet the input requirements of the deep neural network model. Select a suitable deep neural network architecture, such as a gated recurrent unit (GRU) or other time series prediction model. Initialize the model parameters, including weights and biases, usually using random initialization or pre-training initialization. Traverse the training set: In each training cycle, the model will traverse the entire training set, usually in a small batch (batch) manner. For each small batch of data, the model performs forward propagation calculations to obtain predicted values. Use a loss function (such as mean absolute error) to calculate the difference between the predicted value and the actual value. According to the gradient of the loss function, the parameters of the model are updated by the back propagation algorithm to reduce the prediction error. After each training cycle, the model is verified using the validation set. By comparing the prediction results of the validation set with the actual data, the loss or other performance indicators (such as accuracy, recall, etc.) on the validation set are calculated. Adjust the model parameters or training strategies according to the verification results, such as adjusting the learning rate, increasing or decreasing the training cycle, etc. Repeat the training and validation process until the model achieves satisfactory performance on the validation set. Use the test set to test the final model and evaluate its performance in real-world applications.

[0090] Suppose there is a task of load forecasting for a power system, the goal is to predict the hourly load demand for the next day. We have a training set containing hourly load data for the past year, and a validation set containing hourly load data for the past month. Normalize the training set and validation set data to scale the load values ​​to the range [0, 1]. Extract time features (such as hours, days of the week, months, etc.) as part of the input feature vector. Select a deep neural network model with two GRU layers. Initialize the model parameters, set the learning rate to 0.001, and the batch size to 64. In each training cycle, the model traverses the entire training set, processing 64 samples each time. Calculate the predicted value through forward propagation, and use the mean absolute error as the loss function. Update the model parameters through backpropagation to reduce the prediction error. After each training cycle, use the validation set to validate the model. Calculate the mean absolute error on the validation set and monitor its changing trend. If you find that the error on the validation set starts to rise, it may be that the model is overfitting, and you need to adjust the learning rate or increase the regularization term. After multiple training and validation, the mean absolute error of the model on the validation set reaches a low level. The final model is tested using the test set to evaluate its predictive performance in practical applications.

[0091] Through repeated forward propagation and back propagation, the model parameters can be gradually adjusted to the optimal value, making the predicted value closer to the actual data. Deep neural networks can automatically learn and extract useful features from training data. The validation step after each training cycle provides real-time feedback, allowing developers to monitor the performance of the model on unseen data. By comparing the performance on the training set and the validation set, it is possible to promptly find out whether the model is overfitting, that is, whether it performs well on the training set but poorly on the validation set. The validation set usually contains data that is different from but related to the training set, which helps the model learn more common features and improve generalization ability. The performance on the validation set can be used as a basis for adjusting hyperparameters (such as learning rate, batch size, network structure, etc.). Through feedback on the validation set, the optimal hyperparameter combination can be found faster, reducing training time and computing resources. By validating on multiple models, the best performing model can be selected for integration to further improve prediction accuracy. Models that have been fully trained and validated are more reliable in practical applications and can reduce decision-making risks caused by prediction errors. Accurate prediction results can enhance users' trust in the model and improve the acceptance and frequency of model use in practical applications.

[0092] In a preferred embodiment of the present invention, the above step 14, based on the prediction results, through reinforcement learning, continuous iterative optimization, and determination of the load frequency control scheme may include:

[0093] Step 144, defining the state space, action space and reward function of reinforcement learning, and initializing the value network parameters;

[0094] Step 145, obtaining the current state of the power system from the state space, determining the control action according to the current state, and initializing the value network;

[0095] Step 146, predicting the impact of the determined control action on future load demand and frequency change trend through the deep neural network model, and calculating the reward value after executing the action;

[0096] Step 147, using the calculated reward value, updates the network parameters of the reinforcement learning, and repeats the state acquisition, action execution and future prediction, reward calculation and parameter update until a preset number of iterations is reached, extracts the control scheme, and uses the control scheme as the load frequency control scheme.

[0097] In an embodiment of the present invention, the state space of the power system is defined, including key variables such as the current load, frequency, and generator state. Possible control actions are determined, such as adjusting the output power of the generator, switching capacitor banks, etc. A reward function is designed to give positive or negative rewards according to the stability and economy of the load frequency control. The value network parameters are initialized, usually using random initialization or pre-training parameters. Hyperparameters such as learning rate and discount factor are set. The real-time state of the current power system is obtained from the state space. The control action is selected according to the current state and the reinforcement learning strategy (such as Q learning, deep Q network, etc.). A deep neural network model is used to predict the impact of the determined control action on the future load demand and frequency change trend. The reward value after the action is executed is calculated according to the prediction results and the reward function. The network parameters of the reinforcement learning are updated using the calculated reward value, such as optimizing the value network by the gradient descent method. The process of state acquisition, action execution, future prediction, reward calculation and parameter update is repeated until the preset number of iterations or convergence conditions are reached. The control scheme after iterative optimization is extracted. The control scheme is applied to the actual load frequency control system for real-time control.

[0098] Assume that there is a power system consisting of multiple generators and loads, and the goal is to optimize the load frequency control scheme through reinforcement learning to maintain the system frequency near the rated value.

[0099] State space, including current system frequency, output power of each generator, load demand, etc.

[0100] The action space includes adjusting the increase or decrease of the generator output power, the number of capacitor banks switched on and off, etc.

[0101] Reward function, designed according to factors such as system frequency deviation and the economic efficiency of control actions (such as power generation cost). Use deep Q network (DQN) as reinforcement learning model and initialize the parameters of DQN. Set the learning rate to 0.001 and the discount factor to 0.99. Obtain the current state of the power system in real time, including frequency, generator output power and load demand. Select control actions according to the DQN strategy, such as increasing or decreasing the output power of a certain generator. Use the deep neural network model to predict the load demand and frequency change trend in the future after the control action is executed. Calculate the reward value after the action is executed according to the prediction results and the reward function. If the frequency deviation is reduced, a positive reward is given, otherwise a negative reward is given. Use the reward value to update the parameters of DQN and optimize the value network by gradient descent. Repeat the above process until the preset number of iterations is reached or the DQN strategy converges. Extract the control scheme after iterative optimization, that is, the DQN strategy. Apply the control scheme to the actual load frequency control system, and maintain the stability of the system frequency by adjusting the generator output power and other control measures in real time.

[0102] By comprehensively defining the state space of the power system, including multiple information such as load, frequency, and generator status, the control scheme can perceive the system status more comprehensively and make more accurate control decisions.

[0103] Exploring in a wide range of action spaces enables the control scheme to learn a variety of possible control strategies, enhancing its adaptability to different operating conditions and disturbances. The deep neural network model is used to predict the impact of control actions on future load demand and frequency change trends, improving the accuracy and reliability of the prediction. By acquiring the power system state in real time and adjusting it according to the prediction results, the control scheme can respond quickly to system changes and improve the control effect. The network parameters of reinforcement learning are continuously updated using the calculated reward value, and the strategy gradually converges to the final solution through iterative optimization. Through the combination of deep neural networks and reinforcement learning, efficient search and optimization of control strategies are achieved, reducing the waste of computing resources. The optimized control scheme can more accurately adjust the generator output power and load demand, reducing the operating cost of the system and improving the economy. The control scheme obtained through reinforcement learning training has intelligent decision-making capabilities and can automatically select the optimal control action according to the power system state. The control scheme is applied to the actual load frequency control system to achieve automatic control, reduce manual intervention, and improve operating efficiency.

[0104] In a preferred embodiment of the present invention, the calculation formula of the reward function is:

[0105] ;

[0106] in, Indicates in status Take action After transfer to state Rewards received when represents the weight of frequency deviation; Indicates the frequency of the power system after the action is performed; Indicates the set frequency of the power system; The weight representing the matching degree between power generation and load; Indicates the power generation output after the action is executed; Indicates the load requirement after performing the action; represents the weight of the stability index; Indicates the weight of voltage stability in the stability index; Indicates the voltage value after the action is performed; represents the weight of current stability in stability calculation; Indicates the current value after the action is performed; represents the weight of the total cost; represents the power generation cost coefficient; Represents the equipment maintenance cost.

[0107] In the reward function The term emphasizes the minimization of frequency deviation. By giving higher rewards to states where the frequency is close to the set value f, the scheme can motivate the controller to take actions to maintain the stability of the system frequency, thereby effectively reducing frequency fluctuations and improving the overall stability of the power system. Encourage power generation output and load demand This matching can reduce unnecessary power generation waste or insufficient load, thereby improving energy efficiency and potentially reducing power generation costs.

[0108] pass The reward function incorporates the voltage and current This helps ensure that the control process not only focuses on frequency stability, but also considers the stable operation of electrical equipment, thereby comprehensively improving the physical stability of the system. The item reflects the cost of power generation and equipment maintenance costs By penalizing high-cost operations, this scheme can guide the controller to choose a more economical and efficient control strategy, which will help reduce the operating cost of the power system in the long run. , , , as well as , Can be adjusted according to actual needs.

[0109] In a preferred embodiment of the present invention, the above step 15, integrating the trained deep neural network model with reinforcement learning to construct a deep reinforcement learning controller to automatically and real-time control the load frequency in the power system, may include:

[0110] Step 155, using the power system model to build a reinforcement learning environment, and performing trial and error learning on the deep reinforcement learning controller in the reinforcement learning environment. During the trial and error process, the deep reinforcement learning controller determines an action according to the current state, and accepts rewards and new state information fed back by the environment to obtain a trial and error learning result;

[0111] Step 156, based on the trial and error learning results, the deep reinforcement learning controller continuously updates the control scheme until the deep reinforcement learning controller learns how to determine the final action based on the current state and future predictions, thereby obtaining a trained deep reinforcement learning controller;

[0112] Step 157, integrating the trained deep reinforcement learning controller into the power system to automatically and in real time control the load frequency in the power system.

[0113] In an embodiment of the present invention, a virtual environment capable of simulating the behavior of a real power system is constructed using power system simulation software or a mathematical model. This environment needs to be able to reflect the dynamic characteristics of the power system, such as frequency changes, load fluctuations, and the response of the generator. In this environment, the state space, action space, and reward function are defined. The state space may include the current frequency, load demand, and generator state of the power system; the action space may include control means such as adjusting the generator output and switching the load; and the reward function is set according to the stability and economy of the power system. A deep neural network model is designed as a policy network for reinforcement learning, which can receive the power system state as input and output control actions. A suitable reinforcement learning algorithm, such as a deep Q network (DQN), is selected for training the deep neural network. In the reinforcement learning environment, the deep reinforcement learning controller is allowed to perform trial and error learning. The controller selects an action based on the current state and accepts rewards and new state information fed back by the environment. Using this information, the controller continuously updates the parameters in its policy network to maximize the long-term cumulative reward. This process involves a large number of iterations and calculations, and usually requires the use of high-performance computing resources. After the deep reinforcement learning controller exhibits good control performance in the reinforcement learning environment, it is integrated into the actual power system.

[0114] The controller monitors the state of the power system in real time and outputs control actions according to the learned strategy to automatically and in real time control the load frequency in the power system.

[0115] Suppose you want to control a power system that includes multiple generators and loads, and the goal is to maintain the system frequency near the set value. The state of the power system includes the current frequency, the output power of each generator, the load demand, etc. The controller's actions can include adjusting the output power of each generator, switching on and off certain loads, etc. The reward function can be designed so that the smaller the absolute value of the system frequency deviation, the higher the reward, while considering factors such as power generation cost and equipment maintenance cost. During the training process, the deep reinforcement learning controller will try different control actions and observe the impact of these actions on the state of the power system. Through continuous trial and error and learning, the controller gradually learns how to determine the optimal control action based on the current state and future predictions (such as load forecasting). After training is completed, the controller is integrated into the power system. When the system is running, the controller will monitor the state of the power system in real time and automatically adjust the output power of the generator or switch on the load according to the learned strategy to maintain the stability of the system frequency. For example, when the load suddenly increases, the controller may increase the output power of some generators to meet the load demand; when the load decreases, the output power of the generator will be reduced to reduce the cost of power generation.

[0116] Through the trial-and-error learning process of deep reinforcement learning, the controller can learn how to determine the optimal control action based on the current state of the power system (such as frequency deviation, load changes, etc.) and future predictions (such as load forecasts, renewable energy output forecasts, etc.). This enables the controller to adjust the output power or load of the generator more accurately to maintain the stability of the system frequency. The deep reinforcement learning controller can quickly respond to changes in the power system and adjust the control strategy in time to minimize the impact of frequency deviation and load fluctuations. This efficient response capability helps to improve the stability and reliability of the power system. Through trial-and-error learning in different scenarios and working conditions, the deep reinforcement learning controller can accumulate rich experience and learn how to adopt the optimal control strategy in different situations. This enables the controller to adapt to the diverse needs of the power system, such as load peak and valley changes, intermittent output of renewable energy, etc. During the trial-and-error learning process, the controller will also encounter various emergencies (such as equipment failures, load mutations, etc.). By learning and responding to these events, the controller can enhance its ability to respond to emergencies and ensure the stable operation of the power system in emergency situations. The deep reinforcement learning controller will continuously optimize the matching degree between power generation and load during the trial-and-error learning process. By adjusting the output power or load of the generator, the controller can ensure the balance of supply and demand in the power system and reduce unnecessary waste of power generation or insufficient load. By optimizing the matching of power generation and load, the deep reinforcement learning controller can also reduce the cost of power generation. On the premise of meeting the load demand, the controller will give priority to the lower-cost power generation method or adjust the load to reduce the overall power generation cost. The deep reinforcement learning controller has the ability to make intelligent decisions and can make the best control decisions based on the real-time status and future predictions of the power system. This intelligent decision-making ability helps to promote the intelligent development of the power system and improve the automation level of the system. The deep reinforcement learning controller has the ability to continuously learn and optimize. As the power system continues to change and develop, the controller can continuously learn and adapt to new situations and optimize its control strategy to improve the performance of the system.

[0117] like Figure 2 As shown, an embodiment of the present invention further provides a system for controlling load frequency of a power system using deep reinforcement learning, comprising:

[0118] A construction module, used to construct a power system model, wherein the power system model includes a power generation unit, a power transmission network, a load unit and a frequency control device;

[0119] An acquisition module is used to acquire the historical operation data of the power system, including load demand records, power generation output logs, and frequency fluctuations, and to pre-process the historical operation data of the power system to obtain the processed historical operation data of the system;

[0120] The prediction module is used to train a deep neural network model based on the processed system historical operation data, extract key features of the system historical operation data, and predict future load demand and frequency change trends to obtain prediction results;

[0121] The processing module is used to determine the load frequency control plan based on the prediction results through reinforcement learning and continuous iterative optimization; the trained deep neural network model is integrated with reinforcement learning to build a deep reinforcement learning controller to achieve automatic and real-time control of the load frequency in the power system.

[0122] It should be noted that the system is a system corresponding to the above method, and all implementation methods in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0123] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the method described above is executed. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0124] The embodiment of the present invention also provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the method described above. All implementations in the above method embodiment are applicable to this embodiment and can achieve the same technical effect.

[0125] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for controlling load frequency of a power system using deep reinforcement learning, characterized in that: The method comprises: Constructing a power system model, wherein the power system model includes a power generation unit, a transmission network, a load unit, and a frequency control device; Obtaining historical operation data of the power system, including load demand records, power generation output logs, and frequency fluctuations, and preprocessing the historical operation data of the power system to obtain processed historical operation data of the system; Based on the processed system historical operation data, a deep neural network model is trained to extract the key features of the system historical operation data, and the future load demand and frequency change trends are predicted to obtain the prediction results; Based on the prediction results, the load frequency control scheme is determined through continuous iterative optimization through reinforcement learning; Merge the trained deep neural network model with reinforcement learning to build a deep reinforcement learning controller to achieve automatic and real-time control of load frequency in the power system; Based on the prediction results, the load frequency control scheme is determined through reinforcement learning and continuous iterative optimization, including: Define the state space, action space and reward function of reinforcement learning, and initialize the value network parameters; Obtain the current state of the power system from the state space, determine the control action based on the current state, and initialize the value network; The deep neural network model is used to predict the impact of the determined control action on future load demand and frequency change trends, and the reward value after the action is executed is calculated; Use the calculated reward value to update the network parameters of reinforcement learning, and repeat state acquisition, action execution and future prediction, reward calculation and parameter update until the preset number of iterations is reached, extract the control scheme, and use the control scheme as the load frequency control scheme; the calculation formula of the reward function is: ； in, Indicates in status Take action After transfer to state Rewards received when represents the weight of frequency deviation; Indicates the frequency of the power system after the action is performed; Indicates the set frequency of the power system; The weight representing the matching degree between power generation and load; Indicates the power generation output after the action is executed; Indicates the load requirement after performing the action; represents the weight of the stability index; Indicates the weight of voltage stability in the stability index; Indicates the voltage value after the action is performed; represents the weight of current stability in stability calculation; Indicates the current value after the action is performed; represents the weight of the total cost; represents the power generation cost coefficient; Representing equipment maintenance costs; integrating the trained deep neural network model with reinforcement learning to build a deep reinforcement learning controller to automatically and in real time control the load frequency in the power system, including: The power system model is used to build a reinforcement learning environment, and the deep reinforcement learning controller is used for trial-and-error learning in the reinforcement learning environment. During the trial-and-error process, the deep reinforcement learning controller determines the action based on the current state, and accepts the reward and new state information fed back by the environment to obtain the trial-and-error learning result. According to the trial-and-error learning results, the deep reinforcement learning controller continuously updates the control scheme until the deep reinforcement learning controller learns how to determine the final action based on the current state and future predictions, thus obtaining a trained deep reinforcement learning controller; The trained deep reinforcement learning controller is integrated into the power system to automatically and in real time control the load frequency in the power system.

2. The method for controlling power system load frequency using deep reinforcement learning according to claim 1, characterized in that: Based on the processed system historical operation data, a deep neural network model is trained to extract the key features of the system historical operation data, and to predict future load demand and frequency change trends, and obtain prediction results, including: The processed system historical operation data is divided into a training set, a validation set, and a test set, and the gated recurrent unit time series prediction model is used as a deep neural network model; Define the architecture of the deep neural network model, including input layer, hidden layer and output layer, and use mean absolute error as the loss function; The training set is input into the deep neural network model to train the deep neural network model. During the training process, the deep neural network model is verified using the verification set data to obtain the verification result. Adjust the deep neural network model architecture and parameters based on the verification results; The verified deep neural network model is used to predict the test set and obtain the future load demand and frequency change trend prediction results.

3. The method for controlling power system load frequency using deep reinforcement learning according to claim 2, characterized in that: The formula for calculating the mean absolute error is: ； in, represents the mean absolute error; represents the total number of time steps for the forecast; Indicates The actual load demand for each time step; Indicates The predicted value for each time step; represents the input feature vector; represents the model parameter set; Represents environment variables; represents the weight coefficient; represents the global adjustment coefficient; The index value representing the time step.

4. The method for controlling power system load frequency using deep reinforcement learning according to claim 3, characterized in that: The training set is input into the deep neural network model to train the deep neural network model. During the training process, the deep neural network model is verified using the verification set data to obtain verification results, including: The training set is input into the deep neural network model for training, and in each training cycle, the deep neural network model traverses the training set; Through forward propagation, the deep neural network model calculates the predicted value, and through back propagation, the deep neural network model parameters are updated; After each training cycle, the deep neural network model is verified using the validation set, and the verification results are obtained by comparing the prediction results of the validation set with the actual data.

Images