Power dispatching optimization method and system based on reinforcement learning

Abstract

The invention provides a power dispatching optimization method and system based on deep reinforcement learning, and the method comprises the steps: building a deep neural network with spatial-temporal characteristics, simulating a power system environment, and training the dispatching process of power equipment and a power transmission line through reinforcement learning, so as to obtain an optimal power dispatching strategy in different application scenes. According to the method, the economic cost, the carbon emission and the system stability are comprehensively considered through the multi-target reward function, automation and intelligence of power dispatching are achieved, and the economical efficiency, the environmental protection property and the stability of a power system are improved. The system comprises an acquisition module, a construction module and a strategy module which are respectively responsible for data acquisition, construction of an action space and a reward function and implementation of a reinforcement learning strategy. According to the method, the limitation of a traditional power dispatching method in coping with a complex and changeable power system environment is effectively solved, and the flexibility and adaptability of power dispatching are improved.

Description

Technical Field

[0001] The present invention relates to the field of power systems, and more specifically, to a power dispatch optimization method and system based on reinforcement learning. Background Art

[0002] With the development of the power industry and the widespread application of renewable energy, power dispatching faces more and more challenges, such as demand fluctuations, supply uncertainty, and increasing environmental protection requirements. In traditional power systems, power dispatching mainly relies on manual experience and simple mathematical models to make decisions, which cannot cope with the increasingly complex power network and changing power demand. With the large-scale access of renewable energy and the rapid development of the power market, the uncertainty and complexity of power system operation have increased significantly, which has brought unprecedented challenges to power dispatching.

[0003] The output of renewable energy such as wind and solar energy is affected by many factors such as weather conditions, and is highly volatile and unpredictable. The peak-to-valley difference in electricity demand and the safe and stable operation requirements of the power grid also put forward higher requirements for power dispatching. The pressure of environmental protection makes reducing carbon emissions one of the important factors that must be considered in power dispatching.

[0004] With the application of artificial intelligence technology in power dispatching, deep learning technology has been introduced into power system state prediction and fault diagnosis due to its advantages in processing large-scale data and complex pattern recognition. However, reinforcement learning technology has not been reasonably and effectively applied to optimize the operation strategy and dispatch plan of the power system due to its adaptability and real-time nature in the decision-making process.

[0005] How to effectively combine deep learning with reinforcement learning so that it can not only accurately perceive and predict the state of the power system, but also make optimal dispatching decisions in a complex and changing environment is still a problem that needs to be solved urgently in the current power dispatching field.

[0006] In view of the above problems, there is an urgent need for a power dispatch optimization method and system based on reinforcement learning. Summary of the invention

[0007] In order to address the deficiencies in the prior art, the present invention provides a power dispatch optimization method and system based on reinforcement learning. By constructing a deep neural network with spatiotemporal characteristics and training the dispatching process of power equipment and transmission lines through reinforcement learning, the optimal power dispatching strategy under different application scenarios is obtained.

[0008] The present invention adopts the following technical solution.

[0009] The first aspect of the present invention relates to a power dispatch optimization method based on reinforcement learning, the method comprising: collecting power system operation data, and constructing a neural network for extracting time features and spatial features respectively and learning, and using the neural network to simulate the power system environment; using power dispatch operations to construct an action space, and constructing a multi-objective reward function with the goals of minimizing economic costs, minimizing carbon emissions and maximizing system stability; using reinforcement learning to input dispatch actions into the power system environment and obtain rewards for the dispatch actions, thereby obtaining optimal power dispatch strategies under different scenarios; the different scenarios are constructed based on feature engineering of the power system operation data in one or more dimensions, and the optimal power dispatch strategy is measured by the degree of change in returns calculated by the multi-objective reward function.

[0010] Preferably, the power system operation data is collected, including: the power system operation data includes power generation, grid load demand, generator set operating status, renewable energy power generation, weather information, weather forecast data, and electricity market price; the power system operation data is obtained by performing feature extraction from the original collected power system data through principal component analysis.

[0011] Preferably, a neural network is constructed for respectively extracting and learning time features and spatial features, and the neural network is used to simulate the power system environment, including: the neural network includes a CNN module for performing spatial feature extraction on spatial feature data in the power system operation data; the neural network includes an LSTM module for performing temporal feature extraction on temporal feature data in the power system operation data; the neural network includes a DNN module for learning the spatial features and time features in the power system operation data and performing environmental modeling on the power system.

[0012] Preferably, the action space is constructed by using the power dispatching operation, and a multi-objective reward function is constructed with the goals of minimizing economic cost, minimizing carbon emissions and maximizing system stability, including: the multi-objective reward function R is:

[0013]

[0014] Alternatively, the multi-objective reward function R is:

[0015] R=-w C ·Cw E ·E+w S ·S

[0016] In the formula, w C 、w E and w S are the weights of economic cost, carbon emission and system stability in the power system, respectively.

[0017] C max 、E max and S max are the maximum limits of economic cost, carbon emission and system stability in the power system under the desired scenario respectively;

[0018] C, E and S are respectively the economic cost, carbon emissions and system stability corresponding to the current power dispatch strategy under the desired scenario.

[0019] Preferably, the economic cost corresponding to the current power dispatching strategy under the expected scenario includes the sum of the fuel cost of each generator set in the power system and the operation and maintenance cost of each generator set; the carbon emissions corresponding to the current power dispatching strategy under the expected scenario include the sum of the carbon emissions of each generator set in the power system; the system stability corresponding to the current power dispatching strategy under the expected scenario includes the sum of the power supply and demand differences in multiple continuous time periods to be calculated in the power system.

[0020] Preferably, the dispatching action is input into the power system environment by reinforcement learning and the reward of the dispatching action is obtained, so as to obtain the optimal power dispatching strategy in different scenarios, including: taking power equipment and transmission lines as intelligent agents, the state dispatching of the intelligent agents as actions, and the multi-objective reward function as rewards; filtering out the current scene data from the power system operation data, and constructing a sub-network of the current scene through the neural network, and using the sub-network to simulate the power system environment in the current scene; using the intelligent agent to interact with the power system environment in the current scene to calculate the state transition probability of the intelligent agent, and calculating the reward of the current action of the intelligent agent through state transition; training the intelligent agent through reinforcement learning to obtain the state transition trajectory of the intelligent agent during the training process, and the reward corresponding to the state transition trajectory; calculating the degree of change of the rewards of all intelligent agents, and taking the state transition trajectory of the intelligent agent whose degree of change of the reward exceeds a preset amount as the optimal power dispatching strategy in the current scenario.

[0021] Preferably, different scenarios are constructed based on feature engineering of the power system operation data of one or more dimensions, including: the scenarios include at least high load demand scenarios and extreme weather scenarios; in the high load demand scenarios, the power system operation data under the high load demand scenarios are obtained by extracting peak power consumption period data from the original power system operation data, or by incrementally improving the grid load demand in the original power system operation data; in the extreme weather scenarios, the power system operation data under the extreme weather scenarios are obtained by extracting extreme weather period data in which the temperature drops to beyond a first threshold within a first preset time period and the wind power generation capacity drops to beyond a second threshold within a second preset time period.

[0022] Preferably, the optimal power dispatching strategy is measured by the degree of change of the reward calculated by the multi-objective reward function, including: pre-setting a preset amount of the degree of change of the reward, and training the intelligent agent through reinforcement learning; if the degree of change of the reward of all intelligent agents exceeds the preset amount, then retaining the state transfer trajectory of the intelligent agent in the current training as the optimal power dispatching strategy in the current scenario; if the degree of change of the reward of all intelligent agents does not exceed the preset amount, then discarding the state transfer trajectory of the intelligent agent in the current training, and continuing to implement the next training until all possible state transfers of the intelligent agent are completed.

[0023] The second aspect of the present invention relates to a power dispatch optimization system based on reinforcement learning, the system comprising an acquisition module, a construction module and a strategy module; wherein the acquisition module is used to collect power system operation data, and construct a neural network for extracting time features and spatial features respectively and learning, and using the neural network to simulate the power system environment; the construction module is used to construct an action space using power dispatch operations, and construct a multi-objective reward function with the goals of minimizing economic costs, minimizing carbon emissions and maximizing system stability; the strategy module is used to input dispatch actions into the power system environment using reinforcement learning and obtain rewards for the dispatch actions, thereby obtaining optimal power dispatch strategies under different scenarios; in the strategy module, the different scenarios are constructed based on feature engineering of the power system operation data in one or more dimensions, and the optimal power dispatch strategy is measured by the degree of change in returns calculated by the multi-objective reward function.

[0024] A third aspect of the present invention relates to a terminal, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method described in the first aspect of the present invention.

[0025] A fourth aspect of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in the first aspect of the present invention.

[0026] The beneficial effect of the present invention is that, compared with the prior art, the power dispatch optimization method and system based on reinforcement learning in the present invention obtains the optimal power dispatch strategy in different application scenarios by constructing a deep neural network with spatiotemporal characteristics and training the dispatch process of power equipment and transmission lines through reinforcement learning. The present invention solves the limitations of traditional power dispatch methods in dealing with complex and changeable power system environments, accurately represents the power system state through deep learning, makes intelligent decisions through reinforcement learning, realizes the automation and intelligence of power dispatch, and improves the economy, environmental protection and stability of the power system.

[0027] The beneficial effects of the present invention also include:

[0028] 1. The present invention constructs a deep neural network to accurately represent the complex state of the power system, including key factors such as the state of generator sets, load demand, weather conditions, and the output of renewable energy; defines a wide range of action spaces, allowing the agent to make decisions in terms of generator set start-up and shutdown, output adjustment, and energy storage equipment operation; designs a multi-objective reward function that comprehensively considers key indicators such as cost minimization, carbon emission reduction, and system stability improvement to ensure the comprehensiveness and sustainability of the power dispatching plan; by applying advanced deep reinforcement learning algorithms such as DQN, DDPG, or PPO, the agent autonomously learns and optimizes the dispatching strategy in the interaction with the power system environment. In addition, the present invention enables the agent to dynamically adjust decisions according to real-time data and environmental changes through an adaptive adjustment mechanism, significantly improving the flexibility and adaptability of power dispatching. By comprehensively applying the above means, the present invention not only improves the economy and environmental protection of the power system, but also enhances the stability and reliability of the system.

[0029] 2. The present invention optimizes the dispatch of the power system intelligently by comprehensively utilizing the powerful data processing capabilities of deep learning and the decision-making capabilities of reinforcement learning to adapt to the complexity and uncertainty in the operation of the power system. The present invention simulates and tests various power system operation scenarios, such as extreme weather, surges in electricity demand, equipment failures, etc., to evaluate the performance of the intelligent agent's dispatch strategy in different situations, and makes necessary adjustments and optimizations. The fully verified intelligent agent is applied to the actual power system dispatch to achieve automation and intelligence in power dispatch. By constructing a simulated test environment, the performance of the intelligent agent is fully verified before actual application, ensuring the reliability and practicality of the dispatch plan. The present invention improves the economic and environmental performance of the power system, and also enhances the stability and security of the system.

[0030] 3. The power dispatch optimization method of the present invention not only improves the operating efficiency and reliability of the power system, but also helps to achieve environmental protection goals. The present invention predicts the output of renewable energy through deep learning technology, and integrates the prediction results into the power dispatch optimization model, effectively solving the problem of renewable energy integration. The method of the present invention has good application capabilities in power systems of different sizes and types, including microgrids, large-scale power systems, and power systems containing a high proportion of renewable energy.

[0031] 4. The present invention utilizes an efficient and intelligent power dispatch optimization method, combines deep learning and reinforcement learning, and significantly improves the dispatch quality and operating efficiency of the power system. First, the deep learning model can accurately predict the future state of the power system and provide reliable data support for dispatch decisions. Secondly, the design of the multi-objective reward function enables power dispatch to achieve the best balance between economy, environmental protection, and system stability. Furthermore, the adaptive adjustment mechanism gives the intelligent agent flexibility in the face of real-time data and environmental changes, ensuring the real-time and adaptability of the dispatch strategy. In addition, the application of deep reinforcement learning algorithms greatly improves the ability to handle large-scale power system optimization problems, shortens calculation time, and improves the accuracy of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A schematic diagram of the steps of a power dispatch optimization method based on reinforcement learning of the present invention;

[0033] Figure 2 A functional structure diagram of an actual deployment and monitoring system in a power dispatch optimization method based on reinforcement learning of the present invention;

[0034] Figure 3 This is a schematic diagram of the module structure of a power dispatching optimization system based on reinforcement learning in the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in the present invention are only embodiments of a part of the present invention, not all embodiments. Based on the spirit of the present invention, all other embodiments not recorded in the present invention obtained by ordinary technicians in this field according to the embodiments recorded in the present invention without making creative work should belong to the protection scope of the present invention.

[0036] Figure 1 The first aspect of the present invention relates to a method for optimizing power dispatching based on reinforcement learning, and the method includes steps 1 to 3.

[0037] Step 1: Collect power system operation data, construct a neural network for extracting time features and spatial features and learning them respectively, and use the neural network to simulate the power system environment.

[0038] The power system is a highly complex and dynamically changing network involving a large number of variables and uncertainties. In order to obtain the optimal dispatching method, the present invention constructs a deep neural network to accurately represent the complex state of the power system. The state of the power system is modeled and represented using deep learning technology, including the design of the network structure, the preparation of training data, and the optimization of the training process.

[0039] Preferably, the power system operation data is collected, including: the power system operation data includes power generation, grid load demand, generator set operating status, renewable energy power generation, weather information, weather forecast data, and electricity market price; the power system operation data is obtained by performing feature extraction from the original collected power system data through principal component analysis.

[0040] In order to characterize the state of the power system, in addition to key factors such as the start and stop status of the generator set, grid load demand, weather conditions, and the output of renewable energy, the market price of power generation raw materials, and the characteristic parameters of each power generation equipment and transmission lines are also needed, such as the charging parameters of the generator set, the operation and maintenance cost of the generator set, the carbon emission coefficient of the generator set, etc.

[0041] During the data collection process, it should be ensured that all components of the power system related to dispatching and analysis are covered, such as generators, transmission lines, energy storage equipment, etc.

[0042] The original collected data needs to go through a preprocessing process before it can be used for analysis. First, the collected data is cleaned to remove outliers and noise to ensure data quality. Secondly, the data is normalized or standardized, such as Z-score, so that data of different magnitudes can be compared and processed on the same scale, eliminating the scale differences of data of different magnitudes. When dealing with missing values, missing data points can be filled by interpolation, mean or median.

[0043] Deep neural networks can process and analyze large amounts of historical and real-time data. After the historical operating data of the power grid is input into the deep neural network for training, the deep neural network can simulate the power system environment.

[0044] Preferably, a neural network is constructed for respectively extracting and learning time features and spatial features, and the neural network is used to simulate the power system environment, including: the neural network includes a CNN module for performing spatial feature extraction on spatial feature data in the power system operation data; the neural network includes an LSTM module for performing temporal feature extraction on temporal feature data in the power system operation data; the neural network includes a DNN module for learning the spatial features and time features in the power system operation data and performing environmental modeling on the power system.

[0045] The network structure design takes into account the ability of convolutional neural networks (CNN) to process spatially correlated data, thereby learning grid topology information such as the distribution map of generator sets, as well as the ability of recurrent neural networks (RNN) or long short-term memory networks (LSTM) to process the correlation of time series data. Through training, the network can accurately predict the future state of the system and provide reliable information support for scheduling decisions. This model effectively learns the intrinsic characteristics of data of different dimensions in the power system, models and represents the state of the intelligent agent, and improves the accuracy of state prediction.

[0046] Since the original data information dimensions are large, the feature engineering method is used to extract representative features of the power system state from the original data. The selected features may include time series analysis results, statistical data, equipment operating parameters, etc. In one embodiment, correlation analysis, principal component analysis (PCA, Principal Components Analysis) or autoencoder and other techniques are used to select and extract the most useful features. In one embodiment, the PCA algorithm selects the principal component that explains more than 95% of the total variance as an alternative feature, deletes other features, and divides the data set obtained after feature extraction into a training set, a validation set, and a test set to ensure that the model has good generalization ability. In one embodiment, the key features extracted by this feature engineering may include the availability of generator sets, the peak and valley periods of load demand, the expected output of renewable energy, etc. These key features can also be used for subsequent feature extraction of typical scenarios of power grid operation.

[0047] The dataset obtained after feature extraction is used to train the deep learning model, and the weight of the model is optimized through the back propagation algorithm to simulate the power system environment. During the training process, the performance of the model is monitored and evaluated using indicators such as accuracy, recall, and F1 score. The trained deep learning model represents the power system environment, can analyze the input data features, and output the state representation of the power system.

[0048] The state representation of the power system can be the dispatching state of the power generation equipment, energy storage equipment and transmission lines in the power system. By further using deep neural networks to implement learning of different dispatching actions in the power system, the various equipment states output by the deep neural network can be obtained, and the state representation can be used to further analyze the future load demand forecast of the power grid, equipment failure risk assessment, electricity market price change trends, etc.

[0049] The pre-trained deep neural network model is used to provide decision-making basis for the intelligent agent. The intelligent agent selects actions according to the state representation and executes the power dispatch strategy to achieve the optimization goal.

[0050] In one embodiment, the CNN network uses 3 convolutional layers, each of which is connected to a maximum pooling layer. The RNN or LSTM network uses 2 hidden layers. Each layer contains 128 neurons, the activation function ReLU is used for the convolutional layer, the activation function tanh is used for the recurrent layer, the loss function uses mean square error, and the optimization algorithm is the Adam optimizer.

[0051] In addition, DNN contains multiple convolutional layers and recurrent layers. The convolutional layer uses a 3×3 convolution kernel with a step size of 1 and an activation function of ReLU. The recurrent layer uses LSTM units, and each recurrent layer contains 256 neurons. The fully connected layer is located at the end of the network and is used to output the predicted state. The loss function uses categorical cross entropy, which is suitable for multi-classification problems. The Adam optimizer is used, and the learning rate is set to 0.0005. DNN is trained using the back propagation algorithm until convergence.

[0052] Step 2: Use power dispatch operations to construct an action space and construct a multi-objective reward function with the goals of minimizing economic costs, minimizing carbon emissions, and maximizing system stability.

[0053] Based on the state representation, the present invention defines the action space of power dispatch. The action space covers all possible dispatch operations, including the start and stop operations of generator sets, output adjustment operations, and the charging and discharging operations of energy storage devices and the implementation operations of demand response, etc. Each action corresponds to a specific system state and may lead to different system changes and results.

[0054] Preferably, the action space is constructed by using the power dispatching operation, and a multi-objective reward function is constructed with the goals of minimizing economic cost, minimizing carbon emissions and maximizing system stability, including: the multi-objective reward function R is:

[0056]

[0057] Alternatively, the multi-objective reward function R is:

[0058] R=-w C ·CwE ·E+w S ·S

[0059] In the formula, w C 、w E and w S are the weights of economic cost, carbon emission and system stability in the power system, C max 、E max and S max are the maximum limits of economic cost, carbon emissions and system stability in the power system under the expected scenario; C, E and S are the economic cost, carbon emissions and system stability corresponding to the current power dispatch strategy under the expected scenario.

[0060] In the actual application of the present invention, the power system will show different states when operating in different weather environments and under different load demand conditions. To this end, the present invention also supports extracting part of the scene data during the training of the deep neural network constructed in step 1 as a basis to obtain a sub-network for a specific scene. When optimizing the power dispatching strategy, the sub-network of the expected scene is used to perform computational analysis to obtain the optimized dispatch under the expected scene.

[0061] To this end, the present invention constructs a multi-objective reward function for the power dispatching strategy with economic cost, carbon emissions and system stability as targets, uses a reinforcement learning algorithm to train the intelligent agent, and optimizes the multi-objective reward function based on the state and action space of the power system to obtain the power dispatching strategy.

[0062] Power dispatching needs to consider multiple conflicting objectives at the same time, such as minimizing costs, reducing carbon emissions, and improving system stability. In order to guide the learning process of the intelligent agent, the design and implementation of the reward function is key, which directly affects the learning direction and optimization goals of the intelligent agent. The present invention designs a reward function that comprehensively considers multiple objectives. This function not only focuses on the economy of power system operation, such as minimizing power generation costs, but also includes environmental factors, such as minimizing carbon emissions, and system stability, such as ensuring supply and demand balance and avoiding potential power shortages. By constructing a multi-objective reward function and applying it to the deep reinforcement learning framework, effective balance and optimization of multiple objectives are achieved.

[0063] In practical applications, designing a multi-objective reward function usually requires considering how to transform different objectives into a unified quantitative indicator and determining the weight of each objective for optimization in the reinforcement learning process.

[0064] The present invention involves three objectives, namely, maximizing economic costs, minimizing carbon emissions, and maximizing system stability. The target reward function is the weighted sum of the three objectives. The weights are used to balance the importance of the objectives. The possible minimum values ​​of the three indicators under specific conditions are normalized so that the output of the reward function is within a reasonable range.

[0065] Preferably, the economic cost corresponding to the current power dispatching strategy in the expected scenario includes the sum of the fuel cost of each generator set in the power system and the operation and maintenance cost of each generator set; the carbon emissions corresponding to the current power dispatching strategy in the expected scenario include the sum of the carbon emissions of each generator set in the power system; the system stability corresponding to the current power dispatching strategy in the expected scenario includes the sum of the power supply and demand differences in multiple continuous time periods to be calculated in the power system.

[0066] Among them, the economic index is measured by the power generation cost, which can be expressed as the sum of the fuel cost and operation and maintenance cost of all generator sets. The quantitative formula is:

[0067]

[0068] In the formula, c i,fuel is the material cost coefficient of the i-th generator set, p i is the real-time power of the generator set, c i,maint is the operation and maintenance cost of the generator set, and N is the number of generator sets in the area.

[0069] Environmental protection index is measured by carbon emissions, which can be expressed as the sum of carbon emissions of all generators. The quantitative formula is:

[0070]

[0071] In the formula, e i is the carbon emission coefficient of the i-th generator set.

[0072] The stability index is measured based on indicators such as system load balance and frequency deviation. In one embodiment, the quantification method is the absolute value of the system supply and demand difference:

[0073]

[0074] Where T is the length of the time period used for analysis, D(t) is the power demand in time period t, and P(t) is the power supply in time period t.

[0075] Step 3: Use reinforcement learning to input dispatch actions into the power system environment and obtain rewards for dispatch actions, so as to obtain the optimal power dispatch strategy under different scenarios. Different scenarios are constructed based on feature engineering of power system operation data in one or more dimensions, and the optimal power dispatch strategy is measured by the degree of change of rewards calculated by the multi-objective reward function.

[0076] As the scale of power systems continues to expand, traditional optimization algorithms face problems such as high computational complexity and slow convergence when dealing with large-scale systems. The present invention uses a deep reinforcement learning algorithm to train an intelligent agent so that it can autonomously learn and optimize in the decision-making process of power system dispatch. The policy network can be a deterministic strategy or a random strategy based on a value function. The selected algorithms include Deep Q-Learning (DQN), Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), etc., which effectively handle high-dimensional inputs and outputs, as well as continuous or discrete action spaces.

[0077] In deep reinforcement learning, the agent learns the optimal strategy by interacting with the environment. Through interaction with the environment, the agent constantly tries different actions and learns according to the reward function, gradually forming the optimal power dispatch strategy.

[0078] Preferably, the dispatching action is input into the power system environment by reinforcement learning and the reward of the dispatching action is obtained, so as to obtain the optimal power dispatching strategy in different scenarios, including: taking power equipment and transmission lines as intelligent agents, the state dispatching of the intelligent agent as the action, and the multi-objective reward function as the reward; filtering out the current scene data from the power system operation data, and constructing a sub-network of the current scene through a neural network, and using the sub-network to simulate the power system environment in the current scene; using the intelligent agent to interact with the power system environment in the current scene to calculate the state transition probability of the intelligent agent, and calculating the reward of the current action of the intelligent agent through the state transition; training the intelligent agent through reinforcement learning to obtain the state transition trajectory of the intelligent agent during the training process, and the reward corresponding to the state transition trajectory; calculating the degree of change of the rewards of all intelligent agents, and taking the state transition trajectory of the intelligent agent whose reward change degree exceeds the preset amount as the optimal power dispatching strategy in the current scenario.

[0079] In the process of reinforcement learning, the parameters of the agent, such as the weights of the neural network, are randomly initialized or obtained through pre-training to complete the initialization of reinforcement learning. Subsequently, the agent needs to find a balance between exploration, such as trying new, unknown actions to find better strategies, and using the currently known best strategy to obtain rewards. In one embodiment, the probability of exploring new strategies is set to ε, and the probability of using the existing best strategy is set to 1-ε, and the probability ε uses a greedy strategy to achieve balance. In this strategy, the agent selects a random action for exploration with a probability of ε, and selects the currently known best action for use with a probability of 1-ε. As learning proceeds, the value of ε usually decreases gradually, which means that the agent will increasingly use the known best strategy while reducing the frequency of exploring new strategies. This method helps the agent to explore the possible action space extensively in the early stages of learning, and gradually converge to the optimal strategy as learning deepens.

[0080] In one embodiment, the DQN algorithm is used to update the state of the agent. In the DQN algorithm, at each step, the agent selects an action from the current state, which is implemented through a Q network, which takes the current state as input and outputs the Q value of each possible action, that is, the expected reward. The agent performs the selected action and then returns a new state, reward, and a signal through environmental interaction to indicate whether the session is over. The agent stores the experience of this interaction in an experience playback buffer. The experience includes the correspondence between the current state, action, reward, and new state.

[0081] In DQN, the agent periodically randomly extracts a batch of experiences from the experience replay buffer and uses them to update the Q network. During the update process, the loss function is calculated:

[0082] L(θ)=E[(yQ(s,a;θ)) 2 ]

[0083] y is the target Q value, Q(s,a;θ) is the Q value calculated using the optimal action value function in the current Q network with parameter θ, s is the state, a is the action, and θ is the network parameter.

[0084] Therefore, it can be understood that yQ(s,a;θ) is the update amount of the Q value, which is usually determined by the reward function under this action, the discount factor, the maximum possible Q value when the next state takes the next action, the learning rate, etc.

[0085] In order to stabilize the training process, DQN uses a target network, whose parameters θ are copied from the main network at regular intervals. The target Q value y is calculated by this target network. The target Q value and the main network here can be the power system environment simulated by the neural network mentioned above. In one embodiment, the parameter θ includes hyperparameters such as the learning rate and exploration rate of the network, and the update of the parameter θ is implemented according to the learning progress of DQN and environmental feedback.

[0086] When calculating the Q value, future rewards are discounted by a factor γ to reduce their influence to reflect that recent rewards are more important than distant rewards. The agent repeats the above steps and gradually improves its strategy through continuous learning.

[0087] In the present invention, the parameters of DQN are reasonably set, the experience replay pool size is 10000, the learning rate is 0.001, the discount factor is 0.95, the initial value of the exploration rate is 1.0, and it gradually decays to 0.01 with iteration.

[0088] In another embodiment, the state of the agent is updated using a PPO algorithm, which is a policy gradient method that optimizes the policy by stochastic gradient ascent and ensures the stability of the policy update by a reward function.

[0089] In the PPO algorithm, the agent first executes the current strategy and collects states, actions, probability ratios, i.e., the ratio of the probabilities of the new and old strategies, as well as the cumulative rewards. For each state-action pair, the algorithm calculates the PPO cut value, which is a value less than 1 that limits the magnitude of the strategy update. Then, the PPO cut value and probability ratio are used to update the strategy through stochastic gradient ascent. At the same time, a value function is updated to estimate the value of the state under the current strategy. Repeat the above steps until the strategy converges.

[0090] In one embodiment, the hyperparameters of the PPO algorithm are reasonably set: the learning rate is 0.001, the discount factor is 0.98, the weight of the KL divergence is 0.01, and the number of iterations is 10,000.

[0091] The learning process of the agent requires tens of thousands of iterations, each step involving a lot of calculations and data processing. In addition, the learning process of the agent needs to be monitored and adjusted to ensure that it is progressing in the right direction.

[0092] Uncertainties and real-time changes in power system operation require dispatch strategies that can respond quickly and adapt to new situations. In order to enable the agent to adapt to real-time changes in power system operation, the present invention introduces an adaptive adjustment mechanism that allows the agent to dynamically adjust its dispatch strategy based on the latest system status and environmental feedback. This mechanism can be implemented based on online learning strategies, policy gradient methods, or exploration and exploitation balance techniques, etc., to ensure that the agent continuously optimizes its decisions in a constantly changing environment.

[0093] Adaptively and continuously monitor the real-time operating status of the power system, collect key performance indicators and operation data, and ensure that the data collection system can effectively capture all important changes in the system, such as load fluctuations, equipment failures, weather changes, etc. On this basis, regularly evaluate the performance of the agent's current dispatch strategy, and use predefined evaluation indicators, such as the cost savings rate, carbon emissions, system stability, etc. mentioned above to evaluate the agent's dispatch strategy. By comparing historical data with expected goals, the effectiveness of the current strategy can be judged.

[0094] The present invention also supports the analysis of feedback information received by the agent, and identifies problems and deficiencies in the execution of the strategy, such as load fluctuations, equipment failures, etc. Statistical analysis and pattern recognition techniques are used to identify the causes of system performance deviations, such as changes in the external environment, model prediction errors, etc., and the scheduling strategy of the agent is adjusted in a targeted manner. Such adjustments may include changing the start and stop sequence of generator sets, optimizing the charging and discharging plans of energy storage equipment, redistributing loads, and so on. These were originally used as optimization steps of algorithms such as DQN, PPO, and genetic algorithms through scheduling actions, and adjusting the order of scheduling actions and selecting the best of multiple scheduling actions actually changed the energy storage plan, load distribution plan, and start and stop method of the power grid.

[0095] Preferably, different scenarios are constructed based on feature engineering of power system operation data in one or more dimensions, including: the scenarios include at least high load demand scenarios and extreme weather scenarios; in high load demand scenarios, the power system operation data in high load demand scenarios is obtained by extracting peak power consumption period data from the original power system operation data, or by incrementally improving the grid load demand in the original power system operation data; in extreme weather scenarios, the power system operation data in the extreme weather scenarios is obtained by extracting extreme weather period data in which the temperature drops to beyond a first threshold within a first preset time period and the wind power generation capacity drops to beyond a second threshold within a second preset time period.

[0096] The adaptive process tests the performance of the agent in different simulation environments. Different scenarios actually represent the operation status of the power grid under different typical conditions, such as high load demand scenarios, extreme weather scenarios, etc.

[0097] High load demand is usually a scenario where the power consumption in the power system suddenly increases due to factors such as industrial production, residents' lives, and commercial activities. Therefore, the simulation can be implemented by adjusting the load model parameters.

[0098] Extreme weather includes natural phenomena such as storms, heavy rains, high temperatures, hail, and snowfall, which may seriously affect the normal operation of the power system. For example, extreme weather may cause abnormal output of renewable energy such as wind and solar energy. Extreme weather may also increase the failure rate of some power equipment, leading to risks in power grid operation. Therefore, it can be simulated by adjusting the meteorological data in the power system.

[0099] In one embodiment of the present invention, historical power system operation data is collected, such as historical operation data covering at least one full calendar year, to ensure that the load demand in different seasons and different extreme weather conditions are included. The data at least includes the aforementioned power generation, load demand, equipment status, weather parameters, etc. The simulation environment will generate different system states, that is, different scenarios, based on the extracted features and the trained model.

[0100] In the embodiment, the simulation of high load demand is achieved by increasing the load demand data by 20% to simulate the peak period. The simulation of extreme weather is achieved by extracting data of a specific period when the temperature suddenly drops by 10°C or the wind power generation capacity is reduced by 30%.

[0101] Preferably, the optimal power dispatching strategy is measured by the degree of change of rewards calculated by a multi-objective reward function, including: pre-setting a preset amount of the degree of change of rewards, and training the intelligent agent through reinforcement learning; if the degree of change of rewards for all intelligent agents exceeds the preset amount, then retaining the state transfer trajectory of the intelligent agent in the current training as the optimal power dispatching strategy for the current scenario; if the degree of change of rewards for all intelligent agents does not exceed the preset amount, then discarding the state transfer trajectory of the intelligent agent in the current training, and continuing to implement the next training until all possible state transfers of the intelligent agent are completed.

[0102] In the adaptive scheduling optimization process, the cost saving rate, carbon emission reduction rate and system stability index are used as evaluation criteria. In one embodiment, the preset amount of the cost saving rate can be set to save 15%, and the preset amount of the carbon emission reduction can be set to reduce 20%. If the cost saving rate or carbon emission reduction of the agent can reach the above goals in a certain training, the training trajectory is judged to be an alternative optimal scheduling method.

[0103] In the above embodiment, according to the performance evaluation results, the weight of the agent's reward function is adjusted, and the weights of economy, environmental protection, and stability may be set to 0.4, 0.3, and 0.3, respectively. In addition, the hyperparameters of the deep learning model can also be adjusted at the same time, such as the learning rate is set to 0.001 and the batch size is 32 in this embodiment to optimize the prediction accuracy of the model.

[0104] While the above algorithm is being optimized, the deep neural network model can be updated synchronously to reflect the adjusted strategy and learn the new system state. Through online learning or incremental learning, the deep neural network model is more adaptable to the power system environment after scheduling optimization. In the actual application process, the optimized strategy is applied to the power system, and the system execution is monitored to verify the effect of the new scheduling strategy and ensure that it achieves the expected performance improvement goals.

[0105] The adaptive adjustment mechanism is regarded as a continuous process, and the above steps are executed in a continuous cycle, so that the intelligent agent can dynamically adjust its power dispatch strategy according to real-time data and environmental changes, which significantly improves the real-time and adaptability of power dispatch.

[0106] Figure 2 The figure is a functional structure diagram of the actual deployment and monitoring system in the power dispatch optimization method based on reinforcement learning of the present invention. Figure 2 As shown, through simulation tests, the dispatching effect of the agent in various scenarios can be evaluated, and the model parameters can be adjusted according to the test results. In the present invention, the algorithm will repeat the test and evaluation process until the performance of the agent in the simulation environment reaches the expected goal. Finally, the optimal model parameters and strategies will be solidified, and the fully tested agent will be deployed to the actual power system for real-time monitoring and continuous optimization to monitor the operation of the agent and ensure that it performs power dispatch as expected. Based on the actual operation data and feedback, the strategy of the agent is continuously optimized and adjusted.

[0107] In one embodiment, the present invention implements an efficient and intelligent power dispatching system that automatically adjusts the power dispatching strategy according to real-time data and environmental changes, achieving a cost savings of 15%, a reduction in carbon emissions of approximately 20%, and significantly improving the stability of the system.

[0108] The power dispatch optimization method of the present invention can achieve efficient power dispatch in practical applications, significantly improving the economy, environmental protection and stability of the power system. In actual cases applied to real power systems, the method can achieve more than 20% cost savings, reduce 30% carbon emissions, and significantly improve the stability and reliability of the system.

[0109] Figure 3 The figure is a schematic diagram of the module structure of a power dispatch optimization system based on reinforcement learning in the present invention. Figure 3 As shown, the second aspect of the present invention relates to a power dispatch optimization system based on reinforcement learning, and the system includes an acquisition module, a construction module and a strategy module; wherein the acquisition module is used to collect power system operation data, and construct a neural network for extracting time features and spatial features respectively and learning, and using the neural network to simulate the power system environment; the construction module is used to construct an action space using power dispatch operations, and construct a multi-objective reward function with the goals of minimizing economic costs, minimizing carbon emissions and maximizing system stability; the strategy module is used to input dispatch actions into the power system environment using reinforcement learning and obtain rewards for dispatch actions, thereby obtaining the optimal power dispatch strategy under different scenarios; in the strategy module, different scenarios are constructed based on feature engineering of power system operation data of one or more dimensions, and the optimal power dispatch strategy is measured by the degree of change of returns calculated by the multi-objective reward function.

[0110] A third aspect of the present invention relates to a terminal, comprising a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method in the first aspect of the present invention.

[0111] A fourth aspect of the present invention relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in the first aspect of the present invention.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A power dispatch optimization method based on reinforcement learning, characterized in that: The method comprises the following steps: Collecting power system operation data, and constructing a neural network for respectively extracting and learning time features and space features, and using the neural network to simulate the power system environment; The action space is constructed using power dispatch operations, and a multi-objective reward function is constructed with the goals of minimizing economic costs, minimizing carbon emissions, and maximizing system stability; Using reinforcement learning to input the dispatching action into the power system environment and obtain the reward of the dispatching action, so as to obtain the optimal power dispatching strategy under different scenarios; The different scenarios are constructed based on feature engineering of the power system operation data in one or more dimensions, and the optimal power dispatch strategy is measured by the degree of change of the return calculated by the multi-objective reward function.

2. The power dispatch optimization method based on reinforcement learning according to claim 1, characterized in that: The collecting of power system operation data includes: The power system operation data includes power generation, grid load demand, generator set operation status, renewable energy power generation, weather information, weather forecast data, and power market price; The power system operation data is obtained by extracting features from the original collected power system data using a principal component analysis method.

3. The power dispatch optimization method based on reinforcement learning according to claim 2 is characterized in that: The method comprises constructing a neural network for respectively extracting time features and space features and learning the neural network, and using the neural network to simulate the power system environment, including: The neural network includes a CNN module for extracting spatial features from the spatial feature data in the power system operation data; The neural network includes an LSTM module for extracting time features from time feature data in the power system operation data; The neural network includes a DNN module, which is used to learn the spatial characteristics and temporal characteristics in the power system operation data and perform environmental modeling on the power system.

4. The power dispatch optimization method based on reinforcement learning according to claim 3 is characterized in that: The method of constructing an action space by utilizing power dispatching operations and constructing a multi-objective reward function with the goals of minimizing economic costs, minimizing carbon emissions, and maximizing system stability includes: The multi-objective reward function R is: Alternatively, the multi-objective reward function R is: R=-w C ·C-w E ·E+w S ·S In the formula, w C 、w E and w S are the weights of economic cost, carbon emission and system stability in the power system, C max 、E max and S max are the maximum limits of economic cost, carbon emission and system stability in the power system under the desired scenario respectively; C, E and S are respectively the economic cost, carbon emissions and system stability corresponding to the current power dispatch strategy under the desired scenario.

5. The power dispatch optimization method based on reinforcement learning according to claim 4 is characterized in that: The economic cost corresponding to the current power dispatch strategy under the desired scenario includes the sum of the fuel cost of each generator set in the power system and the operation and maintenance cost of each generator set; The carbon emissions corresponding to the current power dispatch strategy under the expected scenario include the sum of the carbon emissions of each generator set in the power system; The system stability corresponding to a current power dispatching strategy in the expected scenario includes the sum of power supply and demand differences in multiple continuous time periods to be calculated in the power system.

6. The power dispatch optimization method based on reinforcement learning according to claim 5 is characterized in that: The method of using reinforcement learning to input the dispatching action into the power system environment and obtain the reward of the dispatching action, thereby obtaining the optimal power dispatching strategy under different scenarios, includes: The power equipment and the transmission line are taken as the intelligent agents, the state scheduling of the intelligent agents is taken as the action, and the multi-objective reward function is taken as the reward; Filtering current scenario data from the power system operation data, constructing a sub-network of the current scenario through the neural network, and using the sub-network to simulate the power system environment under the current scenario; Using an intelligent agent to interact with the power system environment in the current scenario to calculate the state transition probability of the intelligent agent, and calculating the reward of the current action of the intelligent agent through the state transition; Training the agent through reinforcement learning to obtain a state transition trajectory of the agent during training and a reward corresponding to the state transition trajectory; The degree of change of the rewards of all agents is calculated, and the state transition trajectory of the agent whose degree of change of the rewards exceeds a preset amount is used as the optimal power dispatching strategy in the current scenario.

7. The power dispatch optimization method based on reinforcement learning according to claim 6 is characterized in that: The different scenarios are constructed based on feature engineering of the power system operation data in one or more dimensions, including: The scenarios include at least high load demand scenarios and extreme weather scenarios; In the high load demand scenario, the power system operation data under the high load demand scenario is obtained by extracting the peak power consumption period data from the original power system operation data, or by incrementally improving the grid load demand in the original power system operation data; In the extreme weather scenario, the power system operation data under the extreme weather scenario is obtained by extracting extreme weather period data in which the temperature drops below a first threshold within a first preset time period and the wind power generation capacity drops below a second threshold within a second preset time period.

8. The power dispatch optimization method based on reinforcement learning according to claim 7 is characterized in that: The optimal power dispatch strategy is measured by the degree of change of the return calculated by the multi-objective reward function, including: Presetting a preset amount of reward variation and training the agent via reinforcement learning; If the change degree of the rewards of all agents exceeds the preset amount, the state transition trajectory of the agents in the current training is retained as the optimal power dispatching strategy in the current scenario; If the degree of change of the rewards of all agents does not exceed the preset amount, the state transition trajectory of the agent in the current training is discarded, and the next training is continued until all possible state transfers of the agent are completed.

9. A power dispatch optimization system based on reinforcement learning, characterized in that: The system includes a collection module, a construction module and a strategy module; wherein, The acquisition module is used to collect power system operation data, and construct a neural network for respectively extracting and learning time features and space features, and use the neural network to simulate the power system environment; The construction module is used to construct an action space using power dispatch operations, and to construct a multi-objective reward function with the goals of minimizing economic costs, minimizing carbon emissions, and maximizing system stability; The strategy module is used to input the dispatching action into the power system environment by using reinforcement learning and obtain the reward of the dispatching action, so as to obtain the optimal power dispatching strategy under different scenarios; In the strategy module, the different scenarios are constructed based on feature engineering of the power system operation data in one or more dimensions, and the optimal power dispatch strategy is measured by the degree of change of the return calculated by the multi-objective reward function.

10. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images