Demand response optimization method based on deep reinforcement learning in random state of power grid
By uploading the computing process of grid demand response optimization to the cloud and using distributed computing resources, combining the generation adversarial network to generate artificial grid data, the large computing resource consumption problems caused by grid scale and complexity are solved, and efficient demand response optimization is achieved.
Patent Information
- Application Number
- CN202411924448.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-25
- Publication Date
- 2025-04-29
AI Technical Summary
The existing application of deep reinforcement learning in grid demand response optimization has the problem of high computing resources consumption, especially due to the scale and complexity of the power grid, the consumption of local computing resources is very large.
A demand response optimization method based on deep reinforcement learning in the random state of the power grid is proposed. By uploading the computing process to the cloud, using distributed computing resources, and using Generative Adversarial Network (GAN) to generate artificial grid data when data is sparse.
Effectively utilize distributed computing resources in the cloud, reduce the consumption of local computing resources, and improve the quality of grid data by generating manual data, ensuring effective training and optimization of DQN models.
Smart Images
Figure CN119904041A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of power grid technology, and in particular to a demand response optimization method based on deep reinforcement learning under random power grid conditions. Background Art
[0002] Deep reinforcement learning can learn the optimal control strategy by establishing interactions between the agent and the environment. In the optimization of power grid demand response, the agent can be the load controller, and the environment is the operating state of the power grid. The agent can observe the state of the environment (such as the load of the power grid, the output of the generator, the electricity price, etc.), take corresponding actions (such as adjusting the load, storing energy, etc.), and evaluate the effects of these actions according to the reward function (such as minimizing costs, maximizing efficiency, etc.), thereby continuously optimizing the control strategy.
[0003] There are still some challenges in the application of existing deep reinforcement learning in power grid demand response optimization. For example, the scale and complexity of the power grid lead to very large consumption of local computing resources.
[0004] Therefore, in view of this, the existing structure and deficiencies are studied and improved, and a demand response optimization method based on deep reinforcement learning under random power grid conditions is proposed. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention provides a demand response optimization method based on deep reinforcement learning under random power grid conditions, which solves the problems raised in the above background technology.
[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a demand response optimization method based on deep reinforcement learning under random power grid conditions, the demand response optimization method based on deep reinforcement learning under random power grid conditions comprises the following operating steps:
[0007] S1. Problem definition:
[0008] Demand response optimization based on DQN first requires problem definition, including goals, environment, state, action, and reward;
[0009] Among them, the objectives: minimize grid operating costs;
[0010] Environment: simulate the power grid system, including but not limited to load fluctuations and power generation cost factors;
[0011] Status: The current status of the power grid, including but not limited to load, power generation cost, and price;
[0012] Action: The user adjusts the power consumption, such as increasing or decreasing the power consumption;
[0013] Rewards: Negative grid operating costs, i.e. the lower the cost, the higher the reward;
[0014] S2. DRL framework construction:
[0015] Agent: Use DQN algorithm;
[0016] Environment: Build a power grid system model and use Python libraries to simulate load fluctuations and power generation costs;
[0017] State space: defines the set of grid states, including but not limited to load range, generation cost range, and price range;
[0018] Action space: defines the range of user power consumption adjustment, such as increasing or decreasing power consumption by a certain percentage. The percentage needs to be set according to actual conditions and can be adjusted in real time.
[0019] Reward function: Design reward function;
[0020] S3. Training DQN model:
[0021] Collect grid status data and user behavior data, including but not limited to historical load data, power generation cost data, and user power consumption data, and let the DQN agent interact with the environment to perform the following steps:
[0022] Observation status: observe the current power grid status;
[0023] Select action: Based on the current state, use the DQN model to select the optimal power consumption adjustment strategy;
[0024] Execute action: Apply the selected action to the environment, such as adjusting the user's electricity usage;
[0025] Receive rewards: Receive rewards based on feedback from the environment, such as negative grid operating costs;
[0026] Update model: Use reward information to update the DQN model, such as using the Q-learning algorithm to update the neural network weights;
[0027] S4. Model evaluation and optimization:
[0028] Use the test data to evaluate the model performance, such as calculating the average cost reduction, as follows:
[0029] Select a test data set containing different grid states and user behaviors, such as different load levels, power generation costs, and price data. The test set should have the same characteristics and distribution as the training set, but it cannot be the data in the training set. Use the trained DQN model to predict each state in the test set and get the best action recommended by the model, that is, the specific user adjustment of electricity consumption;
[0030] Then calculate the cost, use the power grid model to simulate the actions predicted by the execution model, and calculate the power grid operation cost in each state. The cost can be calculated using the same method as when training the model, such as calculating the power grid operation cost based on load, power generation cost, and price factors;
[0031] Calculate the average cost of all states in the test set. Calculate the average cost of all states in the test set. If using the DQN model for optimization, calculate the average cost of all states.
[0032] By adjusting model parameters, including but not limited to neural network structure and learning rate, the model performance can be improved;
[0033] S5. Distributed computing:
[0034] The computing processes involved in the above S1-S4 are all uploaded to the cloud. The cloud divides the computing data to be processed into multiple shards and assigns them to different computing nodes for processing. Tasks are assigned to different computing nodes based on the computing power of the computing nodes and the computing amount of the tasks, so that they have load balancing and high availability. The computing nodes perform corresponding computing processing according to the assigned tasks and transmitted data, such as data processing, model training, and model reasoning. The results of the computing node processing are collected to the main node for data aggregation and processing, such as summary, analysis, and visualization of the results, and then the results are fed back.
[0035] Furthermore, in step S1, DQN refers to a DQN algorithm which is a reinforcement learning algorithm based on deep learning.
[0036] Furthermore, in step S2, the simplest reward function is to use the negative value of the grid operation cost as the reward, that is:
[0037] Reward = - Grid operating costs
[0038] This reward function encourages the DQN model to choose actions that reduce grid operating costs.
[0039] Furthermore, the reward function introduces a discount factor γ to discount future costs in consideration of the time factor, namely:
[0040] Reward = -(Current Cost + γ * Future Cost)
[0041] Among them, γ is the discount factor, ranging from 0 to 1. The closer γ is to 1, the greater the impact of future costs on rewards.
[0042] Furthermore, the reward function can also introduce a load fluctuation penalty term to avoid excessive fluctuations in user power consumption, namely:
[0043] Reward = -(grid operating cost + α*load fluctuation)
[0044] Among them, α is the load fluctuation penalty coefficient, which ranges from 0 to 1. The larger the α is, the greater the impact of load fluctuation on the reward.
[0045] Furthermore, the reward function needs to take into account user satisfaction. For example, users may not be willing to significantly reduce electricity consumption during peak hours. Therefore, the user satisfaction score is introduced to adjust the reward, that is:
[0046] Reward = -(grid operating cost + β * user satisfaction score)
[0047] Among them, β is the user satisfaction rating coefficient, ranging from 0 to 1. The larger β is, the greater the impact of user satisfaction on rewards.
[0048] Furthermore, the reward function is to more comprehensively consider the grid operation cost, time factor, load fluctuation, and user satisfaction factor, namely:
[0049] Reward = -(grid operating cost + γ*future cost + α*load fluctuation + β*user satisfaction score).
[0050] Furthermore, in step S3, after collecting the grid status data and user behavior data, if the grid data is not sparse enough to meet the training requirements, a generative adversarial network, i.e., GAN, is used to generate artificial grid data. GAN consists of two neural networks: a generator and a discriminator. The generator attempts to generate artificial data similar to real data, while the discriminator attempts to distinguish between real data and generated data. Through continuous training, the generator can generate grid data that is highly similar to real data.
[0051] Furthermore, in step S4, the average cost reduction degree is calculated using the following formula:
[0052] Average cost reduction degree = (average cost - average cost after optimization) ÷ average cost * 100%.
[0053] The present invention provides a demand response optimization method based on deep reinforcement learning under random power grid conditions, which has the following beneficial effects:
[0054] 1. The demand response optimization method based on deep reinforcement learning under the random state of the power grid uploads the calculation process to the cloud, which is divided into slices and distributed to different computing nodes for processing. The free computing resources of each computing node and the computing amount of the computing task are used for allocation. In this way, distributed computing is used in the cloud to calculate each computing process, making full use of the computing resources of each node to avoid idleness, while solving the problem of large consumption of local computing resources due to the scale and complexity of the power grid.
[0055] 2. The demand response optimization method based on deep reinforcement learning under the random state of the power grid. When collecting power grid data, if the data is sparse, artificial power grid data can be generated through a generative adversarial network, namely GAN. GAN consists of two neural networks: a generator and a discriminator. The generator tries to generate artificial data similar to real data, while the discriminator tries to distinguish between real data and generated data. Through continuous training, the generator can generate power grid data that is highly similar to real data, thereby improving the power grid data to ensure that it meets the needs of DQN model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 It is a flow chart of a demand response optimization method based on deep reinforcement learning under random power grid state of the present invention. DETAILED DESCRIPTION
[0057] The following embodiments of the present invention are described in further detail in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.
[0058] like Figure 1 As shown, the present invention provides a technical solution: a demand response optimization method based on deep reinforcement learning under random power grid state, comprising the following steps:
[0059] S1. Problem definition:
[0060] Demand response optimization based on DQN first requires problem definition, including goals, environment, state, action, and reward;
[0061] Among them, the objectives: minimize grid operating costs;
[0062] Environment: simulate the power grid system, including but not limited to load fluctuations and power generation cost factors;
[0063] Status: The current status of the power grid, including but not limited to load, power generation cost, and price;
[0064] Action: The user adjusts the power consumption, such as increasing or decreasing the power consumption;
[0065] Rewards: Negative grid operating costs, i.e. the lower the cost, the higher the reward;
[0066] Among them, DQN is the DQN algorithm, which is a reinforcement learning algorithm based on deep learning;
[0067] S2. DRL framework construction:
[0068] DRL framework is the deep reinforcement learning framework;
[0069] Agent: Use DQN algorithm;
[0070] Environment: Build a power grid system model and use Python libraries to simulate load fluctuations and power generation costs;
[0071] State space: defines the set of grid states, including but not limited to load range, generation cost range, and price range;
[0072] Action space: defines the range of user power consumption adjustment, such as increasing or decreasing power consumption by a certain percentage. The percentage needs to be set according to actual conditions and can be adjusted in real time.
[0073] Reward function: Design reward function;
[0074] Among them, the simplest reward function is to use the negative value of the grid operation cost as the reward, that is:
[0075] Reward = - Grid operating costs
[0076] This reward function encourages the DQN model to choose actions that can reduce the cost of grid operation;
[0077] The reward function introduces a discount factor γ to discount future costs when taking into account the time factor, that is:
[0078] Reward = -(Current Cost + γ * Future Cost)
[0079] Among them, γ is the discount factor, ranging from 0 to 1. The closer γ is to 1, the greater the impact of future costs on rewards;
[0080] The reward function can also introduce a load fluctuation penalty term to avoid excessive fluctuations in user power consumption, namely:
[0081] Reward = -(grid operating cost + α*load fluctuation)
[0082] Among them, α is the load fluctuation penalty coefficient, which ranges from 0 to 1. The larger α is, the greater the impact of load fluctuation on the reward;
[0083] The reward function needs to take into account user satisfaction. For example, users may not be willing to significantly reduce electricity consumption during peak hours, so the user satisfaction score is introduced to adjust the reward, namely:
[0084] Reward = -(grid operating cost + β * user satisfaction score)
[0085] Among them, β is the user satisfaction rating coefficient, ranging from 0 to 1. The larger β is, the greater the impact of user satisfaction on rewards;
[0086] The reward function takes into account the grid operation cost, time factor, load fluctuation, and user satisfaction factor more comprehensively, namely:
[0087] Reward = -(grid operating cost + γ*future cost + α*load fluctuation + β*user satisfaction score);
[0088] S3. Training DQN model:
[0089] Collect grid status data and user behavior data, including but not limited to historical load data, power generation cost data, and user power consumption data, and let the DQN agent interact with the environment to perform the following steps:
[0090] Observation status: observe the current power grid status;
[0091] Select action: Based on the current state, use the DQN model to select the optimal power consumption adjustment strategy;
[0092] Execute action: Apply the selected action to the environment, such as adjusting the user's electricity usage;
[0093] Receive rewards: Receive rewards based on feedback from the environment, such as negative grid operating costs;
[0094] Update model: Use reward information to update the DQN model, such as using the Q-learning algorithm to update the neural network weights;
[0095] Among them, after collecting grid status data and user behavior data, if the grid data is not sparse enough to meet the training, the generative adversarial network (GAN) is used to generate artificial grid data. GAN consists of two neural networks: the generator and the discriminator. The generator tries to generate artificial data similar to the real data, while the discriminator tries to distinguish between the real data and the generated data. Through continuous training, the generator can generate grid data that is highly similar to the real data.
[0096] S4. Model evaluation and optimization:
[0097] Use the test data to evaluate the model performance, such as calculating the average cost reduction, as follows:
[0098] Select a test data set containing different grid states and user behaviors, such as different load levels, power generation costs, and price data. The test set should have the same characteristics and distribution as the training set, but it cannot be the data in the training set. Use the trained DQN model to predict each state in the test set and get the best action recommended by the model, that is, the specific user adjustment of electricity consumption;
[0099] Then calculate the cost, use the power grid model to simulate the actions predicted by the execution model, and calculate the power grid operation cost in each state. The cost can be calculated using the same method as when training the model, such as calculating the power grid operation cost based on load, power generation cost, and price factors;
[0100] Calculate the average cost of all states in the test set. Calculate the average cost of all states in the test set. If using the DQN model for optimization, calculate the average cost of all states.
[0101] The average cost reduction is calculated using the following formula:
[0102] Average cost reduction degree = (average cost - average cost after optimization) ÷ average cost * 100%;
[0103] By adjusting model parameters, including but not limited to neural network structure and learning rate, the model performance can be improved;
[0104] Specific formula:
[0105] Assume that there are N states in the test set, and the cost of each state is C1, C2, ..., CN. The costs after optimization using the DQN model are C1', C2', ..., CN'.
[0106] but:
[0107] Average cost = (C1+C2+...+CN) / N
[0108] Average cost after optimization = (C1'+C2'+...+CN') / N
[0109] Average cost reduction = ((C1+C2+...+CN)÷N-(C1'+C2'+...+CN')÷N)÷((C1+C2+...+CN)÷N)*100%;
[0110] Example:
[0111] Assume that there are 10 states in the test set, and the cost of each state is 100, 110, 120, ..., 190, and the cost after optimization using the DQN model is 90, 100, 110, ..., 180 respectively;
[0112] but:
[0113] Average cost = (100 + 110 + 120 + ... + 190) / 10 = 145
[0114] Average cost after optimization = (90+100+110+...+180) / 10 = 135
[0115] Average cost reduction = (145-135) / 145*100% = 6.9%
[0116] This means that the DQN model is able to reduce grid operating costs by an average of 6.9%;
[0117] Using test data to evaluate the performance of the DQN model and calculate the average cost reduction can help us understand the effectiveness of the model and evaluate whether the model can effectively reduce grid operating costs;
[0118] S5. Distributed computing:
[0119] The computing processes involved in the above S1-S4 are all uploaded to the cloud. The cloud divides the computing data to be processed into multiple shards and assigns them to different computing nodes for processing. Tasks are assigned to different computing nodes based on the computing power of the computing nodes and the computing amount of the tasks, so that they have load balancing and high availability. The computing nodes perform corresponding computing processing according to the assigned tasks and transmitted data, such as data processing, model training, and model reasoning. The results of the computing node processing are collected to the main node for data aggregation and processing, such as summary, analysis, and visualization of the results, and then the results are fed back.
[0120] Based on the above description, the present invention uploads the process involving calculation to the cloud, which is then divided and distributed to different computing nodes for processing, and the free computing resources of each computing node and the computing amount of the computing task are used for distribution, thereby using distributed computing in the cloud to calculate each computing process, making full use of the computing resources of each node to avoid idleness, while solving the problem of very large consumption of local computing resources due to the scale and complexity of the power grid;
[0121] When collecting power grid data, the present invention can generate artificial power grid data through a generative adversarial network, namely GAN, if the data is sparse. GAN consists of two neural networks: a generator and a discriminator. The generator attempts to generate artificial data similar to real data, while the discriminator attempts to distinguish between real data and generated data. Through continuous training, the generator can generate power grid data that is highly similar to real data, thereby improving the power grid data to ensure that it meets the requirements of DQN model training.
[0122] In summary, the demand response optimization method based on deep reinforcement learning under the random state of the power grid has the following specific process when used:
[0123] Define the environment: Use Python libraries to build a power grid environment model, such as the Gym library;
[0124] Define state space: define the set of grid states, such as load, generation cost, price, etc.
[0125] Define action space: define the range of user power consumption adjustment, such as increasing or decreasing power consumption by a certain percentage;
[0126] Define reward function: Define reward function, for example, reward = -(grid operating cost);
[0127] Build a DQN model: Use deep learning frameworks such as PyTorch or TensorFlow to build a DQN model;
[0128] Training model: Use the collected data to train the DQN model and use the test data to evaluate the model performance;
[0129] Optimize the model: adjust model parameters, such as neural network structure, learning rate, etc., to improve model performance;
[0130] Deployment model: Deploy the trained DQN model to the actual environment to achieve demand response optimization.
[0131] The embodiments of the present invention are given for the purpose of illustration and description, and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.
Claims
1. A demand response optimization method based on deep reinforcement learning under random power grid conditions, characterized by: The demand response optimization method based on deep reinforcement learning under random power grid conditions includes the following steps: S1. Problem definition: Demand response optimization based on DQN first requires problem definition, including goals, environment, state, action, and reward; Among them, the objectives: minimize grid operating costs; Environment: simulate the power grid system, including but not limited to load fluctuations and power generation cost factors; Status: The current status of the power grid, including but not limited to load, power generation cost, and price; Action: The user adjusts the power consumption, such as increasing or decreasing the power consumption; Rewards: Negative grid operating costs, i.e. the lower the cost, the higher the reward; S2. DRL framework construction: Agent: Use DQN algorithm; Environment: Build a power grid system model and use Python libraries to simulate load fluctuations and power generation costs; State space: defines the set of grid states, including but not limited to load range, generation cost range, and price range; Action space: defines the range of user power consumption adjustment, such as increasing or decreasing power consumption by a certain percentage. The percentage needs to be set according to actual conditions and can be adjusted in real time. Reward function: Design reward function; S3. Training DQN model: Collect grid status data and user behavior data, including but not limited to historical load data, power generation cost data, and user power consumption data, and let the DQN agent interact with the environment to perform the following steps: Observation status: observe the current power grid status; Select action: Based on the current state, use the DQN model to select the optimal power consumption adjustment strategy; Execute action: Apply the selected action to the environment, such as adjusting the user's electricity usage; Receive rewards: Receive rewards based on feedback from the environment, such as negative grid operating costs; Update model: Use reward information to update the DQN model, such as using the Q-learning algorithm to update the neural network weights; S4. Model evaluation and optimization: Use the test data to evaluate the model performance, such as calculating the average cost reduction, as follows: Select a test data set containing different grid states and user behaviors, such as different load levels, power generation costs, and price data. The test set should have the same characteristics and distribution as the training set, but it cannot be the data in the training set. Use the trained DQN model to predict each state in the test set and get the best action recommended by the model, that is, the specific user adjustment of electricity consumption; Then calculate the cost, use the power grid model to simulate the actions predicted by the execution model, and calculate the power grid operation cost in each state. The cost can be calculated using the same method as when training the model, such as calculating the power grid operation cost based on load, power generation cost, and price factors; Calculate the average cost of all states in the test set. Calculate the average cost of all states in the test set. If using the DQN model for optimization, calculate the average cost of all states. By adjusting model parameters, including but not limited to neural network structure and learning rate, the model performance can be improved; S5. Distributed computing: The computing processes involved in the above S1-S4 are all uploaded to the cloud. The cloud divides the computing data to be processed into multiple shards and assigns them to different computing nodes for processing. Tasks are assigned to different computing nodes based on the computing power of the computing nodes and the computing amount of the tasks, so that they have load balancing and high availability. The computing nodes perform corresponding computing processing according to the assigned tasks and transmitted data, such as data processing, model training, and model reasoning. The results of the computing node processing are collected to the main node for data aggregation and processing, such as summary, analysis, and visualization of the results, and then the results are fed back.
2. According to claim 1, a demand response optimization method based on deep reinforcement learning under random power grid state is characterized by: In step S1, DQN is a DQN algorithm which is a reinforcement learning algorithm based on deep learning.
3. The demand response optimization method based on deep reinforcement learning under random power grid state according to claim 1 is characterized by: In step S2, the simplest reward function is to use the negative value of the grid operation cost as the reward, that is: Reward = - Grid operating costs This reward function encourages the DQN model to choose actions that reduce grid operating costs.
4. The demand response optimization method based on deep reinforcement learning under random power grid state according to claim 3 is characterized by: The reward function introduces a discount factor γ to discount future costs in consideration of time, namely: Reward = -(Current Cost + γ * Future Cost) Among them, γ is the discount factor, ranging from 0 to 1. The closer γ is to 1, the greater the impact of future costs on rewards.
5. The method for optimizing demand response based on deep reinforcement learning under random power grid conditions according to claim 4, characterized in that: The reward function can also introduce a load fluctuation penalty term to avoid excessive fluctuations in user power consumption, namely: Reward = -(grid operating cost + α*load fluctuation) Among them, α is the load fluctuation penalty coefficient, which ranges from 0 to 1. The larger the α is, the greater the impact of load fluctuation on the reward.
6. The demand response optimization method based on deep reinforcement learning under random power grid state according to claim 5 is characterized by: The reward function needs to take into account user satisfaction. For example, users may not be willing to significantly reduce electricity consumption during peak hours. Therefore, the user satisfaction score is introduced to adjust the reward, namely: Reward = -(grid operating cost + β * user satisfaction score) Among them, β is the user satisfaction rating coefficient, ranging from 0 to 1. The larger β is, the greater the impact of user satisfaction on rewards.
7. The method for optimizing demand response based on deep reinforcement learning under random power grid conditions according to claim 6, characterized in that: The reward function is designed to more comprehensively consider the grid operation cost, time factor, load fluctuation, and user satisfaction factor, namely: Reward = -(grid operating cost + γ*future cost + α*load fluctuation + β*user satisfaction score).
8. The demand response optimization method based on deep reinforcement learning under random power grid state according to claim 1 is characterized by: In step S3, after collecting the grid state data and user behavior data, if the grid data is not sparse enough to meet the training, a generative adversarial network, i.e., GAN, is used to generate artificial grid data. GAN consists of two neural networks: a generator and a discriminator. The generator attempts to generate artificial data similar to real data, while the discriminator attempts to distinguish between real data and generated data. Through continuous training, the generator can generate grid data that is highly similar to real data.
9. The method for optimizing demand response based on deep reinforcement learning under random power grid conditions according to claim 1, characterized in that: In step S4, the average cost reduction degree is calculated using the following formula: Average cost reduction degree = (average cost - average cost after optimization) ÷ average cost * 100%.