Power grid load side response intelligent decision-making system and method adopting deep reinforcement learning

Abstract

The invention relates to the technical field of power grid load adjustment, and relates to a power grid load side response intelligent decision-making system and method adopting deep reinforcement learning, and the method comprises the steps: setting a central server, dividing the service coverage of a power grid side into N service areas, and setting a cloud end for each service area; collecting user historical power consumption data, weather data, real-time electricity prices and power grid states in the N service areas, and inputting the data to the cloud of the service area; and the central server initializes the global decision model and the user demand prediction model, and sends the global decision model and the user demand prediction model to the cloud of the service area. According to the method, a power grid load scheduling decision has self-learning and self-adaptive capabilities, and a scheduling strategy can be continuously optimized according to real-time data and historical data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power grid load adjustment, and more particularly to a power grid load side response intelligent decision system and method using deep reinforcement learning. BACKGROUND

[0002] Power grid load response is an important part of power system operation and management. The power system is a typical large-scale linear system, and its operation must follow the principle of supply and demand balance. With the development of society, population growth and the popularization of electrical appliances have led to increasingly complex power grid loads. On the one hand, various new types of loads such as data centers and electric vehicles are growing rapidly, and their dynamic characteristics make load forecasting more difficult. On the other hand, the rapid integration of renewable energy sources also brings new challenges to load management. Traditionally, the power grid adjusts the output of generator units to track the load and maintain the frequency and rated voltage. However, due to the instability of renewable energy output, it is difficult to quickly respond to load changes by adjusting power generation. This requires the introduction of load side technology to compensate for the instability of renewable energy by consciously controlling certain loads with certain rapid response capabilities.

[0003] The power supply scale of the power system is growing rapidly, but due to the limitations of traditional load characteristics and power supply methods, the load response level of local power grids is still not high. At present, by increasing user participation awareness and exploring related technologies, it is the best choice to realize load side flexibility under the smart grid system.

[0004] For example, the Chinese patent with the authorization announcement number CN111342446B discloses a three-dimensional home smart energy use ubiquitous scheduling control method and system, which is based on "time scale, spatial scale, and operating state" three-dimensional scheduling management, and realizes efficient interaction between the power grid side and the user side through a mobile phone APP platform, and realizes encrypted transmission of interactive data through blockchain technology. It has the function of fully tapping the load demand response potential of the user side, reducing the pressure of load scheduling on the power grid side, and can simultaneously meet the requirements of improving the satisfaction and comfort of the power grid side load scheduling and the user side, and has high safety and reliability.

[0005] However, the above-mentioned technology has the following problems:

[0006] In actual operation, more emphasis is placed on the demand of the user side, and the huge data processing demand of the power grid side is often ignored. This imbalance leads to difficulties in overall planning when load scheduling is performed on the power grid side, which is specifically manifested in that when the power grid side processes and analyzes a large amount of data, it cannot respond or accurately predict the demand changes of the user side in a timely manner, resulting in a large deviation between the scheduling strategy and the actual demand of the user, which not only affects the accuracy of load scheduling, but also causes the power grid operation efficiency to decline, and even leads to the problem of waste or uneven distribution of power resources. SUMMARY

[0007] To solve the above problems, the application provides a power grid load side response intelligent decision method using deep reinforcement learning, which comprises the following steps:

[0008] A central server is established, and the service coverage of the power grid side is divided into N service areas, and a cloud is established in each service area;

[0009] Collect the user historical power consumption data, weather data, real-time electricity price and power grid state in the N service areas, and input them into the cloud of the service area;

[0010] The central server initializes the global decision model and the user demand prediction model, and sends the global decision model and the user demand prediction model to the cloud of the service area;

[0011] In the cloud, the user historical power consumption data, weather data, real-time electricity price and power grid state in the service area are used to perform localized training of the global decision model and the user demand prediction model in the service area, and the model parameters are output to the blockchain network, including the model parameters of the user demand prediction model and the model parameters of the global decision model;

[0012] The central server calls the model parameters in the blockchain network through the smart contract calling interface, aggregates the model parameters, updates the global decision model and the user demand prediction model with the aggregated model parameters, and sends the updated global decision model and the user demand prediction model to all clouds;

[0013] Input the user historical power consumption data, weather data and real-time electricity price into the user demand prediction model, and output the power consumption demand; input the power consumption demand and the power grid state into the global decision model, and output the load scheduling strategy.

[0014] Further, the method for obtaining the number of service areas is to determine the basic partition demand by the ratio of the number of users to the maximum number of users that each service area can handle, to amplify the basic partition number in combination with the power consumption mode diversity factor to adapt to the complexity of the power consumption mode, and to dynamically optimize the division of the service coverage of the power grid side by using the average constraint of the computing resource demand factor and the data transmission cost factor, to obtain the number of service areas.

[0015] Further, the initialization and construction method of the global decision model comprises:

[0016] Define the state space:

[0017] The state s includes: the power consumption demand is represented by a real number, the power grid state includes the current load, power generation, energy storage state and power transmission line state of the power grid, the current load and power generation of the power grid are represented by real numbers, and the energy storage state and power transmission line state are represented by 0-1 standardized real numbers;

[0018] Define the action space:

[0019] The action a is the next load scheduling value, the sign of each load scheduling value indicates addition or subtraction, and the absolute value of the real number indicates the percentage of the amount of load scheduling;

[0020] Design the Actor network structure:

[0021] The Actor network structure includes a first input layer, M hidden layers, and a second output layer, and the activation functions of the hidden layers are ReLu; the second output layer has no activation function;

[0022] Design the Critic network structure:

[0023] The Critic network structure includes a second input layer, M hidden layers, and a third output layer; the activation functions of the hidden layers are ReLu; the third output layer includes at least one node and has no activation function;

[0024] Define the Actor network structure and the Critic network structure using the Keras library or the reinforcement learning framework.

[0025] Further, the localization training method of the user demand prediction model comprises:

[0026] Divide the user historical power consumption data, weather data, real-time electricity price, and power consumption demand into a training set and a validation set; input the historical power consumption data, weather data, and real-time electricity price, and generate a power consumption demand prediction value through forward propagation; update the user demand prediction model parameters using the gradient descent algorithm, and after the loss function converges, encrypt and upload the final user demand prediction model parameters to the blockchain network.

[0027] Further, the localization training method of the global decision-making model includes the methods of Actor network training and Critic network training:

[0028] Calculate the action probability distribution using forward propagation, calculate the Actor network loss function based on the action probability distributions of the new and old strategies and the advantage function, update the Actor network parameters through the Actor network loss function; the Critic network training and the Actor network training are performed synchronously, and the training method is the same as the localization training method of the user demand prediction model. After convergence, the Actor network and Critic network parameters are used as the model parameters of the global decision-making model, and are encrypted and uploaded to the blockchain network.

[0029] Further, the aggregation method is a weighted average method, which aggregates the model parameters of each cloud through the weighted average method, and the weight is determined by the ratio of the data volume of the cloud to the total number of clouds.

[0030] Further, it further comprises:

[0031] collecting weather forecast information, the weather forecast information including weather data and extreme weather warning information;

[0032] inputting the weather forecast information into a pre-constructed power grid state prediction model, outputting a power grid predicted state, and replacing a power grid state with the power grid predicted state.

[0033] Further, the method for constructing the power grid state prediction model comprises:

[0034] using the weather forecast information as analysis data, collecting a group of power grid predicted states corresponding to the analysis data in advance, and converting the analysis data and the corresponding power grid predicted states into a corresponding group of feature vectors;

[0035] using each group of feature vectors as an input of the power grid state prediction model, using a group of power grid predicted states corresponding to each group of analysis data as an output of the power grid state prediction model, using an actual power grid predicted state corresponding to each group of analysis data as a prediction target, the actual power grid predicted state being the power grid predicted state collected in advance corresponding to the analysis data, using minimization of a sum of prediction errors of all analysis data as a training target, training the power grid state prediction model until the sum of prediction errors reaches convergence, and stopping the training; the power grid state prediction model being a deep neural network model.

[0036] An intelligent decision-making system for power grid load side response based on deep reinforcement learning, which implements the intelligent decision-making method for power grid load side response based on deep reinforcement learning, and comprises:

[0037] a cloud creation module: a central server is set up, a service coverage of a power grid side is evenly divided into N service areas, and one cloud is set up in each service area;

[0038] a collection module: user historical power consumption data, weather forecast information and real-time electricity prices in the N service areas are collected, the weather forecast information is inputted into a pre-constructed power grid state prediction model, a power grid predicted state is outputted, and the user historical power consumption data, the weather forecast information, the real-time electricity prices and the power grid predicted state are inputted into the cloud of the service area;

[0039] a model initialization module: a global decision-making model and a user demand prediction model are initialized using the central server, and the global decision-making model and the user demand prediction model are sent to the cloud of the service area;

[0040] a local training module: the global decision-making model and the user demand prediction model are locally trained using the user historical power consumption data, the weather forecast information, the real-time electricity prices and the power grid state in the cloud, and model parameters are outputted;

[0041] The model updating module: the central server calls the model parameters in the blockchain network through the smart contract calling interface, aggregates the model parameters, updates the global decision model and the user demand prediction model with the aggregated model parameters, and sends the updated global decision model and the user demand prediction model to all cloud ends;

[0042] The load scheduling module: input the user historical power consumption data, weather forecast information and real-time electricity price into the global decision model, and output the power consumption demand; input the power consumption demand and the power grid prediction state into the global decision model, and output the load scheduling strategy.

[0043] The power grid load side response intelligent decision method and system provided by the application have the following technical effects and advantages:

[0044] Technical effects:

[0045] The application adopts a distributed cloud computing architecture, divides the service coverage range into multiple service areas, sets up independent clouds in each service area for localized processing, encrypts and stores data in a distributed manner through blockchain technology, realizes data simplification and automatic processing, which not only reduces the computing pressure and deployment cost of the central server, but also improves the overall operation efficiency and reliability of the system, so that the power grid side can respond or accurately predict the demand change of the user side in time when processing and analyzing a large amount of data, so that the scheduling strategy and the actual demand of the user are more matched, the deep reinforcement learning model is introduced, the user demand is predicted by collecting user historical power consumption data, weather data, real-time electricity price and other multi-source data, the waste of power resources is reduced, the uneven distribution is avoided, the updated reinforcement learning model is used as the output of the continuous precise load scheduling strategy, the data interaction and cooperation between the power grid side and the user side are realized, and the precision of the load scheduling and the overall stability of the system are ensured.

[0046] The user demand is predicted in advance and dynamically adjusted to ensure that the power grid can quickly respond to load changes and improve user satisfaction.

[0047] Advantages:

[0048] Through the deep reinforcement learning model, the user's electricity demand can be predicted in real time, and the optimization scheduling is combined with the grid state to improve the efficiency of the grid load management. By introducing user historical electricity data, weather data, real-time electricity price and other data sources, the model can more accurately predict electricity demand and reduce prediction error. Improved blockchain technology is used to encrypt and transmit model parameters to ensure data security and privacy, effectively preventing data leakage and tampering. By calculating the number of service areas and the corresponding resource demand factor, the optimization allocation of grid resources is realized to ensure that the computing resources of each service area can be reasonably utilized. By introducing weather forecast information and major event data, the model can respond to emergencies in a timely manner and adjust the grid load scheduling strategy to improve the emergency handling capacity of the grid. The deep reinforcement learning algorithm enables the grid load scheduling decision to have self-learning and adaptive capabilities, which can continuously optimize the scheduling strategy based on real-time data and historical data. By setting up multiple clouds, distributed processing of data in each service area reduces the computing pressure on the central server and improves the overall processing efficiency and reliability of the system. By using the method of local training and global decision model parameter aggregation, the global decision model and user demand prediction model are continuously updated and optimized to ensure that the model always remains efficient and accurate. Through accurate prediction and intelligent scheduling, power waste is reduced and the utilization of power resources is optimized, which helps to save energy and promote green energy development. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 Flow chart of the method for intelligent decision-making of grid load side response using deep reinforcement learning in embodiment 1;

[0050] Figure 2 Timing chart of the method for intelligent decision-making of grid load side response using deep reinforcement learning in embodiment 1;

[0051] Figure 3 Connection diagram of the system for intelligent decision-making of grid load side response using deep reinforcement learning in embodiment 3. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0053] Embodiment 1

[0054] Please refer to Figures 1-2 The method for intelligent decision-making of grid load side response using deep reinforcement learning described in this embodiment, the method comprises:

[0055] A central server is established, and the service coverage of the power grid side is evenly divided into N service areas, and a cloud is established in each service area;

[0056] The user historical power consumption data, weather data, real-time electricity price and power grid state in the N service areas are collected and input into the cloud of the corresponding service area;

[0057] The central server initializes a global decision-making model and a user demand prediction model, and sends the global decision-making model and the user demand prediction model to the cloud of the service area;

[0058] The user historical power consumption data, weather data, real-time electricity price and power grid state in the cloud are used to perform localized training of the global decision-making model and the user demand prediction model in the service area, and the model parameters are output to the blockchain network, including the model parameters of the user demand prediction model and the model parameters of the global decision-making model;

[0059] The central server calls the model parameters in the blockchain network through the smart contract calling interface, aggregates the model parameters, updates the global decision-making model and the user demand prediction model with the aggregated model parameters, and sends the updated global decision-making model and the user demand prediction model to all clouds;

[0060] The user historical power consumption data, weather data and real-time electricity price are input into the user demand prediction model to output the power consumption demand, and the power consumption demand and the power grid state are input into the global decision-making model to output the load scheduling strategy.

[0061] The service coverage of the power grid side is evenly divided into N service areas, which not only disperses the computing power pressure of the central server, but also balances the scheduling. The actual power consumption of each area is completely different, even with a large difference, such as industrial areas and residential areas. The power consumption of industrial areas is much larger than that of residential areas, and the load level is always high and stable. Residential areas are also divided into ordinary residential areas and high-end residential areas. The ordinary residential areas are greatly affected by the real-time electricity price fluctuations, while the high-end residential areas are mainly affected by the time period, and the influence of the real-time electricity price is small. The objective gap leads to the larger the service coverage, the more training data the model needs, and the greater the risk of overfitting. That is, the more inaccurate the data, the higher the cost of power scheduling, which affects the accuracy of subsequent load scheduling strategy derivation. Therefore, reasonable division is necessary.

[0062] The service area number acquisition method is that the ratio of the number of users U and the maximum number of users that each service area can handle determines the basic partition demand, combines the diversity factor D of the power consumption mode to amplify the basic partition number to adapt to the complexity of the power consumption mode, and uses the average constraint of the computing resource demand factor C and the data transmission cost factor T to dynamically optimize the division of the service coverage range of the power grid side to obtain the service area number N; finally, flexible optimization of service coverage division is realized, and the efficiency of power grid load response decision, the balance of resource allocation and the scalability of the system are considered. The specific calculation example of the service area number is as follows:

[0063]

[0064] In the formula, U represents the total number of users; U max is the maximum number of users that each service area can handle, which is a preset value, and the ratio of U and U max represents the basic service area number; D is the diversity factor of the power consumption mode, which is a preset value, for example, the set of D is (ordinary area D=0, complex area D=0.5, very complex area D=1), (1+D), which reflects the diversity adjustment, and the basic service area demand is amplified by the diversity factor D to cope with the complexity and difference of the user power consumption mode; C is the computing resource demand factor, which is a preset value, for example, the set of C is (basic demand C=1, high demand C=2); T is the data transmission cost factor, which is a preset value, for example, the set of T is (low cost T=1, high cost T=2), The average of the computing resource demand C and the data transmission cost T is taken to avoid single factor dominance, balance the constraints of calculation and communication, dynamically integrate user scale, diversity and resource cost, optimize service area division, and ensure the efficiency of power grid load response and system scalability.

[0065] The user historical power consumption data is recorded by smart meters for each user's power consumption in a certain time period, and the meter sends the power consumption data to the power company periodically (e.g. every hour), the user historical power consumption data reflects the user's power consumption habits and patterns at different times and situations, by analyzing these data, the user's power demand in the future similar situation can be predicted, for example, a certain user's power consumption is higher every summer evening in the past, which indicates that the user may also have higher power demand in the future summer at the same time period; weather data includes temperature, humidity, precipitation and wind speed information, the power company can obtain future weather data through cooperation with meteorological agencies or using public weather API, weather has a significant impact on power demand, high temperature weather usually leads to the increase of the use of air conditioning and other refrigeration equipment, thereby increasing the power demand, similarly, cold weather will increase the use of heating equipment, in addition, meteorological conditions such as precipitation and wind speed will also affect the user's power consumption behavior; real-time electricity price is the current price of electricity in the electricity market, which is usually updated at an hourly or shorter interval, the power company can obtain information through the electricity trading platform, the fluctuation of real-time electricity price will affect the user's power consumption behavior, the user may choose to use electricity at a lower price period to save costs, for example, under the peak-valley electricity price mechanism, the user may carry out high-power consumption activities such as washing and charging at night when the electricity price is lower; the grid state includes the current load of the grid, the power generation capacity, the energy storage state and the transmission line state, the values of the energy storage state and the transmission line state represent the degree of good or bad of the energy storage and the line state, the power company obtains the data in real time through the monitoring system, the grid state directly affects the formulation of the dispatching strategy, for example, if a transmission line fails, the transmission path of electricity needs to be adjusted, if the power generation capacity of a power station is insufficient, electricity needs to be dispatched from other power stations or standby power sources need to be started, if a main transmission line fails, the load of other lines needs to be adjusted quickly, or even standby lines need to be started, to ensure uninterrupted power supply; power demand is a key factor of load dispatching strategy, during high power demand period, power supply needs to be increased to avoid power shortage, during low power demand period, power supply needs to be reduced to avoid waste and unnecessary cost, for example, during peak period (assuming 6pm to 9pm), power generation capacity needs to be increased or energy storage system needs to be used to provide electricity; during valley period (assuming 2am to 5am), power generation capacity can be reduced, or even some generators can be shut down for maintenance.

[0066] The initialization construction method of the user demand prediction model includes:

[0067] Designing an initial model structure:

[0068] The design input shape is (time step, feature number), the time step refers to the difference between the two time steps, which is used to determine the time concept reference of the whole model, for example, defined as 24 hours; the feature number is 3, that is, the user's historical power consumption data, weather data and real-time electricity price.

[0069] The LSTM layer is designed, and two initial LSTM layers are defined, including a first LSTM layer and a second LSTM layer.

[0070] The full connection layer is designed, and two initial full connection layers are defined, including a first full connection layer and a first output layer; the activation function of the first full connection layer is set to ReLu; and the activation function of the first output layer is set to linear.

[0071] The model is compiled.

[0072] The loss function is MSE, the optimizer is adam, and the model is configured to training mode using the compile function.

[0073] The above design is defined using the Keras library or deep learning framework.

[0074] It is particularly pointed out that: using LSTM as the main structure of the model can be used to predict the power demand in the future period, and the dynamic change of the power demand is predicted in advance, thereby laying a foundation for subsequent load scheduling.

[0075] The initialization construction method of the global decision model comprises:

[0076] The state space is defined.

[0077] The state s includes: the power demand is represented by a real number, the power grid state includes the current load, power generation, energy storage state and power transmission line state of the power grid, the current load and power generation of the power grid are represented by real numbers, and the energy storage state and power transmission line state are represented by 0-1 standardized real numbers; the state vector dimension is set to 4.

[0078] The action space is defined.

[0079] The action a is the next load scheduling value, the sign of each load scheduling value indicates addition or subtraction, and the absolute value of the real number indicates the percentage of the scheduled load.

[0080] The Actor network structure is designed.

[0081] The Actor network structure includes a first input layer, M hidden layers and a second output layer, M is a preset value, and the activation functions of the hidden layers are all ReLu; the second output layer has no activation function.

[0082] The Critic network structure is designed.

[0083] The Critic network structure comprises a second input layer, M hidden layers and a third output layer; the activation functions of the hidden layers are ReLu; the third output layer comprises one node and has no activation function.

[0084] The Actor network structure and the Critic network structure are defined using the Keras library or a reinforcement learning framework.

[0085] It is particularly pointed out that:

[0086] The complete state space and action space are defined, which can well describe the nature of the problem and provide a basis for training. The state space and action space are the visualized interpretation of data storage. Since reinforcement learning does not have a black box effect, it is necessary to construct a space for storing basic data in advance. The state space is the space for storing "state" data, and the action space is the space for storing "action" data. The closer the values of the energy storage state and the power transmission line state are to 1, the better the state is, and the closer the values are to 0, the worse the state is. The Actor network is a module for generating actions. It can gradually improve the effect of actions through learning. The Critic network evaluates the value of the actions generated by the Actor network and gives feedback to help the Actor network training. Multiple hidden layers can fit more complex mapping relationships between states and actions. Using reinforcement learning can prioritize electricity demand and supplement power grid state to output load scheduling strategies, thereby accurately achieving load scheduling based on the dynamic changes in user electricity demand. At the same time, the reinforcement learning model is not used to directly output the strategy, but is processed by two models. The principle is that if the electricity demand is not forward-looking, the strategy output will lose real-time performance, causing errors in strategy adjustment.

[0087] The local training method of the user demand prediction model comprises:

[0088] The cloud-stored user historical electricity consumption data, weather data, real-time electricity price and electricity demand are divided into a training set and a validation set. The input Xt (i.e. historical electricity consumption data, weather data and real-time electricity price) is forward propagated to generate a electricity demand prediction value The user demand prediction model parameters are updated using a gradient descent algorithm (learning rate controls step size). When the loss function converges, the final user demand prediction model parameters are encrypted and uploaded to the blockchain network.

[0089] The forward propagation calculates the prediction value

[0090]

[0091] In the formula, f L is a function of the user demand prediction model.

[0092] The loss function Ls(θl) is calculated:

[0093]

[0094] where p is the total number of training sets, i represents the number of training sets, Yt i is the measured value of electricity demand, is the predicted value of electricity demand.

[0095] The model parameters θl are updated using the gradient descent algorithm:

[0096]

[0097] where ε is a preset learning rate, represents the gradient of the user demand prediction model, and the formula is a computer language expression.

[0098] When the loss function Ls(θl) converges, the cloud stops local training and uploads θl as the model parameters of the user demand prediction model to the blockchain network and encrypts it.

[0099] It is particularly pointed out that:

[0100] Forward propagation calculates the predicted value, which can realize the comparison between the real demand and the predicted demand, provide a guidance signal for model training, use the mean square loss function to describe the prediction error, and directly reflect the prediction ability of the model; the gradient descent update algorithm can make the model parameters approach the real demand distribution, effectively improving the prediction accuracy.

[0101] The local training method of the global decision model includes Actor network training and Critic network training, wherein the training method of the Actor network (i.e. the policy network) training includes:

[0102] The action probability distribution is calculated by forward propagation, the action probability distribution and the advantage function of the new and old policies are used to calculate the Actor network loss function, and the Actor network parameters are updated through the Actor network loss function; the Critic network training is performed synchronously with the Actor network training, and the training method is the same as the local training method of the user demand prediction model. After convergence, the Actor network and Critic network parameters are used as the model parameters of the global decision model and are encrypted and uploaded to the blockchain network. Specifically as follows:

[0103] The action probability distribution π(a|s; Qp) calculated by forward propagation:

[0104] π(a|s; Qp) = fp(s; Qp);

[0105] where fp is a function representation of the policy network, and Qp represents the parameters of the policy network.

[0106] The computing strategy loss function is Lp(Qp):

[0107]

[0108] In the formula, E t represents the expectation at time step t; π(a t |s t ) represents the probability distribution of action a old and state s old at time step t under the old strategy network parameter Qp t ; t is an advantage function, representing the advantage estimation at time step t. The strategy loss function is constructed by multiplying the ratio of new and old strategy action probabilities and the advantage function, limiting the amplitude of policy update (preventing mutation), and using the advantage function to guide the optimization direction, that is, retaining the stability of the old strategy and dynamically adjusting the action weight through the advantage value, thereby balancing exploration and utilization.

[0109] The strategy network parameters are updated using the gradient descent algorithm:

[0110]

[0111] In the formula, represents the gradient of the global decision model, and the formula is a computer language expression.

[0112] The Critic network training is synchronized with the Actor network training, and the training method is the same as the localization training method of the user demand prediction model, which will not be described here. The Critic network parameter is set as Qv.

[0113] When Lp(Qp) converges, the cloud stops local training, and Qp and Qv are used as model parameters of the global decision model and uploaded to the blockchain network and encrypted.

[0114] It is particularly pointed out that:

[0115] The Actor network calculates the action probability distribution through forward propagation, describes how the strategy selects actions, and provides update guidance for training. The expected reward strategy loss function is used to effectively achieve the optimization of the objective function, and the reinforcement model selects high-value action ability; the gradient descent algorithm can effectively improve the strategy in a complex space, so that the Actor network approximates the optimal strategy; the Critic network evaluates the action value through the advantage function, guides the Actor network training, realizes the neural reinforcement learning idea of Actor-Critic, and synchronously learns Actor and Critic to ensure effective cooperation of model training.

[0116] ​The method of the aggregation processing is a weighted average method, the method of the aggregation processing is a weighted average method, the model parameters of each cloud are aggregated by the weighted average method, the weight is determined by the ratio of the data amount of the cloud to the total number of clouds, aiming to dynamically adjust the influence of each cloud on the global model through the data amount - the larger the data amount, the higher the weight, so as to more fully reflect its contribution; at the same time, the distribution difference of multi-source data is balanced through the averaging operation, and the single cloud is avoided to dominate the model update. The design of the formula takes into account the fairness of distributed cooperation and the weight allocation of data driving, and finally the global parameters after aggregation are encrypted and uploaded to the blockchain, ensuring the security and credibility of model updating. The specific calculation method includes:

[0117] The model parameters of each cloud are obtained, and the model parameters of each cloud are set as represents the jth model parameter of the nth cloud; including Qp, Qv and θl, the model parameters are the bias and weight of the model;

[0118] The calculation method of the aggregated model parameters

[0119]

[0120] In the formula, F is the data amount of the cloud in the blockchain network; n is the number of the cloud, the total number of clouds is equal to the number of services N, j is the number of the model parameters in the cloud, and Fn is the data amount of the nth cloud in the blockchain network; is the jth global model parameter, including Qp, Qv and θl;

[0121] The role of the aggregation processing is indirect overall planning, and the pressure and deployment cost of direct overall planning are too large, which is easy to lose the accuracy of scheduling under the erosion of complex data. After the model parameters are aggregated, they are used for new global decision model and user demand prediction model, so that the load scheduling strategy can be suitable for precise load scheduling of multi-party service area and the ability of automatic processing.

[0122] Embodiment 2

[0123] As shown in Figures 1-2 , the embodiment further improves the design on the basis of embodiment 1, and the difference is that:

[0124] The power grid load side response intelligent decision method of embodiment 1 using deep reinforcement learning ignores the influence of extreme weather on the state of the power grid in actual operation, for example, hail and cold wave will cause serious influence on the state of the power grid; the extreme weather needs to be known in advance, otherwise the load scheduling when the extreme weather comes is easy to cause overload of the power grid. Based on this, the power grid load side response intelligent decision method using deep reinforcement learning provided by the embodiment further includes: ​

[0125] Collect weather forecast information, including weather data and extreme weather warning information;

[0126] Input weather forecast information into a pre-built power grid state prediction model, output the power grid predicted state, and replace the power grid state with the power grid predicted state.

[0127] The extreme weather warning information includes warnings of extreme weather events such as hail and cold waves. The method of acquisition is the same as that of weather data acquisition. The extreme weather warning information is used to predict the power grid status using a power grid status prediction model. The predicted power grid status is used to replace the power grid status in Example 1, so that subsequent adjustments are more predictable and further losses are avoided.

[0128] The method for constructing the power grid state prediction model includes:

[0129] Using weather forecast information as analysis data, we pre-collected the power grid prediction status corresponding to the c sets of analysis data, and converted the analysis data and the corresponding power grid prediction status into a set of feature vectors.

[0130] Each set of feature vectors is used as input to the power grid state prediction model. The model outputs a set of predicted power grid states corresponding to each set of analyzed data, and uses the actual predicted power grid state corresponding to each set of analyzed data as the prediction target. The actual predicted power grid state is the pre-collected predicted power grid state corresponding to the analyzed data. The training objective is to minimize the sum of prediction errors for all analyzed data. The prediction error is calculated using the formula Z. K =(α K -μ K ) 2 Z K The prediction error is represented by K, where K is the group number of the feature vector corresponding to the analyzed data, and α is the prediction error. K For the predicted power grid state corresponding to the Kth set of analysis data, μ K The actual power grid prediction state corresponds to the Kth set of analysis data; the power grid state prediction model is trained until the sum of prediction errors converges and training stops.

[0131] The power grid state prediction model is a deep neural network model.

[0132] Taking the impact of extreme weather on the power grid into account when generating load dispatching strategies will improve the accuracy of load dispatching.

[0133] Example 3

[0134] like Figure 3 As shown, a smart decision-making system for power grid load-side response employing deep reinforcement learning is presented. The system includes:

[0135] Cloud creation module: set up a central server, evenly divide the service coverage of the power grid side into N service areas, and set up a cloud in each service area;

[0136] Collection module: collect user historical power consumption data, weather forecast information and real-time electricity price in N service areas, input weather forecast information into the pre-constructed power grid state prediction model, output power grid prediction state, and input user historical power consumption data, weather forecast information, real-time electricity price and power grid prediction state into the cloud of the service area;

[0137] Model initialization module: initialize the global decision model and user demand prediction model using the central server, and send the global decision model and user demand prediction model to the cloud of the service area;

[0138] Local training module: use user historical power consumption data, weather forecast information, real-time electricity price and power grid state in the cloud to perform local training of the global decision model and user demand prediction model, and output model parameters;

[0139] Model updating module: the central server calls the model parameters in the blockchain network through the smart contract calling interface, aggregates the model parameters, updates the global decision model and user demand prediction model with the aggregated model parameters, and sends the updated global decision model and user demand prediction model to all clouds;

[0140] Load scheduling module: input user historical power consumption data, weather forecast information and real-time electricity price into the user demand prediction model, output power consumption demand; input power consumption demand and power grid prediction state into the global decision model, output load scheduling strategy.

[0141] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any skilled person in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be limited by the protection scope of the claims.

[0142] Finally: the above is only a preferred embodiment of the present application and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for smart decision making for grid load side response using deep reinforcement learning, the method comprising: The application relates to a power grid service area division method and a power grid service area division system. ​ The central server divides the service coverage of the power grid side into N service areas, and each service area is provided with a cloud; The user historical power consumption data, weather data, real-time electricity price and power grid state in the N service areas are collected and input into the cloud of the service area; The central server initializes a global decision model and a user demand prediction model, and sends the global decision model and the user demand prediction model to the cloud of the service area; The user historical power consumption data, weather data, real-time electricity price and power grid state in the service area are used to perform localized training of the global decision model and the user demand prediction model in the service area, and the model parameters are output to the blockchain network, wherein the model parameters include the model parameters of the user demand prediction model and the model parameters of the global decision model; The central server calls the model parameters in the blockchain network through an intelligent contract calling interface, aggregates the model parameters, updates the global decision model and the user demand prediction model with the aggregated model parameters, and sends the updated global decision model and the user demand prediction model to all the clouds; The user historical power consumption data, weather data and real-time electricity price are input into the user demand prediction model, and the power consumption demand is output; the power consumption demand and the power grid state are input into the global decision model, and the load scheduling strategy is output. 2.The grid load side response intelligent decision method using deep reinforcement learning according to claim 1, characterized in that, The number of service areas is obtained by determining the basic partition demand according to the ratio of the number of users to the maximum number of users that can be processed by each service area, amplifying the basic partition number according to the diversity factor of the power consumption mode to adapt to the complexity of the power consumption mode, and dynamically optimizing the division of the service coverage of the power grid side according to the average constraint of the computing resource demand factor and the data transmission cost factor. 3.The grid load side response intelligent decision method using deep reinforcement learning of claim 1, wherein, The initialization construction method of the global decision model comprises: Defining a state space: The state s includes: the power consumption demand is represented by a real number, the power grid state includes the current load, power generation, energy storage state and power transmission line state of the power grid, the current load and power generation of the power grid are represented by real numbers, and the energy storage state and power transmission line state are represented by 0-1 standardized real numbers; Defining an action space: The action a is the next load scheduling value, the sign of each load scheduling value indicates addition or subtraction, and the absolute value of the real number indicates the percentage of the scheduled load; Designing an Actor network structure: The Actor network structure comprises a first input layer, M hidden layers and a second output layer, the hidden layer activation functions are all ReLu, and the second output layer has no activation function; Designing a Critic network structure: The Critic network structure comprises a second input layer, M hidden layers and a third output layer; the hidden layer activation functions are all ReLu; and the third output layer comprises at least one node and has no activation function; The Actor network structure and the Critic network structure are defined by using a Keras library or a reinforcement learning framework.

4. The method of claim 2, wherein, The localized training method of the user demand prediction model comprises: The user historical power consumption data, weather data, real-time electricity price and power consumption demand are divided into a training set and a verification set; the historical power consumption data, weather data and real-time electricity price are inputted, and a power consumption demand prediction value is generated through forward propagation; the user demand prediction model parameters are updated by using a gradient descent algorithm, and when the loss function converges, the final user demand prediction model parameters are encrypted and uploaded to a blockchain network.

5. The method of claim 3, wherein, The local training method of the global decision model comprises Actor network training and Critic network training, wherein the method of Actor network training comprises: An action probability distribution is calculated by using forward propagation, an Actor network loss function is calculated based on the action probability distributions of new and old strategies and advantage functions, and Actor network parameters are updated through the Actor network loss function; Critic network training is performed synchronously with Actor network training, and the training method is the same as the local training method of the user demand prediction model; after convergence, the Actor network and Critic network parameters are used as model parameters of the global decision model, and are encrypted and uploaded to a blockchain network.

6. The power grid load side response intelligent decision method using deep reinforcement learning according to claim 4 or 5, characterized in that: The aggregation method is a weighted average method, and the model parameters of each cloud are aggregated by using the weighted average method, and the weight is determined by the ratio of the data volume of the cloud to the total number of clouds.

7. The method of claim 1, wherein, Further comprising: Collecting weather forecast information, wherein the weather forecast information comprises weather data and extreme weather warning information; Inputting the weather forecast information into a pre-constructed power grid state prediction model, outputting a power grid prediction state, and replacing the power grid state with the power grid prediction state.

8. The method of claim 7, wherein, The construction method of the power grid state prediction model comprises: Using the weather forecast information as analysis data, pre-collecting a group of analysis data corresponding to the power grid prediction state, and converting the analysis data and the corresponding power grid prediction state into a corresponding group of feature vectors; Using each group of feature vectors as the input of the power grid state prediction model, using a group of power grid prediction states corresponding to each group of analysis data as the output of the power grid state prediction model, using the actual power grid prediction state corresponding to each group of analysis data as the prediction target, and using the actual power grid prediction state as the power grid prediction state pre-collected corresponding to the analysis data; taking the sum of the prediction errors of all analysis data as the training target; training the power grid state prediction model until the sum of the prediction errors converges, and then stopping the training; and the power grid state prediction model is a deep neural network model.

9. A smart decision system for grid load side response using deep reinforcement learning, characterized in that, Implementing the power grid load side response intelligent decision method using deep reinforcement learning according to any one of claims 1-8, and the system comprises: A cloud creation module: setting up a central server, dividing the service coverage range of the power grid side into N service areas, and setting up one cloud in each service area; A collection module: collecting user historical power consumption data, weather forecast information and real-time electricity prices in N service areas, inputting the weather forecast information into a pre-constructed power grid state prediction model to output a power grid prediction state, and inputting the user historical power consumption data, weather forecast information, real-time electricity price and power grid prediction state into the cloud of the service area; The model initialization module initializes the global decision model and the user demand prediction model using the central server, and sends the global decision model and the user demand prediction model to the cloud of the service area; The local training module uses the user historical power consumption data, weather forecast information, real-time electricity price and power grid state in the service area in the cloud to perform localized training of the global decision model and the user demand prediction model, and outputs model parameters; The model updating module calls the model parameters in the blockchain network through the smart contract calling interface, aggregates the model parameters, updates the global decision model and the user demand prediction model with the aggregated model parameters, and sends the updated global decision model and the user demand prediction model to all clouds; The load scheduling module inputs the user historical power consumption data, weather forecast information and real-time electricity price into the global decision model, and outputs power consumption demand; and inputs the power consumption demand and the predicted state of the power grid into the global decision model, and outputs a load scheduling strategy.

Images