A method and device for obtaining energy scheduling strategy

Abstract

The embodiment of the present invention provides a method and device for obtaining an energy scheduling strategy, which relates to the technical field of energy management. The method includes: obtaining equipment data and environmental data; establishing an energy scheduling model based on the equipment data and the environmental data; constructing a Markov decision process MDP hierarchical control framework based on the energy scheduling model, wherein the MDP hierarchical control framework includes an upper layer and a lower layer; using the upper layer of the MDP hierarchical control framework to obtain a prediction strategy for energy scheduling, wherein the prediction strategy is obtained by predicting future energy conditions based on the environmental state at the current moment; using the lower layer of the MDP hierarchical control framework to obtain an energy scheduling strategy based on the prediction strategy. Through the present invention, the prediction scheme for energy scheduling is determined according to the upper layer of the MDP, and the lower layer solves the energy scheduling strategy according to the prediction scheme, which can improve energy utilization efficiency and reduce carbon emissions.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of energy management and intelligent technology, and in particular, to a method and device for acquiring an energy scheduling strategy. Background Art

[0002] With the growth of energy demand and the increasing prominence of environmental issues, distributed microgrids, as an important part of renewable energy systems, have received widespread attention. The generation, storage and consumption of distributed microgrid energy are more complicated than traditional energy processing methods, and traditional energy scheduling methods usually use rules, heuristics or optimization algorithms, which are difficult to adapt to complex distributed microgrid systems.

[0003] In the prior art, the problem of distributed microgrid energy scheduling is that calculations are made based on real-time environmental conditions, and the energy scheduling efficiency is low. Therefore, how to efficiently obtain energy scheduling strategies is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for acquiring an energy scheduling strategy, so as to at least solve the problem of low energy scheduling efficiency.

[0005] According to an embodiment of the present invention, a method for obtaining an energy scheduling strategy is provided, comprising:

[0006] Obtain equipment data and environmental data;

[0007] Establishing an energy scheduling model according to the equipment data and the environmental data;

[0008] Based on the energy scheduling model, a Markov decision process MDP hierarchical control framework is constructed, wherein the MDP hierarchical control framework includes an upper layer and a lower layer;

[0009] The upper layer of the MDP hierarchical control framework is used to obtain a prediction strategy for energy scheduling, wherein the prediction strategy is obtained by predicting future energy conditions based on the current environmental conditions;

[0010] The lower layer of the MDP hierarchical control framework is utilized to obtain an energy scheduling strategy based on the prediction strategy.

[0011] Optionally, the using the upper layer of the MDP hierarchical control framework to obtain a prediction strategy for energy scheduling includes:

[0012] Establishing a prediction model of MPC according to the environmental state and control variables at the current moment;

[0013] The prediction model of the MPC is used to generate a prediction strategy for the energy scheduling.

[0014] Optionally, the using the prediction model of the MPC to generate the prediction strategy for the energy dispatch includes:

[0015] Using the MPC prediction model, predicting the energy state within a future preset time period to obtain a predicted energy state;

[0016] Based on the predicted energy state, a prediction strategy for the energy scheduling is generated.

[0017] Optionally, the lower layer of the MDP hierarchical control framework adopts a DDPG algorithm; and the using the lower layer of the MDP hierarchical control framework to obtain an energy scheduling strategy based on the prediction strategy includes:

[0018] Inputting environmental information into the DDPG algorithm, wherein the environmental information includes the prediction strategy and the environmental state at the current moment;

[0019] The lower layer of the MDP hierarchical control framework is used to generate an energy scheduling strategy using the DDPG algorithm.

[0020] Optionally, the DDPG algorithm includes a target network; after obtaining a prediction strategy for energy scheduling using an upper layer of the MDP hierarchical control framework, the method further includes:

[0021] Integrating the energy scheduling prediction strategy into the DDPG algorithm;

[0022] The target network in the DDPG algorithm is trained using the prediction strategy of the energy scheduling, and the parameters of the target network are updated.

[0023] According to an embodiment of the present invention, a device for acquiring an energy scheduling strategy is provided, comprising:

[0024] A first acquisition module, used to acquire device data and environment data;

[0025] An establishment module is used to establish an energy scheduling model according to the equipment data and the environmental data;

[0026] A construction module, used for constructing a Markov decision process MDP hierarchical control framework based on the energy scheduling model, wherein the MDP hierarchical control framework includes an upper layer and a lower layer;

[0027] A second acquisition module is used to obtain a prediction strategy for energy scheduling using an upper layer of the MDP hierarchical control framework, wherein the prediction strategy is obtained by predicting future energy conditions based on the current environmental conditions;

[0028] The third acquisition module is used to acquire the energy scheduling strategy based on the prediction strategy by utilizing the lower layer of the MDP hierarchical control framework.

[0029] According to another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps in any embodiment of the above-mentioned energy scheduling strategy acquisition method when running.

[0030] According to another embodiment of the present invention, there is also provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any embodiment of the above-mentioned method for obtaining the energy scheduling strategy.

[0031] Through the present invention, the prediction scheme of energy scheduling is determined according to the upper layer of the MDP hierarchical control framework, and the lower layer solves the energy scheduling strategy according to the prediction scheme, which can improve energy utilization efficiency and reduce carbon emissions. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a hardware structure diagram of a mobile terminal of a method for acquiring an energy scheduling strategy according to an embodiment of the present invention;

[0033] Figure 2 is a flow chart of a method for obtaining an energy scheduling strategy according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the interaction between an agent and an environment in an MDP of an embodiment of the present invention;

[0035] Figure 4 It is a schematic diagram of the algorithm framework of an embodiment of the present invention;

[0036] Figure 5 The Actor network and the Critic network in the DDPG algorithm of the embodiment of the present invention;

[0037] Figure 6 is a flow chart of a method for obtaining an energy scheduling strategy according to a specific embodiment of the present invention;

[0038] Figure 7 It is a structural diagram of a device for acquiring an energy scheduling strategy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0041] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for acquiring an energy scheduling strategy according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more (only one is shown in the figure) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0042] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as a computer program corresponding to a method for obtaining an energy scheduling strategy in an embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0043] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0044] This embodiment provides an energy scheduling method, wherein: Figure 2 is a flow chart of an energy scheduling method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0045] Step S201: Acquire device data and environment data.

[0046] Among them, equipment data includes information about power equipment, such as generators, energy storage equipment, and load equipment, describing the technical parameters, rated capacity, working status, real-time power, and other information of the equipment; environmental data includes information about environmental conditions and external factors related to energy production and consumption, such as weather conditions (temperature, humidity, wind speed, sunshine intensity, etc.), energy prices, user needs, and other data. The above equipment data and environmental data can be obtained in the smart energy management system.

[0047] Step S202: Establish an energy scheduling model based on the equipment data and the environmental data.

[0048] An energy dispatch model can be established based on the above data. For ease of understanding, this embodiment takes a distributed microgrid as an example to establish a distributed microgrid energy dispatch model. In practical applications, it can also be applied to other energy sources. The example here should not limit other energy models. The above model can include an energy storage system model and a cost model. The establishment of the energy dispatch model can be performed in the following steps:

[0049] Step 1: Establish a distributed microgrid energy storage system model. The distributed microgrid energy storage system model is as follows:

[0050]

[0051] Among them, B t is the energy stored in the energy storage system during the time interval t, is the charging efficiency coefficient, is the discharge efficiency coefficient, c t Charging power for energy storage system, d t is the discharge power of the energy storage system, c t ≥0,d t ≤0.

[0052] Since the energy storage system cannot discharge from a point above its maximum capacity or below the minimum energy level, the constraints of the distributed microgrid energy dispatching energy storage system model are set as follows:

[0053]

[0054] Among them, B min For minimum capacity discharge, B maxCharge to maximum capacity, c max is the maximum charging power in the energy storage system, d max is the maximum discharge power in the energy storage system.

[0055] In order to avoid simultaneous charging and discharging of the energy storage system, the following constraints are set:

[0056]

[0057] Based on the above constraints, at least one of the charging or discharging values ​​must be 0.

[0058] Step 2: Establish a distributed microgrid cost model, which includes interruptible load compensation cost, operation cost and environmental cost.

[0059] 1. Interruptible load compensation cost

[0060] There are mainly two types of loads: fixed load and interruptible load. Fixed load is the basic load demand of users. This load does not need to be adjusted and needs to be basically met. Interruptible load can be conditionally interrupted by the external environment. Some equipment can be stopped or reduce power usage under certain conditions to adapt to certain changes in the external environment.

[0061] Usually users are compensated based on the scale and impact of the interrupted load. The compensation cost is calculated as follows:

[0062]

[0063]

[0064] in is the initial operating power after the load is interrupted at the tth time interval, is the operating power at the tth time interval, m is the interrupt level, C 1,t Compensation costs for interruptible loads, is the compensation price for load interruption of level m, It is the m level of load interruption.

[0065] In order to ensure the safe operation of the power system and prevent overload, the maximum and minimum limits of the total load are determined by analyzing the power supply capacity of the power system and the rated load of the equipment. The total load constraints are as follows:

[0066]

[0067] in, is the power of the fixed load at time t, P t is the load power at time t, is the maximum allowable power of the total load.

[0068] 2. Operating costs

[0069] The operating cost includes energy cost, energy storage system depreciation cost and maintenance cost. Based on the buying and selling prices of energy and the variables reflecting the buying and selling of electricity, the overall cost of energy is calculated as follows:

[0070]

[0071] Among them, C 2-1,t is the energy cost, v t is the purchase price, u t is the selling price, g t A variable that reflects the buying and selling of electricity.

[0072] Frequent discharge or charge will have an adverse effect on the life of the energy storage system. Based on this, the depreciation cost of the energy storage system is calculated as follows:

[0073]

[0074] Where φ is the depreciation coefficient of the energy storage system, C 2-2,t is the depreciation cost of the energy storage system.

[0075] Based on the above analysis, the operating cost model is:

[0076] C 2,t =C 2-1,t +C 2-2,t +C 2-3,t

[0077] Among them, C 2-3,t For maintenance costs.

[0078] 3. Environmental costs

[0079] Environmental costs mainly consider the carbon emissions of the units. Since renewable energy is clean energy and does not produce polluting gases during operation, the environmental costs of renewable energy are not considered. Specifically, the carbon emissions of distributed microgrids are mainly generated by gas turbines. The actual carbon emissions of distributed microgrids are:

[0080]

[0081] Among them, E c Expressed as the actual carbon emissions of distributed microgrids, σ i is the carbon emission intensity coefficient of the i-th gas turbine, is the total carbon emission of the i-th gas turbine, T is the time period, and N is the number of gas turbines;

[0082] The calculation method of the carbon emission quota of distributed microgrid is as follows:

[0083]

[0084] Among them, E D is the carbon emission quota, ε is the load factor correction factor, η is the carbon emission benchmark coefficient per unit electricity, is the total power generation of the microgrid.

[0085] The formula for calculating environmental cost is:

[0086] C 3,t =λ c (E c -E D )

[0087] Among them, C 3,t is the cost of carbon emissions, λ c is the cost per unit of carbon emissions.

[0088] In summary, the cost model C = C 1,t +C 2,t +C 3,t .

[0089] Based on the established distributed microgrid energy dispatching model, the objective function P1 of deep reinforcement learning is constructed as follows:

[0090]

[0091] Among them, E is the expectation operator, which takes over the system parameters and possible random control actions in each time interval; (the system parameters are renewable energy power generation, load, carbon emissions, and buying and selling electricity prices; random control actions: storage system charging and discharging power).

[0092] Step S203: constructing a Markov decision process (MDP) hierarchical control framework based on the energy scheduling model, wherein the MDP hierarchical control framework includes an upper layer and a lower layer.

[0093] In this step, the distributed microgrid energy scheduling problem is described as a Markov decision process (MDP), which can be performed in the following steps.

[0094] Step 1: Consider the dispatch of distributed microgrid energy storage system and the control of carbon emissions as MDP, and describe the continuous decision-making problem in the smart energy management system. Since the components of renewable energy generation and electricity price in the environmental state do not belong to MDP in time, the MDP formula is an approximate description of the energy management problem.

[0095] Step 2: In order to clearly describe the various elements and relationships of the problem, a standardized framework is provided to fully understand and model the decision-making process. The MDP is defined as a quintuple, which is as follows:

[0096] M=(S,A,P,R,γ)

[0097] P:S*A*S→[0,1]

[0098] R:S*A→R

[0099] Where S is the set of environmental states, A is the set of actions, P is the transition probability function, which simulates the uncertainty of system state evolution based on the actions taken by the smart energy management system. R is the reward function, and γ→[0,1] is the discount factor.

[0100] Step 3: Establish the interaction between the agent and the environment. The agent represents the learner and decision maker. The environment is composed of multiple objects other than the agent. The interaction between the agent and the environment is as follows: Figure 3 As shown. The agent observes the environment state S t , then take action a t , and the environment state changes to S t+1 , and returns the reward R t+1 .

[0101] Specifically, the multiple objects include loads, energy storage systems, renewable energy, non-renewable energy, and carbon emissions;

[0102] Step 4: Define the environment state space. The environment state consists of seven types of information: macro energy control strategy information (i.e., energy scheduling prediction strategy), energy generation, load, storage power, carbon emissions, electricity price, and a certain day time index. The environment state is as follows:

[0103] S t =(k t ,P t ,b t ,B t ,C t ,V t ,t')

[0104] Where S t is the environmental state, k t is the macro energy control strategy information, P t is the energy generation, b t is the load, B t is the stored power, C t is carbon emissions, V t is the electricity price, and t' is the time period index of a certain day.

[0105] Step 5: Define the behavior space. The behavior space of MDP consists of the charging power and discharging power of the distributed microgrid energy storage system. The goal of the smart energy management system is to determine the energy dispatch g(t) of the optimal microgrid control center and set the MDP action to the charging and discharging power of the energy storage system to c t and d t , according to c t and d t We can get g(t).

[0106] Step 6: Define the activation function. According to MDP theory, execute action a t-1 Can trigger environmental states t-1 →s t The conversion finally gets the reward R t , since the goal of the smart energy management system is to minimize the cost under the premise of controlling carbon emissions within a certain range. The corresponding reward consists of three parts, namely, operating cost, interruption load compensation cost and environmental cost. Since the interruption load compensation cost is penalized as C at the t-1 time interval 1,t-1 , so R t The first part uses C 1,t-1 (s t-1 , a t-1 ) indicates that; since the operating cost penalty in the t-1 time interval is C 2,t-1 , so R t The second part uses C 2,t-1 (s t-1 , a t-1 ) indicates that the environmental cost is C in the time interval t-1. 3,t-1 , so R t The third part adopts C 3,t-1 (s t-1 , a t-1 )express.

[0107] So the reward function is:

[0108] R t = -β(C 1,t-1 (S t-1 ,a t-1 )+C 2,t-1 (S t-1 ,a t-1 )+C 3,t-1 (S t-1 ,a t-1 ))

[0109] Among them, R t is the reward function, and β is the weight coefficient.

[0110] Step 7: Define the action value function. When the microgrid intelligent management system and the energy storage system are jointly controlled at time interval t, the goal of the smart energy management system is to obtain the maximum expected benefit in the future, and the benefit is defined as the sum of discounted rewards:

[0111]

[0112] At the same time, define Q Π (s,a) is the action-value function under strategy Π, (strategy is the probabilistic mapping from state to selection of each possible action), which means that under strategy Π, if s t =s, at a t = the expected return of the action taken when a, at which point the optimal behavior value function Q * (s,a) is the maximum Π strategy Q Π (s t ,a t ), the optimality equation is obtained by recursive calculation and using the Bellman equation:

[0113] Q * (s,a)=E[R t+1 +γmax a' Q * (s',a')|s t =s,a t =a]

[0114] =∑ s′，r P(s',r|s,a)[r+γmax a' Q(s',a')]

[0115] =∑s',r

[0116] Among them, Q * (s,a) is the optimal behavior value function, s is the state, a is the action, s t is the state at time t, a t is the action at time t; E[] is the expected return expectation, R t+1 is the reward obtained at time t+1, γ is the discount factor, max a' is the value selected in the next time interval t+1 so that Q * (s', a') is the action a' that reaches the maximum value, s' is the state at the next moment, a' is the action at the next moment, Q * (s',a') is the optimal behavior value function obtained by selecting a specific action in the next state, r is the reward obtained by selecting action a; P(s',r|s,a) is the state transition probability.

[0117] Step S204: Utilize the upper layer of the MDP hierarchical control framework to obtain a prediction strategy for energy scheduling, wherein the prediction strategy is obtained by predicting future energy conditions based on the current environmental conditions.

[0118] A hierarchical control framework is set up, in which the upper layer uses the MPC (Model Predictive Control) algorithm to generate a macro energy control strategy, that is, a prediction strategy for energy scheduling. The MPC algorithm uses a mathematical model and the current system state to predict the system behavior in the future (system behavior refers to the future evolution of the system predicted by the mathematical model) and formulate a prediction strategy for energy scheduling. The prediction strategy for energy scheduling considers the optimization indicators of the overall system and guides the DDPG (Deep Deterministic Policy Gradient) algorithm in the lower layer.

[0119] Optionally, the using the upper layer of the MDP hierarchical control framework to obtain a prediction strategy for energy scheduling includes:

[0120] Establishing a prediction model of MPC according to the environmental state and control variables at the current moment;

[0121] The prediction model of the MPC is used to generate a prediction strategy for the energy scheduling.

[0122] In this implementation, the MPC algorithm is used to generate a prediction strategy for energy scheduling. By setting the MPC prediction model, taking the current environmental state and control variables as input, a macro energy control strategy, i.e., a prediction strategy for energy scheduling, is obtained through a series of operations.

[0123] In specific implementation, the energy scheduling prediction strategy can be used as a separate vector and spliced ​​together with the current environmental state vector as the environmental state, and the environmental state is passed as input to the Actor network in the DDPG algorithm. The Actor network is responsible for outputting real-time energy scheduling decisions. During the training process, the DDPG algorithm adjusts the real-time strategy parameters according to the environmental state and reward signal.

[0124] Optionally, the using the prediction model of the MPC to generate the prediction strategy for the energy dispatch includes:

[0125] Using the MPC prediction model, predicting the energy state within a future preset time period to obtain a predicted energy state;

[0126] Based on the predicted energy state, a prediction strategy for the energy scheduling is generated.

[0127] In this implementation, the MPC is divided into discrete time periods, and an energy scheduling strategy is generated in each time period, and each time period generally represents a discrete time point of the system. In each time period, the MPC algorithm recalculates the optimal energy scheduling plan, taking into account the current energy supply and demand conditions, constraints and objective functions, and predicts future energy conditions. In this way, the MPC algorithm can gradually determine the energy scheduling plan in each time period to meet demand, optimize costs, ensure system stability, etc. Since energy demand and supply conditions may change in different time periods, it is necessary to calculate and optimize each time period to obtain the energy scheduling strategy that best suits the time period. Gradually achieve the goals and requirements of long-term energy scheduling.

[0128] Using the MPC prediction model, a prediction strategy for energy dispatch is generated. The specific steps can be performed as follows:

[0129] Step 1: Based on the current environmental status and control variables, a prediction model for energy demand, energy utilization and carbon emissions is established using multiple linear regression.

[0130] Specifically, the environmental status and control variables at the current moment are taken as independent variables, normalization is used to construct new features to express the relationship between variables, and a multivariate linear regression model is established, in which energy demand, energy utilization and carbon emissions are taken as dependent variables, and the environmental status and control variables are taken as independent variables, and the optimal regression equation is fitted.

[0131] Among them, the control variables are the activation or stop of energy production equipment (controlling the start or shutdown status of energy generation equipment such as solar photovoltaic panels, wind turbines and gas turbines), the charge and discharge control of energy storage equipment (charging rate, discharge rate, charging time, discharge time), and load management strategies (adjusting the use of loads and elasticity requirements).

[0132] Step 2: Make a prediction based on the established prediction model to predict the energy situation in the next day (just an example here), including energy demand, energy utilization, and carbon emissions.

[0133] Step 3: Optimize strategy. According to the formulated objective function and constraints, combined with the established prediction model, select the linear programming method for optimization and solution to obtain the prediction strategy for energy scheduling. The optimization algorithm will calculate the energy scheduling strategy for each time period, and analyze the results obtained by the optimization algorithm, including key indicators such as energy scheduling plan, energy efficiency, carbon emissions, and operating status of energy storage system in each time period. According to the analysis results, update the control strategy to further optimize the long-term energy scheduling decision and improve decision-making efficiency.

[0134] Step S205: Utilize the lower layer of the MDP hierarchical control framework to obtain an energy scheduling strategy based on the prediction strategy.

[0135] In this step, the lower layer uses the DDPG algorithm to make real-time energy scheduling decisions based on the prediction strategy obtained by MPC. Among them, the DDPG algorithm is an online deep reinforcement learning algorithm under the Actor-Critic framework, which includes an Actor network and a Critic network. Each network is updated according to its own update rules to maximize the cumulative expected return.

[0136] Optionally, the lower layer of the MDP hierarchical control framework adopts a DDPG algorithm; and the using the lower layer of the MDP hierarchical control framework to obtain an energy scheduling strategy based on the prediction strategy includes:

[0137] Inputting environmental information into the DDPG algorithm, wherein the environmental information includes the prediction strategy and the environmental state at the current moment;

[0138] The lower layer of the MDP hierarchical control framework is used to adopt the DDPG algorithm to solve the energy scheduling strategy.

[0139] In this implementation, the lower layer uses the DDPG algorithm to interact with the environment, combine the given energy scheduling prediction strategy and the current environmental state to generate real-time scheduling decisions. In addition, the DDPG algorithm can also learn and adapt strategies for different environmental states, and optimize according to the reward signal to achieve the optimal strategy for microgrid energy scheduling.

[0140] like Figure 4 As shown, the upper layer uses the MPC algorithm to establish a prediction model. The current environmental state and control variables are input into the MPC algorithm, and the prediction model outputs macro energy control information (i.e., the prediction strategy for energy scheduling) according to the loss function constraints.

[0141] The MDP architecture outputs the energy scheduling prediction strategy, current environment state and control variables to the lower-level DDPG algorithm, calculates the output action through the Actor network, and the agent executes the selected action and interacts in the environment. After the action is executed, the environment returns a reward signal for evaluation. The environment state, the executed action, the reward obtained, and the environment state predicted by the MPC algorithm are stored in the experience replay mechanism. The Actor network samples a part of the experience samples from the experience replay mechanism buffer to train the Actor and Critic networks and update the network parameters.

[0142] In the DDPG algorithm, the target network is updated, the Actor network and the Critic network are updated by sampling in the experience replay mechanism, the loss function of the Critic network is calculated using the current target Q value, and the Actor network is updated using the policy gradient method to improve the performance of the policy and value function.

[0143] Optionally, the DDPG algorithm includes a target network; after using the MPC prediction model to generate the prediction strategy for energy scheduling, the method further includes:

[0144] Integrating the energy scheduling prediction strategy into the DDPG algorithm;

[0145] The target network in the DDPG algorithm is trained using the prediction strategy of the energy scheduling, and the parameters of the target network are updated.

[0146] In this implementation, the information of the prediction strategy for energy scheduling is integrated into the experience replay mechanism of the DDPG algorithm. The Actor network samples a portion of the experience samples from the experience replay mechanism cache to train the Actor and Critic networks (i.e., the target network) and update the network parameters.

[0147] See also Figure 5 , the above process can be performed as follows:

[0148] Step 1: Build an Actor-Critic model. Combine the Actor network and the Critic network in the DDPG energy scheduling algorithm to learn approximate values ​​and transform the environmental state s t As the input of the Actor network, output action a, and s t and a as the input of the Critic network, outputting the behavior value function Q(s t ,a),Calculate the policy gradient, which is used to update the weight of the Actor network. Before calculating the behavior value function, the weight of the Critic network is updated based on the experience replay mechanism and the target network mechanism;

[0149] Step 2: Train and update the neural network. Train the Actor network weights. In each time interval, the Actor network updates the weights according to the current environment state s. t , select an action for charging and discharging power of distributed microgrid energy storage system, and execute a t Action, the environment state is converted to s t+1 , and at the same time, get the reward R t+1 Specifically, the experience replay mechanism memory D with a capacity of N is initialized, in which the conversion tuple (s t ,a t ,Rt+1 ,s t+1 ), introduce Preprocessing function to normalize the input data.

[0150] Each group in the environmental state at time interval t is normalized in the range [0,1], the normalized expression is:

[0151]

[0152] Among them, k t Indicates the original value of the time interval t, min t k t Indicates the minimum value within the time interval t, max t k t It represents the maximum value in the time interval t;

[0153] Randomly initialize the Critic network and the Actor network, and initialize the Critic network weight to θ Q and the weight of the Actor network is θ μ ;

[0154] Initialize the Critic network as:

[0155]

[0156] Initialize the Actor network as:

[0157]

[0158] Among them, in the DDPG energy scheduling algorithm, the Actor network has 2 hidden layers and the Critic network has 4 hidden layers;

[0159] Initialize the target critic network and the target Actor network The weight θ Q’ and θ μ’ is: Q’ ←θ Q and θ μ’ ←θ μ .

[0160] Specifically, the weights of the target Critic network and the target Actor network are initialized by replication, and an action is selected from the expression in each time period of each group:

[0161]

[0162] in, is the preprocessing function, θ μis the weight of the Actor network, N t To detect noise;

[0163] Get a t After being obtained, it is applied to the energy storage system. At the end of the time interval t, the new state s t+1 and reward R t+1 Return from the environment, the tuple will be converted Stored in memory, used for training the Actor network and the Critic network, randomly sampling K samples to train the neural network, namely the Actor network, the Critic network, the target Actor network and the target Critic network. And calculate the mean square error loss. Update the weights of the Critic network by minimizing the loss function; calculate the sampling strategy gradient.

[0164] Among them, s i Represents the state of the i-th sample, a i represents the action of the i-th sample;

[0165] The policy gradient is used to update the weights of the Actor network, and the experience replay mechanism and the target network mechanism are used to update the weights of the Critic network. This is an iterative process. During this process, the Critic network is trained to take the approximate sum of the actual values ​​(that is, what we want to predict is the total reward after the current state). The Actor network fine-tunes the action it provides to the Critic network based on the feedback of this value.

[0166] Specifically, the policy gradient is calculated to update the Actor network weights. When calculating the policy gradient, the Bellman equation is used to calculate the return value corresponding to each state based on the Critic network. Therefore, the Actor network performs gradient descent according to the output of the Critic network, so the interface is required to be differentiable (continuous action space). The Actor network updates parameters according to the gradient direction to better control each action given to the system by the Agent.

[0167] Step 3: Model evaluation and parameter adjustment. Use test data to evaluate the model, and continuously optimize and adjust parameters according to actual application conditions to achieve optimal control of the smart microgrid energy management system.

[0168] Step 4: Dynamically adjust the electricity price based on the current state and the trained model. Based on the strategies learned by the MPC algorithm and the DDPG algorithm, according to the current energy supply and demand and carbon emission targets, calculate the appropriate electricity price and adjust it to achieve energy scheduling optimization.

[0169] In the DDPG algorithm, the target network is updated, the Actor network and the Critic network are updated by sampling in the experience replay mechanism, the loss function of the Critic network is calculated using the current target Q value, and the Actor network is updated using the policy gradient method to improve the performance of the policy and value function.

[0170] By storing the environmental state information predicted by the MPC algorithm in the experience replay mechanism and using it to update the DDPG algorithm, not only the current energy demand but also the subsequent energy demand is taken into account, thereby improving the performance and accuracy of the strategy.

[0171] See also Figure 6 ,The energy scheduling scheme of distributed microgrid can include the following steps:

[0172] 1. Obtain device data and environmental data.

[0173] 2. Construct a distributed microgrid energy intelligent scheduling model, including a distributed microgrid energy storage system model and a cost model.

[0174] 3. Express the distributed microgrid energy intelligent scheduling problem as MDP, and set the state space, action space, reward function and action value function.

[0175] 4. Set up a hierarchical control architecture. The upper layer uses the MPC algorithm to generate a macro energy dispatch strategy (i.e., the prediction strategy for energy dispatch). The lower layer uses the DDPG algorithm to make real-time energy dispatch decisions. The DDPG algorithm interacts with the environment and combines the given energy dispatch prediction strategy and the current environmental state to make corresponding operations. It also optimizes according to the reward signal to achieve the optimal strategy for microgrid energy dispatch.

[0176] 5. Determine the dispatch plan and adjust the electricity price dynamically.

[0177] The embodiments of the present invention, combined with the MPC and DDPG algorithms, can fully utilize the advantages of long-term prediction and real-time state feedback. The MPC algorithm can determine the approximate energy scheduling plan for a period of time in the future through long-term prediction, while the DDPG algorithm can be fine-tuned and optimized according to the specific situation in a real-time environment, while considering the dynamic and continuous characteristics to achieve a more accurate, flexible and intelligent energy scheduling balance. Enable the system to adjust the energy scheduling strategy in time to meet actual needs. A more comprehensive and integrated target optimization is achieved to meet the needs of different aspects. This will help improve the energy utilization efficiency, load balance and stability of the microgrid system, thereby reducing costs and improving reliability.

[0178] By introducing the impact mechanism of carbon emissions, it helps to reduce dependence on traditional energy, improve energy cleanliness and the utilization rate of renewable energy, better reflect the impact of energy use on the environment, enhance environmental protection awareness, and promote low-carbon production and consumption.

[0179] Based on information such as carbon emissions, costs and load, the electricity price can be adjusted dynamically. The electricity price can be flexibly adjusted according to real-time conditions to better balance the supply and demand relationship and improve energy efficiency.

[0180] The application of this method covers various distributed microgrid scenarios, such as cities, industrial parks, etc. It can be used to optimize energy distribution and scheduling, achieve optimal carbon emission management, improve energy utilization efficiency, reduce dependence on traditional energy, and contribute to sustainable development.

[0181] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0182] In this embodiment, a device for acquiring an energy scheduling strategy is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0183] Figure 7 is a structural diagram of a device for acquiring an energy scheduling strategy according to an embodiment of the present invention, such as Figure 7 As shown, the device comprises:

[0184] The first acquisition module 701 is used to acquire device data and environment data;

[0185] Establishing module 702, used to establish an energy scheduling model according to the equipment data and the environmental data;

[0186] A construction module 703 is used to construct a Markov decision process MDP hierarchical control framework based on the energy scheduling model, wherein the MDP hierarchical control framework includes an upper layer and a lower layer;

[0187] A second acquisition module 704 is used to acquire a prediction strategy for energy scheduling using an upper layer of the MDP hierarchical control framework, wherein the prediction strategy is obtained by predicting future energy conditions based on the current environmental conditions;

[0188] The third acquisition module 705 is used to acquire an energy scheduling strategy based on the prediction strategy by utilizing the lower layer of the MDP hierarchical control framework.

[0189] Optionally, the second acquisition module includes:

[0190] Establishing a submodule for establishing a prediction model of MPC according to the environmental state and control variables at the current moment;

[0191] The generation submodule is used to generate the prediction strategy of the energy scheduling by using the prediction model of the MPC.

[0192] Optionally, the generating submodule includes:

[0193] A prediction unit, used to predict the energy state within a preset time period in the future by using the prediction model of the MPC, to obtain a predicted energy state;

[0194] A generating unit is used to generate a prediction strategy for the energy scheduling based on the predicted energy state.

[0195] Optionally, the lower layer of the MDP hierarchical control framework adopts the DDPG algorithm; the generating submodule includes:

[0196] An input unit, used to input environmental information into the DDPG algorithm, wherein the environmental information includes the prediction strategy and the environmental state at the current moment;

[0197] A solving unit is used to generate an energy scheduling strategy by using the DDPG algorithm in the lower layer of the MDP hierarchical control framework.

[0198] Optionally, the DDPG algorithm includes a target network; and the device further includes:

[0199] A storage module, used to integrate the prediction strategy of the energy scheduling into the DDPG algorithm;

[0200] A training module is used to train the target network in the DDPG algorithm and update the parameters of the target network by using the prediction strategy of the energy scheduling.

[0201] Through the present invention, the prediction scheme of energy scheduling is determined according to the upper layer of MDP, and the lower layer solves the energy scheduling strategy according to the prediction scheme, which can improve energy utilization efficiency and reduce carbon emissions.

[0202] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0203] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0204] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0205] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0206] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0207] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0208] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0209] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for obtaining an energy scheduling strategy, characterized in that: include: Obtain equipment data and environmental data; An energy scheduling model is established according to the device data and the environmental data, wherein the energy scheduling model includes a distributed microgrid cost model, and the distributed microgrid cost model includes interruptible load compensation cost, operation cost and environmental cost, wherein the interruptible load compensation cost is calculated by the following formula: ； ； in is the initial operating power after the load is interrupted at the tth time interval, is the operating power for the tth time interval, m is the interrupt level, Compensation costs for interruptible loads, is the compensation price for load interruption of level m, It is the m level of load interruption; Based on the energy scheduling model, a Markov decision process MDP hierarchical control framework is constructed, wherein the MDP hierarchical control framework includes an upper layer and a lower layer; The upper layer of the MDP hierarchical control framework is used to obtain a prediction strategy for energy scheduling, wherein the prediction strategy is obtained by predicting future energy conditions based on the current environmental conditions; The lower layer of the MDP hierarchical control framework is utilized to obtain an energy scheduling strategy based on the prediction strategy.

2. The method according to claim 1, characterized in that The method of using the upper layer of the MDP hierarchical control framework to obtain a prediction strategy for energy scheduling includes: According to the environmental state and control variables at the current moment, a prediction model of model predictive control (MPC) is established; The prediction model of the MPC is used to generate a prediction strategy for the energy scheduling.

3. The method according to claim 2, characterized in that The method of using the MPC prediction model to generate the energy scheduling prediction strategy includes: Using the MPC prediction model, predicting the energy state within a future preset time period to obtain a predicted energy state; Based on the predicted energy state, a prediction strategy for the energy scheduling is generated.

4. The method according to claim 1, characterized in that The lower layer of the MDP hierarchical control framework adopts a deep deterministic policy gradient DDPG algorithm; the lower layer of the MDP hierarchical control framework is used to obtain an energy scheduling strategy based on the prediction strategy, including: Inputting environmental information into the DDPG algorithm, wherein the environmental information includes the prediction strategy and the environmental state at the current moment; The lower layer of the MDP hierarchical control framework is used to generate an energy scheduling strategy using the DDPG algorithm.

5. The method according to claim 4, characterized in that The DDPG algorithm includes a target network; after the upper layer of the MDP hierarchical control framework is used to obtain a prediction strategy for energy scheduling, the method further includes: Integrating the energy scheduling prediction strategy into the DDPG algorithm; The target network in the DDPG algorithm is trained using the prediction strategy of the energy scheduling, and the parameters of the target network are updated.

6. A device for acquiring an energy scheduling strategy, characterized in that: include: A first acquisition module, used to acquire device data and environment data; An establishment module is used to establish an energy scheduling model according to the device data and the environmental data, wherein the energy scheduling model includes a distributed microgrid cost model, and the distributed microgrid cost model includes interruptible load compensation cost, operation cost and environmental cost, wherein the interruptible load compensation cost is calculated by the following formula: ; ； in is the initial operating power after the load is interrupted at the tth time interval, is the operating power for the tth time interval, m is the interrupt level, Compensation costs for interruptible loads, is the compensation price for load interruption of level m, It is the m level of load interruption; A construction module, used for constructing a Markov decision process MDP hierarchical control framework based on the energy scheduling model, wherein the MDP hierarchical control framework includes an upper layer and a lower layer; A second acquisition module is used to obtain a prediction strategy for energy scheduling using an upper layer of the MDP hierarchical control framework, wherein the prediction strategy is obtained by predicting future energy conditions based on the current environmental conditions; The third acquisition module is used to acquire the energy scheduling strategy based on the prediction strategy by utilizing the lower layer of the MDP hierarchical control framework.

7. The device according to claim 6, characterized in that The second acquisition module includes: Establishing a submodule, for establishing a prediction model of the model predictive control MPC according to the environmental state and control variables at the current moment; The generation submodule is used to generate the prediction strategy of the energy scheduling by using the prediction model of the MPC.

8. The device according to claim 7, characterized in that The generating submodule comprises: A prediction unit, used to predict the energy state within a preset time period in the future by using the prediction model of the MPC, to obtain a predicted energy state; A generating unit is used to generate a prediction strategy for the energy scheduling based on the predicted energy state.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.

Images