Micro-grid energy optimization management method, device, equipment and medium

Abstract

The invention provides a micro-grid energy optimization management method and device, equipment and a medium. The method comprises the steps that a micro-grid model constructed for a micro-grid system and state data of the micro-grid system at the current moment are acquired; wherein the micro-grid model comprises an energy storage system model of a target energy storage subsystem, an equipment model corresponding to any temperature control load TCL equipment in a temperature control load TCL equipment cluster, and a residence load model corresponding to any family in a plurality of families; determining an optimal execution action of the micro-grid system at the current moment through a reinforcement learning algorithm according to the state data at the current moment in combination with a micro-grid model; and controlling the micro-grid system to operate according to the optimal execution action, so that the operating profit of the micro-grid system is the highest. Therefore, the micro-grid system can be controlled to operate in real time according to the economic optimization operation strategy, so that the system always operates in the optimal state, and maximization of economic benefits is realized.

Description

Technical Field

[0001] The present application relates to the field of smart grid management technology, and in particular to a microgrid energy optimization management method, device, equipment and medium. Background Art

[0002] Energy supply and management has always been one of the major challenges facing modern society. With the rapid development of renewable energy and the widespread adoption of distributed energy, microgrid (MG, also known as microgrid system) has attracted widespread attention as an emerging paradigm in energy systems. A microgrid is a small power system that is usually composed of distributed energy resources such as solar energy, wind energy, micro gas turbines, and energy storage systems. It can operate independently or be connected to the main grid to provide electricity to users. Microgrids are highly reliable, sustainable, and have improved energy efficiency, making them ideal for supporting peak load management of the main grid.

[0003] However, faced with the impact of diverse energy resource constraints and changing electricity demand, it is considered necessary to optimize the energy management of microgrids and introduce novel and intelligent energy optimization management strategies for microgrids. Summary of the invention

[0004] The present application provides a microgrid energy optimization management method, device, equipment and medium to at least solve one of the technical problems in the related art to a certain extent. The technical solution of the present application is as follows:

[0005] According to one aspect of the present application, a microgrid energy optimization management method is provided, comprising:

[0006] Acquire a microgrid model constructed for the microgrid system and the state data of the microgrid system at the current moment; wherein the microgrid model includes an energy storage system model of a target energy storage subsystem, a device model corresponding to any temperature control load TCL device in a temperature control load TCL device cluster, and a residential load model corresponding to any of a plurality of households;

[0007] According to the state data at the current moment, combined with the microgrid model, determining the optimal execution action of the microgrid system at the current moment through a reinforcement learning algorithm;

[0008] The microgrid system is controlled to operate according to the optimal execution action to maximize the operating profit of the microgrid system.

[0009] According to another aspect of the present application, a microgrid energy optimization management device is provided, the device comprising:

[0010] An acquisition module is used to acquire a microgrid model constructed for a microgrid system and state data of the microgrid system at a current moment; wherein the microgrid model includes an energy storage system model, a device model corresponding to any temperature control load TCL device in a temperature control load TCL device cluster, and a residential load model corresponding to any of a plurality of households;

[0011] A determination module, configured to determine the optimal execution action of the microgrid system at the current moment through a reinforcement learning algorithm according to the state data at the current moment in combination with the microgrid model;

[0012] The control module is used to control the microgrid system to operate according to the optimal execution action so as to maximize the operating profit of the microgrid system.

[0013] According to another aspect of the present application, a non-transitory computer-readable storage medium of computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the microgrid energy optimization management method proposed in the above-mentioned aspect of the present application.

[0014] According to another aspect of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the microgrid energy optimization management method proposed in the above aspect of the present application.

[0015] The technical solution provided by the embodiments of the present application brings at least the following beneficial effects:

[0016] Obtain the microgrid model constructed for the microgrid system and the state data of the microgrid system at the current moment; wherein the microgrid model includes the energy storage system model of the target energy storage subsystem, the device model corresponding to any temperature control load TCL device in the temperature control load TCL device cluster, and the residential load model corresponding to any family in multiple families; according to the state data at the current moment, combined with the microgrid model, determine the optimal execution action of the microgrid system at the current moment through the reinforcement learning algorithm; control the microgrid system to operate according to the optimal execution action to maximize the operating profit of the microgrid system. In this way, the microgrid system can be controlled to operate in real time according to the economic optimization operation strategy, so that the system always operates in the optimal state and maximizes economic benefits.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A schematic diagram of a flow chart of a microgrid energy optimization management method provided in Example 1 of the present application;

[0020] Figure 2 A schematic diagram of a flow chart of a microgrid energy optimization management method provided in Embodiment 2 of the present application;

[0021] Figure 3 A comparison chart of daily profits under different energy management methods or strategies provided for this application;

[0022] Figure 4 A comparison chart of the total profits provided for this application under different energy management methods or strategies;

[0023] Figure 5 This is a structural diagram of the microgrid energy optimization management device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0024] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0025] The following describes the microgrid energy optimization management method, device, equipment and medium of the embodiments of the present application with reference to the accompanying drawings.

[0026] Figure 1 A flow chart of the microgrid energy optimization management method provided in Example 1 of the present application.

[0027] The embodiment of the present application takes the microgrid energy optimization management method as an example in which it is configured in a microgrid energy optimization management device. The microgrid energy optimization management device can be applied to any electronic device so that the electronic device can perform the microgrid energy optimization management function.

[0028] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0029] like Figure 1 As shown, the microgrid energy optimization management method may include the following steps:

[0030] Step 101, obtaining a microgrid model constructed for the microgrid system and state data of the microgrid system at the current moment.

[0031] Among them, the microgrid model may include, for example, an energy storage system model of the target energy storage subsystem, a device model corresponding to any TCL device in a TCL (Temperature Control Load) device cluster, a residential load model corresponding to any household in multiple households, etc., and this application does not impose any restrictions on this.

[0032] Among them, the target energy storage subsystem can be an energy storage system of the target administrative level, wherein the target administrative level can be, for example, a community level, an administrative village, etc., and this application does not impose any restrictions on this.

[0033] Among them, TCL equipment refers to equipment that can adjust the load by controlling the output power, such as air conditioners, heat pumps, water heaters, refrigerators, etc. This application does not impose any restrictions on this.

[0034] Among them, the residential load associated with the residential load model refers to the power load that is affected by the activities of family members and is difficult to accurately control through automated equipment. This type of power load includes various electrical appliances used in daily household use, such as washing machines, lighting, vacuum cleaners, televisions, etc. This application does not impose any restrictions on this.

[0035] Among them, the microgrid system may include, for example, a distributed power generation subsystem, a target energy storage subsystem, etc., which is not limited in this application.

[0036] It should be noted that the current time in this application is in hours.

[0037] Optionally, the energy storage system model of the target energy storage subsystem can be expressed as:

[0038]

[0039] Among them, B t represents the electric energy stored in the target energy storage subsystem at time t, and B t ∈[0,B max ], B max Represents the maximum capacity of the target energy storage subsystem; η c Represents the charging efficiency coefficient, η d Indicates the discharge efficiency coefficient; C t represents the charging power of the target energy storage subsystem, and C t ∈[0,C max ], C max represents the upper limit of the charging power of the target energy storage subsystem at time t; D t represents the discharge power of the target energy storage subsystem at time t, and D t ∈[0,D max ], D max Indicates the upper limit of the discharge power of the target energy storage subsystem; BBSC,t Represents the state of charge SOC of the target energy storage subsystem at time t.

[0040] Optionally, when the TCL device is an air conditioner, the device model corresponding to the i-th air conditioner can be expressed as:

[0041]

[0042] Among them, T t i represents the indoor temperature corresponding to the i-th air conditioner at time t, represents the temperature of the air output by the i-th air conditioner at time t, T t 0 represents the outdoor temperature corresponding to the i-th air conditioner at time t, represents the thermal mass of the air corresponding to the i-th air conditioner at time t, represents the thermal mass of the building material corresponding to the i-th air conditioner at time t, q i represents the uncontrollable heat load in the building corresponding to the i-th air conditioner at time t, represents the rated output power of the i-th air conditioner, A binary variable representing the on or off action of the i-th temperature control load TCL device in the temperature control load TCL device cluster, where i is an integer.

[0043] Optionally, the residential load model corresponding to the jth household is expressed as:

[0044]

[0045] in, represents the power load of the jth household at time t, L b,t represents the daily base load power at time t, β j It is used to indicate the ratio of the load power change of the jth household to the price fluctuation, δ t Indicates the change in electricity price at time t compared with the electricity price at time t-1, represents the load transferred by the jth household at time t, It represents the load amount transferred from the time period corresponding to time t-1 to the time period corresponding to time t, and j is an integer.

[0046] In a possible implementation of an embodiment of the present application, the status data may include the average state of charge of the TCL device cluster, the state of charge of the target energy storage subsystem, the state of the public electricity meter in the microgrid system, the indoor temperature or equipment temperature corresponding to any temperature-controlled load TCL device, the power generation of the microgrid system, the electricity price in the power market, the time step t and at least one of the daily basic load power.

[0047] Step 102, based on the current state data and in combination with the microgrid model, the optimal execution action of the microgrid system at the current moment is determined by a reinforcement learning algorithm.

[0048] Step 103, controlling the microgrid system to operate according to the optimal execution action to maximize the operating profit of the microgrid system.

[0049] Optionally, the operating profit of the microgrid system may be determined by using a reward function, wherein the reward function is expressed as:

[0050] r t =C I,t -C cost,t ; (7)

[0051] in,

[0052]

[0053] r t represents the operating profit of the microgrid system at time t, C I,t represents the income from selling electricity to users and the main grid at time t, C cost,t represents the cost at time t, represents the price of electricity sold to the main grid at time t, represents the price of purchasing electricity from the main grid at time t, G t represents the amount of electrical energy generated at time t, represents the amount of electricity sold to the main grid at time t, represents the amount of electricity purchased from the main grid at time t, C gen represents the cost of electricity generation charged to the customer controlling the load, Indicates the binary variable corresponding to the on or off action of the i-th temperature control load TCL device in the temperature control load TCL device cluster, represents the cost of importing electricity from the main grid, represents the cost of exporting electricity to the main grid, represents the power load of the ith household at time t, P t represents the household electricity price at time t.

[0054] The microgrid energy optimization management method of the embodiment of the present application obtains the microgrid model constructed for the microgrid system and the state data of the microgrid system at the current moment; wherein the microgrid model includes the energy storage system model of the target energy storage subsystem, the device model corresponding to any temperature control load TCL device in the temperature control load TCL device cluster, and the residential load model corresponding to any family in multiple families; according to the state data at the current moment, combined with the microgrid model, the optimal execution action of the microgrid system at the current moment is determined by the reinforcement learning algorithm; the microgrid system is controlled to operate according to the optimal execution action to maximize the operating profit of the microgrid system. Thus, the microgrid system can be controlled to operate in real time according to the economic optimization operation strategy, so that the system always operates in the optimal state and maximizes economic benefits.

[0055] In order to clearly explain how, in the above-mentioned embodiment of the present application, the optimal execution action of the microgrid system at the current moment is determined by a reinforcement learning algorithm based on the status data at the current moment in combination with the microgrid model, the present application also proposes a microgrid energy optimization management method.

[0056] Figure 2 A flow chart of the microgrid energy optimization management method provided in Example 2 of the present application.

[0057] like Figure 2 As shown, the microgrid energy optimization management method may include the following steps:

[0058] Step S201, obtaining a microgrid model constructed for the microgrid system and state data of the microgrid system at the current moment.

[0059] The execution process of step S201 can refer to the execution process of any embodiment of the present application, and will not be repeated here.

[0060] Step S202, based on the current state data and microgrid model, the SAC (Soft Actor-Critic) algorithm is used to determine the optimal execution action of the microgrid system at the current moment.

[0061] It is important to understand that the SAC algorithm uses the non-strategy Actor-Critic architecture of the maximum entropy reinforcement learning framework, in which the Actor-Critic architecture contains two neural networks, namely Actor (strategy network) and Critic (value function network); Actor is responsible for outputting the probability distribution of each action in the current state, while Critic is responsible for estimating the value of the state-action pair. Under this framework, the algorithm aims to maximize the expected reward and entropy, that is, to maximize both the reward and the randomness of the action. The optimal strategy can maximize the expected reward and maximize the maximum entropy term in each state. The optimal strategy π * The expression can be:

[0062]

[0063] Among them, π(·|s t ) represents the state S of the policy function π at time t t The probability of taking an action under the condition t, rt represents the value determined by the reward function at time t, H(π(·|s t )) represents the Shannon entropy term, which can indicate the uncertainty of the agent when taking random actions; α is the regularization coefficient, which determines the importance of the Shannon entropy term relative to the reward; γ represents the discount factor;

[0064] At the same time, the SAC algorithm has three network functions, which include the state value function parameterized by ψ as V ψ (s t ), the soft Q function is parameterized by θ as Q θ (s t ,a t ), and by The parameterized policy function is

[0065] The state value function V can be trained by minimizing the error expressed by the following formula (11):

[0066]

[0067] Among them, α can be 1;

[0068] The policy function π can be trained by minimizing the error expressed in the following formula (12):

[0069]

[0070] Among them, D KL is the relative entropy (also known as Kullback-Leibler (abbreviated as KL) divergence, or information divergence); Z is the normalization function;

[0071] It should be noted that by finding the minimum value of the objective function of the policy function shown in formula (12), it can be ensured that the distribution of the policy function π becomes more similar to the exponential distribution form of the normalized function Q through the normalized function Z.

[0072] In order to improve the performance of the SAC algorithm and minimize the value of the objective function shown in formula (12), the following formula (13) is introduced for reparameterization:

[0073]

[0074] Among them, ε t is the noise sampled from a Gaussian distribution;

[0075] Therefore, after introducing formula (13) into formula (12), the following error is obtained after sorting:

[0076]

[0077] Since the normalization function Z and the parameter It is irrelevant, therefore, the normalization function Z can be discarded. Thus, the unbiased estimate of the gradient of the objective function shown in formula (14) can be expressed as:

[0078]

[0079] Therefore, the gradient estimate determined by formula (15) can be used to estimate the parameters of the policy network The updated policy network is then used to determine the optimal action.

[0080] Step S203, controlling the microgrid system to operate according to the optimal execution action to maximize the operating profit of the microgrid system.

[0081] The execution process of step S203 can refer to the execution process of any embodiment of the present application, and will not be repeated here.

[0082] In summary, the SAC algorithm uses an entropy-based function to encourage action exploration, which helps to explore a wider range of action spaces and avoids excessive preference for any specific part of the action space, thereby improving the robustness and generalization ability of the algorithm.

[0083] In order to clearly illustrate the microgrid energy optimization management method of the present application, a detailed description is given below with reference to examples.

[0084] As an example, the microgrid energy optimization management method of the present application is applied to an energy management system (EMS). Specifically, the following steps can be used to implement the microgrid energy optimization management method of the present application:

[0085] Step 1: Simulation modeling of microgrid system

[0086] In this application, the microgrid system includes several key components: a distributed generation system (referred to as a distributed generation subsystem in this application), an energy storage system (ESS) (referred to as a target energy storage subsystem in this application), a TCL equipment cluster and residential loads.

[0087] Among them, the TCL device cluster includes multiple TCL devices;

[0088] Among them, the microgrid is interconnected with the main grid, and continuous electric energy exchange is carried out in the power market through the main grid.

[0089] Among them, effective two-way communication is established between each component in the microgrid and the energy management system EMS to exchange important information, including electricity prices, battery charging / discharging status and power generation statistics; EMS efficiently uses intelligent agents to send control signals to each component, manage the on / off operation of TCL equipment, process the charging / discharging procedures of ESS, and send or receive control signals for purchasing / selling electricity to the main grid.

[0090] 1. EMS Intelligent Agent

[0091] Energy Management System (EMS) intelligent agents determine the best balance strategy between energy supply and demand by using information provided by different components of the power network and the grid operating environment. Intelligent agents play a vital role in microgrids to achieve comprehensive energy management by implementing multiple control mechanisms, which can include:

[0092] 1. Control the microgrid through TCL equipment;

[0093] 2. The optimal price strategy, that is, the optimal global profit strategy;

[0094] 3. Control strategies in case of energy shortage: strategies with optimal energy supply and optimal benefits;

[0095] 4. Control strategy in case of energy surplus: strategy with optimal energy consumption and optimal benefits.

[0096] Among them, regarding the control of TCL devices, the intelligent agent allocates a specific amount of power for the operation of the TCL device cluster at each time step t. This part of power is then transmitted to each TCL device through an intermediate energy aggregator. The aggregator determines the on / off operation of each TCL device based on their power consumption status priority. In addition, the aggregator provides the intelligent agent with real-time average energy consumption updates of the TCL device cluster.

[0097] In addition, the intelligent agent needs to determine the residential load price δ to be used at each time step t t Since the microgrid operator does not have a monopoly on energy demand, the electricity price will fluctuate around an average value while maintaining the daily average price K avg and close to the rates K offered by the electric utility MARKET Changes in electricity prices are used as a means to encourage the shifting of part of the load from periods of high demand to periods when the supply of electrical energy is higher.

[0098] When the distributed energy corresponding to the distributed generation system cannot meet the energy needs of users, the microgrid can use the electricity stored in the energy storage, or purchase electricity from the main grid and store it in the energy storage system for future use. At each time step t, the intelligent agent determines the priority of using the electricity purchased from the main grid and the electricity stored in the energy storage system. In the case of a voltage drop in the microgrid due to insufficient power supply, power supply can be automatically provided from the priority power source. If the priority power source is an energy storage system, and the power provided by the energy storage system is insufficient to meet the power gap, the insufficient energy demand will be automatically purchased and supplied from the main grid.

[0099] Since the electricity generated by local distributed energy may exceed the energy demand of users, the excess electricity can be stored in the energy storage system or sold to the main grid. The intelligent agent pre-specifies the priority options for the use of excess energy. The process of supplying excess energy is similar to the case of power shortage. If the priority option selected is to store the excess energy in the energy storage system, and there is still surplus power after the corresponding battery of the energy storage system is fully charged, the surplus power will be automatically sold back to the main grid.

[0100] 2. Energy storage system

[0101] In this application, the energy storage system studied in the microgrid refers to the community-level energy storage system, rather than the battery energy storage system of a single residence. The capacity of the energy storage system used is designed to meet the energy demand of all users in the microgrid for at least 2 hours, and the energy storage system model corresponding to the energy storage system is shown in formula (1) and formula (2).

[0102] The behavior of the ESS in response to the charge / discharge control signal depends on the amount of electrical energy provided or requested by the battery. When the energy management system EMS issues a charge signal, the ESS agent receives information about the battery state of charge and verifies the feasibility of the charging operation based on the maximum capacity and maximum charging power, stores energy, and sells any excess energy back to the main grid. During ESS discharge, the ESS agent receives a power supply request from the EMS. The ESS then checks the discharge conditions and releases the available electrical energy. If the ESS cannot provide the required electrical energy exactly as requested by the EMS, the remaining power will be automatically provided by the main grid.

[0103] 3. Distributed power generation system

[0104] The microgrid is equipped with a distributed generation system that can generate different amounts of electricity depending on weather conditions. In this application, the distributed generation system used does not rely on generative model analysis, but instead uses actual distributed generation data, such as actual wind power generation data. The distributed generation system includes multiple distributed generation units, which can share real-time power generation data with the energy management system EMS and can directly supply power to the local main grid.

[0105] 4. Main power grid

[0106] Due to the intermittent and uncontrollable nature of generators in distributed generation systems, it is not feasible to achieve energy balance in microgrids only through the power supplied by distributed generation systems. When the power supply of distributed generation systems is insufficient, the main grid can quickly provide power to the microgrid. When the power of distributed generation systems is in excess, the main grid can also receive excess power from the microgrid. Transactions between the main grid and the microgrid are carried out by adjusting the fluctuations in electricity prices in the power market in real time, where and They represent the price increase and price decrease respectively, that is, they represent the price of selling electricity to the main grid at time t and the price of buying electricity from the main grid at time t respectively.

[0107] In order to establish a priority source of electricity when there is a shortage of electricity supply and a priority source of discharge when there is a surplus of electricity, the energy management system EMS specifically manages the electrical switches of the main grid. After each time step t, the EMS receives information about the amount of electricity purchased or sold to the main grid E. t Information. t When it is positive, it means purchasing electricity from the market, and E t A negative value means selling electricity to the market.

[0108] 5. TCL equipment cluster

[0109] Any TCL device in a TCL device cluster follows the principle of heat conservation. The TCL device cluster can serve as an important source of flexibility in energy supply. Assuming that most households in the microgrid are equipped with TCL devices such as air conditioners, heat pumps, water heaters, or refrigerators, the TCL devices can be directly controlled by the control signals sent by the cluster aggregator at each time step, and the energy costs incurred by users to maintain indoor comfort are included in Cgen. The controller of the TCL device receives the switching operation signal from the cluster aggregator. Among them, the controller checks the temperature constraints and performs the switching operation according to the following conditions:

[0110]

[0111] in, represents the switch operation command given by the controller corresponding to the i-th TCL device at time step t; T t i represents the operating temperature of the ith TCL device at time step t (referred to as the device temperature in this application), or the space temperature of the space where it is located (referred to as the room temperature in this application); Indicates when The switch operation command of the controller corresponding to the i-th TCL device; and Respectively represent the upper and lower temperature limits set by the end user.

[0112] In the present application, for the TCL air-conditioning type device, its corresponding device model can be expressed by formula (3) and formula (4), which will not be elaborated here.

[0113] 6. Residential load

[0114] In this application, residential load analysis refers to household power demand that cannot be directly controlled in the microgrid. The residential load model can be expressed using formula (5) and formula (6), which will not be described in detail here.

[0115] Step 2: Optimization management calculation based on SAC algorithm

[0116] The state data is collected, and based on the state data, the system model of the microgrid system is assembled, the SAC algorithm is used to update the strategy function, and the updated strategy network is used to determine the optimal execution action.

[0117] It should be noted that the specific process can be found in the execution process in step 202 of this application and will not be elaborated here.

[0118] Application process of the microgrid energy optimization management method of this application:

[0119] The microgrid energy optimization management method of the present application is implemented in the microgrid operating environment described above through target tools, such as the OpenAI Gym tool.

[0120] In this microgrid operating environment, the energy management system using the microgrid energy optimization management method was run for a total of 10 days. The process was divided into multiple stages, each lasting one day. In each stage, one day was randomly selected from the 10-day data set. Each time step represents one hour, so there are 24 time steps per day.

[0121] First, collect status data.

[0122] Optionally, the status data may include the average state of charge of the TCL device cluster, the state of charge of the energy storage system, the status of the utility meter in the microgrid system, the space temperature or operating temperature corresponding to any TCL device, the power generation of the microgrid system, the electricity price in the power market, the current time and daily basic load power.

[0123] Next, set up the reward function.

[0124] It should be noted that the reward function is as shown in formulas (7), (8) and (9), which will not be described in detail in this application.

[0125] Finally, based on the state data, combined with the system model and reward function of the microgrid system, the SAC algorithm is used to determine the optimal execution action of the microgrid system.

[0126] In order to evaluate the effectiveness of the proposed microgrid energy optimization management method, it is compared with two other energy management strategies:

[0127] i) Strategy 1: Microgrid energy optimization management strategy based on genetic algorithm, which can obtain the optimal information about power generation, consumption, electricity price and temperature;

[0128] ii) Strategy 2: Theoretical retailers. These retailers purchase a precise amount of electricity in the day-ahead electricity market and sell it to a simulated customer base at market prices.

[0129] The above comparison is to evaluate the performance of the microgrid energy optimization management method of the present application relative to these alternative energy management strategies.

[0130] At the same time, the energy management results of the microgrid energy optimization management method of the present application are compared with the other two deep reinforcement learning algorithms to evaluate the performance of the microgrid energy optimization management method proposed in the present application in microgrid energy management relative to the other two deep reinforcement learning algorithms. Among them, the other two deep reinforcement learning algorithms are the Deep Q-Network (DQN) algorithm and the State-Action-Reward-State-Action (SARSA) algorithm.

[0131] The method of the present application (i.e., the microgrid energy optimization management method of the present application), the DQN algorithm, the SARSA algorithm, strategy 1, and strategy 2 were tested in a simulated microgrid, and the daily average profit data for 10 days from the 50th day to the 59th day were recorded. Figure 3 Shows the daily profits for these 10 days under different energy management methods or strategies. Figure 4 It shows the total profit for these 10 days under different energy management methods or strategies. Figure 3 As shown in Figure 2, the average profit performance of the proposed method in ten days is better than that of the DQN algorithm, SARSA algorithm, strategy 1 and strategy 2. Although strategy 1 also achieves good results in microgrid energy management, it requires a deep understanding of the current and subsequent states of the network dynamics; Figure 4 As shown, the average profitability of running the method of the present application exceeds that of running the DQN algorithm, the SARSA algorithm, strategy 1, and even strategy 2.

[0132] In summary, the microgrid energy optimization management method of the present application adopts the SAC algorithm to encourage action exploration with an entropy-based function without assigning a significantly higher probability to any part of the action space. It can effectively simplify the analysis process of the variability and uncertainty of the distributed power generation process, and effectively solve the convergence fragility problem of the high-dimensional mathematical model of microgrid energy management.

[0133] With the above Figure 1 to Figure 2 Corresponding to the microgrid energy optimization management method provided in the embodiment, the present application also provides a microgrid energy optimization management device. Since the microgrid energy optimization management device provided in the embodiment of the present application is consistent with the above Figure 1 to Figure 2 The microgrid energy optimization management method provided in the embodiment corresponds to the embodiment, so the implementation method of the microgrid energy optimization management method is also applicable to the microgrid energy optimization management device provided in the embodiment of the present application, and will not be described in detail in the embodiment of the present application.

[0134] Figure 5 This is a structural diagram of the microgrid energy optimization management device provided in Example 3 of the present application.

[0135] like Figure 5 As shown, the microgrid energy optimization management device 500 may include: an acquisition module 501 , a processing module 502 , an optimization module 503 and a control module 504 .

[0136] Among them, the acquisition module 501 is used to obtain the microgrid model constructed for the microgrid system and the status data of the microgrid system at the current moment; wherein the microgrid model includes the energy storage system model of the target energy storage subsystem, the device model corresponding to any temperature control load TCL device in the temperature control load TCL device cluster, and the residential load model corresponding to any household in multiple households.

[0137] The determination module 502 determines the optimal execution action of the microgrid system at the current moment through a reinforcement learning algorithm according to the current state data and the microgrid model.

[0138] The control module 503 controls the microgrid system to operate according to the optimal execution action to maximize the operating profit of the microgrid system.

[0139] In a possible implementation of the embodiment of the present application, the operating profit of the microgrid system is determined by using a reward function, wherein the expression of the reward function is:

[0140] r t =C I,t -C cost,t ;

[0141] in,

[0142]

[0143] r t represents the operating profit of the microgrid system at time t, C I,t represents the income from selling electricity to users and the main grid at time t, C cost,t represents the cost at time t, represents the price of electricity sold to the main grid at time t, represents the price of purchasing electricity from the main grid at time t, represents the amount of electricity sold to the main grid at time t, represents the amount of electricity purchased from the main grid at time t, C gen represents the cost of electricity generation charged to the customer controlling the load, represents the binary variable corresponding to the on or off action of the ith temperature control load TCL device in the temperature control load TCL device cluster at time t, represents the cost of importing electricity from the main grid, represents the cost of exporting electricity to the main grid, represents the power load of the ith household at time t, P t represents the household electricity price at time t, where i is an integer.

[0144] In a possible implementation of the embodiment of the present application, the energy storage system model is expressed as:

[0145]

[0146] Among them, B t represents the electric energy stored in the target energy storage subsystem at time t, and B t ∈[0,B max ], B max Represents the maximum capacity of the target energy storage subsystem; η c Represents the charging efficiency coefficient, η d Indicates the discharge efficiency coefficient; C t represents the charging power of the target energy storage subsystem, and C t ∈[0,C max ], C max represents the upper limit of the charging power of the target energy storage subsystem at time t; D t represents the discharge power of the target energy storage subsystem at time t, and D t ∈[0,D max ], D max Indicates the upper limit of the discharge power of the target energy storage subsystem; B BSC,t Represents the state of charge SOC of the target energy storage subsystem at time t.

[0147] In a possible implementation of the embodiment of the present application, the temperature control load TCL device is an air conditioner, and the device model corresponding to the i-th air conditioner is expressed as:

[0148]

[0149] Among them, T t i represents the indoor temperature corresponding to the i-th air conditioner at time t, represents the temperature of the air output by the i-th air conditioner at time t, T t 0 represents the outdoor temperature corresponding to the i-th air conditioner at time t, represents the thermal mass of the air corresponding to the i-th air conditioner at time t, represents the thermal mass of the building material corresponding to the i-th air conditioner at time t, q i represents the uncontrollable heat load in the building corresponding to the i-th air conditioner at time t, represents the rated output power of the i-th air conditioner, It represents the binary variable corresponding to the on action or off action of the i-th temperature control load TCL device in the temperature control load TCL device cluster at time t, where i is an integer.

[0150] In a possible implementation of the embodiment of the present application, the residential load model corresponding to the j-th household is expressed as:

[0151]

[0152] in, represents the power load of the jth household at time t, L b,t represents the daily base load power at time t, β j It is used to indicate the ratio of the load power change of the jth household to the price fluctuation, δ t Indicates the change in electricity price at time t compared with the electricity price at time t-1, represents the load transferred by the jth household at time t, It represents the load amount transferred from the time period corresponding to time t-1 to the time period corresponding to time t, and j is an integer.

[0153] In a possible implementation of an embodiment of the present application, the status data includes the average charge state of the temperature control load TCL device cluster, the charge state of the target energy storage subsystem, the status of the public electricity meter in the microgrid system, the indoor temperature or device temperature corresponding to any temperature control load TCL device, the power generation of the microgrid system, the electricity price in the power market, and at least one of the current time and daily basic load power.

[0154] In a possible implementation of the embodiment of the present application, the determination module 502 is used to: determine the optimal execution action of the microgrid system at the current moment based on the state data and the microgrid model at the current moment, using the Soft Actor-Critic algorithm.

[0155] The microgrid energy optimization management device of the embodiment of the present application obtains the microgrid model constructed for the microgrid system and the state data of the microgrid system at the current moment; wherein the microgrid model includes the energy storage system model of the target energy storage subsystem, the device model corresponding to any temperature control load TCL device in the temperature control load TCL device cluster, and the residential load model corresponding to any family in multiple families; according to the state data at the current moment, combined with the microgrid model, the optimal execution action of the microgrid system at the current moment is determined by the reinforcement learning algorithm; the microgrid system is controlled to operate according to the optimal execution action to maximize the operating profit of the microgrid system. Thus, the microgrid system can be controlled to operate in real time according to the economic optimization operation strategy, so that the system always operates in the optimal state and maximizes economic benefits.

[0156] In order to implement the above-mentioned embodiments, the present application also proposes an electronic device, wherein the electronic device can be a server or detection device in the aforementioned embodiments; it includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, it implements the microgrid energy optimization management method proposed in any of the aforementioned embodiments of the present application.

[0157] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the microgrid energy optimization management method proposed in any of the aforementioned embodiments of the present application.

[0158] In order to implement the above embodiments, the present application also proposes a computer program product. When the instructions in the computer program product are executed by a processor, the microgrid energy optimization management method proposed in any of the aforementioned embodiments of the present application is executed.

[0159] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0160] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0161] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0162] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0163] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0164] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0165] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0166] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A microgrid energy optimization management method, characterized in that: The method comprises: Acquire a microgrid model constructed for the microgrid system and the state data of the microgrid system at the current moment; wherein the microgrid model includes an energy storage system model of a target energy storage subsystem, a device model corresponding to any temperature control load TCL device in a temperature control load TCL device cluster, and a residential load model corresponding to any of a plurality of households; According to the state data at the current moment, combined with the microgrid model, determining the optimal execution action of the microgrid system at the current moment through a reinforcement learning algorithm; The microgrid system is controlled to operate according to the optimal execution action to maximize the operating profit of the microgrid system.

2. The method according to claim 1, characterized in that The operating profit of the microgrid system is determined by using a reward function, wherein the expression of the reward function is: r t =C I,t -C cost,t ; r t represents the operating profit of the microgrid system at time t, C I,t represents the income from selling electric energy to users and the main grid at time t, C cost,t represents the cost at time t, represents the price of electricity sold to the main grid at time t, represents the price of buying electricity from the main grid at time t, represents the amount of electric energy sold to the main grid at time t, represents the amount of electricity purchased from the main grid at time t, C gen represents the cost of electricity generation charged to the customer controlling the load, represents a binary variable corresponding to the on action or off action of the i-th temperature control load TCL device in the temperature control load TCL device cluster at the time t, represents the cost of importing electricity from the main grid, represents the cost of exporting electricity to the main grid, represents the power load of the ith household at time t, P t represents the household electricity price at the time t, and i is an integer.

3. The method according to claim 1, characterized in that The energy storage system model is expressed as: Among them, B t represents the electric energy stored in the target energy storage subsystem at time t, and B t ∈[0,B max ], B max represents the maximum capacity of the target energy storage subsystem; η c Represents the charging efficiency coefficient, η d Indicates the discharge efficiency coefficient; C t represents the charging power of the target energy storage subsystem, and C t ∈[0,C max ], C max represents the upper limit of the charging power of the target energy storage subsystem at the time t; D t represents the discharge power of the target energy storage subsystem at the time t, and D t ∈[0,D max ], D max represents the upper limit of the discharge power of the target energy storage subsystem; B BSC,t Represents the state of charge SOC of the target energy storage subsystem at the time t.

4. The method according to claim 1, characterized in that: The temperature control load TCL device is an air conditioner, and the device model corresponding to the i-th air conditioner is expressed as: in, represents the indoor temperature corresponding to the i-th air conditioner at the time t, represents the temperature of the air output by the i-th air conditioner at time t, represents the outdoor temperature corresponding to the i-th air conditioner at the time t, represents the thermal mass of the air corresponding to the i-th air conditioner at the time t, represents the thermal mass of the building material corresponding to the i-th air conditioner at time t, q i represents the uncontrollable heat load of the i-th air conditioner in the building corresponding to the time t, represents the rated output power of the i-th air conditioner, A binary variable representing the start action or the shutdown action of the i-th temperature control load TCL device in the temperature control load TCL device cluster at the time t, where i is an integer.

5. The method according to claim 1, characterized in that The residential load model corresponding to the jth household is expressed as: in, represents the power load of the jth household at time t, L b,t represents the daily base load power at the time t, β j It is used to indicate the ratio of the load power change of the jth household to the price fluctuation, δ t Indicates the change in electricity price obtained by comparing the electricity price at time t with the electricity price at time t-1, represents the load transferred by the j-th household at time t, It represents the load amount transferred from the time period corresponding to the time period of time t-1 to the time period corresponding to the time period of time t, and j is an integer.

6. The method according to claim 1, characterized in that The status data includes the average charge state of the temperature control load TCL device cluster, the charge state of the target energy storage subsystem, the status of the public electricity meter in the microgrid system, the indoor temperature or device temperature corresponding to any of the temperature control load TCL devices, the power generation of the microgrid system, the electricity price in the power market, and at least one of the current time and daily basic load power.

7. The method according to any one of claims 1 to 6, characterized in that: The determining the optimal execution action of the microgrid system at the current moment by a reinforcement learning algorithm based on the state data at the current moment and in combination with the microgrid model includes: Based on the state data at the current moment and the microgrid model, a Soft Actor-Critic algorithm is used to determine the optimal execution action of the microgrid system at the current moment.

8. A microgrid energy optimization management device, characterized in that: The device comprises: An acquisition module is used to acquire a microgrid model constructed for a microgrid system and state data of the microgrid system at a current moment; wherein the microgrid model includes an energy storage system model, a device model corresponding to any temperature control load TCL device in a temperature control load TCL device cluster, and a residential load model corresponding to any of a plurality of households; A determination module, configured to determine the optimal execution action of the microgrid system at the current moment through a reinforcement learning algorithm according to the state data at the current moment in combination with the microgrid model; The control module is used to control the microgrid system to operate according to the optimal execution action so as to maximize the operating profit of the microgrid system.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7.

Images