A multi-park microgrid demand response method and system

Abstract

This invention relates to the field of smart grid technology and discloses a demand response method and system for multi-campus microgrids. The method includes constructing a campus system environment model, comprising a flexible load model, an electrochemical energy storage model, a power utility model, and a cooling and heating energy storage model; dynamically selecting information shared by other campuses that requires attention based on multi-head attention feature extraction and utilizing a multi-head attention mechanism; and using deep reinforcement learning to coordinate the control of flexible loads, cooling and heating energy storage by the campuses, thereby reshaping the overall power demand curve. This invention can obtain optimal control strategies through autonomous learning, effectively improving the flexibility and energy efficiency of multi-campus microgrids while reducing electricity costs. The effectiveness of the current system's reinforcement learning strategy is ensured through comparative evaluation with existing system baselines.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart grids, and in particular relates to a multi-park microgrid demand response method and system. Background Art

[0002] Multi-park microgrid demand response refers to the integration of park renewable energy resources and the regulation of park controllable load storage, and the acquisition of the optimal strategy for each park through model optimization methods or autonomous learning methods without system dynamics models, thereby improving the flexibility and energy efficiency of multi-park microgrid demand and reducing the electricity cost of the park. However, to achieve multi-park microgrid demand response, two obstacles need to be overcome. One is the random power input of renewable energy, and the other is the use of unknown and complex energy scheduling strategies by other parks. Therefore, it is necessary to coordinate the multi-park system to participate in the demand response plan of the complex park microgrid system structure.

[0003] At present, there are mainly the following methods for demand response of campus microgrids: multi-campus microgrid demand response based on model predictive control, multi-campus microgrid demand response based on hybrid shaping linear programming, and multi-campus microgrid demand response based on single-agent deep reinforcement learning.

[0004] In view of the above existing technologies, the multi-park microgrid demand response based on model predictive control relies on a large amount of training information in a specific environment, and it is difficult to accurately predict the uncertain parameters in the system; the multi-park microgrid demand response based on hybrid shaping linear programming has the defects of easy modeling errors and lack of adaptability; in the multi-park microgrid demand response method based on single-agent deep reinforcement learning, the algorithm can obtain the optimal control strategy through autonomous learning without understanding the system dynamics, but the single-agent reinforcement learning algorithm has limitations in high dimensions and selecting appropriate reward functions, and it is difficult to formulate effective energy management strategies to promote cooperation between parks to reduce overall load demand. At the same time, the above algorithms lack an evaluation system to ensure the effectiveness of the current system working strategy. Summary of the invention

[0005] In order to make up for the shortcomings of the prior art, the present invention proposes a multi-park microgrid demand response method and system, which can use mathematical modeling to construct a park microgrid system environment model, including a flexible load model, an energy storage model and a power utility model. At the same time, the multi-head attention mechanism is used to enable the park to dynamically select information shared by other parks that need to be paid attention to, and through a deep reinforcement learning algorithm, coordinate the park's control of flexible loads, electric cooling and heating energy storage, reshape the overall curve of power demand, and ensure the effectiveness of the reinforcement learning working strategy of the current system by comparing and evaluating with the existing system baseline.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A multi-park microgrid demand response method, comprising:

[0008] Construction of the park system environment model, including the flexible load model, electrochemical energy storage model, power utility model and hot and cold energy storage model;

[0009] Based on multi-head attention feature extraction, the multi-head attention mechanism is used to dynamically select the information shared by other parks that need to be paid attention to, so as to improve the performance and efficiency of the system;

[0010] Deep reinforcement learning is used to coordinate the park's control over flexible loads, electric cooling and heat energy storage to adjust flexible loads and energy storage systems and reshape the overall curve of electricity demand.

[0011] Preferably, the flexible load model is used to adjust the power load according to power system demand, market electricity price or other factors. The formula of the flexible load model is as follows:

[0012]

[0013] st0≤P0≤......≤P G ≤Pmax;

[0014] G∈N;

[0015] Different flexible loads are set with different load levels g, where 0 means the load stops working, G is the maximum load working level, and p k is the load working power of the corresponding level, where k = 0, 1, 2, …, G.

[0016] Preferably, the electrochemical energy storage model is charged and discharged in the form of a battery, and is used for charging during the low voltage period and discharging during the peak voltage period. The formula of the electrochemical energy storage model is as follows:

[0017] SOC t+1 =SOC t ·(1-η loss )+E D / C (t);

[0018]

[0019] Among them, soc t represents the hot and cold energy storage during period t, η loss is the self-discharge efficiency of electrochemical energy storage, E D / C (t) represents the energy selected for charging and discharging during time period t, η es is the charge and discharge efficiency, E batteryIt is the calculated actual charge and discharge capacity of the battery.

[0020] Preferably, the power utility model is used to construct a satisfaction utility function of the park load power consumption based on the hyperbolic tangent according to the properties of the utility function. The formula of the power utility model is as follows:

[0021]

[0022] stω>0

[0023] K>0;

[0024] Among them, E(t) is the power consumption of the load in the park during period t, K and ω are the satisfaction parameters of the utility function. After verification, the formula of the power utility model satisfies the first-order partial derivative greater than or equal to zero and the second-order partial derivative less than or equal to zero and passes through the origin, satisfying the properties of the utility function.

[0025] Preferably, the cold and hot energy storage model is to store the energy generated by the air-water heat pump, work at night or during non-production periods, and release the stored energy during production periods. The formula of the cold and hot energy storage model is as follows:

[0026]

[0027]

[0028]

[0029] soc t+1 =soc t ·(1-η loss )+Q C -Q D ;

[0030] make

[0031] Q C =Q mid , if Q mid >0

[0032] Q D =|Q mid |, if Q mid <0

[0033]

[0034] Among them, soc t represents the hot and cold energy storage during period t, η loss is the self-loss efficiency, a t For the reinforcement learning of hot and cold storage actions, η es Indicates the charging and discharging efficiency, Indicates the maximum thermal power of the heat pump device in period t, Q cap Indicates the capacity of cold and hot energy storage, that is, the maximum energy storage capacity of the system, Q use It represents the actual demand for cooling and heating in the park, that is, the cooling and heating energy required by the park, Q dem It represents the demand of the cold and hot energy storage system, that is, the energy that the cold and hot system needs to release, Q c represents the charging energy of the cold and hot energy storage system in the t period, that is, the energy charged by the cold and hot energy storage system, Q D represents the energy released by the cold and hot energy storage system in period t, that is, the energy released by the cold and hot energy storage system, Q in Indicates the actual cold and hot energy entering the energy storage system, Q out Indicates the actual heat and cold energy released by the energy storage system;

[0035] Unlike electrochemical energy storage, cold and hot energy storage requires constraints on charging and discharging, that is, the maximum energy discharge cannot exceed the cooling and heating demand Q of the park. dem The maximum charging capacity cannot exceed the maximum heat production capacity of the air-water heat pump minus the current use

[0036] Preferably, the multi-head attention feature extraction method includes the following multi-head attention feature extraction module:

[0037] Observation-action input representation module: using a single-layer feedforward neural network The observed state-action vectors of different dimensions in time period t Mapped to the same dimension d, the output matrix of shape (n, d) is represented as x:

[0038] x={x1,x2,…,x n};

[0039]

[0040] x represents the input observation state-action vector matrix, where each row represents the observation state and action of a period, and x i represents the i-th row in the x matrix, that is, the observed state and action vector corresponding to the i-th time period.

[0041] By mapping the observed state and action vectors to the same dimension, the state and action of the park at different times can be uniformly represented, which facilitates subsequent feature extraction and analysis. Such processing can help the system better understand and process the state and behavior of the park microgrid system, providing support for subsequent decision-making and optimization.

[0042] Each head query, key, value matrix calculation module:

[0043] For the matrix input representation output in the observation-action input representation module, three weight matrices are introduced For each head h, the output query matrix Q is calculated by the rectified linear unit h , key matrix K h Sum value matrix V h , in preparation for subsequent feature extraction and calculation, where the query matrix Q h , key matrix K h Sum value matrix V h The calculation formula is as follows:

[0044]

[0045] Weighted value matrix calculation module:

[0046] For each query, key, and value matrix, the weight matrix output in the calculation module is calculated by querying the matrix Q h And the key value matrix K h The inner product of is used to calculate the similarity, divided by Prevent gradient explosion, then normalize the output attention score matrix and add it to the value matrix V h Perform weighted summation and output the weighted value matrix as Z h , the formula of the weighted value matrix is ​​as follows:

[0047]

[0048] Multi-head merge output module:

[0049] For the weighted matrix Z output in the weighted matrix calculation module h , concatenated according to the last dimension, and passed through two layers of feedforward neural network At the same time, x is embedded and the output matrix e after feature extraction is shown as follows:

[0050]

[0051] Preferably, deep reinforcement learning includes distributed execution, which allows multiple agents to learn and make decisions in parallel in their respective environments, and achieve efficient learning and behavior through cooperation and collaboration;

[0052] Distributed execution consists of the following steps:

[0053] The first step is to determine the policy function neural network: deterministic policy function Input the time, weather and energy storage observation values ​​for period t in park i Deterministic actions that should be taken under the coordinated power, cooling, heating, and storage control of the output park This is achieved through a neural network that includes parameters θ that are updated offline through deep reinforcement learning;

[0054] The state input expression of the policy function neural network is determined as follows:

[0055]

[0056] Among them, calendar The meanings are: basic calendar information, including month, day of the week and time period of the day;

[0057] o weather The meanings are: indoor and outdoor temperature, indoor and outdoor humidity, direct solar radiation, and scattered solar radiation;

[0058] o carbon The meaning is: the carbon dioxide emission intensity per KWH in different time periods;

[0059] o load The meanings are: the energy consumption of fixed load and flexible load in the current period;

[0060] o solar The meanings are: photovoltaic power generation value and forecast value in the current period;

[0061] o demand The meanings are: indoor temperature setting value, cooling demand, heating demand and maximum satisfaction of electricity consumption;

[0062] o consumption The meanings are: net energy consumption, cooling energy consumption and heating energy consumption in the current period;

[0063] Step 2: Action Exploration:

[0064] Given the deterministic policy function and park observations in the first step, Gaussian noise is added to the output deterministic action, and a new exploration action is output. The action expression in the action exploration step is:

[0065]

[0066] Among them, a cooling_storage The meaning is: the ratio of the cold storage or cold energy value to the total cold storage capacity, that is,

[0067] a heating_storage The meaning of represents is: the ratio of heat storage or heat energy value to the total heat storage capacity, that is,

[0068] a electrical_storage It means: the amount of electricity charged or discharged by electrochemical energy storage in time period t, that is, EDIC (t);

[0069] a flex_load It means: the energy consumption of the flexible load after scheduling for period t, that is, E flex_load (t);

[0070] Step 3: Reward calculation:

[0071] The actions taken by the deep reinforcement learning algorithm steps are evaluated by calculating the park's electricity utility and net electricity cost, or by calculating the park's carbon emissions. As a reward, the reward is calculated by weighted sum in the following way:

[0072] (a) The electricity consumption of fixed loads and flexible loads in the park, i.e., U(E(t), ω);

[0073] (b) electricity charges for the net electricity consumption of the park;

[0074] (c) Settlement costs of the park's cooling, heating, electricity and energy storage;

[0075] (d) Carbon emissions of the park;

[0076] Step 4: Deposit into the experience replay pool:

[0077] Given the observations of the first step and actions Get the next period of observations through the environment Then go through the third step of reward calculation to get the reward Will Deposit into the experience replay pool.

[0078] Preferably, deep reinforcement learning also includes centralized training,

[0079] Focused training includes the following steps:

[0080] The first step is to obtain the training data set: given an experience replay pool containing enough data, randomly extract batches of data sets, record the data set locations, and request the control center. The control center collects the connected data sets and uses a multi-head attention feature extraction method to return the data set to be trained.

[0081] The second step is to evaluate the function neural network: function Input multi-head attention target features and output park observation values Next select action The scores sc for all parks are implemented through a neural network, which contains parameters This parameter is updated offline through deep reinforcement learning;

[0082] The third step is offline optimizer: According to the Adam algorithm, the parameters of the evaluation function neural network and the parameters of the deterministic strategy function neural network are optimized respectively.

[0083] A multi-park microgrid demand response system, comprising:

[0084] The construction module of the park system environment model is used to construct the park microgrid system environment model including the flexible load model, electrochemical energy storage model, power utility model and cold and hot energy storage model;

[0085] Based on the multi-head attention feature extraction module, it is used to utilize the multi-head attention mechanism to dynamically select the information shared by other parks that need to be paid attention to, so as to improve the performance and efficiency of the system;

[0086] The deep reinforcement learning module is used to coordinate the park's control over flexible loads, electric cooling and heat energy storage to adjust flexible loads and energy storage systems and reshape the overall curve of electricity demand.

[0087] As a preferred embodiment, the initialization phase includes the following: Initializing the deterministic strategy network Comment Network Where k = 1, 2, experience replay pool D; set target parameters

[0088] The operation phase includes the following steps:

[0089] (1) Starting from the period t = 1, the observed state of each park agent i is Enter the distributed execution step;

[0090] (2) Each park agent i enters the action exploration step and performs the action with Gaussian noise added

[0091] (3) Reward calculation step observes the next state And calculate the reward

[0092] (4) The experience replay pool storage step will be Store the experience in the replay pool D and enter the centralized training step;

[0093] (5) The training data set acquisition step randomly extracts a batch data set (o, a, o′, r) from the experience replay pool D, records the data set index, and requests the control center. The control center collects the connected data sets and returns the training data set (e, e′, y′) after the multi-head attention feature extraction step. -i ), start training the model;

[0094] (6) Park agent i calculates the training target value yi :

[0095]

[0096] (7) Calculate the loss function of the review network

[0097]

[0098] (8) Offline optimizer updates and evaluates network parameters

[0099]

[0100] (9) The offline optimizer updates the deterministic policy network parameters θ i ,J(θ i ) is the loss function:

[0101]

[0102] (10) Soft update target network parameters and

[0103]

[0104]

[0105] (11) All n campus agents complete the training of period t;

[0106] (12) Enter the t+1 period until reaching the last period T, completing one iteration;

[0107] (13) The evaluation module evaluates the trained model, calculates the park cost for the entire period, and compares it with the existing baseline of the system. If the effect is good, the iteration is stopped; otherwise, a new round of iterative learning is carried out.

[0108] Compared with the prior art, the technical effects and advantages of the present invention are:

[0109] The multi-campus microgrid demand response method and system combines mathematical modeling, multi-head attention feature extraction and deep reinforcement learning algorithm;

[0110] By establishing a campus microgrid system environmental model, including flexible load, energy storage and power utility models, the energy demand and supply of the campus can be accurately reflected, and accurate modeling and monitoring of the system status can be achieved.

[0111] By using the multi-head attention mechanism, the system can dynamically select information shared by other parks that needs to be paid attention to, thereby improving the perception of the overall status of the system and the attention to key information.

[0112] Through deep reinforcement learning algorithms, the park coordinates the control of flexible loads, electric cooling and heat energy storage, and adjusts the power demand curve to optimize system operating efficiency and energy utilization.

[0113] The system can intelligently adjust charging and discharging behaviors according to the specific needs and status of the park, achieve personalized response to flexible loads and energy storage equipment, and improve the flexibility of the system.

[0114] Through multi-head attention feature extraction and deep reinforcement learning, the system can quickly and accurately respond to and adjust the system status, shorten decision-making time and improve response speed.

[0115] By optimizing energy management and response mechanisms, the system can effectively reduce energy waste, improve energy utilization efficiency, reduce energy costs, and achieve overall performance improvement of the park's energy system.

[0116] In summary, the multi-park microgrid demand response method and system have the advantages of improving system flexibility, accelerating the decision-making process, and optimizing energy management, which helps to realize the intelligent operation and energy efficiency improvement of the park microgrid. BRIEF DESCRIPTION OF THE DRAWINGS

[0117] Figure 1 This is a diagram of the multi-park microgrid system architecture of the present invention;

[0118] Figure 2 It is the overall structural block diagram of the present invention. DETAILED DESCRIPTION

[0119] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0120] The following is combined with Figure 1-2 To further explain this application in detail,

[0121] The present application embodiment discloses a multi-park microgrid demand response method, including:

[0122] Construction of the park system environment model, including the flexible load model, electrochemical energy storage model, power utility model and hot and cold energy storage model;

[0123] Based on multi-head attention feature extraction, the multi-head attention mechanism is used to dynamically select the information shared by other parks that need to be paid attention to, so as to improve the performance and efficiency of the system;

[0124] Deep reinforcement learning is used to coordinate the park's control over flexible loads, electric cooling and heat energy storage to adjust flexible loads and energy storage systems and reshape the overall curve of electricity demand.

[0125] The flexible load model is used to adjust the power load according to the power system demand, market electricity price or other factors. The formula of the flexible load model is as follows:

[0126]

[0127] st0≤P0≤......≤P G ≤Pmax;

[0128] G∈N(Formula 1);

[0129] Different flexible loads are set with different load levels g, where 0 means the load stops working, G is the maximum load working level, and p k is the load working power of the corresponding level, where k = 0, 1, 2, …, G.

[0130] The electrochemical energy storage model is charged and discharged in the form of a battery, which is used for charging during the low-voltage period and discharging during the peak-voltage period. The formula of the electrochemical energy storage model is as follows:

[0131] SOC t+1 =SOC t ·(1-η loss )+E D / C (t)(Formula 2);

[0132]

[0133] Among them, soc t represents the hot and cold energy storage during period t, η loss is the self-discharge efficiency of electrochemical energy storage, E DIC (t) represents the energy selected for charging and discharging during time period t, η es is the charge and discharge efficiency, E battery It is the calculated actual charge and discharge capacity of the battery.

[0134] The cold and hot energy storage model is to store the energy generated by the air-water heat pump, work at night or during non-production periods, and release the stored energy during production periods. The formula of the cold and hot energy storage model is as follows:

[0135]

[0136]

[0137] s oct+1 =soct ·(1-η loss )+Q C -Q D (Formula 5);

[0138] make

[0139] Q C =Q mid , if Q mid >0

[0140] Q D =|Q mid |, if , Q mid <0

[0141]

[0142] Among them, soc t represents the hot and cold energy storage during period t, η loss is the self-loss efficiency, a t For the reinforcement learning of hot and cold storage actions, η es Indicates the charging and discharging efficiency, Indicates the maximum thermal power of the heat pump device in period t, Q cap Indicates the capacity of cold and hot energy storage, that is, the maximum energy storage capacity of the system, Q use It represents the actual demand for cooling and heating in the park, that is, the cooling and heating energy required by the park, Q dem It represents the demand of the cold and hot energy storage system, that is, the energy that the cold and hot system needs to release, Q c represents the charging energy of the cold and hot energy storage system in the t period, that is, the energy charged by the cold and hot energy storage system, Q D represents the energy released by the cold and hot energy storage system in period t, that is, the energy released by the cold and hot energy storage system, Q in Indicates the actual cold and hot energy entering the energy storage system, Q out Indicates the actual heat and cold energy released by the energy storage system;

[0143] Unlike electrochemical energy storage, cold and hot energy storage requires constraints on charging and discharging, that is, the maximum energy discharge cannot exceed the cooling and heating demand Q of the park. dem The maximum charging capacity cannot exceed the maximum heat production capacity of the air-water heat pump minus the current use

[0144] Formulas (5) and (6) show the charging and discharging energy size after selecting the cold and hot energy storage actions and considering the charging and discharging energy efficiency η es The actual incoming and outgoing heat and cold energy Q in and Q out .

[0145] The power utility model is used to construct a satisfaction utility function of the park load power consumption based on the hyperbolic tangent according to the properties of the utility function. The formula of the power utility model is as follows:

[0146]

[0147] stω>0

[0148] K>0;

[0149] Among them, E(t) is the power consumption of the load in the park during period t, K and ω are the satisfaction parameters of the utility function. After verification, the formula of the power utility model satisfies the first-order partial derivative greater than or equal to zero and the second-order partial derivative less than or equal to zero and passes through the origin, satisfying the properties of the utility function.

[0150] The park system environment model is a mathematical model that describes the internal and external environmental conditions of the park, usually including the access of renewable energy in the park, changes in power load, grid electricity prices, weather and other factors. The model can provide various environmental parameters required for system operation and provide basic data for the formulation and optimization of system demand response strategies.

[0151] The flexible load model describes the behavior and response capabilities of various flexible load devices in the park (such as electric vehicles, air conditioners, industrial equipment, etc.). Flexible loads can adjust energy consumption according to system needs. By utilizing the elastic response of these devices, demand response can be more flexible, improving the energy utilization and economic benefits of the system.

[0152] The electrochemical energy storage model describes the performance characteristics, charging and discharging efficiency, energy storage capacity and other parameters of the battery energy storage system in the park. Electrochemical energy storage can provide the ability to quickly adjust and balance energy supply and demand, which plays a key role in smoothing out fluctuations in renewable energy and improving system stability.

[0153] The power utility model takes into account the energy consumption characteristics, usage requirements and utility curves of various types of power consumption equipment in the park. Through the power utility model, the energy consumption of different equipment can be evaluated, providing a reference for system energy regulation and optimization.

[0154] The cold and hot energy storage model describes the demand characteristics of the cold and hot loads in the park and the operating parameters of the cold and hot energy storage system. Using cold and hot energy storage technology, the energy supply and demand balance in the park can be managed more effectively, and the energy utilization efficiency and heating / cooling effect of the system can be improved.

[0155] In summary, the construction and application of the park system environment model, flexible load model, electrochemical energy storage model, power utility model and hot and cold energy storage model help to deeply understand and model the park microgrid system, provide key data and parameter support, and provide an accurate and reliable basis for the formulation of demand response strategies, optimization algorithm design and system performance analysis of multi-park microgrid systems.

[0156] The multi-head attention feature extraction method includes the following multi-head attention feature extraction modules:

[0157] Observation-action input representation module: using a single-layer feedforward neural network The observed state-action vectors of different dimensions in time period t Mapped to the same dimension d, the output matrix of shape (n, d) is represented as x:

[0158] x={x1,x2,…,x n} (Formula 8);

[0159]

[0160] x represents the input observation state-action vector matrix, where each row represents the observation state and action of a period, and x i represents the i-th row in the x matrix, that is, the observed state and action vector corresponding to the i-th time period.

[0161] By mapping the observed state and action vectors to the same dimension, the state and action of the park at different times can be uniformly represented, which facilitates subsequent feature extraction and analysis. Such processing can help the system better understand and process the state and behavior of the park microgrid system, providing support for subsequent decision-making and optimization.

[0162] Each head query, key, value matrix calculation module:

[0163] For the matrix input representation output in the observation-action input representation module, three weight matrices are introduced For each head h, the output query matrix Q is calculated by the rectified linear unit h , key matrix K h Sum value matrix V h , in preparation for subsequent feature extraction and calculation, where the query matrix Q h , key matrix K h Sum value matrix V h The calculation formula is as follows:

[0164]

[0165] In neural networks, LeakyReLU is a variant of the rectified linear unit, which is an activation function used to introduce nonlinear characteristics and solve the problem of gradient disappearance. LeakyReLU allows the negative input value of the activation function to have a small non-zero slope, usually a small constant, while the positive input value remains unchanged. This means that when the input value is negative, LeakyReLU will not return the slope to zero, but retain a small slope to maintain a certain degree of gradient flow, which helps to avoid the problem of gradient disappearance during training, thereby improving the learning effect and stability of the neural network. The introduction of LeakyReLU can help the neural network better learn the feature representation of the data, while reducing the sensitivity to hyperparameter adjustment, making the model easier to train and optimize.

[0166] Weighted value matrix calculation module:

[0167] For each query, key, and value matrix, the weight matrix output in the calculation module is calculated by querying the matrix Q h And the key value matrix K h The inner product of is used to calculate the similarity, divided by Prevent gradient explosion, then normalize the output attention score matrix and add it to the value matrix V h Perform weighted summation and output the weighted value matrix as Z h , the formula of the weighted value matrix is ​​as follows:

[0168]

[0169] Multi-head merge output module:

[0170] For the weighted matrix Z output in the weighted matrix calculation module h , concatenated according to the last dimension, and passed through two layers of feedforward neural network At the same time, x is embedded and the output matrix e after feature extraction is shown as follows:

[0171]

[0172] The multi-head attention mechanism enables the model to focus on information at different positions in the input at the same time, so as to better capture the correlation and importance between the input data, improve the model's attention to the data and learning efficiency. Through the multi-head attention mechanism, the model can learn multiple different attention weights, so as to extract richer and more accurate feature representations, and enhance the model's ability to represent input data. The multi-head attention mechanism can associate and calculate the input data in multiple dimensions, effectively integrating information from different aspects, and helping the model to understand and process the input data more comprehensively.

[0173] The observation-action input representation module is responsible for representing and encoding the observed input data and providing it to the subsequent attention mechanism for processing.

[0174] Each query, key, and value matrix calculation module performs query, key, and value calculations on the input data, and extracts the query vector, key vector, and value vector respectively to calculate the attention weight.

[0175] The weighted value matrix calculation module calculates the attention weight according to the similarity between the query vector and the key vector, and applies the attention weight to the value vector to obtain the weighted value vector representation.

[0176] The multi-head merge output module merges the weighted value matrices calculated by multiple attention heads to obtain the output feature representation of the multi-head attention, which is provided to the subsequent network layers for processing and learning.

[0177] In general, the observation-action input representation module, per-head query, key, value matrix calculation module, weighted value matrix calculation module, multi-head merge output module, etc. constitute the key components of the multi-head attention mechanism. Through the collaborative work of these modules, multi-dimensional attention and feature extraction of input data can be achieved, thereby improving the expressiveness and learning efficiency of the model.

[0178] Deep reinforcement learning includes distributed execution, which allows multiple agents to learn and make decisions in parallel in their respective environments, and achieve efficient learning and behavior through cooperation and collaboration;

[0179] Specifically, distributed execution includes the following steps:

[0180] The first step is to determine the policy function neural network: deterministic policy function Input the time, weather and energy storage observation values ​​for period t in park i Deterministic actions that should be taken under the coordinated power, cooling, heating, and storage control of the output park This is achieved through a neural network that includes parameters θ that are updated offline through deep reinforcement learning;

[0181] The state input expression of the policy function neural network is determined as follows:

[0182]

[0183] Among them, calendar The meanings are: basic calendar information, including month, day of the week and time period of the day;

[0184] o weather The meanings are: indoor and outdoor temperature, indoor and outdoor humidity, direct solar radiation, and scattered solar radiation;

[0185] ocarbon The meaning is: the carbon dioxide emission intensity per KWH in different time periods;

[0186] o load The meanings are: the energy consumption of fixed load and flexible load in the current period;

[0187] o solar The meanings are: photovoltaic power generation value and forecast value in the current period;

[0188] o demand The meanings are: indoor temperature setting value, cooling demand, heating demand and maximum satisfaction of electricity consumption;

[0189] o consumption The meanings are: net energy consumption, cooling energy consumption and heating energy consumption in the current period;

[0190] Step 2: Action Exploration:

[0191] Given the deterministic policy function and park observations in the first step, Gaussian noise is added to the output deterministic action, and a new exploration action is output. The action expression in the action exploration step is:

[0192]

[0193] Among them, a cooling_storage The meaning is: the ratio of the cold storage or cold energy value to the total cold storage capacity, that is,

[0194] a heating_storage The meaning of represents is: the ratio of heat storage or heat energy value to the total heat storage capacity, that is,

[0195] a electrical_storage It means: the amount of electricity charged or discharged by electrochemical energy storage in time period t, that is, E DIC (t);

[0196] a flex_load It means: the energy consumption of the flexible load after scheduling for period t, that is, E flex_load (t);

[0197] Step 3: Reward calculation:

[0198] The actions taken by the deep reinforcement learning algorithm steps are evaluated by calculating the park's electricity utility and net electricity cost, or by calculating the park's carbon emissions. As a reward, the reward is calculated by weighted sum in the following way:

[0199] (a) The electricity consumption of fixed loads and flexible loads in the park, i.e., U(E(t), ω);

[0200] (b) electricity charges for the net electricity consumption of the park;

[0201] (c) Settlement costs of the park's cooling, heating, electricity and energy storage;

[0202] (d) Carbon emissions of the park;

[0203] Step 4: Deposit into the experience replay pool:

[0204] Given the observations of the first step and actions Get the next period of observations through the environment Then go through the third step of reward calculation to get the reward Will Deposit into the experience replay pool.

[0205] Specifically, deep reinforcement learning also includes centralized training.

[0206] Focused training includes the following steps:

[0207] The first step is to obtain the training data set: given an experience replay pool containing enough data, randomly extract batches of data sets, record the data set locations, and request the control center. The control center collects the connected data sets and uses a multi-head attention feature extraction method to return the data set to be trained.

[0208] The second step is to evaluate the function neural network: function Input multi-head attention target features and output park observation values Next select action The scores sc for all parks are implemented through a neural network, which contains parameters This parameter is updated offline through deep reinforcement learning;

[0209] The third step is offline optimizer: According to the Adam algorithm, the parameters of the evaluation function neural network and the parameters of the deterministic strategy function neural network are optimized respectively.

[0210] Deep reinforcement learning is a method that combines deep learning and reinforcement learning to solve the problem of how an agent learns to make optimal decisions in the process of interacting with the environment. In deep reinforcement learning, the agent gradually learns how to maximize the expected cumulative reward by interacting with the environment, trying different actions and observing the feedback from the environment.

[0211] Deep reinforcement learning includes steps such as environmental observation, action selection, reward calculation, experience replay, learning and optimization, and goal achievement. These steps are repeated, and the agent gradually improves its decision-making ability through interaction with the environment and optimization of the strategy network to achieve the best action plan. Deep reinforcement learning can be used to coordinate the park's control of flexible loads, electric cooling and heat storage systems, etc. Through deep reinforcement learning, the agent can learn the optimal strategy to adjust the flexible load and energy storage system to meet the needs of the park and optimize the overall power demand curve. This method can help the park manage and optimize electricity consumption more effectively, maintain system stability, and maximize the use of resources such as renewable energy. The application of deep reinforcement learning can improve the energy efficiency and economy of the park, and play a key role in the operation and management of the energy system.

[0212] A multi-park microgrid demand response system includes an initialization phase and an operation phase, and is used to execute the multi-park microgrid demand response method.

[0213] As a preferred embodiment, the initialization phase includes the following: Initializing the deterministic strategy network Comment Network Where k = 1, 2, experience replay pool D; set target parameters

[0214] The operation phase includes the following steps:

[0215] (1) Starting from the period t = 1, the observed state of each park agent i is Enter the distributed execution step;

[0216] (2) Each park agent i enters the action exploration step and performs the action with Gaussian noise added

[0217] (3) Reward calculation step observes the next state And calculate the reward

[0218] (4) The experience replay pool storage step will be Store the experience in the replay pool D and enter the centralized training step;

[0219] (5) The training data set acquisition step randomly extracts a batch data set (o, a, o′, r) from the experience replay pool D, records the data set index, and requests the control center. The control center collects the connected data sets and returns the training data set (e, e′, y′) after the multi-head attention feature extraction step. -i ), start training the model;

[0220] (6) Park agent i calculates the training target value yi :

[0221]

[0222] (7) Calculate the loss function of the review network

[0223]

[0224] (8) Offline optimizer updates and evaluates network parameters

[0225]

[0226] (9) The offline optimizer updates the deterministic policy network parameters θ i ,J(θ i ) is the loss function:

[0227]

[0228] (10) Soft update target network parameters and

[0229]

[0230] (11) All n campus agents complete the training of period t;

[0231] (12) Enter the t+1 period until reaching the last period T, completing one iteration;

[0232] (13) The evaluation module evaluates the trained model, calculates the park cost for the entire period, and compares it with the existing baseline of the system. If the effect is good, the iteration is stopped; otherwise, a new round of iterative learning is carried out.

[0233] The multi-park microgrid demand response method and system can achieve autonomous learning through deep reinforcement learning algorithms to obtain optimal control strategies without the need to label a large amount of training data in advance. It has good adaptability and intelligence. Through the multi-head attention mechanism, the system can better focus on the correlation between different parks, achieve flexibility in demand response and improve energy efficiency of the park microgrid system, and optimize the control strategy to effectively reduce the park's electricity costs and improve the economic benefits of the system. Through systematic learning and optimization, it can better coordinate cooperation between multiple parks and improve the efficiency and synergy of the overall park microgrid system.

[0234] In general, the present invention can obtain the optimal control strategy through autonomous learning, effectively improve the flexibility and energy efficiency of multi-park microgrids, and reduce electricity costs. Compared with other methods, this method does not rely on a large amount of training information in a specific environment, is not prone to modeling errors or lack of adaptability, and is relatively more suitable for coordinating multi-park systems to participate in complex demand response plans. It can effectively improve the operation of multi-park microgrid systems, improve the intelligence and adaptability of the system, and provide an effective method and system for achieving efficient use of clean energy and coordinated management of power grids.

[0235] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-park microgrid demand response method, characterized in that: include: Construction of the park system environment model, including the flexible load model, electrochemical energy storage model, power utility model and hot and cold energy storage model; Based on multi-head attention feature extraction, the multi-head attention mechanism is used to dynamically select the information shared by other parks that need to be paid attention to, so as to improve the performance and efficiency of the system; Deep reinforcement learning is used to coordinate the park's control of flexible loads, electric cooling and heat storage to adjust flexible loads and energy storage systems and reshape the overall curve of electricity demand; The cold and hot energy storage model is to store the energy generated by the air-water heat pump, work at night or during non-production periods, and release the stored energy during production periods. The formula of the cold and hot energy storage model is as follows: friend t+1 =soc t ·(1-η loss )+Q C -Q D ; make Q C =Q mid ,if Q mid >0 Q D =|Q mid |,if Q mid =0 Among them, soc t represents the hot and cold energy storage during period t, η loss is the self-loss efficiency, a t For the reinforcement learning of hot and cold storage actions, η es Indicates the charging and discharging efficiency, Indicates the maximum thermal power of the heat pump device in period t, Q cap Indicates the capacity of hot and cold energy storage, Q use Indicates the actual demand for cooling and heating in the park, Q dem represents the demand for the hot and cold energy storage system, Q c represents the charging capacity of the hot and cold energy storage system in period t, Q D represents the energy released by the hot and cold energy storage system in period t, Q in Indicates the actual cold and hot energy entering the energy storage system, Q out Indicates the actual heat and cold energy released by the energy storage system.

2. A multi-park microgrid demand response method according to claim 1, characterized in that: The flexible load model is used to adjust the power load according to the power system demand and the market electricity price. The formula of the flexible load model is as follows: s.t.0≤P0≤......≤P G ≤Pmax; G∈N; Different flexible loads are set with different load levels g, where 0 means the load stops working, G is the maximum load working level, and p k is the load working power of the corresponding level, where k = 0, 1, 2,···, G.

3. A multi-park microgrid demand response method according to claim 1, characterized in that: The electrochemical energy storage model is charged and discharged in the form of a battery, which is used for charging during the low-voltage period and discharging during the peak-voltage period. The formula of the electrochemical energy storage model is as follows: SOC t+1 =SOC t ·(1-η loss )+E D / C (t); Among them, soc t represents the hot and cold energy storage during period t, η loss is the self-consumption efficiency, E DIC (t) represents the energy selected for charging and discharging during time period t, η es is the charging and discharging efficiency, E battey It is the calculated actual charge and discharge capacity of the battery.

4. A multi-park microgrid demand response method according to claim 1, characterized in that: The power utility model is used to construct a satisfaction utility function of the park load power consumption based on the hyperbolic tangent according to the properties of the utility function. The formula of the power utility model is as follows: Among them, E(t) is the power consumption of the load in the park during period t, and K and ω are the satisfaction parameters of the utility function.

5. A multi-park microgrid demand response method according to claim 1, characterized in that: The multi-head attention feature extraction method includes the following multi-head attention feature extraction modules: Observation-action input representation module: using a single-layer feedforward neural network The observed state-action vectors of different dimensions in time period t Mapped to the same dimension d, the output matrix of shape (n, d) is represented as x: x={x1,x2,···,x n }; x represents the input observation state-action vector matrix, where each row represents the observation state and action of a period, and x i represents the i-th row in the x matrix; Each query, key, and value matrix calculation module: For the matrix input representation output in the observation-action input representation module, three weight matrices are introduced For each head h, the output query matrix Q is calculated by the rectified linear unit h , key matrix K h Sum value matrix V h , where the query matrix Q h , key matrix K h Sum value matrix V h The calculation formula is as follows: Weighted value matrix calculation module: For each query, key, and value matrix, the weight matrix output in the calculation module is calculated by querying the matrix Q h and key matrix K h The inner product of is used to calculate the similarity, divided by Prevent gradient explosion, then normalize the output attention score matrix and add it to the value matrix V h Perform weighted summation and output the weighted value matrix as Z h , the formula of the weighted value matrix is ​​as follows: Multi-head merge output module: For the weighted matrix Z output in the weighted matrix calculation module h , concatenated according to the last dimension, and passed through two layers of feedforward neural network At the same time, x is embedded and the output matrix e after feature extraction is shown as follows:

6. A multi-park microgrid demand response method according to claim 1, characterized in that: Deep reinforcement learning includes distributed execution, which allows multiple agents to learn and make decisions in parallel in their respective environments, and achieve efficient learning and behavior through cooperation and collaboration; Distributed execution consists of the following steps: The first step is to determine the policy function neural network: deterministic policy function Input the time, weather and energy storage observation values ​​for period t in park i Deterministic actions that should be taken under the coordinated power, cooling, heating, and storage control of the output park This is achieved through a neural network that includes parameters θ that are updated offline through deep reinforcement learning; The state input expression of the policy function neural network is determined as follows: Among them, calendar The meanings are: basic calendar information, including month, day of the week and time period of the day; Ο weather The meanings are: indoor and outdoor temperature, indoor and outdoor humidity, direct solar radiation, and scattered solar radiation; Ο carbon The meaning is: the carbon dioxide emission intensity per KWH in different time periods; Ο load The meanings are: the energy consumption of fixed load and flexible load in the current period; Ο solar The meanings are: photovoltaic power generation value and forecast value in the current period; Ο deman The meanings are: indoor temperature setting value, cooling demand, heating demand and maximum satisfaction of electricity consumption; Ο onsumption The meanings are: net energy consumption, cooling energy consumption and heating energy consumption in the current period; Step 2: Action Exploration: Given the deterministic policy function and park observations in the first step, Gaussian noise is added to the output deterministic action, and a new exploration action is output. The action expression in the action exploration step is: Among them, a cooling_storage The meaning is: the ratio of the cold storage or cold energy value to the total cold storage capacity, that is, a heating_storage The meaning of represents is: the ratio of heat storage or heat energy value to the total heat storage capacity, that is, a electrical_storage It means: the energy charged or discharged by electrochemical energy storage in time period t, that is, E D / C (t); a flex_load It means: the energy consumption of the flexible load after scheduling for period t, that is, E flex_load (t); Step 3: Reward calculation: The actions taken by the deep reinforcement learning algorithm steps are evaluated by calculating the park's electricity utility and net electricity cost, or by calculating the park's carbon emissions. As a reward, the reward is calculated by weighted sum in the following way: (a) The electricity consumption of fixed loads and flexible loads in the park, i.e., U(E(t),ω); (b) electricity charges for the net electricity consumption of the park; (c) Settlement costs of the park's cooling, heating, electricity and energy storage; (d) Carbon emissions of the park; Step 4: Deposit into the experience replay pool: Given the observations of the first step and actions Get the next period of observations through the environment Then go through the third step of reward calculation to get the reward Will Deposit into the experience replay pool.

7. A multi-park microgrid demand response method according to claim 6, characterized in that: Deep reinforcement learning also includes centralized training. Focused training includes the following steps: The first step is to obtain the training data set: given an experience replay pool containing enough data, randomly extract batches of data sets, record the data set locations, and request the control center. The control center collects the connected data sets and uses a multi-head attention feature extraction method to return the data set to be trained. The second step is to evaluate the function neural network: function Input multi-head attention target features and output park observation values Next select action The scores sc for all parks are implemented through a neural network, which contains parameters This parameter is updated offline through deep reinforcement learning; The third step is offline optimizer: According to the Adam algorithm, the parameters of the evaluation function neural network and the parameters of the deterministic strategy function neural network are optimized respectively.

8. A multi-campus microgrid demand response system applied to the multi-campus microgrid demand response method according to any one of claims 1 to 7, characterized in that: include: The construction module of the park system environment model is used to construct the park microgrid system environment model including the flexible load model, electrochemical energy storage model, power utility model and cold and hot energy storage model; Based on the multi-head attention feature extraction module, it is used to utilize the multi-head attention mechanism to dynamically select the information shared by other parks that need to be paid attention to, so as to improve the performance and efficiency of the system; The deep reinforcement learning module is used to coordinate the park's control over flexible loads, electric cooling and heat energy storage to adjust flexible loads and energy storage systems and reshape the overall curve of electricity demand.

9. A multi-park microgrid demand response system according to claim 8, characterized in that: The initialization phase includes the following: Initialize the deterministic policy network Comment Network Where k = 1, 2, experience replay pool D; set target parameters The operation phase includes the following steps: (1) Starting from the period t = 1, the observed state of each park agent i is Enter the distributed execution step; (2) Each park agent i enters the action exploration step and performs the action with Gaussian noise added (3) Reward calculation step observes the next state And calculate the reward (4) The experience replay pool storage step will be Store it in the experience replay pool D and enter the centralized training step; (5) The training data set acquisition step randomly extracts a batch data set (o, a, o′, r) from the experience replay pool D, records the data set index, and requests the control center. The control center collects the connected data sets and returns the training data set (e, e′, y′) after the multi-head attention feature extraction step. -i ), start training the model; (6) Park agent i calculates the training target value y i : (7) Calculate the loss function of the review network (8) Offline optimizer updates and evaluates network parameters (9) The offline optimizer updates the deterministic policy network parameters θ i ,J(θ i ) is the loss function: (10) Soft update target network parameters and (11) All n campus agents complete the training of period t; (12) Enter the t+1 period until reaching the last period T, completing one iteration; (13) The evaluation module evaluates the trained model, calculates the campus cost for the entire period, and compares it with the existing baseline of the system.

Images