Multi-park integrated energy system scheduling method based on personalized federal reinforcement learning

Abstract

The invention relates to the technical field of power systems, and particularly discloses a multi-park integrated energy system scheduling method based on personalized federal reinforcement learning, and the method comprises the steps: building a multi-park integrated energy system low-carbon economic scheduling model with the minimum daily operation cost as the target, the constraint conditions comprise park equipment power output upper and lower limit constraints, storage battery operation constraints, energy supply equality constraints and energy supply inequality constraints; solving the low-carbon economic dispatching model of the multi-park integrated energy system by adopting a personalized federal reinforcement learning method; and solving to obtain a scheduling scheme. The method has the advantages that the efficient low-carbon collaborative scheduling of the multi-park integrated energy system is realized on the premise of strictly protecting the data privacy of each park through a personalized federal reinforcement learning framework.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a multi-park integrated energy system scheduling method based on personalized federated reinforcement learning. Background Art

[0002] To meet the practical needs of improving economic efficiency and reducing carbon emissions, the energy sector is gradually building integrated energy systems that coordinate multiple energy forms, thereby achieving a lower-carbon economy. However, compared to operating a single integrated energy system in a single park, interconnecting multiple integrated energy systems within the same distribution area can further tap into flexible resources. Through the exchange of energy and information, this improves the complementarity of each park's operating methods and reduces its dependence on higher-level, high-carbon energy networks.

[0003] However, traditional model-driven approaches are struggling to adapt to the increasingly complex demands of optimizing and scheduling multi-park integrated energy systems. At the modeling level, high-precision physical modeling presents challenges, and fixed scheduling schemes lack the flexibility to respond to dynamic changes. At the scenario level, the strong coupling between multiple energy devices and the combined effects of uncertainty severely limit algorithm robustness. At the decision-making level, the urgent need for high-performance computing and rapid response in real-time decision-making scenarios also makes existing methods inadequate. Faced with these multiple challenges, traditional model-driven approaches have reached a bottleneck.

[0004] Therefore, deep reinforcement learning has been widely used in the field of optimizing and scheduling multi-campus integrated energy systems. As a model-free approach, deep reinforcement learning leverages historical data for adaptive learning, addressing the uncertainties of loads and renewable energy sources, and meeting the complex scheduling requirements of offline training and online decision-making. However, as multiple stakeholder parks are reluctant to share local data during collaborative training, data silos and privacy issues are becoming increasingly prominent. In this context, federated reinforcement learning, as a distributed training framework, is gradually being applied to multi-agent scenarios, leveraging its advantage of enabling multi-agent collaborative training without sharing raw data.

[0005] Although existing research has made useful explorations into heterogeneity, the collaborative scheduling of multi-park integrated energy systems still faces the mutually exclusive dilemma of privacy protection and personalized scheduling needs. Furthermore, existing optimization scheduling research often uses the optimal operating cost as the objective function, and the environmental costs of system operation are not fully considered. With the promotion and improvement of carbon trading market policy mechanisms, a fixed carbon trading mechanism has been introduced to further promote the deep integration of multi-energy complementarity and low-carbon scheduling goals. Although this mechanism has reduced carbon emissions to a certain extent, its ability to control carbon emissions is still limited, making it difficult to effectively incentivize energy conservation and emission reduction in a multi-park collaborative scheduling environment. Therefore, a tiered carbon trading mechanism is considered to achieve more efficient and coordinated low-carbon economic scheduling. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a multi-park integrated energy system scheduling method based on personalized federated reinforcement learning, which is used to solve the problems of multi-agent heterogeneity, privacy protection and real-time decision-making faced in the current multi-park integrated energy system scheduling process.

[0007] The object of the present invention is achieved through the following technical solution: a multi-park integrated energy system scheduling method based on personalized federated reinforcement learning, the method comprising:

[0008] A low-carbon economic dispatch model for a multi-park integrated energy system is constructed with the goal of minimizing daily operating costs. Its objective function is:

[0009] (1)

[0010] Where, The cost of purchased energy; is the distributed transaction cost; for tiered carbon trading costs; Operation and maintenance costs for energy storage equipment, etc.

[0011] The constraints of the multi-park integrated energy system low-carbon economic dispatch model include upper and lower limits of park equipment power output, battery operation constraints, energy supply equality constraints, and energy supply inequality constraints.

[0012] A personalized federated reinforcement learning method is used to solve the low-carbon economic dispatch model of multi-park integrated energy system;

[0013] At the single-park level, an improved deep deterministic policy gradient reinforcement learning method is adopted; the improvement includes introducing embedding vectors into the deep deterministic policy gradient neural network, which are used to perceive and adapt to the operating characteristics of the specific park;

[0014] At the multi-campus level, a personalized federated framework is constructed for collaborative training. Each park only passes model parameters other than the embedding vector to the aggregation center for aggregation, and the aggregated global model parameters are distributed to each park. Each park uses the aggregated global model parameters to update the local model, but retains the local embedding vector. Through this personalized federated framework, distributed collaborative training is achieved while protecting the privacy data of each park, addressing the challenges of multi-campus heterogeneity.

[0015] Specifically, the upper and lower limits of the power output of the park equipment are:

[0016] (7)

[0017] Where, and are the upper and lower limits of the power output of the cogeneration unit, and They are the upper and lower limits of the thermal output of the gas boiler respectively; and They are the upper and lower limits of the thermal output of the electric boiler respectively.

[0018] Specifically, the battery operation constraints are:

[0019] (8)

[0020] Where, and are the upper and lower limits of the battery energy storage state, and They are the upper and lower limits of the battery operating power respectively.

[0021] Specifically, the energy supply equation constraint is:

[0022] (9)

[0023] Where, 、 、 、 and They represent the amount of electricity traded between the system and the upper main grid at time t, the photovoltaic output, the power consumed by the electric boiler to produce heat energy, the electric load, and the electric energy traded in the distributed energy trading market, all in units of ; and are the thermal power output of the cogeneration unit and the system thermal load at time t, respectively, in units of ; 、 and They represent the total amount of natural gas supplied to the system at time t, the natural gas consumption of the cogeneration unit, and the natural gas consumption of the gas boiler respectively; It is the battery operating power; Provides electrical output for cogeneration units; Provide heat output for gas boilers; Provides heat output for electric boilers.

[0024] Specifically, the energy supply inequality constraint is:

[0025] (10)

[0026] Where, and are the upper and lower limits of electricity purchase and sale, and The upper and lower limits of natural gas supply respectively.

[0027] Specifically, the improved deep deterministic policy gradient reinforcement learning method defines the state space of the Markov decision process for each park agent as follows:

[0028] (11)

[0029] Where, 、 、 and are the photovoltaic output, electrical load, thermal load and battery state of charge of the i-th park respectively; 、 、 、 、 They are the on-grid electricity purchase price, on-grid electricity sales price, local electricity purchase price at the last moment, local electricity sales price at the last moment and dispatch time;

[0030] The action space is:

[0031] (12)

[0032] Where, 、 and They represent the CHP unit output power, electric boiler power consumption and battery operating power of the i-th park at time t respectively;

[0033] The reward function is:

[0034] (13)

[0035] Where, Scaling factor to control costs; is the scaling factor of the constraint penalty; To constrain the penalty cost, it is expressed as:

[0036] (14)

[0037] Where, Store / discharge excess power to the battery.

[0038] The present invention has the following advantages:

[0039] This paper uses a personalized federated reinforcement learning framework to achieve efficient, low-carbon coordinated scheduling of multi-campus integrated energy systems while strictly protecting the data privacy of each park. Based on embedded vectors that perceive the heterogeneous nature of parks, federated aggregation only shares model parameters while isolating sensitive data, achieving privacy protection with "data remains static, model remains dynamic." BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a structural diagram of the multi-park integrated energy system;

[0041] Figure 2 Diagram of the personalized federated reinforcement learning framework for multi-campus integrated energy systems. DETAILED DESCRIPTION

[0042] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for the purpose of explaining the present invention and are not intended to limit the present invention. That is, the embodiments described herein are only some embodiments of the present invention, not all embodiments. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein can be arranged and designed in various different configurations.

[0043] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0044] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0045] The present invention will be further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

[0046] like Figures 1 to 2 As shown in FIG, a multi-park integrated energy system scheduling method based on personalized federated reinforcement learning includes:

[0047] The present invention aims to meet the needs of multi-agent heterogeneity, privacy protection and real-time decision-making in the collaborative optimization of multi-park integrated energy systems, consider the operating characteristics of single-park electric and thermal loads, time-of-use electricity prices and tiered carbon trading, consider the participation of multi-park integrated energy systems in direct inter-system electricity trading through a distributed energy trading market, take the realization of low-carbon economic operation costs of the system as the scheduling goal, consider constraints such as equipment operation and raw material supply, and propose a multi-park integrated energy system scheduling method that takes into account personalized decision-making and privacy protection.

[0048] Multi-park integrated energy system scenario based on distributed energy trading: The multi-park integrated energy system consists of n park integrated energy systems, a distributed energy trading market, the upper-level power grid, the natural gas grid and the regional carbon trading market.

[0049] Under this structure, the park's integrated energy system can purchase and sell electricity through the upper-level power grid and purchase natural gas as fuel from the gas grid to achieve internal autonomy. At the same time, each park can also realize inter-park energy transactions through distributed energy transactions to meet the multi-energy complementary needs of multiple parks. In addition, n parks also trade carbon emission quotas through the regional carbon trading market to incentivize each to achieve low-carbon operation. To ensure the information security of each park and avoid the leakage of internal operation data, the transaction power and internal transaction electricity prices are transmitted in real time between parks through a trusted distributed energy market, and the purchase and sale of carbon emission quotas are transmitted through a trusted regional carbon trading market. Each park is equipped with smart meters and smart plugs to provide basic data input for model training. Through collaborative training, each park ensures its own internal energy balance and achieves optimal low-carbon economic operation;

[0050] A low-carbon economic dispatch model for a multi-park integrated energy system is constructed with the goal of minimizing daily operating costs. Its objective function is:

[0051] (1)

[0052] Where, The cost of purchased energy; is the distributed transaction cost; for tiered carbon trading costs; Operation and maintenance costs for energy storage equipment, etc.

[0053] The constraints of the multi-park integrated energy system low-carbon economic dispatch model include upper and lower limits of park equipment power output, battery operation constraints, energy supply equality constraints, and energy supply inequality constraints.

[0054] The upper and lower limits of the park equipment power output are:

[0055] (7)

[0056] Where, and are the upper and lower limits of the power output of the cogeneration unit, and They are the upper and lower limits of the thermal output of the gas boiler respectively; and They are the upper and lower limits of the thermal output of the electric boiler respectively.

[0057] The battery operation constraints are:

[0058] (8)

[0059] Where, and are the upper and lower limits of the battery energy storage state, and They are the upper and lower limits of the battery operating power respectively.

[0060] The energy supply equality constraint is:

[0061] (9)

[0062] Where, 、 、 、 and They represent the amount of electricity traded between the system and the upper main grid at time t, the photovoltaic output, the power consumed by the electric boiler to produce heat energy, the electric load, and the electric energy traded in the distributed energy trading market, all in units of ; and are the thermal power output of the cogeneration unit and the system thermal load at time t, respectively, in units of ; 、 and They represent the total amount of natural gas supplied to the system at time t, the natural gas consumption of the cogeneration unit, and the natural gas consumption of the gas boiler respectively; It is the battery operating power; Provides electrical output for cogeneration units; Provide heat output for gas boilers; Provides heat output for electric boilers.

[0063] The energy supply inequality constraint is:

[0064] (10)

[0065] Where, and are the upper and lower limits of electricity purchase and sale, and The upper and lower limits of natural gas supply respectively.

[0066] A personalized federated reinforcement learning method is used to solve the low-carbon economic dispatch model of multi-park integrated energy system;

[0067] At the single-park level, an improved deep deterministic policy gradient reinforcement learning method is adopted; the improvement includes introducing embedding vectors into the deep deterministic policy gradient neural network, which are used to perceive and adapt to the operating characteristics of the specific park;

[0068] At the multi-campus level, a personalized federated framework is constructed for collaborative training. Each park only passes model parameters other than the embedding vector to the aggregation center for aggregation, and the aggregated global model parameters are distributed to each park. Each park uses the aggregated global model parameters to update the local model, but retains the local embedding vector. Through this personalized federated framework, distributed collaborative training is achieved while protecting the privacy data of each park, addressing the challenges of multi-campus heterogeneity.

[0069] Construct a Markov decision process:

[0070] Common Markov decision processes usually consist of the tuple ,in Represents a set of states, Represents a set of actions, is the set of state transition probabilities, is the reward function, represents the discount factor;

[0071] The state information observed by each park agent from the environment consists of two categories: global observation state and local observation state. In each decision cycle, the improved deep deterministic policy gradient reinforcement learning method defines the state space of the Markov decision process for each park agent as follows:

[0072] (11)

[0073] Where, 、 、 and are the photovoltaic output, electrical load, thermal load and battery state of charge of the i-th park respectively; 、 、 、 、 They are the on-grid electricity purchase price, on-grid electricity sales price, local electricity purchase price at the previous moment, local electricity sales price at the previous moment and dispatch time; it is worth noting that the global observation space of each intelligent agent is the same, while the local observation space reflects the heterogeneity of multiple agents to a certain extent.

[0074] The action space is:

[0075] (12)

[0076] Where, 、 and They represent the CHP unit output power, electric boiler power consumption and battery operating power of the i-th park at time t respectively;

[0077] As a guiding strategy The updated indicator can guide the agent to take actions within the constraints to minimize the operating cost. At the same time, considering that the daily operation balance of the battery is a strong time-coupled constraint, the agent cannot guarantee that it will always meet the constraint. The reward function is:

[0078] (13)

[0079] Where, Scaling factor to control costs; is the scaling factor of the constraint penalty; To constrain the penalty cost, it is expressed as:

[0080] (14)

[0081] Where, Store / discharge excess power to the battery.

[0082] The personalized federation algorithm mainly includes the following three main internal steps:

[0083] 1) Use a parallel approach. Each park uses Gaussian embedding to initialize its own embedding vector and obtains the state from the environment as input. The Actor network outputs a set of actions and acts on the park's integrated energy system. The internal energy flow is calculated based on the park's integrated energy system model for scheduling management. The current state ,action ,award and the next moment state Save to the experience replay pool;

[0084] 2) Each park's integrated energy system agent extracts a small batch of experiences from the experience replay pool to train and update the local model;

[0085] 3) After a communication interval, the aggregation center collects and aggregates the average of n local DDPG model parameters to obtain the global model parameters And it will be distributed to each park for update.

[0086] Among them, the improved DDPG method of local differential perception is as follows:

[0087] 1) Improve the DDPG algorithm: In the initialization phase, add Gaussian embedding vectors and As a unique parameter of the local DDPG model;

[0088] 2) In the state preprocessing stage, the embedding vector is concatenated with the state space according to different network types to form a new state space:

[0089] (2)

[0090] During training, the online Actor network by As a parameter, the state space of agent i As input value, output action Online Critic Network by As a parameter, the state space of agent i and actions As input, it outputs a scalar estimate of the Q value to perform the policy evaluation task.

[0091] Specifically, an experience replay pool is used to store the experience tuples obtained from the environment , extract small batches Perform training and calculate the mean square error of the online critic network:

[0092] (3)

[0093] Where, and Represent the target Critic network and the target Actor network respectively. Then the deterministic policy gradient theory is used to obtain the policy gradient of the online Actor network:

[0094] (4)

[0095] Then, the four networks are updated according to the following formula:

[0096] (5)

[0097] Where: 、 Represents the learning rate of the online Actor network and the online Critic network, is the target network soft update rate. and Use a learning rate of Gradient descent update:

[0098] (6)

[0099] Based on personalized federated reinforcement learning problem solving, a single campus adopts an actor-critic architecture based on an improved DDPG method. The actor network outputs continuous actions, and the critic network corrects the weights of the actor network. The embedding vector is a multi-dimensional vector representation, and the dimensionality can be adjusted as a hyperparameter, such as two, three, or four dimensions. Its function is to perceive the operating characteristics of each campus and then abstract them into a vector representation. There are some differences in operating characteristics and equipment types between different campuses. To adapt to the diverse and heterogeneous multi-campus environment, the embedding vector also perceives and learns these differences during the training process through gradient descent. Combining the personalized federated framework with the improved DDPG method, distributed control is achieved while protecting privacy. The final solution model is used to obtain a scheduling solution.

[0100] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make many possible changes and modifications to the technical solution of the present invention using the above technical content, or modify it into an equivalent embodiment with equivalent changes. Therefore, any changes, modifications, equivalent changes, and modifications made to the above embodiments based on the technology of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the present technical solution.

Claims

1. A multi-park integrated energy system scheduling method based on personalized federated reinforcement learning, characterized by: The method includes: A low-carbon economic dispatch model for a multi-park integrated energy system is constructed with the goal of minimizing daily operating costs. Its objective function is: （1） Where, The cost of purchased energy; is the distributed transaction cost; for tiered carbon trading costs; Operation and maintenance costs for energy storage equipment, etc. The constraints of the multi-park integrated energy system low-carbon economic dispatch model include upper and lower limits of park equipment power output, battery operation constraints, energy supply equality constraints, and energy supply inequality constraints. A personalized federated reinforcement learning method is used to solve the low-carbon economic dispatch model of multi-park integrated energy system; At the single-park level, an improved deep deterministic policy gradient reinforcement learning method is adopted; the improvement includes introducing embedding vectors into the deep deterministic policy gradient neural network, which are used to perceive and adapt to the operating characteristics of the specific park; At the multi-campus level, a personalized federated framework is constructed for collaborative training. Each park only passes the model parameters except the embedding vector to the aggregation center for aggregation, and the aggregated global model parameters are distributed to each park. Each park uses the aggregated global model parameters to update the local model but retains the local embedding vector. The scheduling plan is solved.

2. The multi-park integrated energy system scheduling method based on personalized federated reinforcement learning according to claim 1 is characterized by: The upper and lower limits of the park equipment power output are: （7） Where, and are the upper and lower limits of the power output of the cogeneration unit, and They are the upper and lower limits of thermal output of gas boilers respectively; and They are the upper and lower limits of the thermal output of electric boilers respectively.

3. The multi-park integrated energy system scheduling method based on personalized federated reinforcement learning according to claim 1 is characterized by: The battery operation constraints are: （8） Where, and are the upper and lower limits of the battery energy storage state, and They are the upper and lower limits of the battery operating power respectively.

4. The multi-park integrated energy system scheduling method based on personalized federated reinforcement learning according to claim 1 is characterized by: The energy supply equality constraint is: （9） Where, 、 、 、 and They represent the amount of electricity traded between the system and the upper main grid at time t, the photovoltaic output, the power consumed by the electric boiler to produce heat energy, the electric load, and the electric energy traded in the distributed energy trading market, all in units of ; and are the thermal power output of the cogeneration unit and the system thermal load at time t, respectively, in units of ; 、 and They represent the total amount of natural gas supplied to the system at time t, the natural gas consumption of the cogeneration unit, and the natural gas consumption of the gas boiler respectively; It is the battery operating power; Provides electrical output for cogeneration units; Provide heat output for gas boilers; Provides heat output for electric boilers.

5. The multi-park integrated energy system scheduling method based on personalized federated reinforcement learning according to claim 1 is characterized in that: The energy supply inequality constraint is: （10） Where, and are the upper and lower limits of electricity purchase and sale, and The upper and lower limits of natural gas supply respectively.

6. The multi-park integrated energy system scheduling method based on personalized federated reinforcement learning according to claim 1 is characterized by: The improved deep deterministic policy gradient reinforcement learning method defines the state space of the Markov decision process for each park agent as: （11） Where, 、 、 and are the photovoltaic output, electrical load, thermal load and battery state of charge of the i-th park respectively; 、 、 、 、 They are the on-grid electricity purchase price, on-grid electricity sales price, local electricity purchase price at the last moment, local electricity sales price at the last moment and dispatch time; The action space is: （12） Where, 、 and They represent the CHP unit output power, electric boiler power consumption and battery operating power of the i-th park at time t respectively; The reward function is: （13） Where, Scaling factor to control costs; is the scaling factor of the constraint penalty; To constrain the penalty cost, it is expressed as: （14） Where, Store / discharge excess power to the battery.

Images