A method and system for optimizing scheduling of a multi-region integrated energy system

Abstract

The application provides a kind of multi-region integrated energy system optimization scheduling method and system, the method is each regional integrated energy system by corresponding local agent management, local agent is based on the idea of reinforcement learning to preset energy optimization scheduling model training obtains local optimization model, and according to the neighbor agent's encryption parameter data packet and the double random matrix based on the topology relationship between integrated energy systems is updated to local optimization model, and after the local optimization model of each local agent is updated, the overall convergence of all local optimization models is evaluated and the collaborative training between local agents is stopped when the preset optimization target is reached, and the local optimization model of each local agent is executed according to the corresponding regional integrated energy system optimization scheduling.The application not only can effectively avoid the risk of data privacy leakage of each regional integrated energy system, but also can improve the real-time performance and reliability of multi-region integrated energy system optimization scheduling.

Description

Technical Field

[0001] The present invention relates to the technical field of optimized scheduling of integrated energy systems, and in particular to an optimized scheduling method and system for a multi-region integrated energy system. Background Art

[0002] The growth of global energy demand makes it difficult for traditional single-energy supply models to meet the needs of modern society. Integrated energy systems, by integrating multiple energy sources such as electricity and natural gas, can complement each other's strengths and overcome the limitations of single-energy technologies. The rational scheduling of multi-regional integrated energy systems is also a key factor affecting the stability of power system operations.

[0003] In existing research on the optimal scheduling of multi-regional integrated energy systems, centralized scheduling methods can achieve global optimization through a central server. However, once sensitive data such as energy production, energy consumption, and energy trading in each region's integrated energy system is centrally stored, it is easily targeted by external attacks or internal leaks, posing a significant risk of privacy leakage. Furthermore, centralized scheduling relies heavily on communication networks and central servers. Failures in the central server or communication links can affect the scheduling efficiency of the entire system. The increased computational burden on the central server can lead to scheduling delays, making it difficult to meet real-time requirements. Although the introduction of distributed federated reinforcement learning can address the data privacy leakage risk in centralized scheduling and avoid the scheduling inefficiency caused by excessive reliance on central servers, scheduling optimization that allows each region's integrated energy system to be managed by an independent agent requires each agent to converge quickly during local training to ensure efficient energy scheduling. Furthermore, reliable model parameter interaction between agents is required to ensure global scheduling optimization. This model parameter interaction between agents carries a significant risk of privacy leakage, and the convergence performance of each agent is also highly susceptible to network delays and parameter update rules. Therefore, there is an urgent need to provide an efficient optimization scheduling method that can avoid the risk of centralized storage of regional integrated energy system data and improve the real-time performance and reliability of multi-regional integrated energy system scheduling. Summary of the Invention

[0004] The purpose of the present invention is to provide an optimization and scheduling method for a multi-regional integrated energy system. By introducing a double random matrix constructed based on the topological relationship between integrated energy systems on a distributed federated learning framework to weightedly aggregate the model parameters of each local intelligent agent with the model parameters of the corresponding neighboring intelligent agent to collaboratively update the local optimization model of each local intelligent agent, the method can not only effectively avoid the privacy leakage risk of the integrated energy system data of each region, but also save network communication overhead, improve the real-time performance of the optimization and scheduling of the multi-regional integrated energy system, and support local intelligent agents to take into account individual learning characteristics and collective knowledge sharing, thereby effectively improving the reliability of the optimization and scheduling of the multi-regional integrated energy system.

[0005] In order to achieve the above objectives, a method and system for optimizing and scheduling a multi-region integrated energy system are provided.

[0006] In a first aspect, an embodiment of the present invention provides a method for optimizing and scheduling a multi-region integrated energy system, wherein the integrated energy system of each region is managed by a corresponding local agent, and the method comprises the following steps:

[0007] Each local agent trains a preset energy optimization scheduling model based on reinforcement learning to obtain a corresponding local optimization model. The local optimization model is then updated based on the encrypted parameter data packets received from each neighboring agent and a pre-constructed double-stochastic matrix. The double-stochastic matrix is ​​constructed based on the topological relationship between the integrated energy systems in each region. The encrypted parameter data packets include the agent ID, encrypted model parameters, and an encrypted timestamp.

[0008] In response to the completion of the update of the local optimization model of each local agent, an overall convergence evaluation is performed on the local optimization models of all local agents. When the corresponding overall convergence evaluation result reaches the preset optimization target, the collaborative training between local agents is stopped, and the integrated energy system optimization scheduling of the corresponding area is executed according to the local optimization model of each local agent.

[0009] Furthermore, the state space and action space of the preset energy optimization scheduling model are respectively defined according to the operating state and scheduling strategy of the integrated energy system; the operating state includes electric load demand, thermal load demand, cooling load demand, photovoltaic power generation, energy storage unit status, energy scheduling interval and future electricity price; the scheduling strategy includes the electric output power of the trigeneration unit, the output power of the photovoltaic generator and the energy storage charging and discharging power; the reward function of the preset energy optimization scheduling model is constructed based on the operating cost of the integrated energy system scheduling and the penalty for the intelligent agent violating the constraints.

[0010] Furthermore, each local agent trains a preset energy optimization scheduling model based on reinforcement learning to obtain a corresponding local optimization model, including the following steps:

[0011] Each local intelligent agent obtains historical data of the integrated energy system of the corresponding area, and preprocesses the historical data of the integrated energy system to obtain a corresponding energy system optimization scheduling data set;

[0012] According to the integrated energy system optimization scheduling data set, the preset energy optimization scheduling model is iteratively trained based on the double-delay deterministic policy gradient algorithm to obtain the local optimization model;

[0013] The model parameters of the local optimization model are encrypted based on a preset encryption algorithm to generate a corresponding encrypted parameter data packet, and the encrypted parameter data packet is sent to each neighboring intelligent agent of the local intelligent agent.

[0014] Furthermore, the step of encrypting the model parameters of the local optimization model based on a preset encryption algorithm to generate a corresponding encrypted parameter data packet includes:

[0015] Arrange the model parameters in column vector form according to the preset parameter sorting rules to obtain the corresponding parameter matrix;

[0016] Using a homomorphic encryption algorithm to perform encryption operation on the parameter matrix to obtain encrypted model parameters;

[0017] The encryption model parameters are combined with the corresponding agent ID and encryption timestamp to generate a data packet to be interacted with, and the data packet to be interacted with is encrypted based on an asymmetric encryption algorithm to generate the encrypted parameter data packet.

[0018] Furthermore, the steps of constructing the double random matrix include:

[0019] Generate an agent adjacency matrix based on the topological relationship between the integrated energy systems of each region;

[0020] Based on the agent adjacency matrix, a neighboring agent set of each local agent is obtained;

[0021] Based on the double random matrix generation principle, the double random matrix is ​​generated according to the neighboring agent set of all local agents; the double random matrix is ​​expressed as:

[0022]

[0023] Where, Indicates the first Rank The element value of the column; and Represents the local agent and local agents The number of neighboring agents; Represents the local agent The set of neighboring agents; Represents the total number of local agents.

[0024] Furthermore, the step of updating the local optimization model according to the received encrypted parameter data packets transmitted by each neighboring agent and the pre-built double random matrix includes:

[0025] Obtain the receiving timestamp of the encrypted parameter data packet of each neighboring agent respectively, and decrypt each encrypted parameter data packet to obtain the corresponding agent ID, encrypted model parameters and encryption timestamp;

[0026] Obtaining a neighbor communication delay between each neighboring agent and the local agent based on a receiving timestamp and an encryption timestamp of an encrypted parameter data packet corresponding to each agent ID;

[0027] Based on the neighbor communication delay between each neighbor agent and the local agent, the weight coefficient of the corresponding position in the double random matrix is ​​corrected to obtain a corresponding local corrected double random matrix;

[0028] According to the local modified double random matrix, the encrypted model parameters of each neighboring intelligent body are weightedly aggregated with the local model parameters of the local optimization model to obtain updated local model parameters, and based on the updated local model parameters, an updated local optimization model is obtained.

[0029] Furthermore, the step of correcting the weight coefficients of corresponding positions in the double random matrix based on the neighbor communication delay between each neighbor agent and the local agent to obtain the corresponding local corrected double random matrix includes:

[0030] Obtaining the corresponding average value of neighbor interaction delay according to the communication delay of all neighbors corresponding to the local agent;

[0031] Calculate the delay deviation rate of each neighbor communication delay and the average value of the neighbor interaction delay respectively;

[0032] When the delay deviation rate is positive, the corresponding weight coefficient reduction value is calculated according to the delay deviation rate and the weight coefficient of the corresponding position of the corresponding neighboring agent in the double random matrix, and the weight coefficient of the corresponding position in the double random matrix is ​​adjusted downward and updated according to the weight coefficient reduction value;

[0033] According to the reduction values ​​of all weight coefficients, the total weight increase target of all neighboring intelligent agents with negative delay deviation rates is obtained, and according to the total weight increase target, the weight coefficients of the corresponding positions of all neighboring intelligent agents with negative delay deviation rates in the double random matrix are adjusted and updated.

[0034] Furthermore, the updated local model parameters are expressed as:

[0035]

[0036] Where, and Respectively represent Local Agents in Round-by-Round Collaborative Training The local model parameters and the corresponding neighbor agents Encrypted model parameters; Indicates the Local agent after round of collaborative training Updated local model parameters; represents the weight coefficient of the local model parameters in the local modified double random matrix; Represents the neighboring agents in the local modified doubly stochastic matrix The weight coefficients of the encryption model parameters; Represents the local agent The set of neighboring agents.

[0037] Furthermore, the step of performing overall convergence evaluation on the local optimization models of all local agents includes:

[0038] Obtain the test accuracy standard deviation and the average parameter update amplitude of each local optimization model, and obtain the corresponding model convergence evaluation index based on the test accuracy standard deviation and the average parameter update amplitude; the model convergence evaluation index is expressed as:

[0039]

[0040] Where, Indicates the Model convergence evaluation indicators of the local optimization model obtained by round-by-round collaborative training; Before The standard deviation of the test accuracy of the local optimized model obtained by the round of collaborative training; Before The average parameter update amplitude of the local optimization model obtained by round-by-round collaborative training; and Indicates the standard deviation of target test accuracy and the target parameter update amplitude; represents the weight coefficient;

[0041] Determine whether the model convergence evaluation indicators of all local optimization models reach the preset model convergence threshold. If so, the corresponding model convergence evaluation result is obtained as converged. Otherwise, the corresponding model convergence evaluation result is determined to be unconverged.

[0042] The overall convergence evaluation result is obtained according to the model convergence evaluation results of all local optimization models.

[0043] In a second aspect, an embodiment of the present invention provides an optimization scheduling system for a multi-region integrated energy system, wherein the integrated energy system of each region is managed by a corresponding local intelligent agent, and the system includes:

[0044] An agent training module is used to train each local agent on a preset energy optimization scheduling model based on reinforcement learning to obtain a corresponding local optimization model. The local optimization model is then updated based on encrypted parameter data packets received from neighboring agents and a pre-constructed double-stochastic matrix. The double-stochastic matrix is ​​constructed based on the topological relationship between the integrated energy systems in each region. The encrypted parameter data packet includes the agent ID, encrypted model parameters, and an encrypted timestamp.

[0045] The energy optimization scheduling module is used to respond to the completion of the local optimization model update of each local intelligent agent, perform an overall convergence evaluation on the local optimization models of all local intelligent agents, and stop the collaborative training between local intelligent agents when the corresponding overall convergence evaluation result reaches the preset optimization target, and perform the integrated energy system optimization scheduling of the corresponding area according to the local optimization model of each local intelligent agent.

[0046] The above-mentioned present invention provides an optimization and scheduling method and system for a multi-region integrated energy system. Through the method, the integrated energy system of each region is managed by the corresponding local intelligent agent. Each local intelligent agent trains the preset energy optimization scheduling model based on the reinforcement learning idea to obtain the corresponding local optimization model. The local optimization model is updated according to the encrypted parameter data packet including the intelligent agent ID, encrypted model parameters and encrypted timestamp transmitted by each neighboring intelligent agent and the double random matrix pre-constructed based on the topological relationship between the integrated energy systems of each region. After the local optimization models of all local intelligent agents are updated, the local optimization models of all local intelligent agents are evaluated for overall convergence. When the corresponding overall convergence evaluation result reaches the preset optimization target, the collaborative training between local intelligent agents is stopped, and the technical solution for optimizing and scheduling the integrated energy system of the corresponding region is executed according to the local optimization model of each local intelligent agent. Compared with the existing technology, the optimization and scheduling method of the multi-regional integrated energy system introduces an intelligent agent interaction and coordination training mechanism on the distributed federated learning framework, which uses a double random matrix constructed based on the topological relationship between the integrated energy systems to weightedly aggregate the model parameters of each local intelligent agent with the model parameters of the corresponding neighboring intelligent agents to collaboratively update the local optimization model of each local intelligent agent. It can not only effectively avoid the privacy leakage risk of the integrated energy system data in each region, but also save network communication overhead, improve the real-time performance of the optimization and scheduling of the multi-regional integrated energy system, but also support local intelligent agents to take into account both individual learning characteristics and collective knowledge sharing, effectively improving the reliability of the optimization and scheduling of the multi-regional integrated energy system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 1 is a flow chart of an optimization scheduling method for a multi-region integrated energy system according to an embodiment of the present invention;

[0048] Figure 2 It is a structural diagram of the optimization scheduling system of the multi-region integrated energy system in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and beneficial effects of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described below are part of the embodiments of the present invention and are only used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0050] The optimization scheduling method for a multi-regional integrated energy system provided by the present invention can be understood as an application status quo in which the optimization scheduling of existing multi-regional integrated energy systems has a large risk of privacy leakage and cannot effectively guarantee the real-time and reliability of the optimization scheduling of multi-regional integrated energy systems. Instead, a method is proposed that introduces a double random matrix constructed based on the topological relationship between integrated energy systems to perform weighted aggregation on the model parameters of each local intelligent agent and the model parameters of the corresponding neighboring intelligent agent in order to collaboratively update the local optimization model of each local intelligent agent, thereby constructing an optimization scheduling method for a multi-regional integrated energy system of local intelligent agents for scheduling and managing each integrated energy system. The following embodiments will explain the optimization scheduling method for a multi-regional integrated energy system of the present invention in detail.

[0051] In one embodiment, Figure 1 As shown, a method for optimizing and scheduling a multi-regional integrated energy system is provided. The integrated energy system of each region is managed by a corresponding local agent. The local agent trains a local optimization model in a distributed environment. Each local agent can independently learn and make decisions based on local data and experience. The local agent can share key information with other local agents without sending the trained local optimization model to a central server. That is, energy scheduling within the region can be managed efficiently and reliably without a central server. Specifically, the method for optimizing and scheduling a multi-regional integrated energy system includes the following steps:

[0052] S11. Each local agent trains the preset energy optimization scheduling model based on the reinforcement learning idea to obtain the corresponding local optimization model, and updates the local optimization model according to the encrypted parameter data packet received from each neighboring agent and the pre-constructed double random matrix; the double random matrix is ​​constructed based on the topological relationship between the integrated energy systems of each region; the encrypted parameter data packet includes the agent ID, encrypted model parameters and encrypted timestamp, and the agent ID is a unique identifier to distinguish each local agent, the encrypted model parameters are the encrypted values ​​of the model parameters of the local optimization model corresponding to the agent ID, and the encrypted timestamp is the moment when the agent ID encrypts the model parameters of its local optimization model.

[0053] The preset energy optimization scheduling model in this embodiment can be understood as a quadratic constraint programming problem model for regional integrated energy system scheduling, which is constructed based on the scheduling management needs of the regional integrated energy system, based on nonlinear optimization theory combined with reinforcement learning ideas, with the real-time operation data of the regional integrated energy system as input and the optimal scheduling strategy of the regional integrated energy system as output. Specifically, the state space and action space of the preset energy optimization scheduling model are respectively defined according to the operating state and scheduling strategy of the integrated energy system. The operating state includes electric load demand, thermal load demand, cooling load demand, photovoltaic power generation, energy storage unit state, energy scheduling interval and future electricity price, etc., among which the future electricity price is also obtained based on the existing relevant electricity price prediction technology, and is not specifically limited here. In order to provide the local intelligent agent with an understanding of the constraints of the operating unit, the various states included in the operating state need to be adjusted to the same scale in actual applications, that is, scaled to the [0,1] interval by normalization; the corresponding state space is expressed as:

[0054]

[0055] Where, 、 、 、 and Respectively Moment The electrical load demand, heating load demand, cooling load demand, photovoltaic power generation and energy storage unit status of each regional integrated energy system; and They represent the energy dispatch interval and the electricity price at future times respectively.

[0056] The scheduling strategy of the preset energy optimization scheduling model includes the output power of the trigeneration unit, the output power of the photovoltaic generator, and the charging and discharging power of the energy storage, so the corresponding action space can be expressed as:

[0057]

[0058] Where, express Moment The dispatch strategy of a regional integrated energy system; 、 and Respectively Moment The output power of the combined heating, cooling and power (CCHP) units, the output power of the photovoltaic generators and the energy storage charging and discharging power of the regional integrated energy system.

[0059] In order to facilitate the evaluation of the rationality of the energy scheduling strategy generation, this embodiment preferably constructs a reward function of the preset energy optimization scheduling model based on the operating cost of the integrated energy system scheduling and the penalty for the agent violating the constraint, which is expressed as:

[0060]

[0061] Where,

[0062]

[0063]

[0064] in, and Respectively Moment The negative value of the operating cost of the regional integrated energy system and the penalty for the agent violating the constraint; is the penalty weight, which can be set according to actual application requirements; Represents the set of constraints violated by the agent. The specific constraint variables in the set can be set according to actual application requirements and are not specifically limited here; express Moment The value of the constraint variable; and Respectively represent The upper and lower limits of the constraint variables; 、 、 and Respectively Moment The cost of power generation by generators in a regional integrated energy system, the cost of electricity purchased from the main power grid and other integrated energy systems, the cost of natural gas purchased from the natural gas grid, and the equipment losses of the units.

[0065] Specifically, The cost of photovoltaic generators in each region It can be expressed as:

[0066]

[0067] in, 、 and is the cost coefficient of the power generation unit, yes Moment The output power of photovoltaic generators in each area.

[0068] No. The cost of electricity purchased from the main grid and other integrated energy systems in each region It can be expressed as:

[0069]

[0070] Where, express The main power grid transaction price at that moment; yes Moment The power traded between the regional integrated energy system and the main grid; yes Moment and Internal transaction electricity price of the integrated energy system IES for regional integrated energy system transactions; yes Moment and The trading power of the integrated energy system IES traded in a regional integrated energy system.

[0071] The natural gas purchase cost of the i-th region to meet the demand of the combined cooling, heating and power unit It can be expressed as:

[0072]

[0073] Where, represents the price of natural gas purchased from the gas network, yes Moment The input power of the CCHP unit of a regional integrated energy system.

[0074] No. Equipment loss cost per area It can be expressed as:

[0075]

[0076] in, is the cost per unit of capacity lost; It is a device Power loss coefficient; yes Moment Equipment for regional integrated energy systems output power.

[0077] It should be noted that the scheduling constraints required by the preset energy optimization scheduling model constructed through the above method steps can be determined based on the actual operating requirements of the integrated energy system combined with the variables involved in the corresponding state space and action space, and are not specifically limited here.

[0078] After initiating the multi-region integrated energy system optimization and scheduling instruction, the preset energy optimization and scheduling model can be distributed to the integrated energy systems in each region based on a distributed federated learning architecture for training the corresponding local intelligent agents. Specifically, each local intelligent agent trains the preset energy optimization and scheduling model based on reinforcement learning to obtain the corresponding local optimization model. The steps include:

[0079] Each local intelligent agent obtains the historical data of the integrated energy system of the corresponding area, and preprocesses the historical data of the integrated energy system to obtain the corresponding energy system optimization scheduling data set; wherein, the historical data of the integrated energy system can be understood as the experience data of a certain period of time collected through the interaction between the intelligent agent and the integrated energy system scheduling environment, including operating status time series data and corresponding scheduling strategy time series data; the corresponding preprocessing process can include outlier removal, missing value filling and normalization processing in sequence, and the corresponding energy system optimization scheduling data set obtained by preprocessing the historical data of the integrated energy system can be stored in the experience replay buffer to provide reliable training data for network updates in subsequent intelligent agent training.

[0080] According to the integrated energy system optimization scheduling dataset, the preset energy optimization scheduling model is iteratively trained based on the dual-delay deterministic policy gradient algorithm to obtain the local optimization model; wherein, the dual-delay deep deterministic policy gradient algorithm (TD3, Twin Delayed Deep Deterministic Policy Gradient) is a reinforcement learning algorithm for continuous action space problems. It improves the DDPG algorithm through innovations such as dual critic networks, delayed update actors and target policy smoothing; this embodiment uses this algorithm for local agent training of integrated energy systems in each region, which can not only make full use of the historical energy data of each region, but also effectively improve the stability and reliability of local optimization model strategy generation; it should be noted that the specific process of each local agent iteratively training the preset energy optimization scheduling model based on the dual-delay deterministic policy gradient algorithm can be referred to the relevant existing technology implementation and will not be described in detail here.

[0081] The model parameters of the local optimization model are encrypted based on a preset encryption algorithm, corresponding encrypted parameter data packets are generated, and the encrypted parameter data packets are sent to each neighbor agent of the local agent; wherein the model parameters can be understood as parameters of a key agent network model shared between agents determined based on actual application requirements, which can include a weight matrix and a bias vector in the agent network.

[0082] In actual applications, after each local agent completes a round of training to obtain a corresponding local optimization model, it needs to interact with the neighbor nodes directly connected thereto to share agent information in a small range, so as to improve the convergence rate and training effect of the model; in order to avoid the privacy leakage risk existing in the interaction of model parameters between local agents, the model parameters to be shared and some key information facilitating subsequent aggregation and use of the model parameters are preferably packaged and encrypted and transmitted to the corresponding neighbor agents; specifically, the step of encrypting the model parameters of the local optimization model based on a preset encryption algorithm to generate corresponding encrypted parameter data packets comprises:

[0083] The model parameters are arranged in the form of column vectors according to a preset parameter sorting rule, to obtain a corresponding parameter matrix; wherein the parameter matrix can be understood as a column vector of h dimensions, h is the total number of all model parameters, and the corresponding preset parameter sorting rule can be understood as a model parameter arrangement order designed in advance for uniform use by each local agent to facilitate the sharing and use of model parameters between adjacent local agents, which is not specifically limited here.

[0084] The parameter matrix is subjected to an encryption operation by using a homomorphic encryption algorithm, to obtain encrypted model parameters; wherein the encrypted model parameters can be understood as an encrypted parameter matrix obtained by respectively encrypting each element in the parameter matrix by using the homomorphic encryption algorithm.

[0085] The encrypted model parameters, the corresponding agent ID and the encryption timestamp are combined to generate a to-be-interacted data packet, the to-be-interacted data packet is subjected to an encryption operation by using an asymmetric encryption algorithm, and the encrypted parameter data packet is generated; wherein the arrangement order of the encrypted model parameters, the agent ID and the encryption timestamp in the to-be-interacted data packet can be set based on a preset inter-agent communication protocol.

[0086] ​The double random matrix in this embodiment can be understood as a reference matrix used to determine the weight coefficients of all model parameters when each local agent uses the encrypted model parameters of its neighboring agents to update the local optimization model. In order to avoid unnecessary waste of communication resources and effectively reduce network load while ensuring that each local agent can indirectly obtain global information, this embodiment preferably constructs the double random matrix based on the neighboring relationship between each agent. Specifically, the steps of constructing the double random matrix include:

[0087] Based on the topological relationship between the integrated energy systems of each region, an agent adjacency matrix is ​​generated; the agent adjacency matrix can be understood as a matrix representing the direct connection relationship between each local agent. For example, if local agent a and local agent b have a direct connection and interaction relationship, the matrix elements corresponding to the a-th row and b-th column and the b-th row and a-th column in the agent adjacency matrix are both 1. Conversely, if there is no direct connection and interaction relationship between local agent a and local agent b, the matrix elements corresponding to the a-th row and b-th column and the b-th row and a-th column in the agent adjacency matrix are both 0.

[0088] Based on the agent adjacency matrix, the neighbor agent set of each local agent is obtained respectively; wherein, the neighbor agent set of a local agent can be understood as the set of other agents corresponding to the elements of 1 in the row vector corresponding to the sequence number of the local agent in the agent adjacency matrix.

[0089] Based on the double random matrix generation principle, the double random matrix is ​​generated according to the neighbor agent set of all local agents. The double random matrix generation principle can be understood as the sum of the elements of the row vector and column vector of the matrix is ​​1. In order to ensure the validity of the weight coefficient used for updating the model parameters of each local agent, this embodiment preferably determines the weight coefficient of the encrypted model parameter of the neighbor agent when updating the local agent based on the number of agents in the neighbor agent set of the local agent and the number of agents in the neighbor agent set corresponding to the neighbor agent of the local agent. Specifically, the double random matrix is ​​expressed as:

[0090]

[0091] Where, Indicates the first Rank The element value of the column; and Represents the local agent and local agents The number of neighboring agents; Represents the local agent The set of neighboring agents; Represents the total number of local agents.

[0092] In this embodiment, a double random matrix is ​​used to determine the encryption model parameters of the neighboring agents for updating the weight coefficients when the local agent is updated. This can achieve purposeful parameter synchronization based on the relative importance and dependency between the agents to adapt to the dynamically changing state of the agents. At the same time, it can also support each local agent to take into account both individual learning characteristics and collective knowledge sharing to optimize its decision-making process, thereby ensuring the convergence effect of the local optimization model while also ensuring the generalization ability and robustness of the local optimization model.

[0093] Specifically, the step of updating the local optimization model according to the received encrypted parameter data packets transmitted by each neighboring agent and the pre-built double random matrix includes:

[0094] Obtain the receiving timestamp of the encrypted parameter data packet of each neighboring intelligent agent respectively, and decrypt each encrypted parameter data packet to obtain the corresponding intelligent agent ID, encryption model parameters and encryption timestamp; among them, decryption of each encrypted parameter data packet can be implemented based on a preset asymmetric encryption algorithm, which will not be repeated here.

[0095] Based on the reception timestamp and encryption timestamp of the encrypted parameter data packet corresponding to each agent ID, the neighbor communication delay between each neighbor agent and the local agent is obtained; that is, the neighbor communication delay between each neighbor agent and the local agent is the difference between the reception timestamp and encryption timestamp of the corresponding encrypted parameter data packet.

[0096] Based on the neighbor communication delay between each neighbor intelligent agent and the local intelligent agent, the weight coefficient of the corresponding position in the double random matrix is ​​corrected to obtain the corresponding local corrected double random matrix; wherein, the local corrected double random matrix can be understood as further correcting the corresponding weight coefficient in the double random matrix obtained by the above method based on the method that the communication delay between intelligent agents affects the convergence efficiency of the local model, taking into account the relative importance and dependency relationship between intelligent agents.

[0097] Specifically, the step of correcting the weight coefficients of corresponding positions in the double random matrix based on the neighbor communication delay between each neighbor agent and the local agent to obtain the corresponding local corrected double random matrix includes:

[0098] According to the communication delays of all neighbors corresponding to the local agent, the corresponding average value of the neighbor interaction delay is obtained; that is, the average value of the neighbor interaction delay of the local agent is the average value of the communication delays of all its neighbors.

[0099] Calculate the delay deviation rate of each neighbor communication delay and the average neighbor interaction delay. The delay deviation rate can be understood as the ratio of the difference between the neighbor communication delay and the average neighbor interaction delay to the average neighbor interaction delay. If the neighbor communication delay is greater than the average neighbor interaction delay, the corresponding delay deviation rate is positive; otherwise, the corresponding delay deviation rate is negative.

[0100] When the delay deviation rate is positive, the corresponding weight coefficient reduction value is calculated based on the delay deviation rate and the weight coefficient of the corresponding neighboring agent at the corresponding position in the dual-stochastic matrix. The weight coefficient of the corresponding position in the dual-stochastic matrix is ​​then adjusted and updated based on the weight coefficient reduction value. The weight coefficient reduction value of the neighboring agent can be understood as the product of the delay deviation rate corresponding to the neighboring agent and the weight coefficient of its corresponding position in the dual-stochastic matrix. After obtaining the weight coefficient reduction value of the neighboring agent, the difference between the weight coefficient of the corresponding position in the existing dual-stochastic matrix and the weight coefficient reduction value is used as the updated weight coefficient value of the corresponding position in the dual-stochastic matrix. This method can reduce the weight coefficient of the local agent model whose convergence efficiency is affected by large communication delays, thereby reducing its contribution to the update of the local optimization model.

[0101] According to the reduction values ​​of all weight coefficients, the total weight increase target of all neighboring agents with negative delay deviation rates is obtained, and the weight coefficients of the corresponding positions of all neighboring agents with negative delay deviation rates in the double random matrix are adjusted and updated according to the total weight increase target; wherein, the total weight increase target can be understood as the cumulative value of the reduction amount of the weight coefficients of all neighboring agents with positive delay deviation rates; in order to reasonably allocate the weight coefficients used for updating the local optimization model on the basis of maintaining the characteristics of the double random matrix, so as to improve the convergence rate of each local agent model, this embodiment preferably allocates the obtained total weight increase target based on the deviation rate proportion of all neighboring agents with negative delay deviation rates. First, the ratio of the deviation rate of all neighboring agents with negative delay deviation rates to the sum of all negative delay deviation rates is calculated. Then, the weight coefficient increase for each neighboring agent is calculated by multiplying the ratio of each neighboring agent with the total weight increase target. After the weight coefficient increase is calculated, the weight coefficient for the corresponding position in the updated double stochastic matrix is ​​calculated by adding the sum of the weight coefficient increase and the existing double stochastic matrix. This method can improve the contribution of neighboring agents with low communication delays to the update of the local optimization model.

[0102] According to the local modified double random matrix, the encrypted model parameters of each neighboring intelligent body are weightedly aggregated with the local model parameters of the local optimization model to obtain updated local model parameters, and based on the updated local model parameters, an updated local optimization model is obtained.

[0103] Specifically, the updated local model parameters are expressed as:

[0104]

[0105] Where, and Respectively represent Local Agents in Round-by-Round Collaborative Training The local model parameters and the corresponding neighbor agents Encrypted model parameters; Indicates the Local agent after round of collaborative training Updated local model parameters; represents the weight coefficient of the local model parameters in the local modified double random matrix; Represents the neighboring agents in the local modified doubly stochastic matrix The weight coefficients of the encryption model parameters; Represents the local agent The set of neighboring agents.

[0106] In this embodiment, on the basis of considering the relative importance and dependency between intelligent agents, and further considering the impact of communication delay between intelligent agents on the convergence efficiency of the local model, the double random matrix is ​​corrected based on the communication delay of the neighboring intelligent agents and then used to update the local model parameters. This can ensure that the convergence effect, generalization ability and robustness of the local optimization model are improved, while also ensuring that the update of the local optimization model can adapt to the dynamically changing intelligent agent status and network conditions at the same time, thereby effectively improving the training efficiency of distributed federated learning, and providing reliable guarantees for the real-time and reliability of the optimization scheduling of multi-regional integrated energy systems.

[0107] S12. In response to the completion of the update of the local optimization model of each local intelligent agent, an overall convergence evaluation is performed on the local optimization models of all local intelligent agents. When the corresponding overall convergence evaluation result reaches the preset optimization target, the collaborative training between the local intelligent agents is stopped, and the integrated energy system optimization scheduling of the corresponding area is executed according to the local optimization model of each local intelligent agent.

[0108] In this embodiment, the overall convergence evaluation can be understood as a comprehensive evaluation of the iterative training convergence performance of the local optimization model of each local agent to determine whether to continue the subsequent inter-agent collaborative training process. In order to determine the effectiveness of the model convergence performance evaluation, this embodiment preferably performs a comprehensive analysis based on the two dimensions of test accuracy and parameter update amplitude. Specifically, the step of performing the overall convergence evaluation of the local optimization models of all local agents includes:

[0109] Obtain the test accuracy standard deviation and the average parameter update amplitude of each local optimization model, and obtain the corresponding model convergence evaluation index based on the test accuracy standard deviation and the average parameter update amplitude; the model convergence evaluation index is expressed as:

[0110]

[0111] Where, Indicates the Model convergence evaluation indicators of the local optimization model obtained by round-by-round collaborative training; Before The standard deviation of the test accuracy of the local optimized model obtained by the round of collaborative training; Before The average parameter update amplitude of the local optimization model obtained by round-by-round collaborative training; and Indicates the target test accuracy standard deviation and target parameter update amplitude, which can be set according to actual application requirements and is a common indicator value for all local agents; Represents the weight coefficient, which can be set as a fixed constant according to actual needs, or dynamically set based on preset dynamic adjustment rules according to actual application needs; It should be noted that when When , the test accuracy standard deviation and the parameter update amplitude average value of each local optimization model can be calculated using the preset initial test accuracy standard deviation and the initial parameter update amplitude average value, respectively. The initial test accuracy standard deviation is smaller than the target test accuracy standard deviation, and the initial parameter update amplitude average value is smaller than the target parameter update amplitude.

[0112] It is determined whether the model convergence evaluation indicators of all local optimization models reach the preset model convergence threshold. If so, the corresponding model convergence evaluation result is obtained as converged. Otherwise, the corresponding model convergence evaluation result is determined as non-converged. The preset model convergence threshold can also be set according to actual application requirements and is not specifically limited here.

[0113] The overall convergence evaluation result is obtained according to the model convergence evaluation results of all local optimization models.

[0114] The preset optimization target in this embodiment is understood as the iterative training stopping condition, which can be set according to actual application needs, such as all local optimization models converge or the corresponding model convergence evaluation results show that the proportion of converged models reaches a preset proportion; when the overall convergence evaluation result reaches the preset optimization target, the collaborative training between local intelligent agents will no longer be performed, and it is considered that the local intelligent agents corresponding to the current regional integrated energy systems have been trained and can be used to perform the optimized scheduling of the integrated energy systems in the corresponding regions; the corresponding local optimization model can be input based on the real-time operation data of the integrated energy systems in each region to obtain the corresponding optimal scheduling strategy.

[0115] The integrated energy system of each region provided by the embodiment of the present invention is managed by the corresponding local intelligent agent, and each local intelligent agent respectively trains the preset energy optimization scheduling model based on the reinforcement learning idea to obtain the corresponding local optimization model, and updates the local optimization model according to the encrypted parameter data packet including the intelligent agent ID, encrypted model parameters and encrypted timestamp transmitted by each neighboring intelligent agent and the double random matrix pre-constructed based on the topological relationship between the integrated energy systems of each region, and after the local optimization models of all local intelligent agents are updated, the local optimization models of all local intelligent agents are evaluated for overall convergence, and when the corresponding overall convergence evaluation result reaches the preset optimization target, the collaborative training between the local intelligent agents is stopped, and the local optimization models of each local intelligent agent are executed according to the local optimization models of each local intelligent agent. A technical solution for optimizing the scheduling of integrated energy systems in corresponding regions is proposed. By introducing a double random matrix constructed based on the topological relationship between integrated energy systems and the network communication delay between agents on the distributed federated learning framework, the model parameters of each local agent and the model parameters of the corresponding neighboring agents are weightedly aggregated to collaboratively update the local optimization model of each local agent. This solution can not only effectively avoid the privacy leakage risk of integrated energy system data in each region, but also save network communication overhead, improve the convergence efficiency of each agent model, ensure the real-time performance of optimized scheduling of multi-regional integrated energy systems, but also support local agents to take into account both individual learning characteristics and collective knowledge sharing, effectively improving the reliability and stability of optimized scheduling of multi-regional integrated energy systems.

[0116] It should be noted that although the steps in the above flowchart are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0117] In one embodiment, Figure 2As shown, an optimization scheduling system for a multi-region integrated energy system is provided, wherein the integrated energy system of each region is managed by a corresponding local intelligent agent, and the system includes:

[0118] Agent training module 1 is used for each local agent to train the preset energy optimization scheduling model based on reinforcement learning to obtain the corresponding local optimization model, and update the local optimization model based on the encrypted parameter data packets transmitted by each neighboring agent and the pre-constructed double random matrix; the double random matrix is ​​constructed based on the topological relationship between the integrated energy systems in each region; the encrypted parameter data packet includes the agent ID, encrypted model parameters and encrypted timestamp;

[0119] Energy optimization scheduling module 2 is used to respond to the completion of the update of the local optimization model of each local intelligent agent, perform an overall convergence evaluation on the local optimization models of all local intelligent agents, and when the corresponding overall convergence evaluation result reaches the preset optimization target, stop the collaborative training between local intelligent agents, and perform the integrated energy system optimization scheduling of the corresponding area according to the local optimization model of each local intelligent agent.

[0120] For the specific definition of the optimization scheduling system of the multi-region integrated energy system, please refer to the definition of the optimization scheduling method of the multi-region integrated energy system above. The corresponding technical effects can also be obtained equivalently, so they will not be repeated here. Each module in the above-mentioned optimization scheduling system of the multi-region integrated energy system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0121] In summary, an embodiment of the present invention provides an optimization scheduling method and system for a multi-region integrated energy system, wherein the optimization scheduling method for the multi-region integrated energy system realizes that the integrated energy systems of each region are managed by corresponding local intelligent agents, and each local intelligent agent respectively trains a preset energy optimization scheduling model based on the reinforcement learning idea to obtain a corresponding local optimization model, and updates the local optimization model according to the encrypted parameter data packet including the intelligent agent ID, encrypted model parameters and encrypted timestamp transmitted by each neighbor intelligent agent and the double random matrix pre-constructed based on the topological relationship between the integrated energy systems of each region, and after the local optimization models of all local intelligent agents are updated, the local optimization models of all local intelligent agents are evaluated for overall convergence, and when the corresponding overall convergence evaluation result reaches the preset optimization target, the local optimization model between the local intelligent agents is stopped. Collaborative training is a technical solution for executing the optimal scheduling of the integrated energy system in the corresponding region based on the local optimization model of each local intelligent agent. This method introduces an intelligent agent interactive coordination training mechanism on the distributed federated learning framework, which uses a double random matrix constructed based on the topological relationship between the integrated energy systems and the network communication delay between intelligent agents to weightedly aggregate the model parameters of each local intelligent agent with the model parameters of the corresponding neighboring intelligent agent to collaboratively update the local optimization model of each local intelligent agent. This can not only effectively avoid the privacy leakage risk of the integrated energy system data in each region, but also save network communication overhead, improve the convergence efficiency of each intelligent agent model, ensure the real-time performance of the optimal scheduling of the multi-regional integrated energy system, but also support local intelligent agents to take into account both individual learning characteristics and collective knowledge sharing, effectively improving the reliability and stability of the optimal scheduling of the multi-regional integrated energy system.

[0122] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the various technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above-described embodiments merely represent several preferred implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent. It should be noted that a person skilled in the art can make several improvements and substitutions without departing from the technical principles of the present invention, and such improvements and substitutions should also be considered within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be based on the scope of protection of the claims.

Claims

1. A method for optimizing and dispatching a multi-region integrated energy system, characterized in that: The integrated energy system of each region is managed by a corresponding local intelligent agent. The method includes the following steps: Each local agent trains a preset energy optimization scheduling model based on reinforcement learning to obtain a corresponding local optimization model. The local optimization model is then updated based on the encrypted parameter data packets received from each neighboring agent and a pre-constructed double-stochastic matrix. The double-stochastic matrix is ​​constructed based on the topological relationship between the integrated energy systems in each region. The weight coefficients in the double-stochastic matrix are determined based on the number of neighboring agents of the local agent and the corresponding neighboring agent. The encrypted parameter data packet includes the agent ID, encrypted model parameters, and an encrypted timestamp. In response to the completion of the update of the local optimization model of each local agent, the local optimization models of all local agents are evaluated for overall convergence. When the corresponding overall convergence evaluation results reach the preset optimization target, the collaborative training between the local agents is stopped, and the integrated energy system optimization scheduling of the corresponding area is performed according to the local optimization model of each local agent; The step of updating the local optimization model according to the received encrypted parameter data packets transmitted by each neighboring agent and the pre-built double random matrix includes: Obtain the receiving timestamp of the encrypted parameter data packet of each neighboring agent respectively, and decrypt each encrypted parameter data packet to obtain the corresponding agent ID, encrypted model parameters and encryption timestamp; Obtaining a neighbor communication delay between each neighboring agent and the local agent based on a receiving timestamp and an encryption timestamp of an encrypted parameter data packet corresponding to each agent ID; Based on the neighbor communication delay between each neighbor agent and the local agent, the weight coefficient of the corresponding position in the double random matrix is ​​corrected to obtain a corresponding local corrected double random matrix; According to the local modified double random matrix, the encrypted model parameters of each neighboring intelligent body are weightedly aggregated with the local model parameters of the local optimization model to obtain updated local model parameters, and based on the updated local model parameters, an updated local optimization model is obtained.

2. The optimization scheduling method of the multi-region integrated energy system according to claim 1, characterized in that: The state space and action space of the preset energy optimization scheduling model are respectively defined according to the operating state and scheduling strategy of the integrated energy system; the operating state includes electric load demand, thermal load demand, cooling load demand, photovoltaic power generation, energy storage unit status, energy scheduling interval and future electricity price; the scheduling strategy includes the electric output power of the trigeneration unit, the output power of the photovoltaic generator and the energy storage charging and discharging power; the reward function of the preset energy optimization scheduling model is constructed based on the operating cost of the integrated energy system scheduling and the penalty for the intelligent agent violating the constraints.

3. The optimization scheduling method for a multi-region integrated energy system according to claim 1, characterized in that: The steps of training each local agent on a preset energy optimization scheduling model based on reinforcement learning to obtain a corresponding local optimization model include: Each local intelligent agent obtains historical data of the integrated energy system of the corresponding area, and preprocesses the historical data of the integrated energy system to obtain a corresponding energy system optimization scheduling data set; According to the integrated energy system optimization scheduling data set, the preset energy optimization scheduling model is iteratively trained based on a double-delay deterministic policy gradient algorithm to obtain the local optimization model; The model parameters of the local optimization model are encrypted based on a preset encryption algorithm to generate a corresponding encrypted parameter data packet, and the encrypted parameter data packet is sent to each neighboring intelligent agent of the local intelligent agent.

4. The optimization scheduling method for a multi-region integrated energy system according to claim 3, characterized in that: The step of encrypting the model parameters of the local optimization model based on a preset encryption algorithm to generate a corresponding encrypted parameter data packet includes: Arrange the model parameters in column vector form according to the preset parameter sorting rules to obtain the corresponding parameter matrix; Using a homomorphic encryption algorithm to perform encryption operation on the parameter matrix to obtain encrypted model parameters; The encryption model parameters are combined with the corresponding agent ID and encryption timestamp to generate a data packet to be interacted with, and the data packet to be interacted with is encrypted based on an asymmetric encryption algorithm to generate the encrypted parameter data packet.

5. The optimization scheduling method for a multi-region integrated energy system according to claim 1, characterized in that: The steps of constructing the double random matrix include: Generate an agent adjacency matrix based on the topological relationship between the integrated energy systems of each region; Based on the agent adjacency matrix, a neighboring agent set of each local agent is obtained; Based on the double random matrix generation principle, the double random matrix is ​​generated according to the neighboring agent set of all local agents; the double random matrix is ​​expressed as: Where, Indicates the first Rank The element value of the column; and Represents the local agent and local agents The number of neighboring agents; Represents the local agent The set of neighboring agents; Represents the total number of local agents.

6. The optimization scheduling method for a multi-region integrated energy system according to claim 1, characterized in that: The step of correcting the weight coefficients of corresponding positions in the double random matrix based on the neighbor communication delay between each neighbor agent and the local agent to obtain the corresponding local corrected double random matrix includes: Obtaining the corresponding average value of neighbor interaction delay according to the communication delay of all neighbors corresponding to the local agent; Calculate the delay deviation rate of each neighbor communication delay and the average value of the neighbor interaction delay respectively; When the delay deviation rate is positive, the corresponding weight coefficient reduction value is calculated according to the delay deviation rate and the weight coefficient of the corresponding position of the corresponding neighboring agent in the double random matrix, and the weight coefficient of the corresponding position in the double random matrix is ​​adjusted downward and updated according to the weight coefficient reduction value; According to the reduction values ​​of all weight coefficients, the total weight increase target of all neighboring intelligent agents with negative delay deviation rates is obtained, and according to the total weight increase target, the weight coefficients of the corresponding positions of all neighboring intelligent agents with negative delay deviation rates in the double random matrix are adjusted and updated.

7. The optimization scheduling method for a multi-region integrated energy system according to claim 1, characterized in that: The updated local model parameters are expressed as: Where, and Respectively represent Local Agents in Round-by-Round Collaborative Training The local model parameters and the corresponding neighbor agents Encrypted model parameters; Indicates the Local agent after round of collaborative training Updated local model parameters; represents the weight coefficient of the local model parameters in the local modified double random matrix; Represents the neighboring agents in the local modified doubly stochastic matrix The weight coefficients of the encryption model parameters; Represents the local agent The set of neighboring agents.

8. The optimization scheduling method for a multi-region integrated energy system according to claim 1, characterized in that: The step of performing overall convergence evaluation on the local optimization models of all local agents comprises: Obtain the test accuracy standard deviation and the average parameter update amplitude of each local optimization model, and obtain the corresponding model convergence evaluation index based on the test accuracy standard deviation and the average parameter update amplitude; the model convergence evaluation index is expressed as: Where, Indicates the Model convergence evaluation indicators of the local optimization model obtained by round-by-round collaborative training; Before The standard deviation of the test accuracy of the local optimized model obtained by the round of collaborative training; Before The average parameter update amplitude of the local optimization model obtained by round-by-round collaborative training; and Indicates the standard deviation of target test accuracy and the target parameter update amplitude; represents the weight coefficient; Determine whether the model convergence evaluation indicators of all local optimization models reach the preset model convergence threshold. If so, the corresponding model convergence evaluation result is obtained as converged. Otherwise, the corresponding model convergence evaluation result is determined to be unconverged. The overall convergence evaluation result is obtained according to the model convergence evaluation results of all local optimization models.

9. An optimization scheduling system for a multi-region integrated energy system, characterized in that: The optimization scheduling method for a multi-region integrated energy system according to claim 1 is applied, wherein the integrated energy system of each region is managed by a corresponding local intelligent agent, and the system includes: An agent training module is used to train each local agent on a preset energy optimization scheduling model based on reinforcement learning to obtain a corresponding local optimization model. The local optimization model is then updated based on encrypted parameter data packets received from neighboring agents and a pre-constructed double-stochastic matrix. The double-stochastic matrix is ​​constructed based on the topological relationship between the integrated energy systems in each region. The encrypted parameter data packet includes the agent ID, encrypted model parameters, and an encrypted timestamp. The energy optimization scheduling module is used to respond to the completion of the local optimization model update of each local intelligent agent, perform an overall convergence evaluation on the local optimization models of all local intelligent agents, and stop the collaborative training between local intelligent agents when the corresponding overall convergence evaluation result reaches the preset optimization target, and perform the integrated energy system optimization scheduling of the corresponding area according to the local optimization model of each local intelligent agent.

Images