A federated reinforcement learning multi-garden scheduling method based on priority experience sharing

Abstract

This invention discloses a federated reinforcement learning multi-campus scheduling method based on priority experience sharing. It constructs a simulation environment by collecting operational data from each campus, and controls intra-campus scheduling by local agents and inter-campus energy interaction by a federated layer, thereby achieving multi-campus collaborative optimization without centrally aggregating raw data. The simulation environment is used to train local agents based on a hybrid reward mechanism, outputting local agent network parameters and experience data. The federated layer then generates a globally shared strategy and collaborative experience package based on these parameters and experience data, distributing it to each campus for gradient updates and fine-tuning. This allows each campus to share and reuse effective experiences formed by other campuses under specific operating conditions. The initial scheduling strategy is combined with the real-time load demands of the campuses for collaborative scheduling control and dynamic correction, outputting a final scheduling scheme. This improves the convergence speed and collaborative efficiency of the multi-campus scheduling strategy, reduces system operating costs, and enhances the level of multi-energy complementary utilization.

Description

Technical Field

[0001] This invention belongs to the field of multi-park integrated energy system optimization and scheduling technology, and more specifically, relates to a federated reinforcement learning multi-park scheduling method based on priority experience sharing. Background Technology

[0002] With the acceleration of energy transition and the proposal of "dual carbon" goals, multi-park integrated energy systems are becoming increasingly common in urban energy interconnection scenarios. The coordinated scheduling of various parks, with their complementary electricity, heat, and cooling systems and energy storage configurations, objectively requires cross-park optimization without centrally exposing park data. Therefore, a scheduling framework combining federated learning and multi-agent reinforcement learning has gradually become a research direction. It achieves collaborative optimization with "data not leaving the domain" through "local training in each park and parameter aggregation in the cloud." However, existing federated algorithms mostly employ homogeneous aggregation mechanisms based on parameter averaging or approximation, making it difficult to effectively share and utilize the local experience gained by each park under specific operating conditions in multi-park collaborative scheduling. On the one hand, key strategy experiences gained by parks under extreme loads and renewable energy fluctuations are easily diluted during aggregation, making it difficult to transform them into reusable collaborative knowledge. On the other hand, significant differences in equipment configuration and load characteristics among different parks make simple aggregation difficult to balance personalized experience with global collaboration, leading to repeated trial-and-error learning by other parks, resulting in slow overall convergence and low collaborative efficiency.

[0003] Existing invention patent CN110866641B proposes a two-level optimization scheduling method and system for a multi-energy complementary system considering source-storage-load coordination. The first level aims for economic optimization while constraining user comfort, using a genetic algorithm to obtain optimal load data. This load is then used as input. The second level aims for minimum operating costs, optimizing equipment output and energy storage status based on stochastic dynamic programming. Through two-level iterative cycles, the optimal load curve and system operation scheduling plan are obtained. However, this scheme belongs to a centralized two-level optimization framework and does not involve multi-campus collaborative training and parameter aggregation mechanisms under federated learning. Therefore, it does not solve the problem that existing federated algorithms cannot share and utilize local experience in multi-campus collaborative scheduling. Summary of the Invention

[0004] To address the problem that existing federated algorithms struggle to share and utilize local experience in multi-campus collaborative scheduling, this invention provides a federated reinforcement learning-based multi-campus scheduling method based on priority experience sharing.

[0005] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows:

[0006] This invention provides a federated reinforcement learning multi-campus scheduling method based on priority experience sharing, comprising the following steps: Construct integrated energy systems for each park and combine them into a multi-agent reinforcement learning simulation environment. The integrated energy systems within a park are controlled by local agents, while the interaction between integrated energy systems in different parks is controlled by a federated layer. The simulation environment is used to train local agents based on a hybrid reward mechanism, and the network parameters and local experience data of local agents in each park are output. The federation layer generates a globally shared strategy and collaborative experience package based on the parameters and experience data, and distributes it to local agents for gradient updates and fine-tuning to obtain a preliminary scheduling strategy. Using the preliminary scheduling strategy and the real-time load demand of the park, we perform coordinated scheduling control and dynamic correction, and output the final scheduling scheme.

[0007] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This solution constructs a simulation environment for a multi-park integrated energy system by collecting real-time operational data from each park. Intra-park scheduling is controlled by local agents, while inter-park energy interaction is controlled by a federated layer. This achieves multi-park collaborative optimization without centrally aggregating raw data. The simulation environment is used for interactive operation, and local agents are trained based on a hybrid reward mechanism. Local agent network parameters and local experience data are output. The federated layer generates a globally shared strategy and collaborative experience package based on these parameters and experience data, which is then distributed to each park for gradient updates and fine-tuning. This allows each park to share and reuse effective experiences gained by other parks under specific operating conditions, avoiding experience dilution and repeated trial and error caused by traditional federated parameter averaging. Finally, the initial scheduling strategy is combined with the real-time load demands of the parks for collaborative scheduling control and dynamic correction, outputting the final scheduling scheme. This improves the convergence speed and collaborative efficiency of multi-park scheduling strategies, reduces system operating costs, and enhances the level of multi-energy complementarity utilization. Attached Figure Description

[0008] To make the objectives and technical solutions of this invention clearer, the following drawings are provided and described: Figure 1 A flowchart of the method provided in an embodiment of the present invention; Figure 2 A cost comparison chart between the present invention's method and existing centralized methods is provided for embodiments of the present invention. Figure 3 This is a performance comparison chart between the method provided in this embodiment of the invention and existing reinforcement learning algorithms. Detailed Implementation

[0009] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0010] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0011] Example 1: This invention provides a federated reinforcement learning multi-campus scheduling method based on priority experience sharing, such as... Figure 1 The diagram shows a flowchart of a federated reinforcement learning multi-campus scheduling method based on priority experience sharing. The specific steps are as follows: S1: Construct integrated energy systems for each park and combine them into a multi-agent reinforcement learning simulation environment. The integrated energy systems within a park are controlled by local agents, while the interaction between integrated energy systems between parks is controlled by the federated layer.

[0012] More specifically, real-time operational data from each park is collected to construct a comprehensive energy system for each park, including the following steps: S1.1: Collect and utilize the operating parameters, energy price data, and physical hard constraints of equipment in each park to construct a local multi-dimensional real physical operating cost model. For each integrated energy system park, establish a local objective function containing four types of core costs. This cost is the real cost of minimum operation in the park and does not contain any algorithmic penalty terms. The expression is as follows:

[0013] in, For the park exist The actual physical cost of operation at any given moment, where obj represents the target and T represents the scheduling cycle. The function representing the external energy purchase cost of park i at time t. This represents the local equipment operation and maintenance cost of park i at time t. This represents the local carbon emission cost of park i at time t. The external energy purchase cost function represents the cross-park P2P transaction settlement cost of park i at time t. The expression is as follows:

[0014] in, For the park exist The electricity price at any given time, where 'e' represents electricity. For the park exist Real-time electricity purchase volume For the park exist The price of natural gas at any given time; GT stands for gas turbine. BO represents the gas purchase volume of the gas turbine at time t in the industrial park, and BO represents the gas boiler. This represents the amount of gas purchased by the gas boiler in park i at time t. Local equipment operation and maintenance cost function The expression is as follows:

[0015] ESS stands for Energy Storage System. For the park The unit operating cost of an energy storage system For the park exist The electrical power output of the energy storage system at any given time; HSS represents thermal energy storage system. For the park The unit operating cost of a thermal energy storage system, where h represents heat. For the park exist The thermal power output of the thermal energy storage system is constantly monitored; CSS represents the cold energy storage system. For the park The unit operating cost of a cold energy storage system, where 'c' represents cold energy. For the park exist The cooling power output of the cold energy storage system at any given time; GT represents the gas turbine. The unit operating cost of the i-gas turbine in the park, The electrical power output of the gas turbine at time t in the industrial park; WHR represents the waste heat recovery device. For the park The unit operating cost of waste heat recovery devices For the park exist The heat output of the waste heat recovery device is constantly monitored; BO represents a gas-fired boiler. For the park The unit operating cost of a gas-fired boiler. The thermal power output of the gas-fired boiler in the park at time t; EC is the electric chiller. For the park The unit operating cost of electric chillers, The refrigeration power output of the electric chiller at time t in the park is represented by ABS; ABS represents a refrigerant chiller. For the park The unit operating cost of absorption chillers For the park exist The cooling power output of the absorption chiller at all times; Local carbon emission cost function The expression is as follows:

[0016] in, For the carbon price of all parks, Carbon emission penalty coefficient for all parks; Cross-park P2P transaction settlement costs The expression is as follows:

[0017] in, For the park exist The electricity price at any given moment Let i be the electrical power traded in the park at time t. For the park exist The hottest trading price at any given moment. The heat power of park i at time t; For the park exist The current trading cold price Let i be the heat power of the transaction in park i at time t.

[0018] S1.2: Construct a virtual penalty cost model based on power balance constraints, ramp rate constraints, and energy storage state constraints.

[0019] More specifically, the virtual penalty cost model The expression is as follows:

[0020] in, The penalty cost for power imbalance in park i at time t. The penalty cost for exceeding the ramp-up rate limit of equipment in park i at time t. The penalty cost for the battery health of park i at time t; The expression is as follows:

[0021] in, To balance the penalty coefficient, These are the absolute differences in electrical power, thermal power, and cooling power, respectively. The expression is as follows:

[0022] in, The penalty cost coefficient for the equipment ramp-up rate. The electrical power output of the gas turbine in the industrial park at time t. The maximum gradeability of the i-gas turbine in the park. The maximum ramp rate for the gas-fired boiler in the park. This represents the maximum ramp rate for the i-electric chiller in the park. The maximum ramp rate for the i-absorption chiller in the park; The expression for the gas turbine ramp rate constraint is shown below:

[0023] in, This represents the lower limit of the electrical power output of the i-gas turbine in the park. This represents the upper limit of the electrical power output of the i-gas turbine in the park. For the park The electrical power output of the gas turbine at time t; The expression for the ramp-up rate constraint of the gas-fired boiler is shown below:

[0024] in, This represents the lower limit of the thermal power output of the gas-fired boiler in the park. This represents the upper limit of the thermal power output of the i-gas boiler in the park. For the park The thermal power output of the gas-fired boiler at time t; The expression for the ramp rate constraint of the electric chiller is as follows:

[0025] in, This is the lower limit of the cooling power output of the i-electric chiller in the park. This is the upper limit of the cooling power output of the i-electric chiller in the park. For the park The cooling power output of the electric chiller at time t; The expression for the ramp rate constraint of the absorption chiller is shown below:

[0026] in, This represents the lower limit of the cooling power output of the i-absorption chiller in the park. This represents the upper limit of the cooling power output of the i-absorption chiller in the park. Let i be the cooling power output of the absorption chiller in the park at time t; Battery health penalty cost at time t in the park The expression is as follows:

[0027] in, This is a penalty coefficient for the energy storage state of the energy storage system. For the park The state of electrical energy storage at time t For the park The limit of electrical energy storage state, For the park Upper limit of electrical energy storage state, For the park Thermal energy storage state at time t For the park The limit of thermal energy storage state, For the park Upper limit of thermal energy storage state, Let i be the cold energy storage state of the park at time t. For the park The limit of cold energy storage state For the park Upper limit of cold energy storage status; The constraint expressions for the energy storage system are as follows:

[0028] in, The time step for system scheduling. For the park Rated capacity of electrical energy storage , These refer to the charging efficiency and discharging efficiency of the energy storage system, respectively. , These represent the charging power and discharging power of the energy storage system in park i at time t, respectively. The constraint expressions for the thermal energy storage system are shown below:

[0029] in, For the park Rated capacity of thermal energy storage , These refer to the heat charging efficiency and heat release efficiency of the thermal energy storage system, respectively. , These represent the charging power and releasing power of the thermal energy storage system in park i at time t, respectively. The constraint expressions for the cold energy storage system are shown below:

[0030] in, For the park Rated capacity of cold energy storage , These refer to the charging efficiency and dissipation efficiency of the cold energy storage system, respectively. , These represent the charging and discharging power of the cold energy storage system in park i at time t, respectively. The expression for the power balance constraint is shown below:

[0031] Where T is the scheduling period and N is the number of parks. Let i be the power purchased by the park at time t. The electrical power output of the gas turbine in the industrial park at time t. For the wind power output of park i at time t, To contribute photovoltaic power to the park at time t. Let i be the electrical power traded in the park at time t. The energy storage output power of park i at time t. The power consumption of park i at time t. Let i be the electrical load demand of the park at time t; The expression for handling thermal equilibrium constraints is shown below:

[0032] in, The thermal power output of the gas-fired boiler in the park at time t is For the park exist The heat output power of the waste heat recovery device at all times. Let i be the heat power of the transaction in the park at time t. Let be the heat consumption power of the absorption chiller in park i at time t. For the park exist The thermal power output of the thermal energy storage system at all times For the park exist The heat consumption power of a constant absorption chiller The heat load demand of park i at time t; The expression for handling cold balance constraints is shown below:

[0033] in, The output cooling power of the electric chiller in the park at time t is For the park exist The cooling power output of the constant absorption chiller. Let i be the cooling power output of the cold energy storage system in the park at time t. The cooling load demand of park i at time t.

[0034] S1.3: Define the total generalized cost based on physical hard constraint penalties. :

[0035] in, Total operating cost With virtual penalty costs sum; Define the physical limit violation detection logic, and calculate it at each scheduling time. and If the difference between the physical constraints and the given values ​​satisfies the following criteria, then the system is determined to have exceeded the physical constraint limit at the given time. The expression for the limit exceeding discrimination model based on physical hard constraints is as follows:

[0036] in, To calculate the error tolerance (in this embodiment, the value is taken as...) ), For virtual penalty costs.

[0037] S1.4: Construct a global objective function for the federation layer using the goal of minimizing the total system cost.

[0038] More specifically, the expression for the global objective function of the federation layer is as follows:

[0039] in, Considering the total cost across all parks, the core task of the federal layer is to reduce the overall cost of the entire system while meeting physical constraints.

[0040] S1.5: Construct a multi-energy flow transmission collaborative constraint model using P2P transaction balance constraints and tie-line thermal stability transmission limit constraints.

[0041] More specifically, the multi-energy flow transmission cooperative constraint model includes a P2P transaction balance constraint model and a tie-line thermal stability transmission limit constraint model. The P2P transaction balance constraint model is designed to ensure that at any scheduling time... The net switching power between all campuses within the entire system must be strictly zero, as shown in the following expression:

[0042] Where N is the total number of parks; The electrical power that park i purchases from the market. The electrical power sold by park i to the market; The heat power that the park purchases from the market. The heat output of the park i sold to the market. The cooling capacity that the park purchases from the market. The cooling capacity that park i sells to the market; The thermal stability transmission limit constraint model for tie lines limits the transmission power of physical lines connecting each park to the public energy exchange network, and its expression is as follows:

[0043] in, The first The maximum transmission capacity of the interconnection line for electrical power in each park, the first The maximum transmission capacity of the interconnection lines for thermal power in each park and the first The maximum transmission capacity of the cold power connection lines in each park.

[0044] S1.6: Construct a physical conversion model for the integrated energy system of each park based on the collected real-time operation data of the park.

[0045] More specifically, the integrated energy system physical conversion model includes a gas turbine model, a gas boiler model, an electric chiller model, an absorption chiller model, and a multi-energy storage state evolution model. The multi-energy storage state evolution model includes an electric energy storage system model, a thermal energy storage system model, and a cold energy storage system model. Constructing the integrated energy system physical conversion model includes the following steps: S1.6.1: The local agent collects real-time operational data of the park through the SCADA system and performs maximum value normalization to adapt to the input requirements of the neural network, thus obtaining the park's... exist Set of variables at time The expression is as follows:

[0046] in, For the park exist Wind power generation capacity at all times For the park exist Photovoltaic power generation at any given time For the park exist The amount of electricity purchased at any given time. For the park exist The amount of gas purchased by the gas turbine at any given time. For the park exist Gas purchase volume for gas boilers at any given time. For the park exist The input power of the electric chiller at any given time. For the park exist The input power of the absorption chiller at any given time. The park exist The state of charge of electrochemical energy storage at any given time, the thermal energy storage system's thermal state, and the cold energy storage system's cold state. For the park exist Constantly exchanging power with other parks, For the park exist The heat exchange capacity with other parks at all times. For the park exist The power of cold exchange with other parks at all times; S1.6.2: Using a set of variables Construct a physical conversion model for an integrated energy system, including a model of the gas turbine. The expression is as follows:

[0047] in, Let i be the gas consumption of the gas turbine in the park at time t. For the park exist The output power of the gas turbine at all times These are the first power generation coefficient, the second power generation coefficient, and the third power generation coefficient of the gas turbine, respectively. The expression for the gas-fired boiler model is shown below:

[0048] in, For the park exist The output power of the gas boiler at all times. Let i be the gas consumption of the gas-fired boiler in the park at time t. The output efficiency of the gas-fired boiler; Electric Refrigeration Model The expression is as follows:

[0049] in, For the park exist The output power of the electric chiller at all times. The output efficiency of the gas-fired boiler; Absorption Refrigeration Model The expression is as follows:

[0050] in, For the park exist The output power of the electric chiller at all times; The output efficiency of the gas-fired boiler; The expression for the energy storage system model is shown below:

[0051] in, The charging efficiency of the energy storage in park i at time t. The charging power of the energy storage system in park i at time t. Let be the discharge power of the electrical energy stored in park i at time t. Let be the discharge efficiency of the electrical energy storage in park i at time t. For the park exist The state of charge of electrical energy storage at any given time. park exist Rated power of the electrical energy storage at all times; The expression for the thermal energy storage system model is shown below:

[0052] in, The thermal energy storage efficiency of park i at time t. The charging power of the thermal energy storage in park i at time t. Let i be the heat release power of the thermal energy storage in the park at time t. Let i be the heat release efficiency of the thermal energy storage in the park at time t. For the park exist The state of charge of thermal energy storage at all times. park exist Rated power of thermal energy storage at all times; The expression for the cold energy storage system model is shown below:

[0053] in, The charging efficiency of the cold energy storage in park i at time t. The charging and cooling power of the cold energy storage in park i at time t. Let i be the cooling power of the cold storage in the park at time t. The cooling efficiency of the cold storage in park i at time t. For the park exist The state of charge of cold energy storage at all times park exist Rated power of the cold energy storage system.

[0054] S1.7: Construct a multi-agent reinforcement learning simulation environment based on the Gym architecture using the predefined model and global objective function.

[0055] S2: Train local agents using a hybrid reward mechanism in a simulation environment, and output the network parameters and local experience data of local agents in each park.

[0056] The specific process is as follows: S2.1: Define the agent at time... Observation state vector This vector is set to 9 dimensions and contains information about the system's internal state and external environment. The expression is as follows:

[0057] in, The park exist The state of charge of electrochemical energy storage, the thermal energy storage system's thermal state, and the cold energy storage system's cold state at any given time. For time information; Wind power output of park i at time t; This represents the maximum output of wind power. The photovoltaic output of the park at time t; This represents the maximum output of the photovoltaic system. For the park exist Real-time electrical load demand; This represents the maximum electrical load demand. For the park At all times, the heat load demand; This represents the maximum heat load demand. For the park exist Constant cooling load demand; This represents the maximum cooling load demand. S2.2: Define the action space The expression is as follows:

[0058] in, For the park exist The proportion of power constantly purchased from the main grid. For the park exist The power output ratio of the gas turbine at any given time. For the park exist The percentage of power consumed by the electric chiller at any given time. For the park exist The power output ratio of the gas-fired boiler at any given time. For the park exist The power ratio of the output of a constant absorption chiller. For the park exist The ratio of electrical exchange power at any given time. For the park exist The ratio of heat exchange power at any given time. For the park exist The ratio of cold exchange power at any given moment; S2.3: Define the park exist Cost-reward function at time step The expression is as follows:

[0059] in, For the park exist The local cost function value at time t. For the park exist The maximum value of the local cost function at time t; S2.4: Define the park exist Time-based penalty function The expression is as follows:

[0060] in, For the park exist The difference in internal electrical power at any given moment. For the park exist The difference in thermal power within the park at any given time. For the park exist The difference in cooling power within the park at any given time, For the park exist The change in charge state at time t. For the park exist Changes in the thermal storage state at any given time. For the park exist Changes in the cold storage state at any given time. This represents the maximum value of the electrical load demand. The maximum value of heat load demand. The maximum value of cooling load demand; S2.5: Construct a local experience replay buffer to cyclically store the set of historical state transition sample variables generated by the agent's interaction with the environment; S2.6: Observe the state vector Action space and the local reward function input SAC algorithm, where the state vector Action vector reward function , For the park exist Penalty coefficient (in this embodiment, λ is 5); To address the "overestimation bias" problem that traditional reinforcement learning often encounters when evaluating the value of high-dimensional energy systems, a dual-Critic network is constructed to evaluate state-action values ​​in parallel. By minimizing the Bellman residual to update network parameters, the value assessment is forced to approximate the actual physical operating costs. Critic network loss function The expression is as follows:

[0061] in, Let i represent the state space of the i-th park at the next time step. This represents the action space of the i-th park at the next moment. Let i be the set of variables stored in the local experience replay buffer for the i-th zone. This represents the mathematical expectation of the set of historical state transition sample variables randomly sampled from the local experience replay buffer. This represents the mathematical expectation of the action ai' at the next moment. Indicates the parameters of the Critic network. , Both represent state-action value functions, used to evaluate states. Execute action Expected returns and assessment status Execute action Expected returns For target value, For the park Local reward function , As a discount factor, For temperature parameters (in this embodiment, The value is 0.99. (Automatic adjustment) For Actor network policy functions, These are the parameters of the Actor network; To prevent the scheduling strategy from getting trapped in local optima, a maximum entropy strategy optimization is performed, and an entropy regularization term is introduced. Through reparameterization techniques, the strategy's entropy is maximized while minimizing the expected system cost, resulting in a scheduling strategy with stronger robustness to fluctuations in photovoltaic and wind power. Specifically, the industrial park... Actor network objective function The expression is as follows:

[0062] in, For the park Actor network parameters, For the park Temperature parameters, This represents the mathematical expectation of performing the corresponding action on the state sampled from the buffer under the current strategy; The training process is iteratively executed until the termination condition is met, and the local agent network parameters and local experience data of each park are output. The termination condition expression is as follows:

[0063] in, The number of rounds for training the SAC algorithm. The termination condition threshold (in this embodiment, ).

[0064] S3: The federation layer generates a global shared strategy and collaborative experience package based on the parameters and experience data, and distributes it to local agents for gradient updates and fine-tuning to obtain a preliminary scheduling strategy.

[0065] The specific process is as follows: The federated layer generates a globally shared policy and collaborative experience package and distributes it to local agents for gradient updates and fine-tuning, including the following steps: S3.1: In each training round, the local agent uploads a dual-channel data packet containing model parameters and high-value data to the federated layer. The dual-channel data packet includes a parameter channel and an experience channel. The parameter channel data packet is the set of parameter packets uploaded by park i to the federated layer. The expression is as follows:

[0066] in, Let i be the Actor network parameters. Critic network parameters for park i For the park Temperature parameters, For intelligent agents exist The local cost statistic for each moment is expressed as follows:

[0067] in, The total number of time steps within a local training cycle. For intelligent agents exist Local time cost; For intelligent agents exist The expression for the local cross-campus power transmission statistics at any given time is as follows:

[0068] in, For intelligent agents exist Real-time local cross-park power transmission; For intelligent agents exist The expression for the local cross-campus heat power transmission statistics at any given time is as follows:

[0069] in, For intelligent agents exist Real-time local cross-park heat power transmission; For intelligent agents exist The expression for the local cross-campus transmission cold power statistics at any given time is as follows:

[0070] in, For intelligent agents exist Real-time local cross-park transmission of cold power; The experience channel data packet is a state transition quadruple extracted from the local experience playback buffer using the limit-crossing discrimination model, as shown in the following expression:

[0071] in, Let i be the gradient of the network parameters in park i. Let i be the set of state spaces in park i. For intelligent agents The physical limit-breaking experience subset selected using the limit-breaking discrimination model contains typical fault samples that cause power imbalance, ramp-up failure, or SOC limit exceeding in the system. ; S3.2: The federation layer uses dual-channel data packets to calculate the global reward function after aggregating all parks. The expression is as follows:

[0072] in, The global cost-reward function, aggregated across all parks, is expressed as follows:

[0073] in, This represents the maximum global cost after aggregating all parks. The total cost for all parks; The global penalty function, which aggregates all parks, is expressed as follows:

[0074] Where N is the total number of parks, The park Cross-regional power transmission, industrial park Cross-regional heat power transmission and park The maximum value of cold power transferred across regions; S3.3: Construct dynamic aggregation weights for each park based on its security contribution. Unlike traditional average aggregation based on data volume, this method employs a dual-factor dynamic weighting system of "security-economy," where the dynamic aggregation weight for park i is... The expression is as follows:

[0075] Where e is a mathematical constant, j is the park index, and N is the number of parks. and These are the local rewards and penalties for park i. and These are the local rewards and penalties for park j. The sensitivity coefficient, For safety preference factor ( The value is 1.0. With a value of 1.2, this mechanism assigns higher aggregation weights to high-quality models that are both economical and safe, thereby accelerating the convergence of the global model to the safe and feasible domain.

[0076] S3.4: The federation layer uses dynamic aggregation weights to generate globally shared policy parameters that incorporate dynamic aggregation weights of security contributions, as shown in the following expression:

[0077] in, This is the revised global sharing strategy. The strategy parameters for park i; S3.5: The federal layer collects all data uploaded from the campus. A global collaboration buffer is constructed to store a global collaboration experience package. The samples in the buffer are high-value "negative examples" of physical limits being exceeded under extreme conditions in other parks. Redundant samples are removed using a clustering algorithm, and the most representative physical limit exceedance samples are retained to generate the global collaboration experience package. And store it in the global collaboration buffer; S3.6: The federation layer will generate globally shared policy parameters and collaboration experience package Simultaneously, it is distributed to each local intelligent agent; S3.7: Execute a risk-avoidance-based mixing ratio sampling strategy, with the local agent utilizing a preset mixing ratio. (In this embodiment, Take the state transition quadruples from the mixed global cooperative buffer and local experience replay buffer (0.3) to construct the training batch. The expression is as follows:

[0078] Where M is the batch size, and the batch includes global alert samples and local exploration samples. A global alert sample from the global collaboration buffer The random sampling forces the local Critic network to significantly lower the Q-value score of dangerous actions by learning from the failures of others before encountering actual failures. A local exploration sample from the local regular buffer Random sampling is used to maintain the ability to fit the characteristics of the local environment.

[0079] S3.8: The local agent utilizes the training batch The SAC algorithm is used, abandoning traditional uniform sampling and adopting a "safety-exploration" hybrid sampling mechanism to perform gradient updates and fine-tuning, resulting in an initial scheduling strategy. During fine-tuning, the objective function explicitly includes a global reward function to achieve a Pareto balance between "local autonomy" and "global cooperation." The fine-tuned objective function... The expression is as follows:

[0080] in, park Local Actor network loss, The global policy's weight to the local policy (in this embodiment, it is set to 0.6, balancing global interests with local characteristics), s is the state space, To randomly select state samples s from the historical experience database of park i, Let be the expectation of all state samples s randomly sampled from the experience replay region Di. Let i be the gradient of the target network parameters with respect to the i-th node. For the global reward function, Let i be the value function of park i. For parameters The action chosen under the strategy.

[0081] S4: Utilize the preliminary scheduling strategy and the real-time load demand of the park to perform collaborative scheduling control and dynamic correction, and output the final scheduling scheme.

[0082] The specific process is as follows: S4.1: Construct the intelligent agent observation state using the preliminary scheduling strategy and the real-time electricity / heat / cooling load demand of the park, input the observation state into the local intelligent agent, output the initial action, and map the initial action into the power control command of the integrated energy system of each park. S4.2: Send the power control command to the park equipment for execution, and output the operating status of the equipment after execution and the cross-park power-heat-cooling exchange power; S4.3: Perform multi-campus energy interaction monitoring on the operating status and the exchange power input federation layer, calculate the cross-campus electricity / heat / cooling supply and demand balance deviation, and if the deviation is greater than a preset threshold (in this embodiment, the value is taken as...). Then a correction signal will be triggered; S4.4: Send the correction signal to the corresponding local intelligent agent in the park, fuse the initial action of the local intelligent agent with the correction signal to obtain the correction action, and combine the action space projection method based on physical constraints to output the secondary fine-tuning command for power control of the integrated energy system of each park. S4.5: Send the secondary fine-tuning instruction to the park equipment for execution, repeat the above process until the deviation is less than the preset threshold, and output the final scheduling scheme.

[0083] Based on the initial scheduling strategy, a hierarchical collaborative scheduling is implemented: the local agent adjusts the output power of energy supply equipment, energy conversion equipment, and energy storage system in real time according to the strategy; the federation layer monitors the energy interaction status of each park through the federation aggregation server and dynamically optimizes the power allocation of cross-park electricity / heat / cold energy exchange; the two-layer architecture achieves linkage through closed-loop feedback, the local agent executes equipment control while feeding back the operating status to the federation layer, and the federation layer corrects the scheduling instructions in real time according to the multi-energy flow balance status, ultimately forming a hierarchical optimization pattern of "local autonomy - global coordination", ensuring that the multi-park integrated energy system achieves day-ahead economic scheduling under all constraints.

[0084] To verify the effectiveness of the present invention, a large-area integrated energy system was divided into three sub-integrated energy systems. The sub-systems exchanged electrical power, thermal power, and cooling power through a P2P market, and their operating costs and reinforcement learning performance were compared.

[0085] like Figure 2 As shown, the horizontal axis represents the number of training rounds, and the vertical axis represents the total daily operating cost of the system. The results show that the traditional centralized method (solid line) converges slowly and the cost curve oscillates significantly under large-scale coupling constraints; this scheme (dashed line), due to the collaborative experience packages issued by the federation layer enabling cross-park experience sharing, can converge to a lower and more stable cost level within about 250 rounds, demonstrating higher optimization efficiency and convergence stability.

[0086] like Figure 3 As shown, the horizontal axis represents the number of training rounds, and the vertical axis represents the overall reward value including safety penalties. A higher value indicates a better strategy. This scheme (dashed line) uses a physical limit violation detection and penalty mechanism to enable the overall reward to increase rapidly in the early stages of training and stabilize in a high range, significantly outperforming the baseline reinforcement learning algorithm (solid line). This demonstrates that this scheme can learn scheduling behaviors that satisfy safety hard constraints more quickly and achieve better economic performance on this basis.

[0087] This solution constructs a simulation environment for a multi-park integrated energy system by collecting real-time operational data from each park. Intra-park scheduling is controlled by local agents, while inter-park energy interaction is controlled by a federated layer. This achieves multi-park collaborative optimization without centrally aggregating raw data. The simulation environment is used for interactive operation, and local agents are trained based on a hybrid reward mechanism. Local agent network parameters and local experience data are output. The federated layer generates a globally shared strategy and collaborative experience package based on these parameters and experience data, which is then distributed to each park for gradient updates and fine-tuning. This allows each park to share and reuse effective experiences gained by other parks under specific operating conditions, avoiding experience dilution and repeated trial and error caused by traditional federated parameter averaging. Finally, the initial scheduling strategy is combined with the real-time load demands of the parks for collaborative scheduling control and dynamic correction, outputting the final scheduling scheme. This improves the convergence speed and collaborative efficiency of multi-park scheduling strategies, reduces system operating costs, and enhances the level of multi-energy complementarity utilization.

[0088] This scheme achieves deep collaboration among multiple campuses in both day-ahead optimization and real-time scheduling phases through a federated learning architecture that introduces a physical violation experience sharing mechanism. In this process, local agents in each campus not only coordinate with the federated aggregation center to respond to the global social welfare maximization goal, but also utilize a violation discrimination model based on physical hard constraints to fully explore the economically optimal solution within the safety boundary, decoupling real operating costs from virtual penalty costs in real time. Furthermore, leveraging the global collaborative experience package distributed by the federated layer, each campus uses its locally constructed dual experience replay buffer to perform hybrid sampling fine-tuning. While considering the overall system economy, this forces the adoption of violation lessons from the entire network to proactively avoid potential physical violation risks, thereby efficiently executing cross-campus energy sharing in a direction that reduces total costs. This experience screening and hybrid gradient update mechanism based on physical criteria effectively overcomes the convergence oscillation problem caused by blind trial and error in traditional methods, achieving zero-violation collaborative optimization of multi-campus systems under complex physical constraints.

[0089] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A federated reinforcement learning multi-garden scheduling method based on priority experience sharing, characterized in that, Includes the following steps: Construct integrated energy systems for each park and combine them into a multi-agent reinforcement learning simulation environment. The integrated energy systems within a park are controlled by local agents, while the interaction between integrated energy systems in different parks is controlled by a federated layer. The simulation environment is used to train local agents based on a hybrid reward mechanism, and the network parameters and local experience data of local agents in each park are output. The federation layer generates a globally shared strategy and collaborative experience package based on the parameters and experience data, and distributes it to local agents for gradient updates and fine-tuning to obtain a preliminary scheduling strategy. Using the preliminary scheduling strategy and the real-time load demand of the park, we perform coordinated scheduling control and dynamic correction, and output the final scheduling scheme.

2. The federated reinforcement learning multi-garden scheduling method based on priority experience sharing according to claim 1, characterized in that, The construction of the multi-agent reinforcement learning simulation environment includes the following steps: Collect and utilize the operating parameters, energy price data, and physical constraints of equipment in each park to construct a local multi-dimensional real physical operating cost model. A virtual penalty cost model is constructed based on power balance constraints, ramp rate constraints, and energy storage state constraints. Construct a limit-crossing discrimination model based on physical hard constraints; Construct a global objective function for the federation layer using the goal of minimizing the total system cost; A multi-energy flow transmission collaborative constraint model is constructed by utilizing the balance constraint of P2P transactions and the thermal stability transmission limit constraint of tie lines. Physical conversion models of the integrated energy systems of each park are constructed based on real-time operational data collected from the parks. A multi-agent reinforcement learning simulation environment is constructed using the predefined model and global objective function described above.

3. The federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 2, characterized in that, The local multi-dimensional real physical operating cost model The expression is as follows: wherein, is a park At the real physical operation cost at time t, obj represents the target, T represents the scheduling period, represents the external energy purchase cost function of park i at time t, represents the local device operation and maintenance cost of park i at time t, represents the local carbon emission cost of park i at time t, represents the park i at time t settlement cost of cross-park P2P transaction, external energy purchase cost function The expression is as follows: in, For the park exist The electricity price at any given time, where 'e' represents electricity. For the park exist Real-time electricity purchase volume For the park exist The price of natural gas at any given time; GT stands for gas turbine. BO represents the gas purchase volume of the gas turbine at time t in the industrial park, and BO represents the gas boiler. This represents the amount of gas purchased by the gas boiler in park i at time t. Local equipment operation and maintenance cost function The expression is as follows: ESS stands for Energy Storage System. For the park The unit operating cost of an energy storage system For the park exist The electrical power output of the energy storage system at any given time; HSS represents thermal energy storage system. For the park The unit operating cost of a thermal energy storage system, where h represents heat. For the park exist The thermal power output of the thermal energy storage system is constantly monitored; CSS represents the cold energy storage system. For the park The unit operating cost of a cold energy storage system, where 'c' represents cold energy. For the park exist The cooling power output of the cold energy storage system at any given time; The unit operating cost of the i-gas turbine in the park, The electrical power output of the gas turbine at time t in the industrial park; WHR represents the waste heat recovery device. For the park The unit operating cost of waste heat recovery devices For the park exist The thermal power output of the waste heat recovery device at all times; For the park The unit operating cost of a gas-fired boiler. The thermal power output of the gas-fired boiler in the park at time t; EC is the electric chiller. For the park The unit operating cost of electric chillers The refrigeration power output of the electric chiller at time t in the park is represented by ABS; ABS represents a refrigerant chiller. For the park The unit operating cost of absorption chillers For the park exist The cooling power output of the absorption chiller at all times; Local carbon emission cost function The expression is as follows: in, For the carbon price of all parks, Carbon emission penalty coefficient for all parks; Cross-park P2P transaction settlement costs The expression is as follows: in, For the park exist The electricity price at any given moment Let i be the electrical power traded in the park at time t. For the park exist The hottest trading price at any given moment. The heat power of the park at time t; For the park exist The current trading cold price Let t be the cold power of the transaction in park i at time t.

4. The federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 2, characterized in that, Virtual penalty cost model The expression is as follows: in, The penalty cost for power imbalance in park i at time t. The penalty cost for exceeding the ramp-up rate limit of equipment in park i at time t. The penalty cost for the battery health of park i at time t; The expression is as follows: in, To balance the penalty coefficient, These are the absolute differences in electrical power, thermal power, and cooling power, respectively. The expression is as follows: in, The penalty cost coefficient for the equipment ramp-up rate. The electrical power output of the gas turbine in the industrial park at time t. The maximum gradeability of the i-gas turbine in the park. The maximum ramp rate for the gas-fired boiler in the park. This represents the maximum ramp rate for the i-electric chiller in the park. The maximum ramp rate for the i-absorption chiller in the park; The expression for the gas turbine ramp rate constraint is shown below: in, This represents the lower limit of the electrical power output of the i-gas turbine in the park. This represents the upper limit of the electrical power output of the i-gas turbine in the park. For the park The electrical power output of the gas turbine at time t; The expression for the ramp-up rate constraint of the gas-fired boiler is shown below: in, This represents the lower limit of the thermal power output of the gas-fired boiler in the park. This represents the upper limit of the thermal power output of the i-gas boiler in the park. For the park The thermal power output of the gas-fired boiler at time t; The expression for the ramp rate constraint of the electric chiller is as follows: in, This is the lower limit of the cooling power output of the i-electric chiller in the park. This is the upper limit of the cooling power output of the i-electric chiller in the park. For the park The cooling power output of the electric chiller at time t; The expression for the ramp rate constraint of the absorption chiller is shown below: in, This represents the lower limit of the cooling power output of the i-absorption chiller in the park. This represents the upper limit of the cooling power output of the i-absorption chiller in the park. Let i be the cooling power output of the absorption chiller in the park at time t; Battery health penalty cost at time t in the park The expression is as follows: in, This is a penalty coefficient for the energy storage state of the energy storage system. For the park The state of electrical energy storage at time t For the park The limit of electrical energy storage state, For the park Upper limit of electrical energy storage state, For the park Thermal energy storage state at time t For the park The limit of thermal energy storage state, For the park Upper limit of thermal energy storage state, Let i be the cold energy storage state of the park at time t. For the park The limit of cold energy storage state For the park Upper limit of cold energy storage status; The constraint expressions for the energy storage system are as follows: in, The time step for system scheduling. For the park Rated capacity of electrical energy storage , These refer to the charging efficiency and discharging efficiency of the energy storage system, respectively. , These represent the charging power and discharging power of the energy storage system in park i at time t, respectively. The constraint expressions for the thermal energy storage system are shown below: in, For the park Rated capacity of thermal energy storage , These refer to the heat charging efficiency and heat release efficiency of the thermal energy storage system, respectively. , These represent the charging power and releasing power of the thermal energy storage system in park i at time t, respectively. The constraint expressions for the cold energy storage system are as follows: in, For the park Rated capacity of cold energy storage , These refer to the charging efficiency and dissipation efficiency of the cold energy storage system, respectively. , These represent the charging and discharging power of the cold energy storage system in park i at time t, respectively. The expression for the power balance constraint is shown below: Where T is the scheduling period and N is the number of parks. Let i be the power purchased by the park at time t. The electrical power output of the gas turbine in the industrial park at time t. For the wind power output of park i at time t, To contribute photovoltaic power to the park at time t. Let i be the electrical power traded in the park at time t. The energy storage output power of park i at time t. The power consumption of park i at time t. Let i be the electrical load demand of the park at time t; The thermal balance constraint processing expression is as follows: in, The thermal power output of the gas-fired boiler in the park at time t is For the park exist The heat output power of the waste heat recovery device at all times. Let i be the heat power of the transaction in the park at time t. Let be the heat consumption power of the absorption chiller in park i at time t. For the park exist The thermal power output of the thermal energy storage system at all times For the park exist The heat consumption power of a constant absorption chiller The heat load demand of park i at time t; The expression for handling cold balance constraints is shown below: in, The output cooling power of the electric chiller in the park at time t is For the park exist The cooling power output of the constant absorption chiller. Let i be the cooling power output of the cold energy storage system in the park at time t. The cooling load demand of park i at time t; The expression for the limit violation detection model is as follows: in, To calculate the error tolerance, For virtual penalty costs.

5. A federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 2, characterized in that, The expression for the global objective function of the federation layer is as follows: in, For the park exist The real physical operating cost at any given moment Considering the total cost across all parks, the core task of the federal layer is to reduce the overall cost of the entire system while meeting physical constraints.

6. A federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 2, characterized in that, The multi-energy flow transmission cooperative constraint model includes a P2P transaction balance constraint model and a tie-line thermal stability transmission limit constraint model. The P2P transaction balance constraint model is applicable at any scheduling time. The net switching power between all campuses within the entire system must be strictly zero, as shown in the following expression: Where N is the total number of parks; The electrical power that park i purchases from the market. The electrical power sold by park i to the market; The heat power that the park purchases from the market. The heat output of the park is sold to the market. The cooling capacity that the park purchases from the market. The cooling capacity that park i sells to the market; The thermal stability transmission limit constraint model for tie lines limits the transmission power of physical lines connecting each park to the public energy exchange network, and its expression is as follows: in, The first The maximum transmission capacity of the interconnection line for electrical power in each park, the first The maximum transmission capacity of the interconnection lines for thermal power in each park and the first The maximum transmission capacity of the cold power connection lines in each park.

7. A federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 2, characterized in that, The integrated energy system physical conversion model includes a gas turbine model, a gas boiler model, an electric chiller model, an absorption chiller model, and a multi-energy storage state evolution model. The multi-energy storage state evolution model includes an electric energy storage system model, a thermal energy storage system model, and a cold energy storage system model. Constructing the integrated energy system physical conversion model includes the following steps: The local agent collects real-time operational data of the park through the SCADA system and performs maximum value normalization to adapt to the input requirements of the neural network, thereby obtaining the park's... exist Set of variables at time The expression is as follows: in, For the park exist Wind power generation capacity at all times For the park exist Photovoltaic power generation at any given time For the park exist The amount of electricity purchased at any given time. For the park exist The amount of gas purchased by the gas turbine at any given time. For the park exist The amount of gas purchased for the gas boiler at any given time. For the park exist The input power of the electric chiller at any given time. For the park exist The input power of the absorption chiller at any given time. The park exist The state of charge of electrochemical energy storage at any given time, the thermal energy storage system's thermal state, and the cold energy storage system's cold state. For the park exist Constantly exchanging power with other parks, For the park exist The heat exchange capacity with other parks at all times. For the park exist The power of cold exchange with other parks at all times; Using variable sets Construct a physical conversion model for an integrated energy system, including a model of the gas turbine. The expression is as follows: in, Let i be the gas consumption of the gas turbine in the park at time t. For the park exist The output power of the gas turbine at all times These are the first power generation coefficient, the second power generation coefficient, and the third power generation coefficient of the gas turbine, respectively. The expression for the gas-fired boiler model is shown below: in, For the park exist The output power of the gas boiler at all times. Let i be the gas consumption of the gas-fired boiler in the park at time t. The output efficiency of the gas-fired boiler; Electric Refrigeration Model The expression is as follows: in, For the park exist The output power of the electric chiller at all times. The output efficiency of the gas-fired boiler; Absorption Refrigeration Model The expression is as follows: in, For the park exist The output power of the electric chiller at all times; The output efficiency of the gas-fired boiler; The expression for the energy storage system model is shown below: in, The charging efficiency of the energy storage in park i at time t. The charging power of the energy storage system in the park at time t. Let be the discharge power of the electrical energy stored in park i at time t. Let be the discharge efficiency of the electrical energy storage in park i at time t. For the park exist The state of charge of electrical energy storage at any given time. park exist Rated power of the electrical energy storage at any given time; The expression for the thermal energy storage system model is shown below: in, The thermal energy storage efficiency of park i at time t. The charging power of the thermal energy storage in park i at time t. Let i be the heat release power of the thermal energy storage in the park at time t. Let i be the heat release efficiency of the thermal energy storage in the park at time t. For the park exist The state of charge of thermal energy storage at all times. park exist Rated power of thermal energy storage at all times; The expression for the cold energy storage system model is shown below: in, The charging efficiency of the cold energy storage in park i at time t. The charging and cooling power of the cold energy storage in park i at time t. Let i be the cooling power of the cold storage in the park at time t. The cooling efficiency of the cold storage in park i at time t. For the park exist The state of charge of cold energy storage at all times park exist Rated power of the cold energy storage system.

8. The federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 1, characterized in that, The initial policy training of a local agent based on a hybrid reward mechanism is performed in a simulation environment, including the following steps: Define the agent at time. Observation state vector The expression is as follows: in, The park exist The state of charge of electrochemical energy storage, the thermal energy storage system's thermal state, and the cold energy storage system's cold state at any given time. For time information; Wind power output of park i at time t; This represents the maximum output of wind power. The photovoltaic output of the park at time t; This represents the maximum output of the photovoltaic system. For the park exist Real-time electrical load demand; This represents the maximum electrical load demand. For the park At all times, the heat load demand; This represents the maximum heat load demand. For the park exist Constant cooling load demand; This represents the maximum cooling load demand. Define action space The expression is as follows: in, For the park exist The proportion of power constantly purchased from the main grid. For the park exist The power output ratio of the gas turbine at any given time. For the park exist The percentage of power consumed by the electric chiller at any given time. For the park exist The power output ratio of the gas-fired boiler at any given time. For the park exist The power ratio of the output of a constant absorption chiller. For the park exist The ratio of electrical exchange power at any given time. For the park exist The ratio of heat exchange power at any given time. For the park exist The ratio of cold exchange power at any given moment; Define the park exist Cost-reward function at time step The expression is as follows: in, For the park exist The local cost function value at time t. For the park exist The maximum value of the local cost function at time t; Define the park exist Time-based penalty function The expression is as follows: in, For the park exist The difference in internal electrical power at any given moment. For the park exist The difference in thermal power within the park at any given time. For the park exist The difference in cooling power within the park at any given time, For the park exist The change in charge state at time t. For the park exist Changes in the thermal storage state at any given time. For the park exist Changes in the cold storage state at any given time. This represents the maximum value of the electrical load demand. The maximum value of heat load demand. The maximum value of cooling load demand; Construct a local experience replay buffer to cyclically store the set of historical state transition sample variables generated by the agent's interaction with the environment; The observed state vector Action space and the local reward function input SAC algorithm, where the state vector Action vector reward function , For the park exist Penalty coefficient; A dual-Critic network is constructed to evaluate state-action value in parallel, and the network parameters are updated by minimizing the Bellman residual. Critic network loss function The expression is as follows: in, Let i represent the state space of the i-th park at the next time step. This represents the action space of the i-th park at the next moment. Let i be the set of variables stored in the local experience replay buffer for the i-th zone. This represents the mathematical expectation of the set of historical state transition sample variables randomly sampled from the local experience replay buffer. This represents the mathematical expectation of the action ai' at the next moment. Indicates the parameters of the Critic network. , Both represent state-action value functions, used to evaluate states. Execute action Expected returns and assessment status Execute action Expected returns For target value, For the park Local reward function , As a discount factor, For temperature parameters, For Actor network policy functions, These are the parameters of the Actor network; The maximum entropy strategy is optimized by implementing a maximum entropy policy and introducing an entropy regularization term. Through reparameterization techniques, the policy's entropy is maximized while minimizing the expected system cost, resulting in a scheduling policy. Among these, the campus... Actor network objective function The expression is as follows: in, For the park Actor network parameters, For the park Temperature parameters, This represents the mathematical expectation of performing the corresponding action on the state sampled from the buffer under the current strategy; The training process is iteratively executed until the termination condition is met, and the local agent network parameters and local experience data of each park are output. The termination condition expression is as follows: in, The number of rounds for training the SAC algorithm. This is the threshold for the termination condition.

9. A federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 1, characterized in that, The federated layer generates a globally shared policy and collaborative experience package and distributes it to local agents for gradient updates and fine-tuning, including the following steps: In each training round, the local agent uploads a dual-channel data packet containing model parameters and high-value data to the federated layer. The dual-channel data packet includes a parameter channel and an experience channel, wherein the parameter channel data packet is the set of parameter packets uploaded by campus i to the federated layer. The expression is as follows: in, Let i be the Actor network parameters. Critic network parameters for park i For the park Temperature parameters, For intelligent agents exist The local cost statistic for each moment is expressed as follows: in, The total number of time steps within a local training cycle. For intelligent agents exist Local time cost; For intelligent agents exist The expression for the local cross-campus power transmission statistics at any given time is as follows: in, For intelligent agents exist Real-time local cross-park power transmission; For intelligent agents exist The expression for the local cross-campus heat power transmission statistics at any given time is as follows: in, For intelligent agents exist Real-time local cross-park heat power transmission; For intelligent agents exist The expression for the local cross-campus transmission cold power statistics at any given time is as follows: in, For intelligent agents exist Real-time local cross-park transmission of cold power; The experience channel data packet is a state transition quadruple extracted from the local experience playback buffer using the limit-crossing discrimination model, as shown in the following expression: in, Let i be the gradient of the network parameters in park i. Let i be the set of state spaces in park i. For intelligent agents The physical limit-breaking experience subset selected using the limit-breaking discrimination model contains typical fault samples that cause power imbalance, ramp-up failure, or SOC limit exceeding in the system. ,in, As of the current assessment status, For the currently executing action, For the assessment status at the next moment, For the action to be performed in the next moment; The federated layer uses dual-channel data packets to calculate the global reward function after aggregating all parks. The expression is as follows: in, The global cost-reward function, aggregated across all parks, is expressed as follows: in, This represents the maximum global cost after aggregating all parks. The total cost for all parks; The global penalty function, which aggregates all parks, is expressed as follows: Where N is the total number of parks, The park Cross-regional power transmission, industrial park Cross-regional heat power transmission and park The maximum value of cold power transferred across regions; Construct dynamic aggregate weights for each park, where the dynamic aggregate weight of park i is... The expression is as follows: Where e is a mathematical constant, j is the park index, and N is the number of parks. and These are the local rewards and penalties for park i. and These are the local rewards and penalties for park j. The sensitivity coefficient, Safety preference factor; The federated layer uses dynamically aggregated weights to generate globally shared policy parameters, as shown in the following expression: in, This is the revised global sharing strategy. The strategy parameters for park i; The federal layer collects all uploaded data from the campus. A global collaboration buffer is constructed, redundant samples are removed using a clustering algorithm, and a global collaboration experience package is generated. And store it in the global collaboration buffer; The federation layer will generate globally shared policy parameters and collaboration experience package Simultaneously, it is distributed to each local intelligent agent; The local agent uses a preset mixing ratio State transition quadruples from both the global collaborative buffer and the local experience replay buffer are used to construct training batches. The expression is as follows: Where M is the batch size, and the batch includes global alert samples and local exploration samples. A global alert sample from the global collaboration buffer Randomly selected from; A local exploration sample from the local experience replay buffer Randomly selected from; The local agent utilizes the training batch Inputting the SAC algorithm, performing gradient update and fine-tuning, yields the initial scheduling strategy and the objective function for fine-tuning. The expression is as follows: in, park Local Actor network loss, The global policy's weights on the local policy, where s is the state space. To randomly select state samples s from the historical experience database of park i, Let be the expectation of all state samples s randomly sampled from the experience replay region Di. Let i be the gradient of the target network parameters with respect to the i-th node. For the global reward function, Let i be the value function of park i. For parameters The action chosen under the strategy.

10. A federated reinforcement learning multi-campus scheduling method based on priority experience sharing according to claim 1, characterized in that, Using a preliminary scheduling strategy model and the real-time load demand of the park, collaborative scheduling control and dynamic correction are performed, including the following steps: The intelligent agent observation state is constructed by using the preliminary scheduling strategy and the real-time electricity / heat / cooling load demand of the park. The observation state is input into the local intelligent agent, the initial action is output, and the initial action is mapped to the power control command of the integrated energy system of each park. The power control command is sent to the park equipment for execution, and the operating status of the equipment after execution and the cross-park power exchange between electricity, heat and cold are output. The operating status and the exchange power input federation layer are used to perform multi-campus energy interaction monitoring, and the cross-campus electricity / heat / cooling supply and demand balance deviation is calculated. If the deviation is greater than a preset threshold, a correction signal is triggered. The correction signal is sent to the local intelligent agent in the corresponding park. The initial action of the local intelligent agent is fused with the correction signal to obtain the correction action. Combined with the action space projection method based on physical constraints, the power control secondary fine-tuning command of the integrated energy system of each park is output. The secondary fine-tuning command is sent to the park equipment for execution, and the above process is repeated until the deviation is less than the preset threshold, and the final scheduling scheme is output.

Images