A hierarchical federated reinforcement learning energy optimization method for zero-carbon park

Abstract

The application discloses a layered federated reinforcement learning energy optimization method for a zero-carbon park, belongs to the technical field of energy optimization, and aims to solve problems of data privacy leakage in traditional centralized optimization and performance decline of a federated learning in processing non-independent and identically distributed data of heterogeneous enterprises, takes a data static model dynamic as a core, constructs a layered federated aggregation architecture of an enterprise layer, a grouping layer and a global layer, adopts a PPO reinforcement learning intelligent agent on the enterprise side, trains and encrypts model parameters based on local data and uploads the model parameters, divides enterprises into groups through JS divergence and K-means clustering on the park side, completes model aggregation through grouped federated averaging and global secondary fusion, and then updates and issues an optimization strategy, the application combines a multi-objective reward function of economic cost, equipment loss and carbon emission cost, realizes distributed autonomous optimization and global collaborative scheduling of the park, improves high-proportion new energy consumption capacity, and takes into account economic and zero-carbon targets.

Description

Technical Field

[0001] This invention belongs to the field of energy optimization technology, specifically a hierarchical federation-based enhanced distributed energy optimization method. Background Technology

[0002] Against the backdrop of global efforts to address climate change and promote energy transition, the construction of zero-carbon industrial parks serves as a crucial vehicle for regional energy transformation. It necessitates adapting to the new demands of large-scale, high-proportion renewable energy sources and emerging energy entities on power dispatch, and comprehensively optimizing and adjusting dispatch methods, mechanisms, and management. This policy direction sets forth clear requirements for the intelligent optimization of zero-carbon industrial park energy systems, demanding the coordinated dispatch of high-proportion renewable energy consumption and diverse flexible loads while ensuring data security.

[0003] Currently, centralized data processing and modeling methods are commonly used for the collaborative optimization of energy systems in industrial parks. The system collects energy consumption data, renewable energy output data, and load characteristics from each enterprise unit to construct a global optimization model. Then, it analyzes the system's frequency response and steady-state recovery characteristics based on classical control theory or frequency domain analysis. For example, existing research quantitatively analyzes the system's frequency deviation by constructing a transfer function model that includes wind, solar, and energy storage frequency regulation units; or it uses frequency domain analysis to establish a power system frequency response model containing large-scale wind power to correct the primary frequency regulation capability evaluation indicators. These methods have good applicability in specific scenario assessments.

[0004] To address data privacy concerns, federated learning, as a distributed machine learning paradigm, has been introduced into the field of energy management. The federated averaging algorithm enables multiple clients to collaboratively learn and share a predictive model, while keeping all training data on local devices, thus balancing model training and data privacy protection to some extent. For situations where there are significant differences in data distribution among different types of enterprises within a park, this algorithm provides a feasible technical path for collaborative modeling with the participation of multiple stakeholders.

[0005] In terms of system collaborative optimization and stability assessment, existing research provides theoretical support for system frequency characteristic analysis and frequency regulation capability assessment by constructing transfer function models and frequency domain analysis models that include wind, solar, and energy storage units. These methods have certain reference value in the stability assessment and optimized scheduling of park-level energy systems. Summary of the Invention

[0006] The purpose of this invention is to provide a hierarchical federated reinforcement learning energy optimization method and system for zero-carbon industrial parks, aiming to propose an advanced and practical technical approach to solve the contradiction between data privacy and global optimization. This method, based on the core principle of "data remains stationary while the model moves," constructs a hierarchical federated aggregation architecture based on enterprise type clustering. It achieves distributed autonomous optimization by deploying local reinforcement learning agents on the enterprise side, and establishes a collaborative optimization platform on the park side that balances privacy protection and knowledge fusion. Furthermore, it designs a multi-objective reward function that incorporates real-time carbon costs, thereby fundamentally resolving the contradictions of existing technologies.

[0007] The specific technical solution is as follows:

[0008] A hierarchical federated reinforcement learning energy optimization method for zero-carbon parks is proposed. This method is applied to a zero-carbon park energy system comprising enterprise-side terminals, a park-side collaborative optimization platform, and a secure communication network. The method includes local reinforcement learning training, model parameter uploading, and federated aggregation steps, comprising the following steps:

[0009] S1 Local Privacy Training Steps: Each enterprise-side terminal collects local energy operation and production status data, trains a local energy optimization agent based on the reinforcement learning algorithm of the Actor-Critic architecture, and optimizes the energy control strategy of local energy storage charging and discharging and adjustable load through a self-developed reward function; during the training process, all raw energy consumption and production data are stored locally on the enterprise side, and only the strategy parameters and gradient information of the model are extracted.

[0010] S2 parameter encryption transmission steps: After each enterprise-side terminal completes a preset round of local training, it encrypts the extracted model strategy parameters and gradient information and uploads them to the park-side collaborative optimization platform through a secure communication network.

[0011] S3 hierarchical federated aggregation steps: The park-side collaborative optimization platform first uses the data distribution statistics uploaded by each enterprise to complete the enterprise clustering and grouping through its self-developed algorithm. Then, it performs weighted aggregation of the model parameters of enterprises within the same group to generate a group-specific optimization model. Finally, it performs global weighted fusion of all group-specific optimization models to generate a global optimization model that is adapted to the overall zero-carbon goal of the park.

[0012] S4 Strategy Distribution and Closed-Loop Training Steps: The park-side collaborative optimization platform distributes the group-specific optimization model to the corresponding enterprise-side terminals, synchronizes the global optimization model to all enterprise-side terminals, completes the local model update of enterprises, and forms a closed-loop training cycle of local training - parameter upload - hierarchical aggregation - strategy distribution until the model converges;

[0013] S5 online optimization deployment steps: Deploy the converged optimization model to each enterprise-side terminal, collect enterprise energy operation status data in real time, and output the optimal energy dispatch control command through the optimization model to achieve minute-level real-time energy optimization control.

[0014] In step S1, the zero-carbon oriented multi-objective reward function is a creatively developed formula, expressed as follows: ;

[0015] in, For instant rewards, This is the economic cost item for purchasing electricity under time-of-use pricing. For energy loss or lifespan loss, For the satisfaction item, The carbon emission cost is calculated based on the real-time grid carbon emission factor. The carbon weight coefficient is used as the first formula to integrate carbon emission costs with economic, equipment, and satisfaction costs in a hierarchical federated reinforcement learning scenario for energy optimization in a park. This directly drives the federated training process to simultaneously achieve park economic cost optimization and zero-carbon goals.

[0016] In step S3, the distribution-adaptive clustering algorithm employs a creative, self-developed clustering objective formula, using JS divergence as a distance metric for the distribution of energy data among enterprises. The expression is: ;

[0017] in, , For the first Clusters, For the first The center of each cluster, The total number of clusters; this formula is designed for the heterogeneous data characteristics of non-independent and identically distributed enterprises in the park.

[0018] The expression for the JS divergence is:

[0019] Where enterprise data distribution statistics are assumed to be ,make .

[0020] in For probability distribution With probability JS divergence between them For probability distribution With probability KL divergence between them For probability distribution With probability KL divergence between them For probability distribution With probability KL divergence between them For distribution In taking values The probability at that location. For distribution In taking values The probability at that location.

[0021] In step S3, the intra-group weighted aggregation uses the amount of local training data of each enterprise in the group as the weight to perform a weighted average of the model parameters uploaded by enterprises in the same group, and generates a group-specific optimization model that is adapted to the energy consumption characteristics of enterprises of the same type.

[0022] In step S3, the global weighted fusion uses the total amount of training data in each group as the weight to perform a weighted average on all the dedicated optimization models of the groups, generating a global optimization model that takes into account both the overall collaborative optimization of the park and the zero-carbon goal.

[0023] In step S1, the local energy optimization agent uses the PPO algorithm to complete the local model training, wherein the Actor network outputs control actions and the Critic network evaluates the value of the current system state.

[0024] A hierarchical federated reinforcement learning energy optimization system for zero-carbon industrial parks includes an enterprise-side terminal, a park-side collaborative optimization platform, and a secure communication network. The enterprise-side terminal includes a data acquisition module, a local reinforcement learning agent, and a model training and parameter uploading module, used to execute the local training, parameter uploading, and real-time control steps. The park-side collaborative optimization platform includes an enterprise clustering module, a federated aggregation engine, and a policy distribution module, used to execute the clustering grouping, hierarchical aggregation, and policy distribution steps. The secure communication network is communicatively connected to both the enterprise-side terminal and the park-side collaborative optimization platform, used for encrypted transmission of model parameters and aggregation policies.

[0025] The advantages of this invention are: compared with the prior art, the design scheme and method provided by this invention take "data does not move, model moves" as the core principle, construct a hierarchical federated aggregation architecture based on enterprise type clustering, realize distributed autonomous optimization by deploying local reinforcement learning agents on the enterprise side, establish a collaborative optimization platform that takes into account privacy protection and knowledge fusion on the park side, and design a multi-objective reward function that integrates real-time carbon cost, thereby fundamentally solving the contradiction between data privacy and global optimization. Attached Figure Description

[0026] Figure 1 This is a flowchart of the present invention;

[0027] Figure 2 This is a step diagram of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Figure 1 This is the main flowchart of the design scheme of this invention. For example... Figure 1 As shown, the hierarchical federated reinforcement learning energy optimization method for zero-carbon parks proposed in this invention includes the following components:

[0030] (1) Enterprise-side terminal, comprising the following components:

[0031] Data acquisition module: Collects real-time data from within the enterprise.

[0032] Local reinforcement learning agent: A reinforcement learning model employing an Actor-Critic architecture. The Actor network outputs control actions based on the current state, while the Critic network evaluates the value of the current state.

[0033] Model training and parameter upload module: This module enables offline or online training using local data. After each training round, the model parameters are encrypted and uploaded to the campus platform via a secure communication network.

[0034] (2) The park-side collaborative optimization platform includes the following components:

[0035] Enterprise Clustering Module: Regularly receives data distribution statistics uploaded by enterprises. This data is not the original data. It uses methods such as JS divergence to calculate the similarity of data distribution among enterprises, and then uses clustering algorithms such as K-means to divide the enterprises into several groups.

[0036] The federated aggregation engine specifically includes the following components:

[0037] Group Aggregator: For each enterprise group, federated aggregation algorithms such as FedAvg are used to perform a weighted average of the model parameters uploaded by all enterprises in the group to generate a dedicated optimized model for that group.

[0038] Global Aggregator: Based on the overall goals of the park, the various group models are merged again to form a global optimization model.

[0039] Policy distribution module: Distributes the updated group model or global model to terminal devices within the corresponding group or all enterprises to update their local models.

[0040] (3) Secure communication network: responsible for encrypted data transmission between the enterprise and the park platform, ensuring the confidentiality and integrity of the uploaded model parameters and the distributed aggregation strategies during the transmission process.

[0041] Figure 2 This is a diagram illustrating the method steps of the present invention. For example... Figure 2 As shown, the hierarchical federated reinforcement learning energy optimization method for zero-carbon parks proposed in this invention includes the following steps:

[0042] (1) Local training step: The enterprise-side terminal trains the local model based on the reinforcement learning algorithm, and then uses local data to optimize its energy equipment control strategy. In this step, a multi-objective federated reinforcement learning reward function oriented towards the zero-carbon goal is innovatively designed, which optimizes the economic cost of the park while quantifying and coordinating the reduction of carbon emissions. Each enterprise Using its local dataset, and leveraging the system's local reinforcement learning agent with an Actor-Critic architecture, such as the PPO algorithm, the system is trained, where the PPO training formula is:

[0043] (1) The formula for PPO clipped surrogate objective is:

[0044]

[0045] in For parameters The PPO loss function value under the following conditions For all time Seeking expectations, For time Importance sampling ratio In time Estimating the advantage of the current action relative to the average action. This is a hyperparameter for the cropping range. For parameters strategy, These are the parameters of the current policy network. For parameters strategy, These are the parameters for the old policy network. For time The actions taken by the company For time The status of the enterprise.

[0046] (2) The commonly used formula for total Actor-Critic loss is:

[0047]

[0048] in For parameters and parameters The total loss function value under the following conditions The value loss coefficient, For value networks State The estimated value, For time steps Target value The entropy coefficient is used to control the weight of entropy reward in the total loss. Entropy.

[0049] The local reinforcement learning agent runs the PPO reinforcement learning algorithm, updates the Actor and Critic network parameters, and is responsible for generating continuous or discrete control action commands based on the current system state observations, such as BES charging and discharging commands and adjustable load adjustment commands. This ensures continuous policy improvement and optimization convergence while maintaining algorithm stability. (The enterprise...) The action vector formula is:

[0050]

[0051] in For the first individual companies in time The action taken to generate power, The charging and discharging power of the battery. The adjustment amount is for the adjustable load. Adjustment amount for charging and discharging power of electric vehicles.

[0052] The enterprise then observes the state at each time step t. Select Action Receive instant rewards after interacting with the environment. The local Actor and Critic network parameters are updated using the experience trajectory. The reward function integrates economic costs, equipment depreciation, production satisfaction, and carbon emission costs; the reward function is defined as:

[0053]

[0054] in For instant rewards, This is the economic cost item for purchasing electricity under time-of-use pricing. For energy loss or lifespan loss, For the satisfaction item, The carbon emission cost is calculated based on the real-time grid carbon emission factor. This is the carbon weighting coefficient.

[0055] (1) Economic cost of purchasing electricity under time-of-use pricing The formula is:

[0056]

[0057] in For the time change, For time steps Purchase electricity For time steps Purchase power from the grid. For time Electricity sales price, For time steps Selling power to the grid.

[0058] (2) Energy loss or lifespan loss The formula is:

[0059]

[0060] in The degradation cost coefficient per unit energy throughput. For time steps Battery charging power, For time steps The battery discharge power.

[0061] (3) Satisfaction Item The formula is:

[0062]

[0063] in This is a weighting coefficient for task completion. For time steps Task completion progress, This is a weighting factor for temperature comfort. For time steps The actual indoor temperature The target temperature is set.

[0064] (4) Carbon emission cost calculated based on real-time grid carbon emission factor The formula is:

[0065]

[0066] in This is the real-time carbon emission factor of the power grid.

[0067] The local reinforcement learning agent processes all raw energy consumption and production data used in training and stores them locally on the enterprise's terminal. Then, by exchanging intermediate model parameters instead of raw data, it fundamentally solves the data security and privacy concerns of enterprises when participating in park collaborative optimization. Enterprises can also train reinforcement learning agents locally using their own fine-grained data to learn and optimize control strategies such as energy storage charging and discharging and adjustable load.

[0068] (2) Parameter Upload: After completing one or more local training cycles, enterprises use the system's model training and parameter upload module to encrypt the model gradients or policy parameters of their Actor-Critic models and then upload them to the park's collaborative optimization platform. Only the model gradients or policy parameters are uploaded; the enterprise's original sensitive data is not transmitted over the network. The park's central server only receives and aggregates the model weights or gradient information uploaded by each enterprise and updates the global or grouped models using an averaging algorithm. The enterprise's original energy consumption data, production plans, and other sensitive information are always kept locally, meeting data security and legal requirements, and forming a system for each time step. The RL interaction loop involves observing the state, selecting an action, receiving an immediate reward, and updating the local Actor / Critic parameters using the experience trajectory. The reinforcement learning interaction formula for local training on the enterprise side is as follows:

[0069]

[0070] in The state of the environment in the next moment. Let be the state transition probability function.

[0071] (3) Grouping and Aggregation: This step pioneered a hierarchical federated aggregation architecture based on enterprise type clustering, solving the performance degradation problem of traditional federated learning when processing heterogeneous enterprise data within the park. This method includes a three-layer structure: enterprise layer, grouping layer, and global layer. The enterprise layer is trained locally using the PPO algorithm reinforcement learning, uploading only the model gradient or policy parameters to ensure the privacy of enterprise data; the grouping layer uses cluster analysis and methods such as JS divergence to calculate the similarity of data distribution among enterprises, and then uses clustering algorithms such as K-means to divide enterprises into several groups, and performs parameter aggregation within the groups to generate dedicated optimization models for specific enterprise types such as high-energy-consuming manufacturing, data centers, and commercial buildings; the global layer is responsible for integrating the aggregation models of each group to form a final scheduling strategy that takes into account the overall goals of the park, achieving a balance between global and local optimization. The steps specifically include:

[0072] 3.1: Aggregation sub-step: Based on the historical or synchronously uploaded data distribution statistics of each enterprise, the similarity measure of JS divergence and the K-means clustering algorithm are used to group enterprises with similar data distribution characteristics into the same group. The relevant formula is as follows:

[0073] (1) The formula for JS divergence as a measure of the distribution similarity of firms (i,j) is:

[0074]

[0075] Where enterprise data distribution statistics are assumed to be ,make .

[0076] in For probability distribution With probability JS divergence between them For probability distribution With probability KL divergence between them For probability distribution With probability KL divergence between them For probability distribution With probability KL divergence between them For distribution In taking values The probability at that location. For distribution In taking values The probability at that location.

[0077] (2) Using JS as the distance K-means clustering objective formula is:

[0078]

[0079] in , For the first Clusters, For the first The center of each cluster, The total number of clusters.

[0080] Similarity is calculated based on enterprise load and photovoltaic distribution characteristics, and JS-KMeans is used to divide enterprises into several groups to visualize enterprise clustering images.

[0081] 3.2: Aggregation Sub-step: For each group, a weighted federated average algorithm is used to aggregate the model parameters of all enterprises within the group, forming a dedicated optimization model for that group. Using the system's group aggregator, the park platform receives data from... After determining the model parameters for each enterprise, clustering is first performed using pre-stored or recently updated enterprise data distribution information. The aggregation model parameters for the k-th group are then used. Calculated using the following formula:

[0082]

[0083] in For enterprises The amount of data, For the first The total amount of data from all companies within the group. For the first A set of groups It is the first Local model weights for each member.

[0084] For each enterprise group, a federated aggregation algorithm such as FedAvg is used to perform a weighted average of the model parameters uploaded by all enterprises within the group. This weighted averaging method takes into account the differences in data size among different enterprises. The standard weighting formula of FedAvg is:

[0085]

[0086] in For the first The aggregation model parameters for each group For the first The total amount of data from all companies within the group. For the first Local model parameters of each member

[0087] (4) Strategy Distribution: The park platform utilizes the system's global aggregator and, based on the overall goals of the park, performs secondary fusion of the parameters of each group optimization model, and then performs a weighted average of the group models to generate a unified global optimization model. The final strategy is thus formed, and the global weighted fusion formula is as follows:

[0088]

[0089] in These are global model parameters. The total number of groups, For the first The aggregate weight of the group This represents the total amount of data across all groups.

[0090] The park platform will then update the grouping model parameters. These parameters are sent to all companies within the group, who then use them to update their local models in preparation for the next round of local training, forming a "local training-upload-aggregation-distribution" loop. This loop will continue until the model converges. The first step... The conceptual formula for the cycle of a wheel is:

[0091] in For the first The company in the The strategy parameters of the wheel, For the first The company in the The value parameters of the wheel For the first The strategy parameters of each enterprise after local training is completed. For the first The value parameters of an enterprise after local training. For the first The strategy parameters uploaded by all members in the group. For the first The global strategy parameters obtained after the group is aggregated.

[0092] (5) Online Deployment: After training, the final optimized model is deployed on the enterprise-side terminal. During real-time operation, the enterprise uses the data acquisition module to collect real-time status data, including but not limited to photovoltaic output, basic production load, energy storage SOC status, adjustable loads such as production line power and air conditioning power, and electric vehicle charging power, to monitor the status of internal energy equipment and loads. The enterprise then uses the collected data to call the trained model and directly outputs the optimal energy dispatch command, thereby achieving minute-level real-time optimization control. The state vector formula is:

[0093]

[0094] in Photovoltaic power generation capacity, For load power energy storage, The state of charge of the energy storage system. For adjustable load power, Charging power for electric vehicles.

[0095] Key points of the invention

[0096] This invention pioneers a hierarchical federated aggregation architecture based on enterprise type clustering, solving the performance degradation problem of traditional federated learning when processing heterogeneous enterprise data within a park. The method comprises a three-layer structure: an enterprise layer, a grouping layer, and a global layer. The enterprise layer is trained locally using reinforcement learning, such as the PPO algorithm, uploading only model gradients or policy parameters to ensure the privacy of enterprise data. The grouping layer uses K-means clustering based on JS divergence to group enterprises with similar data distributions and aggregates parameters within each group, generating specialized optimization models for specific enterprise types such as high-energy-consuming manufacturing, data centers, and commercial buildings. The global layer is responsible for fusing the aggregation models from each group to form a final scheduling strategy that considers the overall goals of the park, achieving a balance between global and local optimization.

[0097] This invention proposes a privacy-enhanced collaborative optimization paradigm of "data remains stationary while the model moves." By exchanging intermediate model parameters instead of raw data, it fundamentally addresses enterprises' data security and privacy concerns when participating in collaborative optimization within industrial parks. Enterprises locally train reinforcement learning agents using their own fine-grained data, learning to optimize control strategies such as energy storage charging and discharging, and adjustable load. The park's central server only receives and aggregates the model weights or gradient information uploaded by each enterprise, updating the global or grouped models using a federated averaging algorithm. Throughout the entire process, sensitive information such as enterprises' original energy consumption data and production plans remains locally, meeting data security and legal requirements.

[0098] This invention designs a multi-objective federated reinforcement learning reward function oriented towards zero-carbon goals, enabling industrial parks to quantify and collaboratively reduce carbon emissions while optimizing economic costs. This technical solution innovatively introduces a carbon emission cost term into traditional economic rewards and user or enterprise satisfaction rewards. This design allows the federated learning process to directly drive enterprises to autonomously participate in carbon emission reduction, achieving the industrial park's zero-carbon goal.

Claims

1. A hierarchical federated reinforcement learning energy optimization method for zero-carbon parks, applied to a zero-carbon park energy system including enterprise-side terminals, a park-side collaborative optimization platform, and a secure communication network, comprising local reinforcement learning training, model parameter uploading, and federated aggregation steps, characterized in that, Includes the following steps: S1 Local Privacy Training Steps: Each enterprise-side terminal collects local energy operation and production status data, trains a local energy optimization agent based on the reinforcement learning algorithm of the Actor-Critic architecture, and optimizes the energy control strategy of local energy storage charging and discharging and adjustable load through a self-developed reward function. During the training process, all raw energy consumption and production data are stored locally on the enterprise side, and only the model's policy parameters and gradient information are extracted. S2 parameter encryption transmission steps: After each enterprise-side terminal completes a preset round of local training, it encrypts the extracted model strategy parameters and gradient information and uploads them to the local park-side collaborative optimization platform through a secure communication network. S3 hierarchical federated aggregation steps: The park-side collaborative optimization platform first uses the data distribution statistics uploaded by each enterprise to complete the enterprise clustering and grouping through its self-developed algorithm. Then, it performs weighted aggregation of the model parameters of enterprises within the same group to generate a group-specific optimization model. Finally, it performs global weighted fusion of all group-specific optimization models to generate a global optimization model that is adapted to the overall zero-carbon goal of the park. S4 Strategy Distribution and Closed-Loop Training Steps: The park-side collaborative optimization platform distributes the group-specific optimization model to the corresponding enterprise-side terminals, synchronizes the global optimization model to all enterprise-side terminals, completes the local model update of enterprises, and forms a closed-loop training cycle of local training - parameter upload - hierarchical aggregation - strategy distribution until the model converges; S5 online optimization deployment steps: Deploy the converged optimization model to each enterprise-side terminal, collect enterprise energy operation status data in real time, and output the optimal energy dispatch control command through the optimization model to achieve minute-level real-time energy optimization control.

2. The hierarchical federated reinforcement learning energy optimization method for zero-carbon industrial parks according to claim 1, characterized in that, In step S1, the zero-carbon oriented multi-objective reward function is a creatively developed formula, expressed as follows: ; in, For instant rewards, This is the economic cost item for purchasing electricity under time-of-use pricing. For energy loss or lifespan loss, For the satisfaction item, The carbon emission cost is calculated based on the real-time grid carbon emission factor. The carbon weight coefficient is used as the first formula to integrate carbon emission costs with economic, equipment, and satisfaction costs in a hierarchical federated reinforcement learning scenario for energy optimization in a park. This directly drives the federated training process to simultaneously achieve park economic cost optimization and zero-carbon goals.

3. The hierarchical federated reinforcement learning energy optimization method for zero-carbon industrial parks according to claim 1, characterized in that, In step S3, the distribution-adaptive clustering algorithm employs a creative, self-developed clustering objective formula, using JS divergence as a distance metric for the distribution of energy data among enterprises. The expression is: ; in, , For the first Clusters, For the first The center of each cluster, The total number of clusters; this formula is designed for the heterogeneous data characteristics of non-independent and identically distributed enterprises in the park.

4. The hierarchical federated reinforcement learning energy optimization method for zero-carbon industrial parks according to claim 3, characterized in that, The expression for the JS divergence is: Where enterprise data distribution statistics are assumed to be ,make ; in For probability distribution With probability JS divergence between them For probability distribution With probability KL divergence between them For probability distribution With probability KL divergence between them For probability distribution With probability KL divergence between them For distribution In taking values The probability at that location. For distribution In taking values The probability at that location.

5. The hierarchical federated reinforcement learning energy optimization method for zero-carbon industrial parks according to claim 1, characterized in that, In step S3, the weighted aggregation within the group uses the amount of local training data of each enterprise in the group as the weight to perform a weighted average of the model parameters uploaded by enterprises in the same group, thereby generating a group-specific optimization model that is adapted to the energy consumption characteristics of enterprises of the same type.

6. The hierarchical federated reinforcement learning energy optimization method for zero-carbon industrial parks according to claim 1, characterized in that, In step S3, the global weighted fusion uses the total amount of training data for each group as the weight to perform a weighted average on all the dedicated optimization models of the groups, generating a global optimization model that takes into account both the overall collaborative optimization of the park and the zero-carbon goal.

7. The hierarchical federated reinforcement learning energy optimization method for zero-carbon industrial parks according to claim 1, characterized in that, In step S1, the local energy optimization agent uses the PPO algorithm to complete the local model training, wherein the Actor network outputs control actions and the Critic network evaluates the value of the current system state.

8. An energy optimization system for implementing the method as described in any one of claims 1-7, characterized in that, include: The data acquisition module is deployed on the enterprise-side terminal to collect real-time data on photovoltaic output, production load, energy storage charge status, equipment operation, and new energy output within the enterprise. The local reinforcement learning agent is deployed on the enterprise-side terminal. It is used to output energy equipment control actions based on the collected real-time status data and the PPO reinforcement learning algorithm with the Actor-Critic architecture. It combines a multi-objective reward function that integrates carbon emission costs to complete local model training and optimize energy dispatch strategies such as energy storage charging and discharging and adjustable load adjustment. The model training and parameter upload module is deployed on the enterprise-side terminal. After the model is trained locally, the model strategy parameters and gradient parameters are encrypted and then uploaded to the park-side collaborative optimization platform. The enterprise's original sensitive data is stored locally throughout the process. The enterprise clustering module is deployed on the collaborative optimization platform on the park side. It is used to receive the data distribution statistics information uploaded by enterprises, calculate the data distribution similarity between enterprises using JS divergence, and divide enterprises with similar data distribution characteristics into corresponding groups using the K-means clustering algorithm. The Federated Aggregation Engine, deployed on the collaborative optimization platform on the park side, is used to perform weighted aggregation of model parameters uploaded by enterprises in the same group using the FedAvg Federated Averaging algorithm to generate a group-specific optimization model. Then, the group-specific optimization models are further weighted and fused to generate a global optimization model that is adapted to the overall zero-carbon goal of the park. The strategy distribution module is deployed on the park-side collaborative optimization platform. It is used to distribute the updated group-specific optimization model and global optimization model to the corresponding enterprise-side terminals to complete the iterative update of the enterprise's local model. A secure communication network is connected to both the enterprise-side terminal and the campus-side collaborative optimization platform. It provides an encrypted transmission channel for model parameters and aggregation strategies between the enterprise and campus sides, ensuring the confidentiality and integrity of the transmitted data.

Images