Zero-carbon park source network load storage coordination control method based on deep reinforcement learning algorithm

Abstract

The invention relates to a zero-carbon park source network load storage coordination control method based on a deep reinforcement learning algorithm, and the method comprises the following steps: S1, collecting data in a park in real time, carrying out the standardization, denoising and feature extraction of the collected data, and forming a data set; s2, modeling the environment state space to comprehensively reflect the current state of the system; s3, defining an action space for adjusting the operation state of the system, realizing real-time regulation and control and forming the action space; s4, designing a reward function to realize dynamic balance among different targets; s5, training a strategy network and a value network by adopting an entropy regularization SAC algorithm, and optimizing a control strategy; and S6, in the system operation process, continuously collecting new operation data, updating the strategy network and the value network in real time by using the online learning capability of the SAC algorithm, and dynamically adjusting the control strategy to adapt to environmental changes. According to the invention, efficient utilization of new energy, low-carbon power purchase optimization and dynamic load regulation and control are realized, and the overall operation efficiency of the park is improved.

Description

Technical Field

[0001] This invention relates to the field of energy management and intelligent control technology for zero-carbon industrial parks, and more specifically, to a source-grid-load-storage coordinated control method for zero-carbon industrial parks based on deep reinforcement learning algorithms. Background Technology

[0002] Currently, with the widespread application of renewable energy in industrial parks and power distribution networks, achieving zero-carbon goals has become an important direction for promoting green and low-carbon development. However, the intermittency and uncertainty of new energy sources make it difficult for traditional rule-based energy dispatching methods to achieve efficient, stable, and low-carbon coordinated control when facing complex scenarios such as variable loads, grid fluctuations, and equipment status. Traditional methods typically rely on precise physical models and rule-based algorithms, which are easily limited by the accuracy of the models and their ability to respond to abnormal situations. They cannot adequately address the dynamic balance issues between multiple dimensions of indicators, such as fluctuations in new energy output, changes in load demand, and carbon emission costs.

[0003] In recent years, deep reinforcement learning (DRL) technology has attracted widespread attention due to its adaptive learning and global optimization capabilities, especially the flexible action-evaluation (SAC) algorithm based on entropy regularization. Due to its advantages in handling continuous action spaces and encouraging policy exploration, it has shown high application potential in the field of energy dispatching and optimization. By establishing policy and value networks and utilizing experience replay and soft update mechanisms, the SAC algorithm effectively addresses the sample complexity and local optima problems of traditional reinforcement learning in continuous control problems. This allows for real-time dynamic adjustment of energy system operating strategies, achieving an effective trade-off between economy, low carbon emissions, and safety.

[0004] However, current research on the coordinated control of source, grid, load and storage in zero-carbon parks mainly focuses on rule-based or model-based optimization methods. When faced with complex and ever-changing actual operating environments, their scheduling strategies often suffer from problems such as slow response, imprecise scheduling or high costs. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a zero-carbon park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm, which realizes dynamic regulation of the multi-energy system in the park and effectively solves the challenges brought about by multiple uncertainties such as the fluctuation of new energy output, changes in load demand and carbon emissions from the power grid.

[0006] The technical solution adopted by this invention to solve its technical problem is: constructing a zero-carbon park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm, including the following steps: S1. Collect data within the park in real time, standardize, denoise, and extract features from the collected data to form a dataset, providing support for subsequent state space modeling; S2. Model the environmental state space to comprehensively reflect the current state of the diverse energy systems within the park; S3. Define the action space used for real-time adjustment of the system's operating status; S4. Design a reward function to achieve a dynamic balance among different objectives; S5. Employ entropy-regularized flexible action-evaluation SAC deep reinforcement learning algorithm, train policy network and value network, and optimize control policy. S6. During system operation, new operational data is continuously collected. The online learning capability of the flexible action-evaluation SAC deep reinforcement learning algorithm is utilized to update the policy network and value network in real time, and the control strategy is dynamically adjusted to adapt to environmental changes.

[0007] According to the above scheme, the data collected in S1 within the park includes: active power, reactive power, power fluctuation, electricity price, electricity carbon emission factor, load demand, meteorological information, and equipment operating status of photovoltaic, wind power, energy storage equipment, power grid, and flexible loads.

[0008] According to the above scheme, in S2, the environmental state space is a collection of real-time operating parameters of equipment in the park, including new energy output, energy storage status, load demand and power grid operating parameters. According to the above scheme, the environmental state space includes:

[0009]

[0010] Source-side state variables: Photovoltaic output Wind power output ; Grid-side state variables: Power purchased by the grid Real-time electricity price Carbon emission factors Voltage level ; Load-side state variables: Total load demand Adjustable load ; Energy storage state variables: Energy storage SOC Energy storage charging power Energy storage and discharge power ; Environment state variables: current timestamp Carbon emission trading price .

[0011] According to the above scheme, the action space in S3 includes: .

[0012] According to the above scheme, in step S4, the reward function The calculation formula is:

[0013] In the formula, To optimize the power purchase strategy based on economic incentives for power purchase costs and benefits, with the goal of minimizing the park's operating costs; As a low-carbon incentive, its goal is to reduce emissions from electricity purchases and optimize electricity purchase strategies; As a security reward, its goal is to ensure that the system operates within security constraints, which is achieved through penalty items; The expressions for the three types of rewards are:

[0014]

[0015]

[0016] In the formula, The operating cost of the energy storage system; and These are penalty weighting factors corresponding to different security constraints; This is the maximum permissible SOC value for the energy storage system to prevent overcharging; This is the upper limit of the stable operating range of the voltage, to prevent the voltage from exceeding the limit.

[0017] According to the above scheme, in S5, the network architecture of the Flexible Action-Evaluation SAC deep reinforcement learning algorithm includes: a dual Q-value network, a target Q-value network, and a policy network; The dual-Q value network consists of two identical value networks used to evaluate the value of performing a specific action in the current state. The target Q-network: the parameters are slowly updated using a "soft update" method to stabilize training and avoid excessive oscillations in Q-value estimation; The strategy network Actor takes the current environmental state as input and outputs the optimal control strategy, namely the charging and discharging power of energy storage devices, the power purchase ratio of the grid, and the load regulation strategy. It belongs to the decision problem in the continuous action space.

[0018] According to the above scheme, the Flexible Action-Evaluation SAC deep reinforcement learning algorithm is trained using an experience replay mechanism to improve sample utilization and entropy regularization to enhance exploration capabilities, thereby optimizing the control strategy of the zero-carbon park. The specific training process is as follows: Experience collection: Based on the current strategy network during the actual operation of the park. Sample the environmental state, actions, and corresponding reward values ​​to form ( , , , And store it in experience pool D; in, Environmental state; To control actions; This is the reward value; The state at the next moment after the action is performed; Policy network optimization: The goal of the policy network is to maximize the weighted sum of the expected Q-value of its output action and the policy entropy. The optimization objective function is as follows:

[0019] In the formula, Represents policy entropy, and the entropy term is the core of SAC, used to encourage policy exploration; The temperature coefficient controls the degree of entropy's influence, enhancing the strategy's exploration capabilities. The value network update expression is:

[0020] The Q-network update uses a soft update mechanism to update the target network parameters, thereby improving the stability of Q-value estimation. The update method is as follows:

[0021] In the formula, The weights are updated softly to smooth the Q-value updates and prevent policy instability caused by excessively rapid gradient updates.

[0022] This invention also provides a zero-carbon industrial park source-grid-load-storage coordinated control device based on a deep reinforcement learning algorithm, comprising: The data acquisition and preprocessing module is used to collect data in the park in real time, standardize, denoise and extract features from the collected data to form a dataset, which provides support for subsequent state space modeling. The environmental state space modeling module is used to model the environmental state space to comprehensively reflect the current state of the system. The action space definition module is used to adjust the system's operating state, achieve real-time control, and constitute the action space; Design a reward function module to achieve dynamic balance among different objectives; The SAC algorithm training module is used to train the policy network and value network and optimize the control policy by employing the entropy-regularized flexible action-evaluation SAC deep reinforcement learning algorithm. The online learning and policy update module is used to continuously collect new operational data during system operation. It utilizes the online learning capability of the Flexible Action-Evaluation SAC deep reinforcement learning algorithm to update the policy network and value network in real time, and dynamically adjust the control policy to adapt to environmental changes.

[0023] The source-grid-load-storage coordinated control method of the zero-carbon park based on deep reinforcement learning algorithm of the present invention has the following beneficial effects: 1. This invention proposes an efficient, intelligent, and adaptive zero-carbon energy park source-grid-load-storage coordinated control technology based on the Flexible Action-Evaluation (SAC) deep reinforcement learning algorithm. Compared with traditional rule-based or optimization algorithm-based scheduling methods, it has significant advantages in terms of renewable energy utilization, low-carbon power purchase optimization, load dynamic regulation, and system security. Through the adaptive capability of deep reinforcement learning, it achieves efficient decision-making in complex and ever-changing environments, breaking through the limitations of traditional energy scheduling methods that rely on fixed optimization objectives and artificially set rules. It can dynamically adapt to the fluctuations in renewable energy output and the real-time changes in the carbon emission factor of the power grid. The SAC algorithm uses the entropy regularization mechanism to enhance the exploration capability of decision-making strategies, enabling it to find the optimal control scheme under different operating conditions, thereby improving the overall operating efficiency of the park. 2. This invention achieves a dynamic balance between economy, low carbon emissions, and safety through a multi-objective optimization mechanism. It comprehensively considers the optimization of new energy utilization rate, grid electricity purchase carbon emissions, and energy storage system operation costs. While ensuring the economic operation of the park's energy system, it effectively reduces carbon emissions and improves the level of new energy consumption. It adopts an online learning and strategy update mechanism, which enables the control strategy to be continuously optimized with environmental changes. Combined with the experience playback mechanism and continuous interactive learning capability of the SAC algorithm, it realizes the real-time update of the strategy network and value network, enabling the system to continuously learn from historical experience and improve the stability and reliability of long-term operation. 3. This invention combines deep reinforcement learning technology to achieve efficient utilization of new energy sources, dynamic load regulation, optimization of carbon emissions from power grid purchases, and system security assurance. It breaks through the limitations of traditional methods and provides a brand-new solution for intelligent energy management in zero-carbon parks, with broad application prospects and significant socio-economic value. Attached Figure Description

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to the present invention. Detailed Implementation

[0025] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] Example 1 like Figure 1 As shown, the zero-carbon park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm of the present invention includes the following steps: S1. Data Acquisition and Preprocessing: Real-time acquisition of data on active power, reactive power, power fluctuation, electricity price, carbon emission factor, load demand, meteorological information, and equipment operating status of photovoltaic, wind power, energy storage equipment, power grid, and flexible loads within the park. The acquired data is standardized, denoised, and feature extracted to form a high-quality dataset.

[0027] Data acquisition includes real-time acquisition of operational data from various devices within the park and external environmental information to ensure the accuracy of environmental status. The data is also standardized, denoised, and feature extracted to ensure the accuracy of the input data for the reinforcement learning model, providing basic data support for reinforcement learning training and decision-making.

[0028] S2. Environmental State Space Modeling: The environmental state space includes the real-time operating parameters of various devices, such as new energy output, energy storage status, load demand, and power grid operating parameters, to comprehensively reflect the current state of the system. The state space includes:

[0029]

[0030] Source-side state variables: Photovoltaic output Wind power output .

[0031] Grid-side state variables: Power purchased by the grid Real-time electricity price Carbon emission factors Voltage level .

[0032] Load-side state variables: Total load demand Adjustable load .

[0033] Energy storage state variables: Energy storage SOC Energy storage charging power Energy storage and discharge power .

[0034] Environment state variables: current timestamp Carbon emission trading price .

[0035] Taking data from a specific moment as an example, =320kW =180kW =980kW =0.62 yuan / kWh =0.70kg / kWh, SOC=0.50, =0.20 yuan / kg.

[0036] This state space comprehensively considers renewable energy output, load demand, grid power purchase, energy storage operation, and environmental factors, providing an accurate state description for reinforcement learning and ensuring the model's generalization ability under different operating environments. Constructing a state space that comprehensively and accurately describes the current operating state of the system is fundamental for reinforcement learning agents to make correct decisions. The agent refers to an autonomous learning decision-making system built using the SAC algorithm. Its core function is to perceive environmental states, output control strategies, and continuously optimize based on reward feedback in complex and ever-changing energy system environments.

[0037] S3. Action Space Definition: Define control variables, including the charging and discharging power of energy storage devices, the power purchase ratio of the grid, and flexible load regulation schemes, to adjust the system operating status, achieve real-time regulation, and constitute the action space. The motion space includes:

[0038] Pick =180kW =40kW, the load after flexible load adjustment is 980-40=940kW =940-320-180-180=260kW =0.

[0039] Precisely defining the regulatory behavior of controllable units within the park constitutes a continuous action space, facilitating fine-grained control by the SAC algorithm. The SAC algorithm is particularly well-suited for handling such continuous action spaces, outputting precise control values ​​rather than discrete commands, which is crucial for the smooth and efficient control of energy systems.

[0040] S4. Reward Function Design: Economic operating costs, carbon emissions, load balancing, and safe operation constraints are incorporated into the reward function through penalty terms to achieve dynamic balance among different objectives.

[0041] reward function The calculation formula is:

[0042] In the formula, To optimize the electricity purchase strategy based on economic rewards derived from electricity purchase costs and revenues, the goal is to minimize the park's operating costs. These costs primarily consist of electricity purchase costs and the operation and maintenance costs of energy storage equipment. Since the objective of reinforcement learning is to maximize cumulative rewards, cost terms typically appear as negative values. As a low-carbon incentive, its goal is to reduce emissions from electricity purchases and optimize electricity purchase strategies; The safety reward mechanism aims to ensure the system operates within safety constraints, achieved through a penalty system. When the system state exceeds a safety threshold, a significant negative reward is given to prevent abnormal situations such as overload, undervoltage, and overcharging of energy storage.

[0043] The expressions for its three rewards are:

[0044]

[0045]

[0046] In the formula, The operating cost of the energy storage system; and These are penalty weighting factors corresponding to different security constraints; This is the maximum permissible SOC value for the energy storage system to prevent overcharging; This is the upper limit of the stable operating range of the voltage, to prevent the voltage from exceeding the limit.

[0047] Pick =0.01 yuan / kg, the value is... =-163.00、 =-234、 =0, take = = =1, then =-397.

[0048] reward function The design of the agent directly determines the learning objectives and the final control effect, and is the core of the entire reinforcement learning framework. S5 and SAC algorithm training: The entropy-regularized flexible action-evaluation (SAC) deep reinforcement learning algorithm is used to train the policy network and value network and optimize the control policy.

[0049] SAC (Strategy-Assisted Reinforcement) is a deep reinforcement learning method based on policy optimization. It has stronger exploration capabilities, a more stable training process, and is more suitable for continuous control. Through the mechanism of maximum entropy reinforcement learning, SAC maintains policy diversity while optimizing policies, avoids the agent from converging to suboptimal solutions too early, and improves exploration efficiency and decision robustness.

[0050] The network architecture of the SAC algorithm includes: two value networks and one policy network. Policy network (Actor): The input is the current environmental state, and the output is the optimal control strategy, namely the charging and discharging power of energy storage equipment, the power purchase ratio of the grid, and the load regulation strategy. It belongs to the decision problem of continuous action space.

[0051] Dual Q-value networks (Critic-1 and Critic-2): Two identical value networks used to evaluate the value (Q-value) of performing a specific action in the current state. By employing a dual Q-value network design, the smaller of the two values ​​is used when calculating the target Q-value, which effectively mitigates the problem of overestimation of the Q-value, thereby improving training stability.

[0052] Target Critic: This is a copy of the Critic network, but its parameters are updated slowly using a "soft update" method to stabilize training and avoid excessive oscillations in Q-value estimation.

[0053] The control policy is trained using the state space, action space, and reward function defined in the preceding steps. The SAC algorithm is employed, whose core advantages lie in its stability and sample efficiency when handling continuous control problems.

[0054] The SAC algorithm employs an experience replay mechanism during training to improve sample utilization and uses entropy regularization to enhance its exploration capabilities, avoiding getting trapped in local optima and optimizing the control strategy for zero-carbon parks. The specific training process is as follows: Experience collection: Based on the current strategy network during the actual operation of the park. Sample the environmental state, actions, and corresponding reward values ​​to form ( , , , And store it in experience pool D.

[0055] in, Environmental state; To control actions; This is the reward value; The state at the next moment after the action is performed. , , For the vector in the above steps, after the execution is complete, This is the new state.

[0056] The above steps are used to obtain The experience tuple is the core data structure in the SAC algorithm training process. This experience tuple comprehensively describes the changes in system response and reward caused by executing control actions in the current state, and serves as the basis for updating the parameters of the policy network and value network.

[0057] Policy network optimization: The goal of the policy network is to maximize the weighted sum of the expected Q-value of its output action and the policy entropy. The optimization objective function is as follows:

[0058] In the formula, Represents policy entropy, and the entropy term is the core of SAC, used to encourage policy exploration; The temperature coefficient controls the degree of entropy's influence, enhancing the strategy's exploration capabilities.

[0059] The value network update expression is:

[0060] The Q-network update uses a soft update mechanism to update the target network parameters, thereby improving the stability of Q-value estimation. The update method is as follows:

[0061] In the formula, The weights are updated softly to smooth the Q-value updates and prevent policy instability caused by excessively rapid gradient updates.

[0062] The SAC deep reinforcement learning algorithm is used to train the policy network and value network to optimize the source-network-load-storage coordination control strategy. Entropy regularization is employed to enhance policy exploration capabilities and avoid getting trapped in local optima. A two-Q network learning method is used to improve the algorithm's convergence speed and make the scheduling strategy more stable.

[0063] SAC's policy process is a continuous control optimization process based on maximum entropy reinforcement learning. At each decision time t, the current state is received. The policy network generates control decisions, i.e., actions, based on the current state. This includes the charging and discharging power of energy storage devices, the power purchased by the grid, and the power of flexible load regulation. The strategy uses a Gaussian distribution for sampling and adjusts its mean and variance through trainable strategy parameters, enabling the decision-making process to be both exploratory and stably convergent.

[0064] S6. Online Learning and Policy Update: During system operation, new operational data is continuously collected. The online learning capability of the SAC algorithm is used to update the policy network and value network in real time, and the control strategy is dynamically adjusted to adapt to environmental changes.

[0065] Online learning and policy updates via the SAC algorithm are key to achieving model adaptability. During system operation, new operational data is continuously collected. This new data constitutes new empirical tuples (...). , , , These data are continuously fed into the experience pool D. Through the online learning capability of the SAC algorithm, the model uses this latest data to fine-tune and update the parameters of the policy network and value network in real time and continuously. This process enables the control strategy to dynamically adapt to various unforeseen changes, such as sudden changes in renewable energy output, temporary changes in load demand, and fluctuations in electricity prices or carbon emission factors. Ultimately, the trained policy network outputs optimal control decisions, including energy storage charging and discharging scheduling, grid power purchase optimization, and flexible load regulation strategies, which are then distributed to the park's control system for execution, thereby achieving intelligent, low-carbon, and efficient operation of the entire park's energy system. Based on the trained policy network, the optimal control decisions are output, including energy storage charging and discharging scheduling, grid power purchase ratio optimization, and load regulation strategies, and deployed to the park's source-grid-load-storage system to achieve intelligent, low-carbon, and efficient operation.

[0066] Example 2 This invention also provides a zero-carbon industrial park source-grid-load-storage coordinated control device based on a deep reinforcement learning algorithm, comprising: The data acquisition and preprocessing module is used to collect data in the park in real time, standardize, denoise and extract features from the collected data to form a dataset, which provides support for subsequent state space modeling. The environmental state space modeling module is used to model the environmental state space to comprehensively reflect the current state of the system. The action space definition module is used to adjust the system's operating state, achieve real-time control, and constitute the action space; Design a reward function module to achieve dynamic balance among different objectives; The SAC algorithm training module is used to train the policy network and value network and optimize the control policy by employing the entropy-regularized flexible action-evaluation SAC deep reinforcement learning algorithm. The online learning and policy update module is used to continuously collect new operational data during system operation. It utilizes the online learning capability of the Flexible Action-Evaluation SAC deep reinforcement learning algorithm to update the policy network and value network in real time, and dynamically adjust the control policy to adapt to environmental changes.

[0067] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for coordinated control of source-grid-load-storage systems in zero-carbon industrial parks based on deep reinforcement learning algorithms, characterized in that, Includes the following steps: S1. Collect data within the park in real time, standardize, denoise, and extract features from the collected data to form a dataset; S2. Model the environmental state space based on the dataset to comprehensively reflect the operating status of the diverse energy systems within the park; S3. Define the action space used for real-time adjustment of the system's operating status; S4. Design a reward function to achieve a dynamic balance among different objectives; S5. Employ entropy-regularized flexible action-evaluation SAC deep reinforcement learning algorithm, train policy network and value network, and optimize control policy. S6. During system operation, new operational data is continuously collected. The online learning capability of the flexible action-evaluation SAC deep reinforcement learning algorithm is utilized to update the policy network and value network in real time, and the control strategy is dynamically adjusted to adapt to environmental changes.

2. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 1, characterized in that, In S1, the data collected within the park includes: active power, reactive power, power fluctuation, electricity price, carbon emission factor, load demand, meteorological information, and equipment operating status of photovoltaic, wind power, energy storage equipment, power grid, and flexible loads.

3. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 2, characterized in that, In S2, the environmental state space is a collection of real-time operating parameters of equipment within the park, including new energy output, energy storage status, load demand, and power grid operating parameters.

4. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 3, characterized in that, The model of the environmental state space is as follows: Source-side state variables: Photovoltaic output Wind power output ; Grid-side state variables: Power purchased by the grid Real-time electricity price Carbon emission factors Voltage level ; Load-side state variables: Total load demand Adjustable load ; Energy storage state variables: Energy storage SOC Energy storage charging power Energy storage and discharge power ; Environment state variables: current timestamp Carbon emission trading price .

5. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 4, characterized in that, In S3, the action space is: 。 6. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 5, characterized in that, In S4, the reward function The calculation formula is: In the formula, To optimize the power purchase strategy based on economic incentives for power purchase costs and benefits, with the goal of minimizing the park's operating costs; As a low-carbon incentive, its goal is to reduce emissions from electricity purchases and optimize electricity purchase strategies; As a security reward, its goal is to ensure that the system operates within security constraints, which is achieved through penalty items; The expressions for the three types of rewards are: In the formula, The operating cost of the energy storage system; and These are penalty weighting factors corresponding to different security constraints; This is the maximum permissible SOC value for the energy storage system to prevent overcharging; This represents the upper limit of the voltage stable operating range.

7. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 1, characterized in that, In S5, the network architecture of the Flexible Action-Evaluation SAC deep reinforcement learning algorithm includes: a dual Q-value network, a target Q-network, and a policy network; The dual-Q value network consists of two identical value networks used to evaluate the value of performing a specific action in the current state. The target Q-network uses a "soft update" method to slowly update its parameters for stable training, avoiding excessive oscillations in Q-value estimation. The strategy network Actor takes the current environmental state as input and outputs the optimal control strategy, namely the charging and discharging power of energy storage devices, the power purchase ratio of the grid, and the load regulation strategy. It belongs to the decision problem in the continuous action space.

8. The zero-carbon industrial park source-grid-load-storage coordinated control method based on deep reinforcement learning algorithm according to claim 7, characterized in that, Flexible Actions - The SAC deep reinforcement learning algorithm is trained using an experience replay mechanism to improve sample utilization and entropy regularization to enhance exploration capabilities, thereby optimizing the control strategy for zero-carbon parks. The specific training process is as follows: Experience collection: Based on the current strategy network during the actual operation of the park. Sample the environmental state, actions, and corresponding reward values ​​to form ( , , , And store it in experience pool D; in, Environmental state; To control actions; This is the reward value; The state at the next moment after the action is performed; Policy network optimization: The goal of the policy network is to maximize the weighted sum of the expected Q-value of its output action and the policy entropy. The optimization objective function is as follows: In the formula, Represents policy entropy, and the entropy term is the core of SAC, used to encourage policy exploration; The temperature coefficient controls the degree of entropy's influence, enhancing the strategy's exploration capabilities. The value network update expression is: The Q-network update uses a soft update mechanism to update the target network parameters, thereby improving the stability of Q-value estimation. The update method is as follows: In the formula, The weights are updated softly with smooth Q-value updates.

9. A zero-carbon industrial park source-grid-load-storage coordinated control device based on deep reinforcement learning algorithm, characterized in that, include: The data acquisition and preprocessing module is used to collect data in the park in real time, standardize, denoise and extract features from the collected data to form a dataset, which provides support for subsequent state space modeling. The environmental state space modeling module is used to model the environmental state space to comprehensively reflect the current state of the system. The action space definition module is used to adjust the system's operating state, achieve real-time control, and constitute the action space; Design a reward function module to achieve dynamic balance among different objectives; The SAC algorithm training module is used to train the policy network and value network and optimize the control policy by employing the entropy-regularized flexible action-evaluation SAC deep reinforcement learning algorithm. The online learning and policy update module is used to continuously collect new operational data during system operation. It utilizes the online learning capability of the Flexible Action-Evaluation SAC deep reinforcement learning algorithm to update the policy network and value network in real time, and dynamically adjust the control policy to adapt to environmental changes.

10. An electronic device, characterized in that, include: The processor, communication interface, memory, and communication bus are connected, with the processor, communication interface, and memory communicating with each other via the communication bus. The memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 8.

Images