Cogeneration system control strategy based on AME-TD3 algorithm

Abstract

The invention discloses a combined heat and power generation system control strategy based on an AME-TD3 algorithm, and belongs to the technical field of combined heat and power generation system optimization control. Comprising the following steps: S1, collecting system parameters and initializing a CHP running state; s2, when a target # imgabs0 # value is calculated, an entropy reward mechanism is introduced; s3, realizing exploration and utilization of an adaptive noise adjustment mechanism and a dynamic balance strategy based on Critic network evaluation; and S4, optimizing the control strategy in real time. Aiming at the problem that an existing TD3 algorithm is insufficient in exploratory performance, an entropy correction item is introduced into calculation of a target # imgabs1 # value, structural correction is conducted on a target value function, strategy updating is more efficient, and the effectiveness of overall strategy optimization is improved; aiming at the problem that random noise in a traditional TD3 algorithm cannot adapt to a complex environment, a dynamic self-adjusting noise generation function is added in a strategy updating process, so that the convergence speed is increased, the stability is enhanced, and the self-adaptive adjustment capability of a system to load fluctuation and environmental condition change is enhanced. The method has good expandability.

Description

Technical Field

[0001] The present invention belongs to the technical field of optimization control of cogeneration systems, and in particular relates to a control strategy for a cogeneration system based on the AME-TD3 algorithm. Background Art

[0002] Combined heat and power (CHP) systems are a highly efficient energy utilization method. By simultaneously producing electricity and heat, they improve energy efficiency and reduce carbon emissions, offering significant economic and environmental advantages. However, in practice, different control strategies can significantly impact operational performance. Deep reinforcement learning-based control strategies for CHP systems, while ensuring stable heat and power supply, improving system efficiency, and reducing operating costs, have become a hot topic of research.

[0003] Twin Delayed Deep Deterministic PolicyGradient TD3 introduces dual Mechanisms such as the network, delayed update policy network, and target policy smoothing effectively alleviate problems such as policy over-estimation and training instability in traditional deep deterministic policy gradient algorithms. However, when faced with nonlinear and complex actual energy systems, the random noise in the traditional TD3 algorithm cannot adapt to the complex environment, resulting in insufficient exploration or instability, affecting the algorithm's global optimization capabilities. The operating environment of the CHP system is complex and changeable. The existing TD3 method has difficulty dynamically adjusting its strategy when faced with different working conditions, resulting in insufficient generalization of its control strategy. Therefore, there is an urgent need for a reinforcement learning algorithm that retains the original advantages of TD3 while further improving its policy exploration capabilities, adaptive adjustment capabilities, and convergence stability to meet the needs of complex nonlinear energy system control tasks. Summary of the Invention

[0004] To address the problems faced by the existing TD3 algorithm in CHP system control, such as unstable strategy convergence, insufficient exploration efficiency, and poor adaptability to disturbances, the present invention provides a combined heat and power system control strategy based on the AME (Adaptive Maximum-Entropy)-TD3 algorithm.

[0005] The technical solution adopted by the present invention is: A control strategy for a combined heat and power system based on the AME-TD3 algorithm includes the following steps: S1: Initialization of acquisition system parameters and CHP operating status; S2: Calculation target When the value is , the entropy reward mechanism is introduced; S3: Implementing an adaptive noise regulation mechanism based on critic network evaluation, and exploring and utilizing dynamic balancing strategies; S4: Real-time optimization of control strategy.

[0006] Compared with the prior art, the present invention has the following beneficial effects: The present invention aims to solve the problem of insufficient exploration of the existing TD3 algorithm. An entropy correction term is introduced into the value calculation to structurally modify the objective value function, making policy updates more efficient and improving the effectiveness of overall policy optimization. To address the problem of random noise in the traditional TD3 algorithm being unable to adapt to complex environments, a dynamic, self-adjusting noise generation function is incorporated into the policy update process to speed up convergence and enhance stability, strengthening the system's ability to adapt to load fluctuations and changing environmental conditions. This invention offers excellent scalability and is easily integrated into the CHP system control architecture, achieving economically optimal system operation control.

[0007] The present invention does not directly output deterministic actions, but outputs the mean and standard deviation of the action distribution through the Actor network, thereby providing the system with flexible and diverse control strategies to meet the thermal power demand under complex load conditions while taking into account the economic optimization of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is the overall flow chart of the present invention; Figure 2 It is a specific flow chart of the present invention. DETAILED DESCRIPTION

[0009] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0010] The present invention provides a control strategy for a combined heat and power system based on the AME-TD3 algorithm. Figure 1 As shown, the method includes the following steps: S1: Initialization of acquisition system parameters and CHP operating status; S2: Calculation target When the value is , the entropy reward mechanism is introduced; S3: Implementing an adaptive noise regulation mechanism based on critic network evaluation, and exploring and utilizing dynamic balancing strategies; S4: Real-time optimization of control strategy.

[0011] like Figure 2 The specific implementation steps are as follows: The AME-TD3 algorithm aims to improve the exploration capability and convergence stability of the TD3 algorithm in complex nonlinear energy systems. Its overall process includes agent-environment interaction, batch training, entropy reward calculation, adaptive noise adjustment based on loss fluctuations, and actor and critic target network updates. The following is a detailed process flow of the AME-TD3 algorithm: S1: Parameter collection and MDP modeling to establish system parameter groups.

[0012] Collect and initialize CHP system operating parameters, modeling the system electrical load, thermal load, and unit power output as a Markov decision process. Also, define the initial network parameters for the AME-TD3 reinforcement learning algorithm, including the state space (including the imbalance of pure generator units, CHP unit imbalance, pure thermal unit imbalance, total thermal load, total electrical load, pure generator unit output power, CHP unit output power, CHP unit output power, and pure thermal unit output power). , Action space (including the output power of pure electric units, the output power of CHP units, and the output power of pure thermal units) , reward function (minimize cost) .

[0013] Where n i represents the number of pure motor groups, c i Indicates the i The cost coefficient of a pure motor group, p i Indicates the i The power output of a pure motor group, n j represents the number of CHP units, c j Indicates the j The comprehensive cost coefficient of a CHP unit, p j Indicates the j Power output of each CHP unit, h j Indicates the j Thermal output of a CHP unit, n k represents the number of pure heat units, c k Indicates the k The cost coefficient of a pure heat unit, h k Indicates the k The thermal output of a pure heat unit.

[0014] S2: Initialize the Actor network ( ), output control actions for the CHP system: According to the CHP system operation requirements, the Actor network is initialized to generate corresponding control actions (i.e., output power adjustment instructions for each unit) given the current system status (such as load demand, current unit output power, etc.).

[0015] The present invention does not directly output deterministic actions, but outputs the mean of action distribution through the Actor network. and standard deviation , thereby providing the system with flexible and diverse control strategies to meet the thermal power demand under complex load conditions while taking into account the economic optimization of the system.

[0016] S3: Initialize the Critic network ( , ), used to evaluate the performance of CHP system control: evaluate the economic and energy efficiency of CHP system operation under specific state-action pairs, and learn accurate Value function, the intelligent agent can identify the actual operating effects under different control strategies and provide a basis for the optimization of the Actor network strategy.

[0017] S4: Initialize three target networks ( , , ): A delayed update target network is established for the Actor network and the Critic network respectively to reduce the gradient oscillation caused by load fluctuations, system disturbances, etc. during training, improve the stability of strategy optimization, and make the CHP system control strategy more robust.

[0018] S5: Initialize the replay experience pool (Replay Buffer) to record the exchange data of the CHP system: Establish an experience pool to store the historical data of the agent's interaction with the system in the CHP environment, including the state ,action ,award , next state and whether the termination flag (done) provides diversified sample support for subsequent efficient training.

[0019] S6: The agent interacts with the CHP environment, stores the experience group into the experience replay pool, and updates the experience pool: The agent is based on the current state and Actor Network Generate an action The agent performs an action , and interact with the CHP environment to get the next state and instant rewards . Current experience Stored in the experience pool, each interaction reflects the operating response of the real CHP system under a specific control strategy, and the experience data continuously enriches the experience pool.

[0020] S7: Calculate target When the value is set, the entropy term of action distribution is introduced to encourage the agent to maintain a high level of strategic exploration under complex load fluctuation conditions and avoid falling into local optimal solutions. Calculate the entropy of action distribution under the current strategy :

[0021] S8: Use entropy as a regular term in the reward function and introduce the entropy regularization coefficient , the modified objective function is expressed as follows:

[0022] Where, c represents the discount factor, Indicates the i The state-action value function of the target critic network, Represents the action output by the target Actor network.

[0023] The immediate reward is expanded from simple cost minimization to a comprehensive goal that balances system operating costs and strategic exploration. By adjusting the entropy regularization coefficient, the agent can flexibly adjust the control strategy when faced with unconventional loads or emergencies (such as unit failures), ensuring the continuity and economic efficiency of the CHP system.

[0024] S9: Batch sampling data , calculate the target through the target Critic network value :

[0025] in, is the action output by the target Actor network, To explore noise.

[0026] S10: Critic network update to improve the accuracy of CHP system strategy evaluation: Using the least squares (MSE) loss function, update the two critic networks and : Gradient descent optimization is performed on the two critic networks to minimize the loss function, which improves the value estimation accuracy of the effects of different control strategies of the CHP system and facilitates the iterative optimization of the strategy.

[0027] S11: Loss Fluctuation Monitoring: Monitor the loss fluctuations of the Critic network in the last 10 rounds of training , calculate the mean change of its loss , observe the convergence trend:

[0028] When the loss fluctuation decreases significantly, it indicates that the CHP system control strategy has become stable and efficient.

[0029] S12: Adaptively adjust noise: Adaptively adjust noise. If the loss fluctuation is greater than a threshold, increase the exploration noise; otherwise, reduce the exploration noise.

[0030] When losses fluctuate Exceeding the set threshold d , increasing noise s To increase exploration, promote extensive exploration of the strategy when the load changes drastically; when the loss fluctuation is small, reduce noise s The convergence of the acceleration strategy can effectively adapt to the operation requirements of the CHP system in different load change cycles.

[0031]

[0032] S13: Critic network-based calculation The gradient of the value is calculated, the gradient of the Actor network is calculated, and the Actor network parameters are updated so that the control actions generated by the Actor network can obtain higher rewards in the future state, that is, to maximize the operating benefits of the CHP system:

[0033] Where, Represents the Actor network parameters i The gradient operator is used to calculate the update direction of the policy network parameters. J represents the objective function of policy optimization, that is, the long-term cumulative reward function of expectation maximization, Indicated by the parameter i A deterministic Actor policy network, responsible for mapping states to action distributions, Indicates action a The gradient operator is used to calculate The rate of change of value relative to the action, represents the state-action value function evaluated by the Critic network, which is used to evaluate the long-term value of performing a specific action in a given state, where is the Critic network parameter.

[0034] S14: Use the gradient ascent algorithm to update the Actor network parameters to maximize value: Improve the contribution of generated actions to system benefits and ensure that the output control strategy can effectively reduce the operating costs of the CHP system. is the learning rate of the Actor network.

[0035] S15: Target network soft update to ensure a smooth learning process: The target network is updated using a weighted average soft update method to ensure slow synchronization of target network parameters, effectively preventing the CHP system control strategy from unstable updates due to sudden load changes during training, and ensuring a smooth overall learning process. The specific update formula is:

[0036] in, are the parameters of the target network, is the soft update coefficient, which is usually smaller (such as t =0.005).

[0037] S16: End training and strategy output: Whether the stopping condition is met, if so, stop training and output the final CHP system control strategy, otherwise go to S6.

[0038] When the number of training rounds reaches the preset upper limit, or when the cumulative reward tends to be stable through monitoring and the temporal difference error of the Critic network converges, the training is considered complete and the final Actor network parameters are output. . Income strategy It can be used in CHP systems to achieve the goal of optimal economic operation.

[0039] It will be understood that the present invention is described by way of some embodiments, and it will be appreciated by those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are intended to be protected by the present invention.

Claims

1. A control strategy for a combined heat and power system based on the AME-TD3 algorithm, characterized by: The following steps are involved: S1: Initialization of acquisition system parameters and CHP operating status; S2: Calculation target When the value is , the entropy reward mechanism is introduced; S3: Implementing an adaptive noise regulation mechanism based on critic network evaluation, and exploring and utilizing dynamic balancing strategies; S4: Real-time optimization of control strategy.

2. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 1, characterized in that: Said S1 comprises the following steps: S11: parameter collection and MDP modeling; S12: Initialize the Actor network and output the actions that control the CHP system; S13: Initialize the critic network to evaluate the performance of the CHP system under control; S14: Initialize the target network and set up delayed update target networks for the Actor network and the Critic network respectively; S15: Initialize the playback experience pool and record the CHP system exchange data; S16: The agent interacts with the CHP environment and stores the experience group into the experience replay pool.

3. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 2, characterized in that: The S2 comprises the following steps: S21: Calculation target When the value is , an entropy reward mechanism is added; S22: Use entropy as a regularization term and add it to the reward function; S23: Batch sampling data, calculating the target through the target critic network value; S24: Critic network update to improve the accuracy of CHP system strategy evaluation.

4. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 3, characterized in that: The S22 is specifically expressed by the formula: Where, γ represents the discount factor, Indicates the i The state-action value function of the target critic network, Represents the action output by the target Actor network.

5. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 4, characterized in that: The formula of S23 is: in, is the action output by the target Actor network, To explore noise.

6. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 5, characterized in that: In S24, the least squares loss function is used to update the Critic network. and Critic Network : .

7. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 3, characterized in that: The S3 includes the following steps: S31: Monitoring loss fluctuations; S32: Adaptively adjust the noise. If the loss fluctuation is greater than a threshold, increase the exploration noise; otherwise, reduce the exploration noise.

8. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 7, characterized in that: The S4 comprises the following steps: S41: Critic network-based calculation The gradient of the value, calculate the gradient of the Actor network; S42: Use the gradient ascent algorithm to update the Actor network parameters to maximize value; S43: Target network soft update to ensure a smooth learning process; S44: Check whether the stopping condition is met. If so, stop the training and output the final CHP system control strategy. Otherwise, go to S16.

9. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 8, characterized in that: The S41 is expressed by the formula: Where, Represents the Actor network parameters θ The gradient operator is used to calculate the update direction of the policy network parameters. J represents the objective function of policy optimization, Indicated by the parameter θ A deterministic Actor policy network, responsible for mapping states to action distributions, Indicates action a The gradient operator is used to calculate The rate of change of value relative to the action, represents the state-action value function evaluated by the Critic network, which is used to evaluate the long-term value of performing a specific action in a given state, where is the Critic network parameter.

10. The control strategy for a cogeneration system based on the AME-TD3 algorithm according to claim 9, characterized in that: The specific process of S43 is: the target network is updated using a weighted average soft update method, and the specific update formula is: in, are the parameters of the target network, is the soft update coefficient.

Images