Supercritical coal-fired unit control method and device based on deep reinforcement learning

Abstract

The invention provides a supercritical coal-fired unit control method and device based on deep reinforcement learning. The method comprises the following steps: constructing an initial supercritical coal-fired unit dynamic model; training the initial supercritical coal-fired unit dynamic model according to the historical operation data, and obtaining the supercritical coal-fired unit dynamic model after training is completed; constructing a control target and a state, an action and a reward of supercritical coal-fired unit coordination control; constructing a reinforcement learning agent, training the reinforcement learning agent according to the state, the action, the reward and the supercritical coal-fired unit dynamic model, and taking the converged reinforcement learning agent as a reinforcement learning controller; and the given control parameters and the state parameters in the actual operation process of the supercritical coal-fired unit are input into the reinforcement learning controller, and the reinforcement learning controller outputs optimization control parameters used for optimizing and controlling the actual operation state of the supercritical coal-fired unit. The method can improve the control robustness and precision of the supercritical coal-fired unit.

Description

Technical Field

[0001] This invention relates to the field of power grid coal-fired unit control technology, and in particular to a control method and device for supercritical coal-fired units based on deep reinforcement learning. Background Technology

[0002] Coal-fired power generating units are a core component of my country's energy supply system, undertaking the important tasks of grid peak and frequency regulation and compensating for fluctuations in renewable energy. The boiler-turbine coordinated control system, as a core component of coal-fired power generating units, directly determines the unit's ability to respond quickly and accurately to grid load commands while maintaining its own safe and efficient operation. However, the strong nonlinearity, large inertia, and multivariable coupling characteristics of coal-fired units under peak and deep-range operating conditions, coupled with frequent fluctuations in fuel quality, changes in equipment status, and environmental disturbances, pose severe challenges to efficient and reliable operation control. Traditional coordinated control strategies mostly employ PID control strategies based on empirical rules or optimization methods based on mechanistic models. For example, patent CN119846940A proposes a decoupling control method for the coordinated PID and model predictive control loops of a coal-fired power unit, including the following steps: Step 1: Analyze the multivariable coupling characteristics of the unit's coordinated control system to determine the unit's multi-loop decoupling control requirements; Step 2: Based on the unit's multi-loop decoupling control requirements, design a PID controller for the unit's coordinated system for the power generation control loop to control the main steam valve opening; Step 3: Design a model predictive controller for the unit's coordinated control system for the dual loops of main steam pressure and water-cooled wall outlet temperature. However, this method suffers from drawbacks such as slow response speed, poor robustness, and insufficient control accuracy when dealing with multivariable coupling, large changes in operating conditions, and strong nonlinearity. Summary of the Invention

[0003] This invention provides a control method and device for supercritical coal-fired power units based on deep reinforcement learning, which can improve the robustness and accuracy of supercritical coal-fired power unit control.

[0004] According to a first aspect of the present invention, a control method for a supercritical coal-fired power unit based on deep reinforcement learning is provided, comprising: Construct an initial dynamic model of a supercritical coal-fired unit; Historical operating data of the supercritical coal-fired power unit is collected, and the initial dynamic model of the supercritical coal-fired power unit is trained based on the historical operating data. After training, the dynamic model of the supercritical coal-fired power unit is obtained. Establish control objectives and define the status, actions, and rewards for coordinated control of supercritical coal-fired power units; A reinforcement learning agent is constructed and trained based on the state, action, reward, and dynamic model of the supercritical coal-fired unit. The converged reinforcement learning agent is then used as a reinforcement learning controller. The given control parameters and the state parameters during the actual operation of the supercritical coal-fired unit are input to the reinforcement learning controller. The reinforcement learning controller outputs optimized control parameters, which are used to optimize the actual operating state of the supercritical coal-fired unit.

[0005] Furthermore, the input data of the supercritical coal-fired unit dynamic model includes fuel quantity, main steam valve opening degree, and feedwater flow rate; the output data of the supercritical coal-fired unit dynamic model includes coal-fired unit power generation, main steam pressure, and steam-water separator outlet enthalpy value.

[0006] Furthermore, the control objectives are to minimize the absolute value of the deviation between the actual value and the given value of the power generation, the absolute value of the deviation between the actual value and the given value of the main steam pressure, and the absolute value of the deviation between the enthalpy value at the outlet of the steam-water separator and the given value.

[0007] Furthermore, the states include the deviation between the power generation of the supercritical coal-fired unit at time t and the given value, the deviation between the actual value of the main steam pressure at time t and the given value, the deviation between the actual value of the steam-water separator outlet enthalpy at time t and the given value, the rate of change of power generation at time t, the rate of change of main steam pressure at time t, the rate of change of steam-water separator outlet enthalpy at time t, the fuel quantity at time t-1, the opening degree of the main steam valve at time t-1, and the feedwater flow rate at time t-1. The actions include the fuel quantity increment, the main steam valve opening increment, and the feedwater flow rate increment at time t; The reward is the sum of the primary and secondary objectives. The primary objectives include the reward values ​​for the absolute values ​​of the deviation between the actual and given values ​​of the power generation, the actual and given values ​​of the main steam pressure, and the absolute values ​​of the deviation between the actual and given values ​​of the steam-water separator outlet enthalpy, under different value ranges. The secondary objectives are the penalty items when the state does not meet the upper and lower limit constraints.

[0008] Furthermore, the reinforcement learning agent includes a policy network and a value function network. The reinforcement learning agent is trained based on the state, action, reward, and the dynamic model of the supercritical coal-fired power unit, including: S41. Initialize the strategy network, value function network, the supercritical coal-fired unit dynamic model, and the experience playback buffer, and create an old strategy network with the same structure as the strategy network. S42. The old strategy network and the supercritical coal-fired unit dynamic model perform interactive sampling based on the state, action and reward to obtain experience data and store it in the experience replay buffer. S43. Optimize the parameters of the policy network and value function network based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update; Repeat steps S42 and S43 until the reinforcement learning agent meets the performance requirements.

[0009] Furthermore, the old strategy network and the supercritical coal-fired unit dynamic model interact and sample based on the state, action, and reward to obtain empirical data and store it in the experience replay buffer, including: Initial action data is constructed and input into the dynamic model of the supercritical coal-fired unit. The dynamic model of the supercritical coal-fired unit outputs the state data at time t and calculates the reward at time t. The state data at time t is input into the old strategy network. The old strategy network selects an action to output based on the state data at time t, which is recorded as the action at time t. The action at time t output by the old strategy network is input into the supercritical coal-fired unit dynamic model and outputs the state data at time t+1. Based on the actions and state data at time t, calculate the probability density at time t under the old policy network; The state data at time t, the action at time t, the reward at time t, the state data at time t+1, and the probability density at time t under the old policy network are stored as experience data in the experience replay buffer.

[0010] Furthermore, the parameters of the policy network and value function network are optimized based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update, including: The experience data in the experience replay buffer is acquired, and the value function network calculates the time difference target and time difference error based on the reward at time t. The advantage estimate is calculated based on the time difference objective and the time difference error. The state data at time t in the experience data of the experience replay buffer is obtained and input into the current policy network. The current policy network selects the corresponding action and outputs it. The probability density of the current policy network is calculated based on the state data at time t and the action output by the current policy network. The probability ratio is calculated based on the probability density of the current policy network and the probability density of the old policy network, and the shearing probability is introduced. The value of the loss function is calculated based on the advantage estimate, the probability ratio, the shearing probability, and the time difference objective. The parameters of the current policy network and the value function network are updated by minimizing the value of the loss function through gradient descent. Update the parameters of the old policy network to the parameters of the current policy network, and clear the experience replay buffer.

[0011] Furthermore, the probability ratio is the ratio of the probability density of the current policy network to the probability density of the old policy network; The loss function is calculated based on the advantage estimate, probability ratio, shearing probability, and temporal difference objective. The parameters of the current policy network and value function network are updated by minimizing the loss function using gradient descent, including: An entropy reward term is introduced, and a first loss function is constructed based on the advantage estimate, probability ratio, shearing probability, and entropy reward term. The policy network updates its parameters by minimizing the value of the first loss function through gradient descent. A second loss function is constructed based on the value prediction of the value function network and the time difference objective. The parameters of the value function network are updated by minimizing the value of the second loss function through gradient descent.

[0012] According to a second aspect of the present invention, a control device for a supercritical coal-fired power unit based on deep reinforcement learning is provided, comprising: The model building module is used to build an initial dynamic model of a supercritical coal-fired unit; The model training module is used to collect historical operating data of the supercritical coal-fired unit, train the initial dynamic model of the supercritical coal-fired unit based on the historical operating data, and obtain the dynamic model of the supercritical coal-fired unit after training. The target construction module is used to construct control targets and the status, actions, and rewards for coordinated control of supercritical coal-fired power units. The agent training module is used to construct a reinforcement learning agent. The reinforcement learning agent is trained based on the state, action, reward and the dynamic model of the supercritical coal-fired unit. The converged reinforcement learning agent is used as a reinforcement learning controller. The operation control module is used to input the given control parameters and the state parameters of the supercritical coal-fired unit during actual operation to the reinforcement learning controller. The reinforcement learning controller outputs optimized control parameters, which are used to optimize the control of the actual operating state of the supercritical coal-fired unit.

[0013] Furthermore, the input data of the supercritical coal-fired unit dynamic model includes fuel quantity, main steam valve opening degree, and feedwater flow rate; the output data of the supercritical coal-fired unit dynamic model includes coal-fired unit power generation, main steam pressure, and steam-water separator outlet enthalpy value.

[0014] Furthermore, the control objectives are to minimize the absolute value of the deviation between the actual value and the given value of the power generation, the absolute value of the deviation between the actual value and the given value of the main steam pressure, and the absolute value of the deviation between the enthalpy value at the outlet of the steam-water separator and the given value.

[0015] Furthermore, the states include the deviation between the power generation of the supercritical coal-fired unit at time t and the given value, the deviation between the actual value of the main steam pressure at time t and the given value, the deviation between the enthalpy value at the outlet of the steam-water separator at time t and the given value, the rate of change of power generation at time t, the rate of change of the main steam pressure at time t, the rate of change of the enthalpy value at the outlet of the steam-water separator at time t, the fuel quantity at time t-1, the opening degree of the main steam valve at time t-1, and the feedwater flow rate at time t-1. The actions include the fuel quantity increment, the main steam valve opening increment, and the feedwater flow rate increment at time t; The reward is the sum of the primary and secondary objectives. The primary objectives include the reward values ​​for the absolute values ​​of the deviation between the actual and given values ​​of the power generation, the actual and given values ​​of the main steam pressure, and the absolute values ​​of the deviation between the actual and given values ​​of the steam-water separator outlet enthalpy, under different value ranges. The secondary objectives are the penalty items when the state does not meet the upper and lower limit constraints.

[0016] Furthermore, the reinforcement learning agent includes a policy network and a value function network. The agent training module trains the reinforcement learning agent based on the state, action, reward, and the dynamic model of the supercritical coal-fired power unit, including: S41. Initialize the strategy network, value function network, the supercritical coal-fired unit dynamic model, and the experience playback buffer, and create an old strategy network with the same structure as the strategy network. S42. The old strategy network and the supercritical coal-fired unit dynamic model perform interactive sampling based on the state, action and reward to obtain experience data and store it in the experience replay buffer. S43. Optimize the parameters of the policy network and value function network based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update; Repeat steps S42 and S43 until the reinforcement learning agent meets the performance requirements.

[0017] Furthermore, the old strategy network and the supercritical coal-fired unit dynamic model interact and sample based on the state, action, and reward to obtain empirical data and store it in the experience replay buffer, including: Initial action data is constructed and input into the dynamic model of the supercritical coal-fired unit. The dynamic model of the supercritical coal-fired unit outputs the state data at time t and calculates the reward at time t. The state data at time t is input into the old strategy network. The old strategy network selects an action to output based on the state data at time t, which is recorded as the action at time t. The action at time t output by the old strategy network is input into the supercritical coal-fired unit dynamic model and outputs the state data at time t+1. Based on the actions and state data at time t, calculate the probability density at time t under the old policy network; The state data at time t, the action at time t, the reward at time t, the state data at time t+1, and the probability density at time t under the old policy network are stored as experience data in the experience replay buffer.

[0018] Furthermore, the agent training module optimizes the parameters of the policy network and value function network based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update, including: The experience data in the experience replay buffer is acquired, and the value function network calculates the time difference target and time difference error based on the reward at time t. The advantage estimate is calculated based on the time difference objective and the time difference error. The state data at time t in the experience data of the experience replay buffer is obtained and input into the current policy network. The current policy network selects the corresponding action and outputs it. The probability density of the current policy network is calculated based on the state data at time t and the action output by the current policy network. The probability ratio is calculated based on the probability density of the current policy network and the probability density of the old policy network, and the shearing probability is introduced. The value of the loss function is calculated based on the advantage estimate, the probability ratio, the shearing probability, and the time difference objective. The parameters of the current policy network and the value function network are updated by minimizing the value of the loss function through gradient descent. Update the parameters of the old policy network to the parameters of the current policy network, and clear the experience replay buffer.

[0019] Furthermore, the probability ratio is the ratio of the probability density of the current policy network to the probability density of the old policy network; The agent training module calculates the value of the loss function based on the advantage estimate, probability ratio, shearing probability, and temporal difference objective. It then updates the parameters of the current policy network and value function network by minimizing the value of the loss function through gradient descent, including: An entropy reward term is introduced, and a first loss function is constructed based on the advantage estimate, probability ratio, shearing probability, and entropy reward term. The policy network updates its parameters by minimizing the value of the first loss function through gradient descent. A second loss function is constructed based on the value prediction of the value function network and the time difference objective. The parameters of the value function network are updated by minimizing the value of the second loss function through gradient descent.

[0020] According to a third aspect of the present invention, an electronic device is provided, comprising a processor and a storage device, the storage device storing a plurality of instructions, the processor being configured to read the instructions and execute the method described above.

[0021] The supercritical coal-fired power unit control method and device based on deep reinforcement learning provided by this invention have at least the following beneficial effects: (1) Based on the actual operating data of supercritical coal-fired units, a dynamic model of supercritical coal-fired units is constructed and used as the environment in the training process of deep reinforcement learning control strategy. The coordinated control problem of coal-fired power generation units is expressed as a Markov decision process. The state space, action space and reward function are constructed, and a reinforcement learning agent is constructed. The control strategy is optimized through its interaction with the environment, thereby obtaining a reinforcement learning controller for actual operation control. This solves the problems of slow response speed, poor robustness and insufficient control accuracy caused by multivariate coupling, large changes in operating conditions and strong nonlinearity in the actual operation of supercritical coal-fired units. It realizes accurate and fast tracking of the power generation of supercritical coal-fired units, while maintaining the main steam pressure and steam enthalpy value of the steam at the outlet of the steam-water separator within the set range, effectively improving the control performance of the unit coordination system. (2) In each training iteration, the reinforcement learning agent samples multiple trajectories in the simulation environment according to the current policy, improves the policy objective function by shearing probability ratio, controls the update step size, and avoids drastic policy deviation; and introduces time difference objective for training value function network to improve the stability and accuracy of policy estimation. Attached Figure Description

[0022] Figure 1 This is a flowchart of one embodiment of the supercritical coal-fired power unit control method based on deep reinforcement learning provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of an embodiment of the supercritical coal-fired power unit dynamic model in the supercritical coal-fired power unit control method based on deep reinforcement learning provided by the present invention.

[0024] Figure 3 This is a flowchart of an embodiment of the reinforcement learning agent training in the supercritical coal-fired power unit control method based on deep reinforcement learning provided by the present invention.

[0025] Figure 4 This is a flowchart illustrating an embodiment of the interaction between the old policy network and the dynamic model of the supercritical coal-fired power unit in the deep reinforcement learning-based control method for supercritical coal-fired power units provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of an embodiment of the interaction between the old policy network and the dynamic model of the supercritical coal-fired power unit in the control method of supercritical coal-fired power units based on deep reinforcement learning provided by the present invention.

[0027] Figure 6 This is a flowchart of an embodiment of the reinforcement learning agent training and parameter updating method for supercritical coal-fired power unit control based on deep reinforcement learning provided by the present invention.

[0028] Figure 7 This is a schematic diagram of one embodiment of the supercritical coal-fired power unit control device based on deep reinforcement learning provided by the present invention. Detailed Implementation

[0029] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0030] refer to Figure 1 In some embodiments, a control method for supercritical coal-fired power units based on deep reinforcement learning is provided, including: S1. Construct an initial dynamic model of a supercritical coal-fired unit; S2. Collect historical operating data of the supercritical coal-fired unit, and train the initial dynamic model of the supercritical coal-fired unit based on the historical operating data. After training, the dynamic model of the supercritical coal-fired unit is obtained. S3. Construct control objectives and the status, actions, and rewards for coordinated control of supercritical coal-fired power units; S4. Construct a reinforcement learning agent, train the reinforcement learning agent based on the state, action, reward and the dynamic model of the supercritical coal-fired unit, and use the converged reinforcement learning agent as a reinforcement learning controller. S5. Input the given control parameters and the state parameters during the actual operation of the supercritical coal-fired unit to the reinforcement learning controller. The reinforcement learning controller outputs optimized control parameters, which are used to optimize the actual operating state of the supercritical coal-fired unit.

[0031] Specifically, in steps S1 and S2, the initial supercritical coal-fired unit dynamic model is constructed based on an autoregressive neural network.

[0032] The input data of the supercritical coal-fired power unit dynamic model includes fuel quantity, main steam valve opening degree, and feedwater flow rate; the output data of the supercritical coal-fired power unit dynamic model includes power generation of the coal-fired power unit, main steam pressure, and steam-water separator outlet enthalpy value.

[0033] In some embodiments, reference Figure 2 The initial dynamic model of the supercritical coal-fired unit includes an input module 1, a delay module 2, a neural network module 3, and a comparison module 4. The input module 1 includes a historical input module 11 and an exogenous input module 12. The delay module 2 includes a historical input buffer 21 and an exogenous input buffer 22. The neural network module 3 includes an input layer 31, a hidden layer 32, and an output layer 33. The historical input module 12 is connected to the historical input buffer 21, the exogenous input module 12 is connected to the exogenous input buffer 22, the historical input buffer 21 and the exogenous input buffer 22 are connected to the input layer 31, the input layer 31 is connected to the hidden layer 32, the hidden layer 32 is connected to the output layer 33, and the output layer 33 is connected to the comparison module 4.

[0034] Specifically, historical operating data such as fuel quantity, main steam valve opening, feedwater flow rate, power output, main steam pressure, and steam-water separator outlet enthalpy value during the historical operation of the supercritical coal-fired unit are collected as a training set. The initial supercritical coal-fired unit dynamic model is trained based on this training set, and the converged supercritical coal-fired unit dynamic model serves as the simulation environment for subsequent reinforcement learning agents.

[0035] In this system, the fuel quantity, main steam valve opening, and feedwater flow rate from the training set are input as exogenous inputs via exogenous input module 12. The power generation, main steam pressure, and steam-water separator outlet enthalpy from the training set are input as historical input data via historical input module 11, serving as the data source for autoregression. The exogenous inputs are sent to exogenous input buffer 22, and the historical input data is sent to historical input buffer 21 via historical input module 11. Historical input buffer 21 and exogenous input buffer 22 systematically store the data provided by input module 1 and construct a complete input vector containing past information for neural network module 3 at each time step t. Input layer 31 receives input vectors from historical input buffer 21 and exogenous input buffer 22. Hidden layer 32 consists of multiple layers of neurons with nonlinear activation functions, used to capture the nonlinear relationship of the input vector. Output layer 33 outputs the final predicted value and transmits it to comparison module 4. Comparison module 4 calculates the error between the final predicted value and the historical input data, and updates the parameters of neural network module 3 by backpropagation based on the error calculation result.

[0036] Furthermore, in step S3, the control objectives are to minimize the absolute value of the deviation between the actual value and the given value of the power generation during the operation of the supercritical coal-fired unit, the absolute value of the deviation between the actual value and the given value of the main steam pressure, and the absolute value of the deviation between the actual value and the given value of the steam-water separator outlet enthalpy.

[0037] The control objective is: (1) Where G is the control objective, M is the dynamic weight matrix, and E is the weight matrix. pressuer E represents the deviation between the actual value and the given value of the main steam pressure. power E represents the deviation between the generated power and a given value. enthalpy This indicates the deviation between the actual value and the given value of the enthalpy at the outlet of the steam-water separator.

[0038] Further, in step S3, the state, actions, and rewards of the coordinated control of the supercritical coal-fired unit are constructed based on a Markov decision process framework. The state includes the deviation between the actual and given main steam pressure at time t, the deviation between the power generation of the supercritical coal-fired unit at time t and the given value, the deviation between the actual and given enthalpy of the steam-water separator outlet at time t, the rate of change of power generation at time t, the rate of change of main steam pressure at time t, the rate of change of enthalpy of the steam-water separator outlet at time t, the fuel quantity at time t-1, the main steam valve opening at time t-1, and the feedwater flow rate at time t-1. The actions include the fuel quantity increment, the main steam valve opening increment, and the feedwater flow rate increment at time t.

[0039] The reward is the sum of the primary and secondary objectives.

[0040] Specifically, state S is as follows: (2) Among them, E pressuer,t E represents the deviation between the actual value and the given value of the main steam pressure at time t. power,t E represents the deviation between the power generation of a supercritical coal-fired unit at time t and a given value. enthalpy,t This represents the deviation between the actual value and the given value of the enthalpy at the outlet of the steam-water separator at time t. This represents the rate of change of power generation at time t. This represents the rate of change of the main steam pressure at time t. u represents the rate of change of the enthalpy at the outlet of the steam-water separator at time t. coal,t-1 U represents the amount of fuel at time t-1. valve,t-1 u represents the main steam valve opening at time t-1. water,t-1 This represents the water flow rate at time t-1.

[0041] Action B is shown below: (3) Wherein, △u coal,t Δu represents the fuel increment at time t. valve,t Δu represents the opening degree of the main steam valve at time t. water,t This indicates the increase in water supply flow rate.

[0042] The reward function is the sum of a primary objective and a secondary objective. The primary objective is a piecewise function with multiple objectives and conditions, including reward values ​​for the absolute values ​​of the deviations between the actual and given power generation, the actual and given main steam pressure, and the actual and given enthalpy at the steam-water separator outlet, within different ranges. The secondary objective is a penalty term for situations where the state does not meet the upper and lower limit constraints.

[0043] The reward values ​​in the primary objective are based on the Gaussian reward model: (4) Where r is the reward value, A is the maximum reward magnitude, G is the control objective, and σ is the rate at which the reward decays as the control objective G increases.

[0044] The reward function in this embodiment is as follows: r t =r 1,t +r 2,t (5) Where, r t Represents the final reward value, r 1,t r represents the reward value of the primary objective. 2,t This represents the reward value for secondary objectives.

[0045] The main objectives are: (6) The secondary objectives are: ; (7) Specifically, among the main objective items, the smaller the absolute values ​​of the deviations between the actual and given values ​​of power generation, main steam pressure, and steam-water separator outlet enthalpy, the greater the reward. Furthermore, the rate of reward decay increases with the increase of the control objective G; a high reward can only be obtained when the control objective G is very small. M represents the dynamic weight matrix, which assigns different levels of attention to different physical quantities at different stages.

[0046] Further, refer to Figure 3 In step S4, the reinforcement learning agent includes a policy network and a value function network; training the reinforcement learning agent based on the state, action, reward, and the dynamic model of the supercritical coal-fired power unit includes: S41. Initialize the strategy network, value function network, the supercritical coal-fired unit dynamic model, and the experience playback buffer, and create an old strategy network with the same structure as the strategy network. S42. The old strategy network and the supercritical coal-fired unit dynamic model perform interactive sampling based on the state, action and reward to obtain experience data and store it in the experience replay buffer. S43. Optimize the parameters of the policy network and value function network based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update; Repeat steps S42 and S43 until the reinforcement learning agent meets the performance requirements.

[0047] Specifically, in step S41, the structure of the old policy network is the same as that of the policy network. Initially, the old policy network copies the parameters of the policy network. After each iteration, the old policy network updates the parameters of the policy network by copying the parameters of the policy network.

[0048] Specifically, refer to Figure 4 and Figure 5 Step S42 specifically includes: S421. Initial action data is constructed and input into the dynamic model of the supercritical coal-fired unit. The dynamic model of the supercritical coal-fired unit outputs the state data at time t and calculates the reward at time t. S422. Input the state data at time t into the old strategy network. The old strategy network selects an action to output based on the state data at time t, which is recorded as the action at time t. The action at time t output by the old strategy network is input into the supercritical coal-fired unit dynamic model and outputs the state data at time t+1. S423. Based on the actions and state data at time t, calculate the probability density at time t under the old policy network. S424. Store the state data at time t, the action at time t, the reward at time t, the state data at time t+1, and the probability density at time t as experience data in the experience replay buffer under the old policy network.

[0049] Wherein, the probability density is the logarithmic probability density: log(a t |s t ), a t s represents the action at time t. t This represents the state at time t.

[0050] Further, refer to Figure 6 Step S43 specifically includes: S431. Obtain the experience data from the experience replay buffer. The value function network calculates the time difference target and time difference error based on the reward at time t. S432. Calculate the advantage estimate based on the time difference target and time difference error; S433. Obtain the state data at time t from the experience data in the experience playback buffer and input it into the current policy network. The current policy network selects the corresponding action and outputs it. Calculate the probability density of the current policy network based on the state data at time t and the action output by the current policy network. S434. Calculate the probability ratio based on the probability density of the current policy network and the probability density under the old policy network, and introduce the shearing probability. Calculate the value of the loss function based on the advantage estimate, probability ratio, shearing probability, and time difference objective. Update the parameters of the current policy network and value function network by minimizing the value of the loss function through gradient descent. S435. Update the parameters of the old policy network to the parameters of the current policy network, and clear the experience replay buffer.

[0051] Specifically, in step S431, the time difference objective is calculated using the following formula: (8) Among them, T d Represents the time-difference objective, r t V(s) represents the reward at time t, γ represents the discount factor, and (1-done) represents the termination condition, indicating whether a round has ended. If there is no further data, the round ends, and (1-done) is 0; otherwise, it is 1. t+1 The value function network predicts the value of the state at time t+1.

[0052] The time difference error is calculated using the following formula: (9) in, T represents the time difference error at time t. d V(s) represents the time-difference objective. t ) represents the value prediction of the state at time t by the value function network.

[0053] The time difference error is the difference between the value function network's value predictions at time t and time t+1. If the time difference error is greater than 0, it indicates that in state s... t The following action a t Better than expected; conversely, if it equals 0, it means it meets expectations.

[0054] Further, in step S432, the advantage estimation is as follows: (10) in, Let λ represent the advantage estimate, γ represent the discount factor, and λ represent the advantage estimate parameter, which is usually set to 0.95.

[0055] Further, in step S433, the probability ratio is the ratio of the probability density of the current policy network to the probability density of the old policy network.

[0056] The loss function is calculated based on the advantage estimate, probability ratio, shearing probability, and temporal difference objective. The parameters of the current policy network and value function network are updated by minimizing the loss function using gradient descent, including: An entropy reward term is introduced, and a first loss function is constructed based on the advantage estimate, probability ratio, shearing probability, and entropy reward term. The policy network updates its parameters by minimizing the value of the first loss function through gradient descent. A second loss function is constructed based on the value prediction of the value function network and the time difference objective. The parameters of the value function network are updated by minimizing the value of the second loss function through gradient descent.

[0057] The first loss function is shown below: (11) In the formula, L1 represents the first loss function. Indicates the probability ratio. This represents the shear probability, which is the ratio of the probability to the shear probability. Limited to the range Inside, C represents the entropy reward term.

[0058] The second loss function is shown below: (12) Wherein, V(s) t L1 represents the value prediction of the state at time t by the value function network, and L2 represents the second loss function.

[0059] refer to Figure 7 In some embodiments, a control device for a supercritical coal-fired power unit based on deep reinforcement learning is provided, comprising: Model building module 201 is used to build an initial dynamic model of a supercritical coal-fired unit. The model training module 202 is used to collect historical operating data of the supercritical coal-fired unit, train the initial dynamic model of the supercritical coal-fired unit based on the historical operating data, and obtain the dynamic model of the supercritical coal-fired unit after the training is completed. The target construction module 203 is used to construct control targets and the status, actions, and rewards of coordinated control of supercritical coal-fired power units. The agent training module 204 is used to construct a reinforcement learning agent, train the reinforcement learning agent according to the state, action, reward and the dynamic model of the supercritical coal-fired unit, and use the converged reinforcement learning agent as a reinforcement learning controller. The operation control module 205 is used to input the given control parameters and the state parameters of the supercritical coal-fired unit during actual operation to the reinforcement learning controller. The reinforcement learning controller outputs optimized control parameters, which are used to optimize the control of the actual operating state of the supercritical coal-fired unit.

[0060] Furthermore, the input data of the supercritical coal-fired unit dynamic model includes fuel quantity, main steam valve opening degree, and feedwater flow rate; the output data of the supercritical coal-fired unit dynamic model includes coal-fired unit power generation, main steam pressure, and steam-water separator outlet enthalpy value.

[0061] Furthermore, the control objectives are to minimize the absolute value of the deviation between the actual value and the given value of the power generation, the absolute value of the deviation between the actual value and the given value of the main steam pressure, and the absolute value of the deviation between the enthalpy value at the outlet of the steam-water separator and the given value.

[0062] Furthermore, the states include the deviation between the power generation of the supercritical coal-fired unit at time t and the given value, the deviation between the actual value of the main steam pressure at time t and the given value, the deviation between the actual value of the steam-water separator outlet enthalpy at time t and the given value, the rate of change of power generation at time t, the rate of change of main steam pressure at time t, the rate of change of steam-water separator outlet enthalpy at time t, the fuel quantity at time t-1, the opening degree of the main steam valve at time t-1, and the feedwater flow rate at time t-1. The actions include the fuel quantity increment, the main steam valve opening increment, and the feedwater flow rate increment at time t; The reward is the sum of the primary and secondary objectives. The primary objectives include the reward values ​​for the absolute values ​​of the deviation between the actual and given values ​​of the power generation, the actual and given values ​​of the main steam pressure, and the absolute values ​​of the deviation between the actual and given values ​​of the steam-water separator outlet enthalpy, under different value ranges. The secondary objectives are the penalty items when the state does not meet the upper and lower limit constraints.

[0063] Furthermore, the reinforcement learning agent includes a policy network and a value function network. The agent training module trains the reinforcement learning agent based on the state, action, reward, and the dynamic model of the supercritical coal-fired power unit, including: S41. Initialize the strategy network, value function network, the supercritical coal-fired unit dynamic model, and the experience playback buffer, and create an old strategy network with the same structure as the strategy network. S42. The old strategy network and the supercritical coal-fired unit dynamic model perform interactive sampling based on the state, action and reward to obtain experience data and store it in the experience replay buffer. S43. Optimize the parameters of the policy network and value function network based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update; Repeat steps S42 and S43 until the reinforcement learning agent meets the performance requirements.

[0064] Furthermore, the old strategy network and the supercritical coal-fired unit dynamic model interact and sample based on the state, action, and reward to obtain empirical data and store it in the experience replay buffer, including: Initial action data is constructed and input into the dynamic model of the supercritical coal-fired unit. The dynamic model of the supercritical coal-fired unit outputs the state data at time t and calculates the reward at time t. The state data at time t is input into the old strategy network. The old strategy network selects an action to output based on the state data at time t, which is recorded as the action at time t. The action at time t output by the old strategy network is input into the supercritical coal-fired unit dynamic model and outputs the state data at time t+1. Based on the actions and state data at time t, calculate the probability density at time t under the old policy network; The state data at time t, the action at time t, the reward at time t, the state data at time t+1, and the probability density at time t under the old policy network are stored as experience data in the experience replay buffer.

[0065] Furthermore, the agent training module 204 optimizes the parameters of the policy network and value function network based on the empirical data using shearing probability, temporal difference objective, backpropagation, and gradient update, including: The experience data in the experience replay buffer is acquired, and the value function network calculates the time difference target and time difference error based on the reward at time t. The advantage estimate is calculated based on the time difference objective and the time difference error. The state data at time t in the experience data of the experience replay buffer is obtained and input into the current policy network. The current policy network selects the corresponding action and outputs it. The probability density of the current policy network is calculated based on the state data at time t and the action output by the current policy network. The probability ratio is calculated based on the probability density of the current policy network and the probability density of the old policy network, and the shearing probability is introduced. The value of the loss function is calculated based on the advantage estimate, probability ratio, shearing probability and time difference objective. The parameters of the current policy network and value function network are updated by minimizing the value of the loss function through gradient descent. Update the parameters of the old policy network to the parameters of the current policy network, and clear the experience replay buffer.

[0066] Furthermore, the probability ratio is the ratio of the probability density of the current policy network to the probability density of the old policy network; The agent training module 204 calculates the value of the loss function based on the advantage estimation, probability ratio, shearing probability, and temporal difference objective, and updates the parameters of the current policy network and value function network by minimizing the value of the loss function through gradient descent, including: An entropy reward term is introduced, and a first loss function is constructed based on the advantage estimate, probability ratio, shearing probability, and entropy reward term. The policy network updates its parameters by minimizing the value of the first loss function through gradient descent. A second loss function is constructed based on the value prediction of the value function network and the time difference objective. The parameters of the value function network are updated by minimizing the value of the second loss function through gradient descent.

[0067] In some embodiments, an electronic device is also provided, including a processor and a storage device, the storage device storing a plurality of instructions, the processor being configured to read the instructions and execute the methods described above.

[0068] The supercritical coal-fired power unit control method and apparatus based on deep reinforcement learning provided in the above embodiments have at least the following beneficial effects: (1) Based on the actual operating data of supercritical coal-fired units, a dynamic model of supercritical coal-fired units is constructed and used as the environment in the training process of deep reinforcement learning control strategy. The coordinated control problem of coal-fired power generation units is expressed as a Markov decision process. The state space, action space and reward function are constructed, and a reinforcement learning agent is constructed. The control strategy is optimized through its interaction with the environment, thereby obtaining a reinforcement learning controller for actual operation control. This solves the problems of slow response speed, poor robustness and insufficient control accuracy caused by multivariate coupling, large changes in operating conditions and strong nonlinearity in the actual operation of supercritical coal-fired units. It realizes accurate and fast tracking of the power generation of supercritical coal-fired units, while maintaining the main steam pressure and steam enthalpy value of the steam at the outlet of the steam-water separator within the set range, effectively improving the control performance of the unit coordination system. (2) In each training iteration, the reinforcement learning agent samples multiple trajectories in the simulation environment according to the current policy, improves the policy objective function by shearing probability ratio, controls the update step size, and avoids drastic policy deviation; and introduces time difference objective for training value function network to improve the stability and accuracy of policy estimation.

[0069] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A supercritical coal-fired unit control method based on deep reinforcement learning, characterized in that, The application relates to a method for constructing a supercritical coal-fired unit dynamic model. The application relates to a method for constructing a supercritical coal-fired unit dynamic model. The application relates to a method for constructing a supercritical coal-fired unit dynamic model. The application relates to a method for constructing a supercritical coal-fired unit dynamic model. The application relates to a method for constructing a supercritical coal-fired unit dynamic model. The input data of the supercritical coal-fired unit dynamic model comprises a fuel quantity, a main steam valve opening degree and a feed water flow; and the output data of the supercritical coal-fired unit dynamic model comprises a coal-fired unit power, a main steam pressure and a steam-water separator outlet enthalpy value.

2. The method of claim 1, wherein, The control target is that the absolute value of the deviation between the actual value and the given value of the power is minimum, the absolute value of the deviation between the actual value and the given value of the main steam pressure is minimum, and the absolute value of the deviation between the actual value and the given value of the steam-water separator outlet enthalpy value is minimum.

3. The method of claim 2, wherein, The state comprises the deviation between the power and the given value at the t time, the deviation between the actual value and the given value of the main steam pressure at the t time, the deviation between the actual value and the given value of the steam-water separator outlet enthalpy value at the t time, the power change rate at the t time, the main steam pressure change rate at the t time, the steam-water separator outlet enthalpy value change rate at the t time, the fuel quantity at the t-1 time, the main steam valve opening degree at the t-1 time and the feed water flow at the t-1 time.

4. The method of claim 2, wherein, The action comprises a fuel quantity increment, a main steam valve opening degree increment and a feed water flow increment at the t time. The reward is the sum of a main target item and a secondary target item, the main target item comprises the absolute value of the deviation between the actual value and the given value of the power, the absolute value of the deviation between the actual value and the given value of the main steam pressure, and the absolute value of the deviation between the actual value and the given value of the steam-water separator outlet enthalpy value in different value ranges; and the secondary target item is a punishment item when the state does not satisfy the upper and lower limit constraint conditions. The reinforcement learning agent comprises a policy network and a value function network, and the reinforcement learning agent is trained according to the state, the action, the reward and the supercritical coal-fired unit dynamic model, and the converged reinforcement learning agent is used as a reinforcement learning controller.

5. The method of claim 1, wherein, S41, initializing the policy network, the value function network, the supercritical coal-fired unit dynamic model and an experience replay buffer, and creating an old policy network, wherein the structure of the old policy network is the same as that of the policy network; S42, the old policy network and the supercritical coal-fired unit dynamic model interactively sample according to the state, the action and the reward, obtain experience data and store the experience data into the experience replay buffer; ​ S43, optimizing parameters of the policy network and the value function network based on a clipping probability, a time difference target, back propagation, and gradient update, according to the experience data; Steps S42 and S43 are repeated until the reinforcement learning agent meets a performance requirement.

6. The method of claim 5, wherein, The old policy network and the supercritical coal-fired unit dynamic model interactively sample according to the state, the action, and the reward, obtain experience data, and store the experience data into the experience replay buffer, including: An initial action data is input into the supercritical coal-fired unit dynamic model, the supercritical coal-fired unit dynamic model outputs state data at time t, and a reward at time t is calculated; The state data at time t is input into the old policy network, the old policy network selects an action output according to the state data at time t, which is denoted as an action at time t, and the action at time t output by the old policy network is input into the supercritical coal-fired unit dynamic model and outputs state data at time t+1; According to the action at time t and the state data at time t, a probability density at time t under the old policy network is calculated; The state data at time t, the action at time t, the reward at time t, the state data at time t+1, and the probability density at time t under the old policy network are stored as experience data into the experience replay buffer.

7. The method of claim 5, wherein, Parameters of the policy network and the value function network are optimized based on a clipping probability, a time difference target, back propagation, and gradient update, according to the experience data, including: Experience data in the experience replay buffer is obtained, and the value function network calculates a time difference target and a time difference error according to the reward at time t; An advantage estimate is calculated according to the time difference target and the time difference error; The state data at time t in the experience data in the experience replay buffer is input into the current policy network, the current policy network selects a corresponding action and outputs, a probability density under the current policy network is calculated according to the state data at time t and the output action of the current policy network; A probability ratio is calculated according to the probability density of the current policy network and the probability density of the old policy network, a clipping probability is introduced, a value of a loss function is calculated according to the advantage estimate, the probability ratio, the clipping probability, and the time difference target, and parameters of the current policy network and the value function network are updated by gradient descent to minimize the value of the loss function; Parameters of the old policy network are updated as parameters of the current policy network, and the experience replay buffer is emptied.

8. The method of claim 7, wherein, The probability ratio is a ratio of the probability density of the current policy network to the probability density of the old policy network; A value of a loss function is calculated according to the advantage estimate, the probability ratio, the clipping probability, and the time difference target, and parameters of the current policy network and the value function network are updated by gradient descent to minimize the value of the loss function, including: An entropy reward term is introduced, a first loss function is constructed according to the advantage estimate, the probability ratio, the clipping probability, and the entropy reward term; The policy network updates its own parameters by gradient descent to minimize the value of the first loss function; A second loss function is constructed according to the value prediction of the value function network and the time difference target, and the self parameters of the value function network are updated by minimizing the value of the second loss function through gradient descent.

9. A supercritical coal-fired unit control device based on deep reinforcement learning, characterized in that, The method comprises the following steps: A model construction module is configured to construct an initial supercritical coal-fired unit dynamic model; A model training module is configured to collect historical operation data of the supercritical coal-fired unit, train the initial supercritical coal-fired unit dynamic model according to the historical operation data, and obtain a supercritical coal-fired unit dynamic model after the training is completed; A target construction module is configured to construct a control target and a state, an action and a reward of the supercritical coal-fired unit coordinated control; An agent training module is configured to construct a reinforcement learning agent, train the reinforcement learning agent according to the state, the action, the reward and the supercritical coal-fired unit dynamic model, and use the converged reinforcement learning agent as a reinforcement learning controller; An operation control module is configured to input a given control parameter and a state parameter in an actual operation process of the supercritical coal-fired unit to the reinforcement learning controller, and output an optimized control parameter by the reinforcement learning controller, wherein the optimized control parameter is used to optimize the control of the actual operation state of the supercritical coal-fired unit.

10. An electronic device, comprising: The device comprises a processor and a storage device, the storage device stores a plurality of instructions, and the processor is used to read the instructions and execute the method according to any one of claims 1-8.

Images