An Optimal Scheduling Method for Pulverizing System Based on Balanced Distribution of Boiler Heat Load

Abstract

The present invention proposes an optimization scheduling method for a pulverizing system based on balanced distribution of boiler heat load. The three indicators are respectively used to represent the economy, the boiler heat load balance as safety, and the outlet NOx concentration as environmental protection. These three indicators are combined as the value function of reinforcement learning. The actor-critic reinforcement learning framework is adopted to reasonably select the state, action and value function of reinforcement learning, and the network is continuously updated so that the reinforcement learning algorithm can learn the optimal control strategy and select the action that makes the long-term return the best, and finally realize the autonomous optimal decision-making optimization scheduling control of the pulverizing system.

Description

Technical Field

[0001] The invention belongs to the field of process control, and in particular relates to an optimization scheduling method for a pulverizing system based on balanced distribution of boiler heat load. Background Art

[0002] In recent years, the growth rate of renewable energy power in my country has been astonishing, and the problem of large-scale renewable energy power grid connection is serious. Thermal power units are responsible for peak load regulation. During the peak load regulation process, it is necessary to change the output of the coal mill when the unit changes load. Due to the different positions of coal mills, the output of different coal mills has different effects on boiler combustion. However, the actual site mainly relies on human subjective selection of coal mill output, lacks consideration of economy, safety and environmental protection, and often misses the best start and stop time, or chooses an inefficient coal distribution method for coal mill combination. Therefore, a method is needed to help operators make independent decisions and choose a reasonable coal mill output. Traditional genetic algorithms are optimized based on historical data. When the unit characteristics or coal quality changes, the optimized coal mill combination output distribution is no longer suitable, and historical data may not cover all situations. Therefore, a method that can learn online and explore autonomously to find the optimal distribution method is needed. Summary of the invention

[0003] The present invention proposes an optimization scheduling method for a pulverizing system based on balanced distribution of boiler heat load. The reinforcement learning algorithm can realize trial and error exploration of coal mill combination distribution, and finally find a strategy to select the optimal action according to the current state so that the long-term return of the action is the highest, that is, the value function is maximized. Boiler efficiency is used to represent economy, boiler heat load balanced distribution is used as safety, and outlet NO x Concentration is used as environmental protection. These three indicators are combined as the value function of reinforcement learning. The state, action and value function of reinforcement learning are reasonably selected, and the network is continuously updated so that the reinforcement learning algorithm can obtain the optimal control strategy and select the action that makes the best long-term return, and finally realize the autonomous optimal decision-making control of the milling system.

[0004] An optimal scheduling method for a pulverizing system based on balanced distribution of boiler heat load forms a value function by comprehensively considering the control effect of optimal distribution of pulverizer combination from the perspective of the economy, safety and environmental protection of boiler combustion. The pulverizer combination is optimized using a reinforcement learning deep deterministic strategy network algorithm. The actor-critic learning framework is adopted to learn the control strategy of pulverizer output, so that the economy, safety and environmental protection of the unit are optimized.

[0005] The value function is: R = a*R a +b*R b +c*R c

[0006] Among them, R aIt is the economic index of boiler combustion, R b It is the safety index of boiler combustion, R c It is the environmental protection index of boiler combustion, the weight of the three indexes a, b and c. The more important the index is, the higher the weight is. The economic index of boiler combustion is expressed by boiler efficiency, and the safety index of combustion is judged by the difference between the wall temperature measured on each heating surface of the boiler and the wall temperature on the heat balance diagram. The environmental protection of boiler combustion is related to the outlet NOx concentration. The weights a, b and c are all 1 / 3.

[0007] In the actor-critic learning framework, the online actor is based on the current state S t and strategy u, perform action a t , and get a return of r t and the next state S t+1 , the generated sequence (s t ,a t ,r t ,s t+1 ) is saved to the experience pool, and samples are taken from the experience pool to update the network parameters; the online critic gives the behavior value function according to the current state; the target critic and actor are used to predict the state s i+1 The behavior value function y under i , the target actor is responsible for the next state s in the experience pool i+1 The optimal action is selected, and the target critic is responsible for calculating the behavior value function after the selected action; the networks of both use soft updates and use the sliding average method to copy the latest network parameters from the current network. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a distribution diagram of a boiler coal mill;

[0009] Figure 2 This is the schematic diagram of the coal mill combination optimization based on reinforcement learning of the present invention. DETAILED DESCRIPTION

[0010] The present invention provides a method for optimizing the scheduling of a pulverizing system based on balanced distribution of boiler heat load, such as Figure 2 As shown in the figure, a coal mill combination optimization method based on reinforcement learning is established.

[0011] In the actual power output system, the coal mill is located in different positions, such as Figure 1As shown, the output of different coal mills has different effects on boiler combustion. However, during the peak load regulation of the unit, the output of the coal mill mainly depends on the subjective selection of the staff, and the best start and stop time of the coal mill is often missed, or the efficiency of the selected coal mill combination coal distribution method is low, lacking consideration of economy, safety and environmental protection. The present invention proposes to comprehensively consider the control effect of the optimized distribution of coal mill combinations from the three aspects of economy, safety and environmental protection.

[0012] The combination and allocation of coal mills affects the combustion conditions of boilers and the operating efficiency of boilers. Therefore, the economic efficiency is expressed by boiler efficiency. The formula is as follows:

[0013]

[0014] Among them, Q1 is the effective heat absorption of the boiler, Q b is the total heat of the boiler, R a It is an index of boiler economy, ranging from 0 to 1, and the closer to 1, the higher the economy.

[0015] Unreasonable coal mill combination makes boiler load unbalanced and even causes boiler coking. Boiler heat load balance is directly reflected in the wall temperature of each heating surface of the boiler. The wall temperature on the heat balance diagram is the standard value when the boiler is designed. The difference between the wall temperature measured on each heating surface and the wall temperature on the heat balance diagram is calculated as the basis for judging the safety of combustion. The formula is as follows:

[0016]

[0017] Where Eu is the maximum temperature error, T(i) is the current wall temperature, and T a (i) is the standard wall temperature on the heat balance diagram, n is the number of wall temperatures, and the purpose of the numerator in the above formula is to obtain the mean value of the wall temperature difference.

[0018] The boiler combustion state will affect the outlet NO x concentration, incomplete combustion in the boiler leads to outlet NO x The increase in concentration affects the environmental performance of the unit. The formula is as follows:

[0019]

[0020] Among them, C nox For the current export NO x Concentration, C min is the minimum concentration, C max is the maximum concentration.

[0021] Combining the above three indicators, a value function statistical formula is formed:

[0022] R=a*R a +b*Rb +c*R c

[0023] According to R a (Boiler efficiency, economic index), R b (wall temperature difference, safety index), R c The importance of (export NOx concentration, environmental protection index) is determined by selecting appropriate weights a, b, and c. The more importance is attached, the higher the weight is, so that the three indicators are unified into the value function R. The default values ​​of a, b, and c are all 1 / 3.

[0024] Reinforcement learning algorithm is a method with online learning and autonomous exploration capabilities, so the present invention uses the reinforcement learning deep deterministic policy network algorithm (DDPG) to optimize the coal mill combination. The actor-critic learning framework is adopted to learn the control strategy of coal mill output, so that the economy, safety and environmental protection of the unit are optimized. The critic uses a neural network to approximate the behavior value function, and the actor updates the policy parameters in the direction of the behavior value function gradient.

[0025] like Figure 2 As shown in the figure, DDPG includes an online policy network, a target policy network, an online valuation network, and a target valuation network. The online network (including the online policy network and the online valuation network) updates the weights in real time and assigns the weights to the target network (including the target policy network and the target valuation network) in a sliding average manner to prevent the non-convergence problem caused by the target network's variable learning and variable actions.

[0026] The role of online actors is based on the current state S t and strategy u, perform action a t , and get a return of r t and the next state S t+1 , the generated sequence (s t ,a t ,r t ,s t+1 ) is saved in the experience pool, and samples are taken from the experience pool to update the network parameters. The state includes the output, load, and water flow of each coal mill, and the action is the change of coal mill output. According to the deterministic policy gradient theorem, the loss function of updating the online actor network is:

[0027]

[0028] Among them, m represents the m samples taken from the experience pool, Q(s i ,a i ,θ u) is the behavior value function after taking action according to the strategy, and the behavior value function is given by the online critic network. During the training process, the value function is expected to be maximized, so the neural network back-propagates to update the online actor network parameters θ u , so that the loss function is minimized.

[0029] The role of the online critic is to give an action value function based on the current state. The loss function of the online critic network is as follows:

[0030]

[0031] y i =r i +Q′(s i+1 ,u′(s i+1 |θ u′ )|θ Q′ )

[0032] Among them, the online critic network parameter is θ Q , the neural network back propagates to update the online critic network parameters θ Q , so that the loss function is minimized. The target critic network parameter is θ Q′ , the target actor network parameter is θ u′ , the main role of the target critic and actor is to predict the state s i+1 The behavior value function y under i , the target actor is responsible for the next state s in the experience pool i+1 The optimal action is selected, and the target critic is responsible for calculating the behavior value function after the selected action. The networks of both use soft update, and the sliding average method is used to copy the latest network parameters from the current network. The formula is as follows.

[0033] θ Q′ ←τθ Q +(1-τ)θ Q′

[0034] θ θ′ ←τθ θ +(1-τ)θ θ′

[0035] Among them, τ is the update coefficient, which is generally set to 0.01. During the training process, the grinding combination processing strategy given by the online actor will become better and better, making the value function converge to the maximum value.

[0036] Due to the different positions of coal mills, different coal mill outputs have different effects on the boiler combustion state. In order to analyze the influence of coal mill output on the boiler combustion state, neural network training is used to establish a boiler burner model. The boiler burner model represents the influence of coal mill output on the boiler combustion state. The input of neural network training is all coal mill outputs, and the output is the boiler wall temperature and outlet nitrogen oxide concentration. The established model can simulate the actual combustion state of the boiler to a certain extent.

[0037] The reinforcement learning network is initially trained on the established boiler burner model. When the training phase converges, the optimal coal feed rate for each coal mill can be obtained. During the actual operation of the reinforcement learning network, while giving the optimal coal feed rate, the reinforcement learning network adds noise to the deterministic strategy to construct a behavioral strategy, and tries the current state with a small probability, thereby continuously updating the reinforcement learning network and adapting the coal mill output state to the current state (s t state).

[0038] The use of reinforcement learning network to learn the autonomous decision-making control method of the pulverizing system can make the coal mill output act in advance, realize the long-term planning of the boiler economy, safety and environmental protection, keep the coal mill in the optimal output state, and finally the pulverizing system achieves autonomous decision-making control.

[0039] This embodiment is only a preferred specific implementation of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for optimizing the scheduling of a pulverizing system based on balanced distribution of boiler heat load, characterized in that: The control effect of coal mill combination optimization allocation is comprehensively considered from the perspective of boiler combustion economy, safety and environmental protection to form a value function, and the coal mill combination output is optimized using a reinforcement learning network. The actor-critic learning framework is adopted to learn the control strategy of coal mill output, so that the economy, safety and environmental protection of the unit are optimized; the value function is: R=a*R a +b*R b +c*R c Among them, R a It is the economic index of boiler combustion, R b It is the safety index of boiler combustion, R c It is the environmental protection index of boiler combustion. a, b, and c are the weights of the three indicators. The more important the indicator is, the higher the weight is. The economic index of boiler combustion is expressed by boiler efficiency, and the formula is as follows: Among them, Q1 is the effective heat absorption of the boiler, Q b is the total heat of the boiler; The combustion safety index is determined by the difference between the wall temperature measured on each heating surface of the boiler and the wall temperature on the heat balance diagram. The formula is as follows: Where Eu is the maximum temperature error, T(i) is the current wall temperature, and T a (i) is the standard wall temperature on the thermal equilibrium diagram, n is the number of wall temperatures; Environmental protection of boiler combustion and outlet NO x The concentration is related to the formula as follows: Among them, C nox For the current export NO x Concentration, C min is the minimum concentration, C max is the maximum concentration.

2. The method for optimizing the scheduling of a pulverizing system based on balanced distribution of boiler heat load according to claim 1, characterized in that: The weights a, b, and c are all 1 / 3.

3. According to the method for optimizing the scheduling of a pulverizing system based on balanced distribution of boiler heat load in claim 1, in the actor-critic learning framework, the online actor is based on the current state S t and strategy u, perform action a t , and get a return of r t and the next state S t+1 , the generated sequence (s t ,a t ,r t ,s t+1 ) is saved in the experience pool, and samples are taken from the experience pool to update the network parameters; the state includes the output, load, and water flow of each coal mill, and the action is the change of coal mill output; according to the deterministic policy gradient theorem, the loss function of the online actor network is updated as: in, m represents m samples taken from the experience pool, Q(s i ,a i ,θ u ) is the behavior value function after taking action according to the strategy, and the behavior value function is given by the online critic network in the actor-critic learning framework; Training process Neural network back propagation updates the online actor network parameters θ u , so that the loss function is minimized.

4. According to the method for optimizing the scheduling of a pulverizing system based on balanced distribution of boiler heat load in claim 3, in the actor-critic learning framework, the online critic gives a behavior value function according to the current state, and the loss function of the online critic network is as follows: y i =r i +Q′(s i+1 ,u′(s i+1 |θ u′ )|θ Q′ ) in, The online critic network parameters are θ Q , the neural network back propagates to update the online critic network parameters θ Q , which minimizes the loss function; the target critic network parameter is θ Q′ , the target actor network parameter is θ u′ , the target critic and actor are used to predict the state s i+1 The behavior value function y under i , the target actor is responsible for the next state s in the experience pool i+1 The target critic is responsible for calculating the behavior value function after selecting the optimal action. The networks of the two adopt soft update and use the sliding average method to copy the latest network parameters from the current network. The formula is as follows: i Q′ ←tth Q +(1-τ)θ Q′ i θ′ ←tth θ +(1-τ)θ θ′ Among them, τ is the update coefficient, which is generally set to 0.01; during the training process, the grinding combination processing strategy given by the online actor will become better and better, making the value function converge to the maximum value.

5. The method for optimizing the scheduling of a pulverizing system based on balanced distribution of boiler heat load according to claim 1, characterized in that: The reinforcement learning network is initially trained on the established boiler burner model. When the reinforcement learning network converges during the training phase, the optimal coal feed rate for each coal mill can be obtained. During the actual operation of the reinforcement learning network, while giving the optimal coal feed rate, the reinforcement learning network tries the current state with a small probability, thereby continuously updating the reinforcement learning network and making the coal mill output state adapt to the current state.

Images