A biomass thermal power station SCR process reinforcement learning control method

Abstract

The present application relates to a kind of biomass thermal power station SCR process reinforcement learning control method, constructs biomass thermal power station SCR process learning model, select several variables as action value, with the denitration processing efficiency of biomass thermal power station as state value, set each parameter variable value range, determine the optimal controllable action parameter set that meets the condition in SCR process;With the optimal controllable action parameter set optimization biomass thermal power station SCR process denitration efficiency and economic cost.The present application makes SCR denitration process NO x Concentration prediction model output meets SCR denitration processing outlet NO x Concentration, denitration processing efficiency is in reasonable range interval and total denitration processing cost minimization, i.e. When can obtain the optimal controllable action parameter set that meets the condition;Without in-depth mastery catalytic reduction reaction mechanism process, only need to be driven by industrial data, realize the optimization control of biomass thermal power station SCR process by the prediction model constructed by artificial intelligence algorithm.

Description

Technical Field

[0001] This invention belongs to the technical field of control or regulation systems for non-electrical variables, and specifically relates to a reinforcement learning control method for SCR processes in biomass thermal power plants. Background Technology

[0002] Biomass energy, as a clean and renewable energy source, boasts advantages such as environmental friendliness and high energy efficiency, making it my country's fourth largest energy source after coal, oil, and natural gas. Biomass combined heat and power (CHP) is the most direct and effective way to efficiently utilize biomass energy. However, the flue gas emitted from biomass power plant boilers contains nitrogen oxides, which can easily cause serious environmental pollution problems such as smog and acid rain, necessitating combined flue gas denitrification treatment.

[0003] Currently, biomass thermal power plants mainly employ Selective Catalytic Reduction (SCR) to treat boiler combustion flue gas for denitrification. In the SCR process control of biomass thermal power plants, the commonly used automatic control technology is PID control. This involves automatically adjusting the flow rate of the reducing agent (such as liquid ammonia, NH3, or urea) through a PID controller to selectively react with NO in the flue gas. x The reaction produces non-toxic and pollution-free N2 and H2O, thus achieving combined denitrification treatment of boiler combustion flue gas in biomass thermal power plants. Therefore, precise control of the reducing agent flow rate is one of the key factors for achieving efficient denitrification treatment of combustion flue gas in the SCR process of biomass thermal power plants.

[0004] NO in the SCR process of biomass thermal power plants x The concentration is affected by operating parameters such as unit load, reducing agent flow rate, O2 content in SCR inlet flue gas, and SCR inlet flue gas temperature. However, biomass fuels are diverse in type and properties, resulting in varying NO concentrations in boiler flue gas. x The large concentration differences, coupled with significant fluctuations in heat and electricity load demands, result in highly variable and rapidly changing operating conditions for biomass thermal power plants, leading to frequent and large fluctuations in boiler flue gas flow. Therefore, existing PID control methods struggle to optimize the flow rate of the reducing agent, failing to guarantee the effectiveness and economy of the SCR process for denitrification in biomass thermal power plants. First-principles modeling-based SCR process control methods require establishing mathematical conservation relationships for the catalytic reduction process based on the chemical reaction. However, the SCR process in a biomass thermal power plant boiler is clearly a complex, nonlinear, and multivariable coupled chemical reaction system. Establishing a precise mechanistic model would be extremely labor-intensive and complex, making it difficult to meet the automatic control requirements of the SCR process in biomass thermal power plants. Summary of the Invention

[0005] This invention solves the problems existing in the prior art and provides a reinforcement learning control method for the SCR process in a biomass thermal power plant. The method is simple to design, easy to operate, and can simultaneously achieve optimal automatic control.

[0006] The technical solution adopted by this invention to solve its technical problem is a reinforcement learning control method for the SCR process of a biomass thermal power plant. This method involves constructing a learning model of the SCR process in the biomass thermal power plant, selecting several variables as action values, using the denitrification efficiency of the biomass thermal power plant as the state value, setting the value range of each parameter variable, and determining the optimal set of controllable action parameters that meet the conditions during the SCR process. The optimal set of controllable action parameters is then used to optimize the denitrification efficiency and economic cost of the SCR process in the biomass thermal power plant.

[0007] Preferably, the variables used as action values ​​include unit load, reductant flow rate, SCR inlet flue gas O2 content, SCR inlet flue gas temperature, and SCR inlet NO content. x The concentration and denitrification efficiency are given by equation (1).

[0008]

[0009] Where η is the denitrification efficiency, I noxin SCR entry NO x Concentration, I noxout NO for SCR exports x concentration.

[0010] Preferably, the denitrification processing cost T is used as the reward function of the model to control the mass flow rate of the reducing agent.

[0011] Preferably, the cost of denitrification treatment is as shown in equation (2).

[0012]

[0013] Among them, P NH3 M is the unit price of the reducing agent. NH3 L represents the total reduction dose, and L represents the unit load.

[0014] Preferably, the SCR process learning model of the biomass thermal power plant is based on the DDPG algorithm, constructing an associated Actor network and Critic network. The DDPG deep reinforcement learning algorithm adopts an Actor-Critic architecture, where the Critic evaluation in state s uses the behavior value function of action a. The parameterized representation of the Critic network is Q(s,a|θ). Q ), θ Q The parameters of the Critic deep neural network are represented by the Actor, which approximates the deterministic action policy a = μ(s), and the output is the action a. The parameterized representation of the Actor network is μ(s|θ). μ ), θμ This represents the parameters of the Actor deep neural network.

[0015] The Actor is responsible for correcting the bias of the action based on the Critic network evaluation, while the Critic is responsible for scoring the actions generated by the Actor.

[0016] The overall network workflow is roughly as follows:

[0017] (1) First, the Actor generates an action based on the current environment;

[0018] (2) The environment provides a corresponding reward r based on the Actor;

[0019] (3) The Critic will evaluate the action;

[0020] (4) The Actor will adjust its strategy and output a new action based on the Critic's evaluation;

[0021] (5) Critic will adjust the evaluation rules based on the return r;

[0022] (6) Repeat steps (1) to (5) until all networks converge or the threshold of the set training cycle is reached.

[0023] In this patented algorithm, both the Actor network (main network and target network) and the Critic network (main network and target network) contain two hidden layer networks, with the number of neurons in each layer set separately;

[0024] The activation function of the last layer of the Actor network is the tanh function, which controls the action output of each layer between [-1, 1]. Finally, the controllable values ​​of each operating parameter of the denitrification process are obtained based on the range of the denitrification efficiency state value.

[0025] The Critic network evaluates the controllable parameters of the denitrification process obtained from the Actor network, using the ReLU activation function.

[0026] Preferably, determining the optimal set of controllable action parameters that meet the conditions during the SCR process includes the following steps:

[0027] Step 1: Randomly initialize the Critic network Q(s,a|θ) μ ) and Actor network μ(s|θ μ ), with initial weights θ Q and θ μ ;

[0028] Step 2: Initialize the target network Q′ and μ′, with initial weights θ. Q′ ←θ Qθ μ′ ←θ μ ;

[0029] Step 3: Initialize the experience replay pool size to R;

[0030] Step 4: For any event m, receive a state value s. t ;

[0031] Step 5: For any time t, select an action a based on the ε-greedy algorithm. t The algorithm uses a greedy approach with probability ε to randomly select an action; otherwise, it uses a... t =μ(s) t |θ μ Select as the current strategy;

[0032] Step 6: Within the current event, let t = t + 1, and return to step 5, where t = 1, ..., T, and T is the maximum number of iterations set; otherwise, proceed to step 7.

[0033] Step 7: Let m = m + 1, and return to step 4, where m = 1, ..., M, and M is the given number of training events.

[0034] Preferably, step 5 includes the following steps:

[0035] Step 5.1 Perform action a t Input into the known NO x In the prediction model, the SCR export NO is predicted. x Concentration, calculate denitrification efficiency, and generate reward r based on the set reward function. t and the new state value s t+1 The reward here is defined by the corresponding action; a positive reward is given if the requirements are met, and a negative reward is given otherwise.

[0036] Step 5.2: Transfer the sample data at time t (s) t ,a t ,r t ,s t+1 Stored in memory;

[0037] Step 5.3: When the memory bank is full, randomly sample N time-stamped data points (s). i ,a i ,r i ,s i+1 The dataset is used as a unit input group for training the Actor and Critic target networks; i = 1, ..., N;

[0038] Step 5.4: Set y i =r i+γQ'(s i+1 ,μ'(s i+1 |θ μ′ )|θ Q′ ), where γ is the discount factor;

[0039] Step 5.5: Update the Critic network parameters by minimizing the loss.

[0040] Step 5.6: Update the Actor network parameters using policy gradients.

[0041] Step 5.7: Update target network parameters Where ν is the update parameter.

[0042] Preferably, νQ << 1.

[0043] This invention relates to a reinforcement learning control method for the SCR process in a biomass thermal power plant. It constructs a learning model of the SCR process in the biomass thermal power plant, selects several variables as action values, uses the denitrification efficiency of the biomass thermal power plant as the state value, sets the value ranges of each parameter variable, and determines the optimal set of controllable action parameters that meet the conditions during the SCR process. The optimal set of controllable action parameters is then used to optimize the denitrification efficiency and economic cost of the SCR process in the biomass thermal power plant.

[0044] The beneficial effects of this invention are as follows: Based on the above-described DDPG algorithm flow, iterative training, i.e., the process continuously adjusts the parameters of the judge network to correct the parameters of the behavior network, until the Actor network and Critic network tend to stabilize, thereby optimizing the controllable operating parameters of the biomass boiler denitrification process, making the SCR denitrification process NO x The concentration prediction model output satisfies the NO concentration at the SCR denitrification treatment outlet. x When the concentration and denitrification efficiency are within a reasonable range and the total denitrification cost is minimized, the optimal set of controllable action parameters that meet the conditions can be obtained. Compared with the first-principles modeling method, the industrial big data-driven modeling method does not require in-depth understanding of the catalytic reduction reaction mechanism. It only needs to be driven by industrial data and a predictive model built by artificial intelligence algorithms to achieve optimized control of the SCR process in biomass thermal power plants. Attached Figure Description

[0045] Figure 1 This is a block diagram of the model structure of the present invention. Detailed Implementation

[0046] The method of the present invention will be further described in detail below with reference to the accompanying drawings.

[0047] This invention relates to a reinforcement learning control method for a biomass thermal power plant's SCR process, the method comprising the following steps:

[0048] (1) Establishment of deep reinforcement learning model

[0049] A general framework for constructing a deep reinforcement learning model for SCR denitrification efficiency based on the DDPG algorithm is presented. In the network structure design, both the Actor network (main network and target network) and the Critic network (main network and target network) contain two hidden layers, with the number of neurons in each layer set accordingly. The activation function of the last layer of the Actor network is the tanh function, which controls the action output of each layer within the range of [-1, 1]. Finally, the controllable values ​​of each operating parameter of the denitrification process are obtained based on the range of the denitrification efficiency state value. The Critic network evaluates the controllable parameters of the denitrification process obtained by the Actor network using the ReLU activation function.

[0050] After repeated experiments and debugging, the learning and training period of the DDPG model, the learning rate of the Actor network, and the learning rate of the Critic network were set.

[0051] The DDPG algorithm model selects unit load, reductant flow rate, SCR inlet flue gas O2 quantity, SCR inlet flue gas temperature, and SCR inlet NO. x Five variables, including concentration, are used as action values, and the denitrification efficiency is used as the state value. The range of values ​​for each parameter variable is also set.

[0052]

[0053] In equation (1), η is the denitrification efficiency, I noxin SCR entry NO x Concentration, I noxout NO for SCR exports x concentration.

[0054] In SCR denitrification, increasing the dosage of reducing agent can improve the denitrification efficiency. However, excessive reducing agent will also increase the cost of denitrification. The dosage of reducing agent is an important indicator for measuring the cost of denitrification. The cost of denitrification is calculated by multiplying the dosage of reducing agent per unit load by the corresponding unit price.

[0055] Typically, each boiler is equipped with two denitrification treatment devices, therefore the formula for calculating the denitrification treatment cost is as shown in equation (2):

[0056]

[0057] In equation (2), T is the total cost, and P is the total cost. NH3The unit price of the reducing agent (approximately 3500 yuan / ton at market price), M NH3 L represents the total reduction dose, and L represents the unit load.

[0058] Therefore, when setting the reward function of the model, the balance between the reduction dosage and the denitrification efficiency should be taken into account. According to expert experience, when the denitrification efficiency (η) is in a reasonable range of 85% to 95% and the total denitrification cost T is minimized, a reward should be given. In other cases, it is considered unreasonable and should be punished.

[0059] (2) Determine the optimal set of controllable action parameters that satisfy the conditions of the SCR process, as follows:

[0060] 2.1 In DDPG, the Actor and Critic are designed using neural networks. The Actor network is updated according to the gradient rule of a deterministic policy, while the Critic network is updated based on the gradient obtained from the temporal difference error. CriticQ(s,a|θ) is randomly initialized. μ ) and Actorμ(s|θ μ The main network parameters, with initial weights θ, are as follows: Q and θ μ ;

[0061] 2.2 Initialize the target network Q′ and μ′, with initial weights θ Q′ ←θ Q θ μ′ ←θ μ ;

[0062] 2.3 Initialize the experience replay pool size to R;

[0063] 2.4 For an event m (m = 1, ..., M), receive a state value s. t ;

[0064] 2.5 For t (t=1,...,T), select an action value a based on the ε-greedy algorithm. t : Select an action randomly with probability ε, otherwise select an action with probability a. t =μ(s) t |θ μ Choose from the current strategy;

[0065] 2.5.1 Perform action a t The input is given to the denitrification efficiency prediction model to predict the SCR outlet NO. x The concentration is then used to calculate the denitrification efficiency η, and finally, a reward r is generated according to the set reward rules. t and the new state value s t+1 ;

[0066] 2.5.2 Sample data at time t (s) t ,a t ,r t ,s t+1 Stored in memory bank b;

[0067] 2.5.3 When the memory bank is full, randomly sample N transformation data (s) i ,a i ,r i ,s i+1 (), used as a unit input dataset for training the Actor and Critic target networks;

[0068] 2.5.4 Setting y i =r i +γQ'(s i+1 ,μ'(s i+1 |θ μ′ )|θ Q′ ), where γ is the discount factor;

[0069] 2.5.5 Update the Critic network parameters by minimizing the loss:

[0070] 2.5.6 Update Actor network parameters using policy gradients:

[0071] 2.5.7 Last update of target network parameters: Where ν is the update parameter, which is set to νQ << 1.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0077] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A reinforcement learning control method for SCR process in a biomass thermal power plant, characterized in that: A learning model for the SCR process of a biomass thermal power plant is constructed. The learning model is based on the DDPG algorithm, selects several variables as action values, takes the denitrification efficiency of the biomass thermal power plant as the state value, sets the value range of each parameter variable, and determines the optimal set of controllable action parameters that meet the conditions in the SCR process. Determining the optimal set of controllable action parameters that meet the conditions during the SCR process includes the following steps: Step 1: Randomly initialize the Critic network and Actor Network The initial weights are respectively and ; Step 2: Initialize the target network Q′ and μ′, with initial weights as follows: , ; Step 3: Initialize the experience replay pool size to R; Step 4: For any event m, receive a state value s. t ; Step 5: For any time t, select an action a based on the ε-greedy algorithm. t The ε-greedy algorithm randomly selects an action with probability ε, otherwise... Choose as the current strategy; Step 6: Within the current event, let t = t + 1, and return to step 5, where t = 1, ..., T, and T is the maximum number of iterations set; otherwise, proceed to step 7. Step 7: Let m = m + 1, and return to step 4, where m = 1, ..., M, and M is the given number of training events; The denitrification efficiency and economic cost of the SCR process in a biomass thermal power plant are optimized using the optimal set of controllable action parameters.

2. The reinforcement learning control method for SCR process in a biomass thermal power plant according to claim 1, characterized in that: The variables used as action values ​​include unit load, reductant flow rate, SCR inlet flue gas O2 content, SCR inlet flue gas temperature, and SCR inlet NO. x The concentration and denitrification efficiency are given by equation (1). (1) Where η is the denitrification efficiency, I noxin SCR entry NO x Concentration, I noxout NO for SCR exports x concentration.

3. The reinforcement learning control method for SCR process in a biomass thermal power plant according to claim 2, characterized in that: The denitrification processing cost T is used as the reward function of the model to control the mass flow rate of the reducing agent.

4. The reinforcement learning control method for SCR process in a biomass thermal power plant according to claim 3, characterized in that: The cost of denitrification treatment is shown in equation (2). (2) Among them, P NH3 M is the unit price of the reducing agent. NH3 L represents the total reduction dose, and L represents the unit load.

5. The reinforcement learning control method for SCR process in a biomass thermal power plant according to claim 1, characterized in that: Construct an associated Actor network and Critic network, each containing two hidden layers, with the number of neurons in each layer set accordingly. The activation function of the last layer of the Actor network is the tanh function, which controls the action output of each layer between [-1, 1]. Finally, the controllable values ​​of each operating parameter of the denitrification process are obtained based on the range of the denitrification efficiency state value. The Critic network evaluates the controllable parameters of the denitrification process obtained from the Actor network, using the ReLU activation function.

6. The reinforcement learning control method for SCR process in a biomass thermal power plant according to claim 1, characterized in that: Step 5 includes the following steps: Step 5.1 Perform action a t Input into the known NO x In the prediction model, the SCR export NO is predicted. x Concentration, calculate denitrification efficiency, and generate reward r based on the set reward function. t and the new state value s t+1 ; Step 5.2: Transfer the sample data at time t (s t , a t , r t , s t+1 Stored in memory; Step 5.3: When the memory bank is full, randomly sample N time-stamped data (s). i , a i , r i , s i+1 The dataset is used as a unit input group for training the Actor and Critic target networks; i=1,···,N; Step 5.4: Settings , where γ is the discount factor; Step 5.5: Update the Critic network parameters by minimizing the loss. ; Step 5.6: Update the Actor network parameters using policy gradients. ; Step 5.7: Update target network parameters ,in It updates the parameters.

7. The reinforcement learning control method for SCR process in a biomass thermal power plant according to claim 6, characterized in that: ＜＜1。

Images