A waste incinerator discharge automatic control method based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's garbage incinerator discharge automatic control method, comprising the following steps: setting garbage incinerator each control parameter and with a control parameter vector x=[v1, v2, d1, d2, v m , h, v l , m] is expressed;Setting toxic emission concentration vector W=[w1, w2,..., w n ] and energy consumption vector e=[e1, e2,..., e j ] is expressed, find a group of control parameter vector x, so that toxic emission concentration vector W and energy consumption vector e minimum: using deep deterministic policy gradient constructs Ev model, Ac model and Q model;With the historical monitoring data in incinerator DCS training Ev model, then with the trained Ev model corresponding data for Ac model and Q model training after action calculation, the trained Ac model can according to the state S of current incinerator Predict the best control parameter vector x, realize the automatic control of garbage incinerator flue gas emission.The application is adjusted by using reinforcement learning algorithm, combined with incinerator historical monitoring data, control parameter in incineration process is automatically adjusted, to realize that flue gas emission reaches environmental protection standard while reducing energy consumption.

Description

TECHNICAL FIELD

[0001] The present application relates to waste incineration flue gas emission control technology, in particular to a waste incinerator emission automatic control method based on reinforcement learning, which is used to ensure that the emitted flue gas meets environmental protection regulations and at the same time reduces the energy consumption of treating flue gas. BACKGROUND

[0002] With the acceleration of urbanization, the amount of garbage is increasing year by year. Waste incineration power generation, as an important means of treating municipal solid waste, has been widely used in many countries and regions. Waste incineration not only effectively reduces the volume of garbage, but also converts heat energy into electricity or other energy resources, with good environmental and economic benefits. However, during the waste incineration process, the flue gas and harmful gases produced have a significant impact on the environment. If the flue gas containing harmful gases (such as nitrogen oxides, sulfur dioxide, carbon monoxide, volatile organic compounds, etc.) during the waste incineration process is not effectively controlled, it will seriously pollute the air, endanger human health, and affect the ecological environment. In order to ensure that the harmful substances emitted during the incineration process meet the standards of environmental protection regulations, the emission control system of the waste incinerator needs to be accurately controlled. According to the relevant environmental protection regulations such as "Air Pollutant Emission Standards" (GB 13271-2014), the concentration of pollutants in waste incineration flue gas needs to be strictly controlled, especially the emission of harmful gases such as sulfur dioxide, nitrogen oxides, and carbon monoxide. These standards require continuous and real-time monitoring of gas emissions during waste incineration and dynamically adjusting the combustion conditions in the furnace based on the data to ensure that the emission concentration does not exceed the standard.

[0003] The existing waste incineration flue gas emission control technology usually relies on traditional control methods, such as PID control, fuzzy control, etc. These methods can stabilize the combustion process in the furnace to some extent, but due to the involvement of multiple variables in the waste incineration process, such as the calorific value, humidity, and composition of the garbage, traditional control methods are difficult to cope with the complex and variable incineration process, which can easily lead to system response lag, low combustion efficiency, and emission concentration exceeding the standard. In addition, traditional manual adjustment and static optimization methods are difficult to cope with complex working condition changes. In actual incineration processes, the composition and characteristics of garbage, weather conditions, and furnace temperature will change, and these factors have a dynamic and complex impact on the incineration process and its emissions. For the control of these uncertain factors, the effect of traditional methods is often not ideal. Therefore, how to realize the automation and intelligent control of the incineration process is the key to improving the efficiency of the waste incineration emission control system, reducing energy consumption, and ensuring compliance with emissions.

[0004] In addition, a large amount of energy is consumed in the treatment and purification of flue gas in the garbage incineration process, including air flow adjustment, temperature control, reaction time control, etc. Although the heat energy generated during garbage incineration can be used for self-energy supply, the energy consumption of flue gas purification equipment and heat energy recovery equipment is still large. Therefore, how to reduce the energy consumption of treating flue gas and optimize the flue gas purification process has become a difficult problem to be solved in garbage incineration technology. SUMMARY

[0005] The purpose of the present application is to solve the problem that harmful gases generated in the garbage incineration process cause serious pollution to the environment, the traditional control method cannot realize the automatic and intelligent control of the garbage incineration process, and the flue gas purification and heat energy recovery equipment consumes a large amount of energy in the treatment process. The present application provides a garbage incinerator emission automatic control method based on reinforcement learning. The method provided by the present application realizes the automatic control of garbage incineration through reinforcement learning. Reinforcement learning (RL) is a paradigm of machine learning, which learns how to make decisions through interaction with the environment. The main difference between reinforcement learning and supervised learning and unsupervised learning is that the learning process in reinforcement learning is completed through the interaction between the agent (Agent) and the environment (Environment). The agent performs actions in the environment, and adjusts the strategy according to the reward (Reward) feedback from the environment to achieve the predetermined goal.

[0006] In the garbage incinerator, the garbage will go through three stages of drying, burning and burning:

[0007] 1) The garbage is first transported to the drying grate by the feeder, and the water in the garbage begins to evaporate under the action of primary air and grate temperature;

[0008] 2) After the drying process is completed, the garbage is sent to the burning grate, and the garbage reaches the ignition point to start burning and release volatile components through the primary air volume. The high-temperature flue gas forms a high degree of turbulence under the action of the secondary air provided by the secondary air entering above the furnace, which fully decomposes the harmful gases in the flue gas;

[0009] 3) Finally, the garbage after the burning stage is sent to the burning grate to ensure that the residual unburned garbage is completely burned out, and the slag is sent to the slag pool by the slag conveying machine.

[0010] In order to optimize the above incineration process, the following working condition parameters must be controlled: primary air volume , secondary air volume , primary air direction , secondary air direction , grate moving speed , grate moving mode , including forward, backward and pause; garbage feeding rate ; ;

[0011] The goal of the control is to make the garbage emission meet the standard while making the energy consumption of the garbage incineration process as small as possible. The garbage emission index is mainly characterized by the concentration of CO, NOx and SOx toxic substances, denoted as a toxic emission concentration vector , wherein represents the concentration of the nth toxic emission, and n is a natural number greater than 0. The energy consumption is also represented by an energy consumption vector , wherein e j represents the electric power or fuel flow under the jth control parameter vector; at this time, the automatic control problem of the garbage incineration flue gas emission is converted into a minimization problem, that is, to find a set of control parameter vectors x, so that the sum of the toxic emission concentration vector W and the energy consumption vector e is minimized:

[0012]

[0013] to act;

[0014] The above minimization problem is a highly nonlinear non-convex optimization problem, and the method of the present application designs a garbage incineration flue gas control algorithm based on deep reinforcement learning architecture, specifically using Deep Deterministic Policy Gradient (DDPG) to construct a control model.

[0015] The technical scheme adopted by the present application is:

[0016] A garbage incinerator emission automatic control method based on reinforcement learning, characterized in that it comprises the following steps:

[0017] Set the garbage incinerator control parameters: primary air volume , secondary air volume , primary air direction , secondary air direction , grate moving speed , grate moving mode , including forward, backward and pause; garbage feeding rate , adsorbent dosage ; the above garbage incinerator control parameters are represented by a control parameter vector ;

[0018] The goal of the control is to make the garbage emission meet the standard while making the energy consumption of the garbage incineration process as small as possible. The garbage emission index is mainly characterized by the concentration of CO, NOx and SOx toxic substances, denoted as a toxic emission concentration vector wherein represents the nth toxic emission concentration, n is a natural number greater than 0, and e is an energy consumption vector represents, wherein e j represents the electric power or fuel flow under the jth control parameter vector; at this time, the automatic control problem of the waste incineration flue gas emission is converted into a minimization problem, that is, to find a set of control parameter vectors x, so that the sum of the toxic emission concentration vector W and the energy consumption vector e is minimized:

[0019]

[0020] to act;

[0021] Building a control model: using deep deterministic policy gradient to build a simulation environment network model, action network model, and critic network Q model; training model using historical monitoring data in the incinerator DCS, and then using the trained model to calculate the corresponding data after the action model and Q model training, the trained model can predict the optimal control parameter vector x according to the current state of the incinerator , realizing the automatic control of waste incineration flue gas emission.

[0022] First, define the environment state , W is the toxic emission concentration vector, e is the energy consumption vector, and action is the control parameter vector ; three neural network models are built respectively: simulation environment network model, action network model, and critic network Q model; the input and output of the three are as follows:

[0023] 1) The input of the simulation environment network model is the current environment state and the current action ; the output is the environment state after the action ; t is the current time;

[0024] 2) The input of the action network model is the current environment state ; the output is the current action ;

[0025] 3) The input of the critic network Q model is the environment state and the action ; the output is the value of the action ;

[0026] The three network models above all adopt the structure of multi-layer perceptron (MLP), and the training of each model is performed twice, the first time to simulate the environment network model, and the second time to simultaneously train the action network model and the critic network Q model.

[0027] The training steps of the environment network model are as follows:

[0028] 1) A simulation environment data set is constructed, which contains pairs of current environment states , current actions , and states after actions ; the data set is constructed using historical monitoring data in the incinerator DCS, and the sample pairs are obtained by differencing the time series of the historical monitoring data in the DCS; the historical monitoring data in the incinerator DCS includes primary air flow , secondary air flow , primary air direction , secondary air direction , grate moving speed , grate moving mode , garbage feeding rate , and adsorbent dosage ; the concentrations of various toxic emissions , where represents the concentration of the nth toxic emission, n is a natural number greater than 0,

[0029] 2) The sample pairs in the data set are divided into a training set and a validation set according to an 8:2 correspondence;

[0030] 3) A sample pair is randomly taken from the training set ;

[0031] 4) The predicted value of the state after the action is calculated using the simulation environment network Ev model ;

[0032]

[0033] 5) The loss function is calculated :

[0034] ;

[0035] 6) Backpropagation is performed to optimize the simulation environment network Ev model ;

[0036] ​7) Randomly take a sample pair from the validation set to verify the error of the optimized simulation environment network Ev model, and save the corresponding parameters of the input Ev model sample pair in this round;

[0037] 8) Repeat steps 3) to 7) for 10000 rounds, and finally take the corresponding parameters of the input Ev model sample pair with the lowest error on the validation set as the training result, and obtain the trained model.

[0038] After the model training is completed, the parameters of the model are frozen, and then the action network model and the critic network Q model are trained.

[0039] The training steps of the action network model and the critic network Q model are as follows:

[0040] 1) Initialize the replay buffer D;

[0041] 2) Randomly sample a current environment state from the data set;

[0042] 3) The current action is calculated by inputting the current environment state into the action network model:

[0043] ;

[0044] 4) The environment state after the action is calculated by using the trained Ev model:

[0045] ;

[0046] 5) Calculate the new action :

[0047] ;

[0048] 6) Calculate the reward function :

[0049] ;

[0050] 7) Store in the replay buffer D, which is used to store experience data during training ;

[0051] 8) Randomly extract an experience data from the replay buffer D;

[0052] 9) Calculate the loss function using Q model :

[0053] , y is an intermediate variable, is the reward function;

[0054] ;

[0055] 10) Backpropagation, optimize Q model;

[0056] 11) Calculate the policy gradient using Q model , where represents the parameters of the model;

[0057] ;

[0058] 12) Update the parameters of the model using gradient ascent , learning rate 0.001;

[0059] ;

[0060] 13) Repeat steps 2) to 12) above until the average change value of within or the number of repetitions exceeds 100,000, end training, get trained model and Q model.

[0061] After the model and Q model training is completed, the trained model can predict the optimal control parameter vector x according to the current state of the incinerator , to achieve automatic control of waste incineration flue gas emission.

[0062] The present application uses reinforcement learning algorithm, combined with historical monitoring data and flame data characteristics, automatically adjusts the control parameters in the incineration process, to achieve flue gas emission to meet environmental standards while reducing energy consumption. DETAILED DESCRIPTION

[0063] The present application will be further described in detail below with specific examples, which are helpful to clearly understand the present application, but they do not constitute limitations to the present application.

[0064] In the waste incinerator, waste will go through drying, burning, burning three stages:

[0065] ​​1) Waste is first delivered to the drying grate by the feeder, under the action of primary air and furnace temperature, the moisture in the waste begins to evaporate;

[0066] 2) After the drying process is completed, the waste is sent to the combustion grate, and the waste reaches the ignition point by the amount of primary air to start combustion and release volatile matter. The high-temperature flue gas forms a high degree of turbulence under the action of the secondary air provided by the secondary air entering from above the furnace, which fully decomposes the harmful gases in the flue gas;

[0067] 3) Finally, the waste after the combustion stage is sent to the burnout grate to ensure that the residual unburned waste is completely burned out, and the residue is sent to the residue pool by the residue delivery machine.

[0068] In order to optimize the above incineration process, the following working condition parameters must be controlled: primary air volume , secondary air volume , primary air direction , secondary air direction , grate moving speed , grate moving mode , including forward, reverse and pause; waste feeding rate , adsorbent dosage ;

[0069] The goal of control is to make the waste emission meet the standard, while making the energy consumption of the waste incineration process as small as possible. The waste emission indicators are mainly represented by the concentrations of CO, NOx, SOx toxic substances, denoted as a toxic emission concentration vector , where represents the concentration of the nth toxic emission, n=9, is the HCl emission concentration, is the HF emission concentration, is the SO2 emission concentration, is the NO x emission concentration, is the CO emission concentration, is the Hg emission concentration, is the Cd+Ti emission concentration, is the Pb+Cr and other heavy metal emission concentration, is the dioxin emission concentration, and the energy consumption is represented by an energy consumption vector , where e j represents the electric power or fuel flow under the jth control parameter vector; At this time, the automatic control problem of waste incineration flue gas emission is converted into a minimization problem, that is, to find a set of control parameter vectors x, so that the sum of the toxic emission concentration vector W and the energy consumption vector e is minimized:

[0070]

[0071] To act;

[0072] The above minimization problem is a highly nonlinear non-convex optimization problem, and the method of the application designs a waste incineration flue gas control algorithm based on a deep reinforcement learning framework, specifically uses a deep deterministic policy gradient (DDPG) to construct a control model.

[0073] The technical scheme adopted by the application is:

[0074] A waste incinerator emission automatic control method based on reinforcement learning, characterized by comprising the following steps:

[0075] Setting waste incinerator control parameters: primary air flow , secondary air flow , primary air direction , secondary air direction , grate moving speed , grate moving mode , including forward, backward and pause; garbage feeding rate ; adsorbent dosage ; the above waste incinerator control parameters are represented by a control parameter vector ;

[0076] Control objective: to make the waste emission meet the standard, and at the same time make the energy consumption of the waste incineration process as small as possible; the waste emission index is represented by the concentration of toxic and harmful emissions, denoted as a toxic and harmful emission concentration vector , wherein represents the concentration of the nth toxic and harmful emission, n is 9, is the HCl emission concentration, is the HF emission concentration, is the SO2 emission concentration, is the NO x emission concentration, is the CO emission concentration, is the Hg emission concentration, is the Cd+Ti emission concentration, is the Pb+Cr and other heavy metal emission concentration, is the dioxin emission concentration, and the control energy consumption is represented by an energy consumption vector , wherein e jrepresents the electric power or fuel flow under the jth control parameter vector; at this time, the automatic control problem of waste incineration flue gas emission is converted into a minimization problem, that is, to find a set of control parameter vectors x, so that the sum of the toxic emission concentration vector W and the energy consumption vector e is minimized:

[0077]

[0078] To act;

[0079] Building control model: using deep deterministic policy gradient to build simulation environment network Model, action network Model and critic network Q model; train Model using historical monitoring data in the incinerator DCS, and then use the trained Model to calculate the corresponding data after action Model and Q model training, trained Model can predict the optimal control parameter vector x according to the current state of the incinerator , realize the automatic control of waste incineration flue gas emission.

[0080] First, define the environment state , W is the toxic emission concentration vector, e is the energy consumption vector, action is the control parameter vector ; three neural network models are built respectively: simulation environment network Model, action network Model and critic network Q model; the input and output of the three are as follows:

[0081] 1) The input of simulation environment network Model is the current environment state And the current action ; the output is the environment state after action ; t is the current time;

[0082] 2) The input of action network Model is the current environment state ; the output is the current action ;

[0083] 3) The input of critic network Q model is environment state And action ; the output is the value of action ;

[0084] The above three network models all adopt the structure of multilayer perceptron (MLP), and the training of each model is carried out twice. The first time, the simulation environment network Model, second time simultaneous training action network Model and critic network Q model.

[0085] Simulation environment network The training steps of the model are as follows:

[0086] 1) Construct a simulation environment data set, which contains pairs of current environment states , current actions , and states after actions ; the data set is constructed using historical monitoring data in the incinerator DCS, and the time series of the historical monitoring data in the DCS is differentially obtained to obtain sample pairs ; the historical monitoring data in the incinerator DCS is primary air flow , secondary air flow , primary air direction , secondary air direction , grate moving speed , grate moving mode , garbage feeding rate , and adsorbent dosage ; HCl emission concentration , HF emission concentration , SO2 emission concentration , NO x emission concentration , CO emission concentration , Hg emission concentration , Cd+Ti emission concentration , Pb+Cr and other heavy metal emission concentration , dioxin emission concentration ;

[0087] 2) Divide the sample pairs in the data set into a training set and a validation set according to an 8:2 correspondence;

[0088] 3) Randomly take a sample pair from the training set ;

[0089] 4) Calculate the predicted value of the state after the action using the simulation environment network Ev model ;

[0090]

[0091] 5) Calculate the loss function :

[0092] ;

[0093] 6) Update the model parameters using the gradient descent method Back propagation, optimize the simulation environment network Ev model;

[0094] 7) randomly take a sample pair from the validation set to verify the error of the optimized simulation environment network Ev model, and save the corresponding parameters of the input Ev model sample pair in this round;

[0095] 8) repeat steps 3) to 7) for 10000 rounds, finally take the corresponding parameters of the input Ev model sample pair with the lowest error on the validation set as the training result, and get the trained model.

[0096] After the model training is completed, the parameters of the model are frozen, and then the action network model and the critic network Q model are trained.

[0097] The training steps of the action network model and the critic network Q model are as follows:

[0098] 1) initialize the replay buffer D;

[0099] 2) randomly sample a current environment state from the data set;

[0100] 3) model takes the current environment state as input to calculate the current action :

[0101] ;

[0102] 4) use the trained Ev model to calculate the environment state after the action:

[0103] ;

[0104] 5) calculate the new action :

[0105] ;

[0106] 6) calculate the reward function :

[0107] ;

[0108] 7) store in the replay buffer D, which is used to store experience data during training ;

[0109] 8) Randomly select one empirical data point from the playback buffer D. ;

[0110] 9) Calculate the loss function using the Q-model. :

[0111] y is an intermediate variable. It is a reward function;

[0112] ;

[0113] 10) To Backpropagation to optimize the Q-model;

[0114] 11) Calculate the policy gradient using the Q-model. ,in represent The parameters of the model;

[0115] ;

[0116] 12) Update using gradient ascent Model parameters, learning rate Take 0.001;

[0117] ;

[0118] 13) Repeat steps 2) to 12) above until... The average change value at End training when the number of repetitions is less than or exceeds 100,000, and you will obtain a well-trained result. Model and Q-model.

[0119] After the training of the model and the Q-model is completed, the trained model is obtained. The model can be based on the current state of the incinerator. Predict the optimal control parameter vector x to achieve automatic control of flue gas emissions from waste incineration.

Claims

1. A method for automatic emission control of waste incinerators based on reinforcement learning, characterized in that... Includes the following steps: Set the control parameters for the waste incinerator: primary air volume Secondary air volume Primary wind direction Secondary wind direction grate moving speed Grate moving mode This includes forward, backward, and pause functions; waste feeding rate. Adsorbent dosage The above-mentioned waste incinerator control parameters are expressed using a single control parameter vector. To indicate; The control objective: Waste emission indicators are characterized by the concentration of toxic substances, denoted as a toxic substance concentration vector. ,in Let n represent the concentration of the nth toxic pollutant, where n is a natural number greater than 0. An energy consumption vector is used to control energy consumption. It means that e j Let x represent the electrical power or fuel flow rate under the j-th control parameter vector. At this point, the automatic control problem of waste incineration flue gas emissions is transformed into a minimization problem, that is, finding a set of control parameter vectors x such that the sum of the toxic emission concentration vector W and the energy consumption vector e is minimized. For action; Building a control model: Constructing a simulation environment network using deep deterministic policy gradients. Models, Action Networks Model and critical network Q-model; trained using historical monitoring data from the incinerator DCS. The model, then using the trained The corresponding data after the model calculation action is Model and Q-model training, well trained The model can be based on the current state of the incinerator. Predict the optimal control parameter vector x to achieve automatic control of flue gas emissions from waste incineration.

2. The automatic emission control method for waste incinerators based on reinforcement learning according to claim 1, characterized in that... The specific steps for constructing a control model are as follows: First, define the environment state. W represents the concentration vector of toxic emissions, e represents the energy consumption vector, and action... For control parameter vector Three neural network models were constructed respectively: a simulated environment network. Models, Action Networks The model and the critical network Q-model; the inputs and outputs of the three are as follows: 1) Simulated environment network The model's input is the current environment state. and current action Output the environment state after the action. ; t represents the current time. 2) Action Network The model's input is the current environment state. Output the current action. ; 3) The input environment state of the critical network Q-model and actions The value of output actions ; All three network models mentioned above employ a multilayer perceptron (MLP) architecture. Training for each model is performed in two stages: the first stage trains a network simulating the environment. The model, the action network was trained simultaneously for the second time. Model and Critical Network Q Model.

3. The automatic emission control method for waste incinerators based on reinforcement learning according to claim 2, characterized in that... Simulated Environment Network The training steps for the model are as follows: 1) Construct a simulated environment dataset containing pairs of current environment states. Current action and the state after the action. ; The dataset was constructed using historical monitoring data from the incinerator's DCS (Distributed Control System). Sample pairs were obtained by differentiating the time series data of the historical monitoring data from the DCS. The historical monitoring data in the incinerator DCS is the primary air volume. Secondary air volume First wind direction Secondary wind direction grate moving speed Grate moving mode Waste feeding rate and adsorbent dosage Concentration of various toxic emissions , This represents the concentration of the nth toxic pollutant, where n is a natural number greater than 0; 2) Divide the sample pairs in the dataset into training and validation sets in an 8:2 ratio; 3) Randomly select a sample pair from the training set. ; 4) Calculate the predicted state value after the action using the simulated environment network Ev model. ; 5) Calculate the loss function : ； 6) Backpropagation to optimize the network Ev model in the simulated environment; 7) Randomly select a sample pair from the validation set to perform error validation on the optimized simulation environment network Ev model, and save the parameters corresponding to the sample pair of input Ev model in this round; 8) Repeat steps 3) to 7) for 10,000 rounds. Finally, take the set of input Ev model samples with the lowest error on the validation set and the corresponding parameters as the training result, and obtain the trained model. Model.

4. The automatic emission control method for waste incinerators based on reinforcement learning according to claim 3, characterized in that... Action Network The training steps for the model and the critical network Q-model are as follows: 1) Initialize the playback buffer D; 2) Randomly sample a current environment state from the dataset. ; 3) The model is based on the current environmental state. Calculate the current action based on the input. : ； 4) Calculate the environmental state after the action using the trained Ev model. : ； 5) Calculate new actions : ； 6) Calculate the reward function : ； 7) The data is stored in the playback buffer D, which is used to store empirical data during the training process. ; 8) Randomly select one empirical data point from the playback buffer D. ; 9) Calculate the loss function using the Q-model. : y is an intermediate variable. It is a reward function; ； 10) To Backpropagation to optimize the Q-model; 11) Calculate the policy gradient using the Q-model. ,in represent The parameters of the model; ； 12) Update using gradient ascent Model parameters, learning rate Take 0.001; ； 13) Repeat steps 2) to 12) above until... The average change value at End training when the number of repetitions is less than or exceeds 100,000, and you will obtain a well-trained result. Model and Q-model; trained The model can be based on the current state of the incinerator. Predict the optimal control parameter vector x to achieve automatic control of flue gas emissions from waste incineration.

Images