Coal ethylene glycol wastewater emission reduction control method based on first principle modeling and reinforcement learning

Abstract

The invention provides a coal ethylene glycol wastewater emission reduction control method based on first principle modeling and reinforcement learning, which comprises the following steps: establishing a coal ethylene glycol dynamic model based on a first principle, adding a controller for key process parameters according to experience, and performing actual working condition simulation by using the dynamic model to obtain a data set; obtaining a control variable global causal relationship graph by using Granger causal test GCT; constructing an agent model based on extreme gradient lifting XGBoost, and using the agent model to select a plurality of optimal control variables from the global causal relationship graph as operation variables; constructing and training an intelligent control model based on reinforcement learning RL to determine an optimal control decision; an intelligent control model based on reinforcement learning RL is used for monitoring uncertain feeding disturbance, an optimal control decision under the current operation condition is determined, an action control signal is transmitted to an intelligent controller to adjust corresponding control variables, and emission reduction of coal-to-ethylene glycol wastewater is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of clean production, and in particular to a method for controlling emission reduction of coal-to-ethylene glycol wastewater based on first principle modeling and reinforcement learning. Background Art

[0002] Coal-to-ethylene glycol (CtEG) is the production of ethylene glycol using coal instead of petroleum ethylene. Its main process routes include direct process, olefin process and oxalate process. Among them, oxalate process has the advantages of low energy consumption, good economy, mature technology and no need for complex catalysts. It is the preferred method for Chinese enterprises to produce ethylene glycol from synthesis gas. However, the wastewater generated in the process contains a variety of organic pollutants, and the chemical oxygen demand (COD) of the discharged wastewater increases due to the fluctuation of raw coal, which exceeds the limit of the actual treatment process and brings a heavy burden to environmental protection.

[0003] In the actual operation of chemical processes, variables change dynamically rather than being stable at a certain set value. Therefore, it is difficult to maintain a stable wastewater discharge, resulting in unsatisfactory effects in the wastewater treatment section. In order to stabilize the wastewater discharge, an effective process parameter control strategy should be added.

[0004] However, traditional proportional integral derivative (PID) control cannot promote coordinated control of the entire system. Therefore, future process systems will need to implement intelligent control models that can adapt to fluctuations in environmental characteristics to ensure the optimal autonomous operation of chemical processes. Reinforcement learning (RL) is a machine learning technology that maximizes benefits and makes optimal decisions through agent-environment interaction. With the continuous expansion of data volume and continuous optimization of algorithms, the significance of intelligent control models based on reinforcement learning has gradually emerged. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a coal-to-ethylene glycol wastewater emission reduction control method based on first principle modeling and reinforcement learning, comprising:

[0006] Step 1: Establish a dynamic model of coal-to-ethylene glycol based on first principles, add controllers for key process parameters based on experience, and use the dynamic model to simulate actual working conditions to obtain a data set;

[0007] Step 2: Use Granger causality test GCT to analyze the causal relationship between the control variables and wastewater, and obtain the global causal relationship diagram of the control variables;

[0008] Step 3: Build a proxy model based on extreme gradient boosting XGBoost, and use the proxy model to select the best several control variables as operation variables from the global causal relationship graph;

[0009] Step 4: construct and train an intelligent control model based on reinforcement learning (RL) to determine an optimal control decision, wherein the optimal control decision includes action control signals corresponding to optimal control variables;

[0010] Step 5. Use the intelligent control model based on reinforcement learning (RL) to monitor uncertain feed disturbances, determine the optimal control decision under current operating conditions, and transmit the action control signal to the intelligent controller to adjust the corresponding control variables to achieve wastewater reduction in coal-to-ethylene glycol production, wherein the intelligent controller is connected to controllers corresponding to several optimal control variables.

[0011] Optionally, the step of using a dynamic model to simulate actual working conditions to obtain a data set includes:

[0012] The dynamic model is used to simulate the operating conditions where disturbances lead to an increase in wastewater pollutants. The historical PID control data from the beginning of the disturbance to the process stabilization stage are selected to analyze the causal relationship between the control variables and the wastewater.

[0013] Optionally, the step 2 includes:

[0014] Granger causality test GCT was used to test the control variables under six disturbances, and the dependent variable that caused the increase of COD in wastewater was screened out from 30 control variables;

[0015] By analyzing the regularity under different interferences and combining the process flow to analyze the spatial information transmission process, the causal relationship under six interferences is integrated, and the transmission information of the process is structured to obtain a global causal relationship diagram.

[0016] Optionally, in step 3, the proxy model based on extreme gradient boosting XGBoost is as follows:

[0017] For each tree in the set, the XGBoost objective function consists of a single data point loss function and a regularization term, as shown in Formula 1. The expected value of each data point is updated according to the contribution of the newly inserted tree. This relationship is shown in Formula 2. The set tree is established by recursively partitioning the feature space, as shown in Formula 3. The predicted value of a leaf node is the weighted average of the training samples belonging to the leaf. By recursively partitioning the data, the established tree can minimize the objective function. The ideal split point for each node is determined by finding the split point with the largest gain, as shown in Formula 4.

[0018]

[0019] The estimator, maximum depth, and learning rate of the XGBoost model were selected as the main hyperparameters for optimization. During the optimization process, grid search and 5-fold cross validation were used to determine the best combination of parameters. Grid search involves traversing all combinations within the initial range, while 5-fold cross validation involves dividing the training dataset into five equal parts and using one of them as a validation set to evaluate the performance of the model. MAPE and R were calculated. 2 The score evaluates the performance of the model, as shown in Equations 5 and 6;

[0020]

[0021] The contribution of the control variables to the prediction results was analyzed according to Shapley Additive Explanations (SHAP). Taking into account process stability and operating costs, five control variables with larger SHAP values ​​were selected as operating variables.

[0022] Optionally, the step 4 includes:

[0023] The intelligent control model based on reinforcement learning (RL) is constructed in the context of Markov decision processes, where the agent directly interacts with the environment in a sequential manner. The agent controls the manipulated variables with causal relationships, and the environment is the agent model. COD and product quality are calculated based on the manipulated variables. The goal of reinforcement learning (RL) is to learn a control strategy that maximizes the sum of expected rewards for the actions taken in the current state.

[0024] In actual PID control, the feed rate is controlled by continuously controlling the valve, rather than opening or closing it discretely. Therefore, the action of the agent is a continuous process, and it takes real-time actions according to changes in the environment. t to t+1 The state transition is represented by s t+1 =f(a t ,s t ,ω t );

[0025] At time t, for the current state, the agent takes an action through the action neural network according to the current strategy. Then, the environment of the CtEG process enters the next state and obtains the reward under the action. Starting from the initial state, the agent repeats the above process to obtain a sequence, that is, the interaction trajectory τ between the agent and the environment, as shown in Formula 7:

[0026] τ=s 0 ,a 0 ,s 1 ,a 1 …s t ,a t ,r,a t+1Formula 7

[0027] Rewards are often used to encourage the selection of actions that achieve a desired outcome and can be calculated using Formula 8:

[0028] Reward=0.01 |Ratio-0.97|-1 -100+0.1 COD-1 Formula 8

[0029] Ratio refers to the purity of ethylene glycol after normalization. The purity of the product in the normal production process is 0.97. The closer the normalized COD is to 0, the smaller the pollutant value in the wastewater.

[0030] The SAC algorithm is used to allow the agent to explore the environment extensively to obtain the best strategy and avoid falling into the local optimum. The SAC algorithm introduces the maximum entropy mechanism to reduce entropy when repeatedly selecting a certain action and encourage exploration of other actions, thereby increasing the range of scheduling strategies explored and improving robustness. During the learning process, two Q networks are established. The SAC algorithm selects the smaller Q value of the two networks as the estimated value of the action value function and continuously and adaptively adjusts the policy parameters to increase the probability of obtaining higher returns. The parameters of the two Q networks are continuously updated during the training process. As the Q estimate increases, the actions selected by the decision network will also increase, thereby selecting better control actions to maximize rewards.

[0031] When the environment step reaches the set value, the environment parameters, actions and uncertainty factors will be reset to perform the next iteration. After all parameters are set, the intelligent control model based on reinforcement learning (RL) is iteratively trained until the average return rate stabilizes at a high level.

[0032] Optionally, the controller includes 6 flow controllers, 8 pressure controllers, 6 liquid level controllers, and 6 temperature controllers, and the flow controllers respectively control the feed flow rate of each material; the pressure controllers respectively control the pressure of the coal gasification reactor, the water-gas shift reactor, the gas separation and purification device, the methyl nitrite synthesis reactor, the dimethyl oxalate synthesis reactor, the flash evaporation device (FLASH), the ethylene glycol refining tower and the methanol recovery tower; the liquid level controllers respectively control the liquid levels of the gas separation and purification device tower bottom, the methyl nitrite synthesis reactor, the dimethyl oxalate synthesis reactor, the flash evaporation device (FLASH), the ethylene glycol refining tower and the methanol recovery tower; the temperature controllers respectively control the temperatures of the coal gasification reactor, the water-gas shift reactor, the methyl nitrite synthesis reactor, the dimethyl oxalate synthesis reactor, the ethylene glycol refining tower and the methanol recovery tower.

[0033] After adopting the above technical solution, the present invention has at least the following beneficial effects:

[0034] 1. The present invention constructs a proxy model based on extreme gradient boosting XGBoost, and uses the proxy model to select the optimal five control variables from the global causal relationship graph;

[0035] 2. From the perspective of the entire process control, the present invention constructs an intelligent control system for the CtEG process based on reinforcement learning, comprehensively considers wastewater discharge, product quality and operating costs, determines the optimal control decision, and realizes intelligent control of CtEG wastewater reduction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0037] Figure 1 A flow chart of a method for reducing and controlling coal-to-ethylene glycol wastewater emissions based on first-principles modeling and reinforcement learning is provided for an embodiment of the present disclosure.

[0038] Figure 2 This is the process flow chart for producing ethylene glycol from dimethyl oxalate.

[0039] Figure 3 It is a basic control system structure diagram of dynamic simulation in the intelligent control method described in the embodiment of the present disclosure.

[0040] Figure 4 It is a global causal relationship diagram of control variables for Granger causality test in the intelligent control method described in the embodiment of the present disclosure.

[0041] Figure 5 This is a structural diagram of the XGBoost model in the intelligent control method described in an embodiment of the present disclosure.

[0042] Figure 6 This is a diagram of prediction results of the agent model in the intelligent control method described in the embodiment of the present disclosure.

[0043] Figure 7 It is the SHAP value of the control parameter in the intelligent control method described in the embodiment of the present disclosure.

[0044] Figure 8 This is a diagram of the SAC network structure in the intelligent control method described in an embodiment of the present disclosure.

[0045] Fig. 9 It is the average reward obtained by the intelligent control model in the intelligent control method described in the embodiment of the present disclosure during the training and testing process.

[0046] Fig.10It is a structural diagram of the intelligent control system under feed disturbance in the intelligent control method described in the embodiment of the present disclosure.

[0047] Fig.11 These are the five optimal control values ​​output by the intelligent control model in the intelligent control method described in the embodiment of the present disclosure when reducing the coal inlet flow rate.

[0048] Fig.12 It is the COD emission reduction effect and product purity of the intelligent control model in the intelligent control method described in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The embodiment of the present disclosure provides a method for reducing the emission of wastewater from coal-to-ethylene glycol based on first principle modeling and reinforcement learning. The specific process is as follows: Figure 1 As shown, the main steps are as follows:

[0051] Step 1: Establish a dynamic model of coal-to-ethylene glycol based on first principles, add controllers for key process parameters based on experience, and use the dynamic model to simulate actual working conditions to obtain a data set;

[0052] The first stage of the CtEG process is to convert coal into synthesis gas. In this process, the coal is pretreated and then enters the coal gasification reactor (CGR) to react with oxygen and steam. The reaction products then pass through the water-gas shift reactor (WGSR) to produce crude synthesis gas (hydrogen, carbon monoxide, carbon dioxide, methane and other hydrocarbons). Through the gas separation and purification unit (GSP), the hydrogen in the crude synthesis gas is purified as a raw material for synthesizing ethylene glycol (EG). Another raw material for the EG process is methyl nitrite (MN), which is produced by reacting methanol with nitric acid in a methyl nitrite synthesis reactor (MNS). R); the second stage of the CtEG process is EG synthesis and refining. First, hydrogen and MN are reacted in a DMO synthesis reactor (DMOSR) to generate dimethoxy oxalate (DMO). Then, hydrogen and dimethylformamide are used to synthesize EG in an ethylene glycol synthesis reactor (EGSR). The resulting product mixture passes through an ethylene glycol refining tower (EGR) and a methanol recovery tower (RET) to refine EG to a purity of 99.8wt%, and recover the byproduct methanol with a purity of 98.0wt%. Finally, wastewater containing various organic substances is discharged from the recovery tower. The entire process is as follows: Figure 2The simulation results of CtEG wastewater are shown in Table 1. As can be seen from Table 1, the wastewater contains a variety of organic pollutants, resulting in a high COD of the wastewater.

[0053] Table 1 Simulation results of CtEG wastewater

[0054]

[0055] In order to more accurately simulate the actual operating conditions of the CtEG process, a dynamic model for wastewater detection and abatement treatment was constructed. The model takes into account that the actual operation of the device may be affected by the dynamic changes in feed flow or composition. In addition, the model can simulate various situations and events that may occur in the actual production process. Controllers are added to key process parameters based on experience. The relevant information of the controllers is shown in Table 2, including 6 flow controllers, 8 pressure controllers, 6 liquid level controllers, and 6 temperature controllers. The flow controllers control the feed flow of each material respectively; the pressure controllers control the gasification reactor and the water-gas shift reactor respectively. The pressure of the reactor, gas separation and purification unit, methyl nitrite synthesis reactor, dimethyl oxalate synthesis reactor, flash evaporation unit (FLASH), ethylene glycol refining tower and methanol recovery tower are controlled by the liquid level controller; the liquid level of the gas separation and purification unit tower kettle, methyl nitrite synthesis reactor, dimethyl oxalate synthesis reactor, flash evaporation unit (FLASH), ethylene glycol refining tower and methanol recovery tower are controlled by the temperature controller; the temperature controller controls the temperature of the coal gasification reactor, water-gas shift reactor, methyl nitrite synthesis reactor, dimethyl oxalate synthesis reactor, ethylene glycol refining tower and methanol recovery tower respectively. The flow chart and control structure of the dynamic simulation are shown in FIG. Figure 3 shown.

[0056] Table 2 CtEG process controller information

[0057]

[0058] The disturbances that occur during the CtEG process include input disturbances, system disturbances and external disturbances. In actual production, the frequency of external disturbances is low, and the study of the impact of input disturbances and system disturbances on wastewater is more valuable. Therefore, a dynamic model is used to simulate the conditions where input disturbances and system disturbances lead to increased pollutants in wastewater. As shown in Table 3, the PID control historical data from the beginning of the disturbance to the process stabilization stage are selected.

[0059] Table 3 Description of six types of interference

[0060]

[0061]

[0062] Step 2: Use Granger causality test GCT to analyze the causal relationship between the control variables and wastewater, and obtain the global causal relationship diagram of the control variables;

[0063] The Granger causality test GCT was used to test the control variables under the six interferences, and the dependent variables that caused the increase of COD in the wastewater were screened out from the 30 control variables. The Granger causality test GCT assumes that there is no causal relationship between the time series, and then tests whether this causal relationship can be overturned. If this causal relationship can be overturned, it verifies that the time series data is effective for the target to be predicted. As a result, the test index values ​​of all parameters are less than 0.05, proving that there is a Granger causal relationship between the variables and the wastewater. Finally, the control variables of the six interferences were tested, and the results are shown in Table 4.

[0064] Table 4 GCT results under six interferences

[0065]

[0066] From the results of GCT, the data of the whole process also presents complex multiple causal relationships. There are multiple causal paths for different interferences. By analyzing the regularity under different interferences and combining the process flow to analyze the spatial information transmission process, the global causal relationship diagram constructed here is based on the spatial information transmission process under six interferences. The variables added with interference are independent variables, and the control variables and the subsequent changes in wastewater characteristics are dependent variables. Therefore, there are multiple causal relationships between control variables, and there are causal relationships between processes. By integrating the causal relationships under six interferences and structuring the transmission information of the process, a global causal relationship diagram is obtained, as shown in Figure 1. Figure 4 shown.

[0067] Step 3: Build a proxy model based on extreme gradient boosting XGBoost, and use the proxy model to select the best several control variables as operation variables from the global causal relationship graph;

[0068] The structure of XGBoost is as follows Figure 5 As shown, for each tree in the set, the XGBoost objective function consists of a single data point loss function and a regular term, as shown in Formula 1. The expected value of each data point is updated according to the contribution of the newly inserted tree. This relationship is shown in Formula 2. The set tree is established by recursively partitioning the feature space, as shown in Formula 3. The predicted value of the leaf node is the weighted average of the training samples belonging to the leaf. By recursively partitioning the data, the established tree can minimize the objective function. The ideal split point of each node is determined by finding the split point with the largest gain, as shown in Formula 4;

[0069]

[0070] The selection of input parameters (15 in total) was based on the global causal relationship diagram. These control variables have a significant impact on the COD of the wastewater and the product quality. Due to the large amount of data, 2500 dynamic process data points were selected here to provide clearer information, which is helpful for understanding and evaluating the trend of operating variables of the CtEG process and revealing information about wastewater changes. Hyperparameter optimization determines the complexity and learning ability of the surrogate model. For the XGBoost model, the estimator, maximum depth, and learning rate were selected as the main hyperparameters to be optimized. During the optimization process, grid search and 5-fold cross validation were used to determine the best combination of parameters. Grid search involves traversing all combinations within the initial range, while five-fold cross validation involves dividing the training dataset into five equal parts and using one of them as a validation set to evaluate the performance of the model. MAPE and R were calculated. 2 The score evaluates the performance of the model, as shown in Equations 5 and 6;

[0071] Table 5 XGBoost model hyperparameter range and best CV score

[0072]

[0073]

[0074]

[0075] Figure 6 The prediction results of the XGBoost model on the test set for the two targets in the design space and the augmented space are shown.

[0076] According to Shapley Additive Explanations (SHAP), the contribution of control variables to the prediction results is analyzed. Positive and negative SHAP values ​​indicate the contribution or inhibition to the target. Control variables are sorted from top to bottom according to their contribution, such as Figure 7 As shown in the figure, the greater the contribution value, the greater the influence of the control variable on the wastewater and the product. The control variable with a smaller SHAP value requires a larger adjustment range to achieve the expected effect. Therefore, the larger the SHAP value of the control variable is, the lower the operating cost is. According to the above analysis, the coal feed controller (FC1) is an uncontrollable environmental change. Considering the process stability and operating cost, the five control variables with larger SHAP values ​​(TC6, PC4, PC1, FC5, PC5) are selected as the operating variables.

[0077] Step 4: construct and train an intelligent control model based on reinforcement learning (RL) to determine an optimal control decision, wherein the optimal control decision includes action control signals corresponding to optimal control variables;

[0078] During reinforcement learning, the agent model is called to calculate the Q value for a given state, and then the decision network decides what action to take in the current state.

[0079] Intelligent control models based on reinforcement learning (RL) are constructed in the context of Markov decision processes, where the agent interacts directly with the environment in a sequential manner. The agent controls the manipulated variables with causal relationships, and the environment is the agent model. COD and product quality are calculated based on the manipulated variables. Given the current state (including online measurements), the controller must decide which operation (or set of operations) to adjust to reduce wastewater discharge. In this example, these operations may include changing the value of the control variables. These variables are selected to maximize the overall goal, which is determined by a set of intermediate rewards, including COD and product quality. The goal of reinforcement learning (RL) is to learn a control policy that enables the actions taken in the current state to maximize the sum of expected rewards.

[0080] In actual PID control, the feed rate is controlled by continuously controlling the valve, rather than opening or closing it discretely. Therefore, the action of the agent is a continuous process, and it takes real-time actions according to changes in the environment. t to t+1 The state transition is represented by s t+1 =f(a t ,s t ,ω t );

[0081] At time t, for the current state, the agent takes an action through the action neural network according to the current strategy. Then, the environment of the CtEG process enters the next state and obtains the reward under the action. Starting from the initial state, the agent repeats the above process to obtain a sequence, that is, the interaction trajectory τ between the agent and the environment, as shown in Formula 7:

[0082] τ=s 0 ,a 0 ,s 1 ,a 1 …s t ,a t ,r,a t+1 Formula 7

[0083] Rewards are usually used to promote the selection of actions that can achieve the expected results. For example, if the product quality is qualified, a positive reward will be given in the last time step; on the contrary, to avoid the purity of product EG being unqualified due to the control of wastewater volume, a negative reward can be given. After a series of operations by the agent, the closer the purity of the generated product is to the process setting value, the less wastewater is discharged, and the higher the reward is. The reward can be calculated by formula 8:

[0084] Reward=0.01|Ratio-0.97|-1 -100+0.1 COD-1 Formula 8

[0085] Ratio refers to the purity of ethylene glycol after normalization. The purity of the product in the normal production process is 0.97. The closer the normalized COD is to 0, the smaller the pollutant value in the wastewater.

[0086] The SAC algorithm is used to allow the agent to explore the environment extensively to obtain the best strategy and avoid falling into the local optimum. The SAC algorithm introduces the maximum entropy mechanism to reduce entropy when repeatedly selecting an action and encourage exploration of other actions, thereby increasing the range of scheduling strategies explored and improving robustness. During the learning process, two Q networks are established, such as Figure 8 As shown in the figure, the SAC algorithm selects the smaller Q value of the two networks as the estimated value of the action value function, and continuously and adaptively adjusts the policy parameters to increase the probability of obtaining higher rewards. The parameters of the two Q networks are continuously updated during the training process. As the Q estimate increases, the actions selected by the decision network will also increase, thereby selecting better control actions to maximize rewards. In the SAC network structure, the linear activation function is used as the activation function of the hidden layer. The policy network and the Q network use relu and tanh activation functions;

[0087] Table 6 lists the initial values ​​and ranges of the action space. The maximum number of iterations is 3000 and the environment step is 20. When the environment step reaches 20, the environment parameters, actions, and uncertainty factors will be reset to perform the next iteration. Table 7 lists the detailed network hyperparameters. After all parameters are set, the intelligent control model based on reinforcement learning RL is trained for 3000 iterations. The average reward is as follows: Fig. 9 shown.

[0088] Table 6 Initial values ​​and ranges of action space

[0089]

[0090] Table 7 Hyperparameters used by SAC

[0091]

[0092] As the number of training times increases, the average rate of return gradually stabilizes at a higher level, indicating that the intelligent control model has learned to effectively balance the two competing goals of product quality and wastewater reduction, and is capable of making correct control decisions to optimize the overall performance of the system.

[0093] Step 5: Use the intelligent control model based on reinforcement learning (RL) to monitor uncertain feed disturbances, determine the optimal control decision under the current operating conditions, and transmit the action control signal to the intelligent controller to adjust the corresponding control variables to achieve wastewater reduction in coal-to-ethylene glycol. The intelligent controller is connected to the controllers corresponding to the optimal five control variables, such as Fig.10 As shown;

[0094] The reduction of coal feed flow rate will significantly increase the COD of wastewater. According to different flow rate reduction conditions, the intelligent control model outputs the optimal control values ​​of 5 control variables in real time, such as Fig.11 shown.

[0095] In the dynamic simulation, two feed disturbances were randomly selected to compare the control effects of the traditional PID control and the intelligent control model. First, the coal content was set to 135.1 kmol / h and 120.8 kmol / h in the dynamic simulation, and the change curve of COD in the wastewater was obtained; then, the disturbance was input into the intelligent control model, and the actions corresponding to the five control variables were obtained, as shown in Table 8; finally, the actions were input into the dynamic simulation to check the wastewater discharge and product quality. The emission reduction effect and product purity after control are shown in Table 8. Fig.12 shown.

[0096] Table 8 Output actions of the intelligent control model under two feed disturbances

[0097]

[0098] Depend on Fig.12 It can be seen that the wastewater discharge of the intelligent control CtEG process is significantly reduced, and the purity of the product EG is always maintained above 99.8% during the entire control process. Each step of the dynamic simulation is 0.01 hour, and the instantaneous value at each moment is regarded as the average value within 0.01 hours. Therefore, the 10-hour COD average value can be calculated according to Formula 10.

[0099]

[0100] Among them, m i It is the COD value corresponding to every 0.01 moment.

[0101] Through calculation, the COD values ​​in the wastewater after intelligent control were reduced by 22.80% and 22.98% respectively within 10 hours. The results show that the constructed intelligent control model can effectively reduce pollutant emissions when dealing with uncertain feed disturbances.

[0102] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the claims.

Claims

1. A coal-to-ethylene glycol wastewater reduction control method based on first-principles modeling and reinforcement learning, characterized in that: include: Step 1: Establish a dynamic model of coal-to-ethylene glycol based on first principles, add controllers for key process parameters based on experience, and use the dynamic model to simulate actual working conditions to obtain a data set; Step 2: Use Granger causality test GCT to analyze the causal relationship between the control variables and wastewater, and obtain the global causal relationship diagram of the control variables; Step 3: Build a proxy model based on extreme gradient boosting XGBoost, and use the proxy model to select the best several control variables as operation variables from the global causal relationship graph; Step 4: construct and train an intelligent control model based on reinforcement learning (RL) to determine an optimal control decision, wherein the optimal control decision includes action control signals corresponding to optimal control variables; Step 5. Use the intelligent control model based on reinforcement learning (RL) to monitor uncertain feed disturbances, determine the optimal control decision under current operating conditions, and transmit the action control signal to the intelligent controller to adjust the corresponding control variables to achieve wastewater reduction in coal-to-ethylene glycol production, wherein the intelligent controller is connected to controllers corresponding to several optimal control variables.

2. The method for reducing and controlling the emission of wastewater from coal-to-ethylene glycol based on first principle modeling and reinforcement learning according to claim 1 is characterized in that: The data set is obtained by simulating the actual working condition using the dynamic model, including: The dynamic model is used to simulate the operating conditions where disturbances lead to an increase in wastewater pollutants. The historical PID control data from the beginning of the disturbance to the process stabilization stage are selected to analyze the causal relationship between the control variables and the wastewater.

3. The method for reducing and controlling the emission of wastewater from coal-to-ethylene glycol based on first principle modeling and reinforcement learning according to claim 1 is characterized in that: The second step comprises: Granger causality test GCT was used to test the control variables under six disturbances, and the dependent variable that caused the increase of COD in wastewater was screened out from 30 control variables; By analyzing the regularity under different interferences and combining the process flow to analyze the spatial information transmission process, the causal relationship under six interferences is integrated, and the transmission information of the process is structured to obtain a global causal relationship diagram.

4. The method for reducing and controlling the emission of wastewater from coal-to-ethylene glycol based on first principle modeling and reinforcement learning according to claim 1 is characterized in that: In step 3, the proxy model based on extreme gradient boosting XGBoost is as follows: For each tree in the set, the XGBoost objective function consists of a single data point loss function and a regularization term, as shown in Formula 1. The expected value of each data point is updated according to the contribution of the newly inserted tree. This relationship is shown in Formula 2. The set tree is established by recursively partitioning the feature space, as shown in Formula 3. The predicted value of a leaf node is the weighted average of the training samples belonging to the leaf. By recursively partitioning the data, the established tree can minimize the objective function. The ideal split point for each node is determined by finding the split point with the largest gain, as shown in Formula 4. The estimator, maximum depth, and learning rate of the XGBoost model were selected as the main hyperparameters for optimization. During the optimization process, grid search and 5-fold cross validation were used to determine the best combination of parameters. Grid search involves traversing all combinations within the initial range, while 5-fold cross validation involves dividing the training dataset into five equal parts and using one of them as a validation set to evaluate the performance of the model. MAPE and R were calculated. 2 The score evaluates the performance of the model, as shown in Equations 5 and 6; The contribution of the control variables to the prediction results was analyzed according to Shapley Additive Explanations (SHAP). Taking into account process stability and operating costs, five control variables with larger SHAP values ​​were selected as operating variables.

5. The method for reducing and controlling the emission of wastewater from coal-to-ethylene glycol based on first principle modeling and reinforcement learning according to claim 1 is characterized in that: The fourth step comprises: The intelligent control model based on reinforcement learning (RL) is constructed in the context of Markov decision processes, where the agent directly interacts with the environment in a sequential manner. The agent controls the manipulated variables with causal relationships, and the environment is the agent model. COD and product quality are calculated based on the manipulated variables. The goal of reinforcement learning (RL) is to learn a control strategy that maximizes the sum of expected rewards for the actions taken in the current state. In actual PID control, the feed rate is controlled by continuously controlling the valve, rather than opening or closing it discretely. Therefore, the action of the agent is a continuous process, and it takes real-time actions according to changes in the environment. t to t+1 The state transition is represented by s t+1 =f(a t ,s t ,ω t ); At time t, for the current state, the agent takes an action through the action neural network according to the current strategy. Then, the environment of the CtEG process enters the next state and obtains the reward under the action. Starting from the initial state, the agent repeats the above process to obtain a sequence, that is, the interaction trajectory τ between the agent and the environment, as shown in Formula 7: τ = s0,a0,s1,a1…s t ,a t ,r,a t+1 Formula 7 Rewards are often used to encourage the selection of actions that achieve a desired outcome and can be calculated using Formula 8: Reward=0.01 Ratio-0.97-1 -100+0.1 COD-1 Formula 8 Ratio refers to the purity of ethylene glycol after normalization. The purity of the product in the normal production process is 0.

97. The closer the normalized COD is to 0, the smaller the pollutant value in the wastewater. The SAC algorithm is used to allow the agent to explore the environment extensively to obtain the best strategy and avoid falling into the local optimum. The SAC algorithm introduces the maximum entropy mechanism to reduce entropy when repeatedly selecting a certain action and encourage exploration of other actions, thereby increasing the range of scheduling strategies explored and improving robustness. During the learning process, two Q networks are established. The SAC algorithm selects the smaller Q value of the two networks as the estimated value of the action value function and continuously and adaptively adjusts the policy parameters to increase the probability of obtaining higher returns. The parameters of the two Q networks are continuously updated during the training process. As the Q estimate increases, the actions selected by the decision network will also increase, thereby selecting better control actions to maximize rewards. When the environment step reaches the set value, the environment parameters, actions and uncertainty factors will be reset to perform the next iteration. After all parameters are set, the intelligent control model based on reinforcement learning (RL) is iteratively trained until the average return rate stabilizes at a high level.

6. The method for reducing and controlling the emission of wastewater from coal-to-ethylene glycol based on first principle modeling and reinforcement learning according to claim 1 is characterized in that: The controller includes 6 flow controllers, 8 pressure controllers, 6 liquid level controllers, and 6 temperature controllers. The flow controllers respectively control the feed flow of each material; the pressure controllers respectively control the pressure of the coal gasification reactor, the water-gas shift reactor, the gas separation and purification device, the methyl nitrite synthesis reactor, the dimethyl oxalate synthesis reactor, the flash evaporation device (FLASH), the ethylene glycol refining tower, and the methanol recovery tower; the liquid level controllers respectively control the liquid levels of the gas separation and purification device tower bottom, the methyl nitrite synthesis reactor, the dimethyl oxalate synthesis reactor, the flash evaporation device (FLASH), the ethylene glycol refining tower, and the methanol recovery tower; the temperature controllers respectively control the temperatures of the coal gasification reactor, the water-gas shift reactor, the methyl nitrite synthesis reactor, the dimethyl oxalate synthesis reactor, the ethylene glycol refining tower, and the methanol recovery tower.

Images