Intelligent dosing method for sewage phosphorus removal based on reinforcement learning

Abstract

The invention provides an intelligent dosing method for sewage dephosphorization based on reinforcement learning in order to solve the problem that the dosing amount is difficult to accurately control in the prior art. The sewage dephosphorization intelligent dosing method based on reinforcement learning comprises the following steps: S1, constructing a PMD environment of a reinforcement learning algorithm; and S2, on the basis of the MDP environment constructed in the step S1, training a DQN algorithm model, and taking the trained DQN algorithm model as an intelligent agent. And S3, based on the MDP environment constructed in the step S1, constructing an MDP environment control system. And S4, deploying the intelligent agent in the step S2 to the MDP environment control system constructed in the step S3, and controlling a dosing pump to perform intelligent dosing on the sewage by the MDP environment control system. The construction of the MDP environment in reinforcement learning can respond and predict in real time, respond to the water quality change in real time, predict the future state and ensure long-term stable operation.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of environmental engineering and automatic control, and in particular to an intelligent dosing method for phosphorus removal in sewage based on reinforcement learning. Background Art

[0002] With the rapid development of my country's economy, the amount of water used in industry, agriculture and life is increasing. Correspondingly, the problem of sewage treatment is becoming increasingly prominent. Sewage treatment is an important means to ensure the sustainable use of water resources and improve the quality of the water environment. In the sewage treatment process, the dosing link plays a vital role in improving the treatment effect and ensuring the effluent quality.

[0003] The existing sewage treatment dosing method mainly relies on manual operation, but it is difficult to accurately control the dosage of manual dosing, which easily causes waste or insufficient dosage, affecting the sewage treatment effect. In addition, manual dosing requires operators to frequently inspect and adjust the dosing equipment, which is labor-intensive. Summary of the invention

[0004] In order to solve the problem in the prior art that it is difficult to accurately control the dosage, the present invention provides an intelligent dosing method for wastewater phosphorus removal based on reinforcement learning.

[0005] The technical solution adopted by the present invention is: A method for intelligent dosing of wastewater phosphorus removal based on reinforcement learning, comprising the following steps: Step S1: construct the MDP environment of the reinforcement learning algorithm; Step S2: Based on the MDP environment constructed in step S1, a DQN algorithm model is trained, and the trained DQN algorithm model is used as an intelligent agent; Step S3: constructing an MDP environment control system based on the MDP environment constructed in step S1; Step S4: deploy the intelligent agent in step S2 to the MDP environmental control system constructed in step S3, and control the dosing pump by the MDP environmental control system to perform intelligent dosing on the sewage.

[0006] Furthermore, the step S1 specifically performs the following steps: Step S11: constructing a state space S in the MDP environment; the state space includes water inlet parameters and water outlet parameters; The inlet water parameters include the instantaneous inlet water flow rate, inlet water COD, inlet water ammonia nitrogen, inlet water TP, inlet water TN, inlet water PH, and inlet water SS; the outlet water parameters include the instantaneous outlet water flow rate, outlet water COD, outlet water ammonia nitrogen, outlet water TP, outlet water TN, outlet water PH, and outlet water SS; All the water inlet parameters and the water outlet parameters are normalized to the interval [0,1]; Step S12: constructing an action space A in the MDP environment; the action space is the dosing flow rate of the PAC dosing pump, and the dosing flow rate is normalized to the interval [0,1]; Step S13: construct the state transition probability P in the MDP environment, expressed as: ; Among them, s t is the state in the state space S at the current time t, a t is the action taken at the current time t, s t+1 is the state in the state space S at the next time t+1. The state transition probability P describes the transition to the state after executing action a in the current state s. possibility; Step S14: constructing a reward function R in the MDP environment; Step S15: Combine the state space S, the action space A, the state transition probability P, and the reward function R to construct the MDP environment.

[0007] Furthermore, the step S2 specifically performs the following steps: Step S21: Initialize two deep neural networks; one of the deep neural networks is used as the main network of the Q network in the DQN algorithm model, and the main network is used to predict the action value function of the Q network. ; Another deep neural network is used as the target network, and the target network parameters Main network parameters Periodic replication of Step S22: Initialize the experience buffer D, which is used to store the experience generated by the interaction between the agent and the MDP environment. ; Wherein s is the current state in the state space S constructed in step S11, a is the action in the action space A constructed in step S12, and r is the reward in the reward function R in step S14, is the next state in the state space S; Step S23: Initialize the Adam optimizer and set the learning rate α in the Adam optimizer; the Adam optimizer is used to update the parameters of the main network. ; Step S24: Calculate the loss function based on the main network, the target network, and the experience buffer D , and output the loss function The value of Step S25: Execute the action a in the MDP environment and observe the reward r and the next state , then the experience Playing back into the experience buffer D; Step S26: Based on the main network, the target network, and the experience replayed from step S25 Extract batch_size samples from the experience buffer D and calculate the target value y of the state in the i-th state space S i ; Step S27: Use the Adam optimizer to update the parameters of the main network , and periodically copy the parameters of the main network to the parameters of the target network ; Each sample includes the PAC dosing rate, pH, temperature, suspended solids, total phosphorus, chemical oxygen demand and flow rate of each high-density pool; Step S28: Circulate the steps S25 to S27 until the preset number of training times or the loss function is reached. Convergence, or stopping when the reward function R converges; wherein each cycle is executed once, the state in the state space S changes once; Step S29: First, based on the process of steps S21 to S28, the DQN algorithm model is constructed and the DQN algorithm model is trained; then, the trained DQN algorithm model is regarded as the intelligent agent.

[0008] Furthermore, the MDP environmental control system constructed in step S3 includes: Controller platform; A dosing pump, the input end of the dosing pump is connected to the output end of the controller platform; A sensor, wherein an output end of the sensor is connected to an input end of the controller platform.

[0009] Furthermore, the controller platform is a PLC platform.

[0010] Furthermore, the step S4 specifically performs the following steps: Step S41: First, the intelligent agent used as the DQN algorithm model in step S2 is connected to the controller platform, and the sensor collects real-time data and transmits it to the controller platform; wherein the collected real-time data includes the PAC dosing rate, pH, temperature, suspended solids, total phosphorus, chemical oxygen demand and flow rate of each high-density pool; Then, a database is constructed, and the collected real-time data is imported into the database, and the intelligent agent stores the collected real-time data from the controller platform into the database; Step S42: The intelligent agent periodically reads the real-time data in the database, and then calculates the dosage corresponding to the current environment, the dosage corresponding to the current environment is predicted data, stores the predicted data in the database, and controls the dosing pump to add medicine through the controller platform.

[0011] Furthermore, the expression of the reward function R is: ; Where d is the deviation between the target outlet TP and the current outlet TP, expressed as: ; Where v is the drug administration acceleration rate, P target is the target outlet water TP value, P current is the current water outlet TP.

[0012] Furthermore, the loss function The expression is: ; in, For the Q network, for the i-th input state s i Take action a i The predicted output Q value is based on the parameters of the current main network. Estimates of state-action values.

[0013] Furthermore, in step S25, the experience When playing back to the experience buffer D, if the experience buffer D is full, the old experience buffer is replaced according to the first-in-first-out FIFO strategy. .

[0014] Furthermore, the target value y i The expression is: ; in, is the discount factor, is the next state in the experience buffer D, For the status Under this condition, all possible actions calculated by the target network The maximum target value in .

[0015] The beneficial effects of the present invention are: The present invention can be adaptively optimized, and the DQN algorithm can automatically adjust the PAC dosage according to changes in water quality, thereby achieving precise control and improving treatment efficiency.

[0016] The loss function is used in the DQN algorithm of the present invention, which can save costs and reduce operating costs by reducing over-dosing and optimizing the use of chemicals.

[0017] The construction of the MDP environment in the reinforcement learning in the present invention can respond and predict in real time, respond to water quality changes in real time, predict future states, and ensure long-term stable operation.

[0018] When training the DQN algorithm model, the present invention uses a large amount of historical data in the experience buffer of the DQN algorithm, and the intelligent dosing method for sewage phosphorus removal based on reinforcement learning of the present invention uses a variety of inlet and outlet water and dosing indicators during operation, so as to effectively process a large amount of data and improve the automation level of the sewage treatment plant. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 The overall flow chart is shown in Figure 1.

[0021] Figure 2 This is the overall technical solution diagram.

[0022] Figure 3 This is the training status of the DQN algorithm model.

[0023] Figure 4 shows the situation after adding chemicals to the intelligent PAC for sewage treatment. DETAILED DESCRIPTION

[0024] The disclosure below provides many different embodiments or examples to realize different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention.

[0025] The embodiments of the invention are described in detail below with reference to the accompanying drawings.

[0026] A method for intelligent dosing of wastewater phosphorus removal based on reinforcement learning. The overall flow chart of this embodiment is shown in the attached figure. Figure 1 As shown, the steps include: Step S1: Construct the MDP environment of the reinforcement learning algorithm.

[0027] Step S2: Based on the MDP environment constructed in step S1, a DQN algorithm model is trained to perform reinforcement learning on the MDP environment, and the trained DQN algorithm model is used as an intelligent agent.

[0028] Step S3: constructing an MDP environment control system based on the MDP environment constructed in step S1.

[0029] Step S4: deploy the intelligent agent in step S2 to the MDP environmental control system constructed in step S3, and control the dosing pump by the MDP environmental control system to perform intelligent dosing on the sewage.

[0030] Furthermore, the step S1 specifically performs the following steps: Step S11: construct a state space S in the MDP environment. The state space includes water inlet parameters and water outlet parameters.

[0031] The inlet water parameters include the instantaneous inlet water flow rate, inlet water COD, inlet water ammonia nitrogen, inlet water TP, inlet water TN, inlet water PH, and inlet water SS. The outlet water parameters include the instantaneous outlet water flow rate, outlet water COD, outlet water ammonia nitrogen, outlet water TP, outlet water TN, outlet water PH, and outlet water SS.

[0032] All the water inlet parameters and the water outlet parameters are normalized to the interval [0, 1].

[0033] Step S12: constructing an action space A in the MDP environment. The action space is the dosing flow rate of the PAC dosing pump, and the dosing flow rate is normalized to the interval [0, 1].

[0034] Step S13: construct the state transition probability P in the MDP environment, expressed as: .

[0035] Among them, s t is the state in the state space S at the current time t, a t is the action taken at the current time t, s t+1 is the state in the state space S at the next time t+1. The state transition probability P describes the transition to the state after executing action a in the current state s. In this embodiment, the transition probability at each moment is set to 1.

[0036] Step S14: construct a reward function R in the MDP environment.

[0037] Step S15: Combine the state space S, the action space A, the state transition probability P, and the reward function R to construct the MDP environment.

[0038] Furthermore, the step S2 specifically performs the following steps: Step S21: Initialize two deep neural networks. One of the deep neural networks is used as the main network of the Q network in the DQN algorithm model, and the main network is used to predict the action value function of the Q network. Another deep neural network is used as the target network, and the target network parameters are Main network parameters Periodic replication.

[0039] Step S22: Initialize the experience buffer D, which is used to store the experience generated by the interaction between the agent and the MDP environment. Where s is the current state in the state space S constructed in step S11, a is the action in the action space A constructed in step S12, and r is the reward in the reward function R in step S14. is the next state in the state space S.

[0040] Step S23: Initialize the Adam optimizer and set the learning rate α in the Adam optimizer. The Adam optimizer is used to update the parameters of the main network. In this embodiment, the learning rate α in the Adam optimizer is set to 0.001.

[0041] Step S24: Calculate the loss function based on the main network, the target network, and the experience buffer D , and output the loss function The value of .

[0042] Step S25: Execute the action a in the MDP environment and observe the reward r and the next state , then the experience Play back into the experience buffer D.

[0043] Step S26: Based on the main network, the target network, and the experience replayed from step S25 Extract batch_size samples from the experience buffer D and calculate the target value y of the state in the i-th state space S i .

[0044] Step S27: Use the Adam optimizer to update the parameters of the main network , and periodically copy the parameters of the main network to the parameters of the target network Each sample includes the PAC dosing rate, pH, temperature, suspended solids, total phosphorus, chemical oxygen demand and flow rate of each high-density pool.

[0045] Step S28: Circulate the steps S25 to S27 until the preset number of training times or the loss function is reached. Convergence, or stop when the reward function R converges. Each time the loop is executed, the state in the state space S changes once. In this embodiment, the loop stops until the reward function R converges.

[0046] In this embodiment, the reason why the reward function R convergence is used as the loop stop condition is that the loss function It is bound to decrease, and the value of the reward function R is equivalent to directly observing the operation of the model in actual conditions. Therefore, it is more realistic to use the value of the reward function R as a condition for stopping the loop.

[0047] Attached Figure 3 This is the training status of the DQN algorithm model. When the training times reached 4000, the reward function R had converged.

[0048] Step S29: First, based on the process of step S21 to step S28, the DQN algorithm model is constructed and trained. Then, the trained DQN algorithm model is used as the intelligent agent.

[0049] Furthermore, the MDP environmental control system constructed in step S3 (as shown in the attached Figure 2 shown), including: Controller platform.

[0050] A dosing pump, wherein the input end of the dosing pump is connected to the output end of the controller platform.

[0051] A sensor, wherein an output end of the sensor is connected to an input end of the controller platform.

[0052] Furthermore, the controller platform is a PLC platform.

[0053] Furthermore, the step S4 specifically performs the following steps: Step S41: First, the intelligent agent used as the DQN algorithm model in step S2 is connected to the controller platform, and the sensor collects real-time data and transmits it to the controller platform. In this embodiment, the collection frequency is once per minute. Among them, the collected data are inlet and outlet water and dosing indicators, and the inlet and outlet water and dosing indicators specifically include the PAC dosing rate of the high-density pool AD, the pH, temperature, suspended solids, total phosphorus, chemical oxygen demand and flow rate of the inlet and outlet water.

[0054] Then, a database is constructed, and the collected real-time data is imported into the database, and the intelligent agent stores the collected real-time data from the controller platform into the database.

[0055] Step S42: The intelligent agent periodically reads the real-time data in the database (in this embodiment, the reading frequency is once every two hours), and then calculates the dosage corresponding to the current environment. The dosage corresponding to the current environment is the predicted data, and the predicted data is stored in the database, and the dosing pump is controlled to add medicine through the controller platform.

[0056] Figure 4 shows the situation after the intelligent PAC for sewage treatment is added. (a) shows the PAC addition situation, and the vertical axis unit is m 3 / h, the horizontal axis unit is h, which shows that compared with manual treatment, the present invention saves up to 61.44% of the medicine. (b) The figure shows the total phosphorus in the effluent, the vertical axis unit is mg / L, the horizontal axis unit is h, and compared with manual treatment, the present invention has a significant improvement.

[0057] Furthermore, the expression of the reward function R is: .

[0058] Where d is the deviation between the target outlet TP and the current outlet TP, expressed as: .

[0059] Where, v is the drug administration acceleration rate, P target is the target outlet water TP value, P current is the current water outlet TP.

[0060] Furthermore, the loss function is the mean square error (MSE), expressed as: .

[0061] in, For the Q network, for the i-th input state s i Take action a i The predicted output Q value is based on the parameters of the current main network. Estimates of state-action values.

[0062] Furthermore, in step S25, the experience When playing back to the experience buffer D, if the experience buffer D is full, the old experience buffer is replaced according to the first-in-first-out FIFO strategy. .

[0063] Furthermore, the target value y i The expression is: .

[0064] in, is the discount factor (in this embodiment, the discount factor is set to 0.99), is the next state in the experience buffer D, For the status Under this condition, all possible actions calculated by the target network The maximum target value in .

Claims

1. A smart dosing method for wastewater phosphorus removal based on reinforcement learning, characterized in that: Includes steps: Step S1: construct the MDP environment of the reinforcement learning algorithm; Step S2: Based on the MDP environment constructed in step S1, a DQN algorithm model is trained, and the trained DQN algorithm model is used as an intelligent agent; Step S3: constructing an MDP environment control system based on the MDP environment constructed in step S1; Step S4: deploy the intelligent agent in step S2 to the MDP environmental control system constructed in step S3, and control the dosing pump by the MDP environmental control system to perform intelligent dosing on the sewage.

2. According to claim 1, a method for intelligent dosing of wastewater phosphorus removal based on reinforcement learning is characterized in that: The step S1 specifically performs the following steps: Step S11: constructing a state space S in the MDP environment; the state space includes water inlet parameters and water outlet parameters; The inlet water parameters include the instantaneous inlet water flow rate, inlet water COD, inlet water ammonia nitrogen, inlet water TP, inlet water TN, inlet water PH, and inlet water SS; the outlet water parameters include the instantaneous outlet water flow rate, outlet water COD, outlet water ammonia nitrogen, outlet water TP, outlet water TN, outlet water PH, and outlet water SS; All the water inlet parameters and the water outlet parameters are normalized to the interval [0,1]; Step S12: constructing an action space A in the MDP environment; the action space is the dosing flow rate of the PAC dosing pump, and the dosing flow rate is normalized to the interval [0,1]; Step S13: construct the state transition probability P in the MDP environment, expressed as: ; Among them, s t is the state in the state space S at the current time t, a t is the action taken at the current time t, s t+1 is the state in the state space S at the next time t+1. The state transition probability P describes the transition to the state after executing action a in the current state S. possibility; Step S14: constructing a reward function R in the MDP environment; Step S15: Combine the state space S, the action space A, the state transition probability P, and the reward function R to construct the MDP environment.

3. According to claim 2, a method for intelligent dosing of wastewater phosphorus removal based on reinforcement learning is characterized in that: The step S2 specifically performs the following steps: Step S21: Initialize two deep neural networks; one of the deep neural networks is used as the main network of the Q network in the DQN algorithm model, and the main network is used to predict the action value function of the Q network. ; Another deep neural network is used as the target network, and the target network parameters Main network parameters Periodic replication of Step S22: Initialize the experience buffer D, which is used to store the experience generated by the interaction between the agent and the MDP environment. ; Wherein s is the current state in the state space S constructed in step S11, a is the action in the action space A constructed in step S12, and r is the reward in the reward function R in step S14, is the next state in the state space S; Step S23: Initialize the Adam optimizer and set the learning rate α in the Adam optimizer; the Adam optimizer is used to update the parameters of the main network. ; Step S24: Calculate the loss function based on the main network, the target network, and the experience buffer D , and output the loss function The value of Step S25: Execute the action a in the MDP environment and observe the reward r and the next state , then the experience Playing back into the experience buffer D; Step S26: Based on the main network, the target network, and the experience replayed from step S25 Extract batch_size samples from the experience buffer D and calculate the target value y of the state in the i-th state space S i ; Step S27: Use the Adam optimizer to update the parameters of the main network , and periodically copy the parameters of the main network to the parameters of the target network ; Each sample includes the PAC dosing rate, pH, temperature, suspended solids, total phosphorus, chemical oxygen demand and flow rate of each high-density pool; Step S28: Circulate the steps S25 to S27 until the preset number of training times or the loss function is reached. Convergence, or stopping when the reward function R converges; wherein each cycle is executed once, the state in the state space S changes once; Step S29: First, based on the process of steps S21 to S28, the DQN algorithm model is constructed and the DQN algorithm model is trained; then, the trained DQN algorithm model is regarded as the intelligent agent.

4. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 3 is characterized in that: The MDP environmental control system constructed in step S3 includes: Controller platform; A dosing pump, the input end of the dosing pump is connected to the output end of the controller platform; A sensor, wherein an output end of the sensor is connected to an input end of the controller platform.

5. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 4 is characterized in that: The controller platform is a PLC platform.

6. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 4 or 5, characterized in that: The step S4 specifically performs the following steps: Step S41: First, the intelligent agent used as the DQN algorithm model in step S2 is connected to the controller platform, and the sensor collects real-time data and transmits it to the controller platform; wherein the collected real-time data includes the PAC dosing rate, pH, temperature, suspended solids, total phosphorus, chemical oxygen demand and flow rate of each high-density pool; Then, a database is constructed, and the collected real-time data is imported into the database, and the intelligent agent stores the collected real-time data from the controller platform into the database; Step S42: The intelligent agent periodically reads the real-time data in the database, and then calculates the dosage corresponding to the current environment, the dosage corresponding to the current environment is predicted data, stores the predicted data in the database, and controls the dosing pump to add medicine through the controller platform.

7. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 3 is characterized in that: The expression of the reward function R is: ; Where d is the deviation between the target outlet TP and the current outlet TP, expressed as: ; Where v is the drug administration acceleration rate, P target is the target outlet water TP value, P current is the current water outlet TP.

8. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 3 is characterized in that: The loss function The expression is: ; in, For the Q network, for the i-th input state s i Take action a i The predicted output Q value is based on the parameters of the current main network. Estimates of state-action values.

9. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 3 is characterized in that: In step S25, the experience When playing back to the experience buffer D, if the experience buffer D is full, the old experience buffer is replaced according to the first-in-first-out FIFO strategy. .

10. The intelligent dosing method for wastewater phosphorus removal based on reinforcement learning according to claim 3 is characterized in that: The target value y i The expression is: ; in, is the discount factor, is the next state in the experience buffer D, For the status Under this condition, all possible actions calculated by the target network The maximum target value in .

Images