A dissolved oxygen concentration self-adaptive PID control method based on deep reinforcement learning

Abstract

This invention proposes an adaptive PID control method for dissolved oxygen (DO) concentration based on deep reinforcement learning, achieving precise online control of DO concentration in wastewater treatment. Addressing the challenges of uncertainty, nonlinearity, strong coupling, and difficulty in determining controller parameters and controlling DO concentration in wastewater treatment, this method combines deep reinforcement learning with traditional PID control. A deep reinforcement learning-based DRL agent enables real-time dynamic adjustment of PID controller parameters, thus solving the problems of low control accuracy and poor adaptability when applying PID controllers to DO concentration control. Experimental results show that this method not only achieves rapid and accurate online control of DO concentration but also possesses good adaptability, quickly stabilizing the DO concentration near the new expected value when the expected concentration changes.

Description

Technical Field

[0001] This invention utilizes an adaptive PID control method based on deep reinforcement learning (DRL) to control dissolved oxygen (DO) concentration during wastewater treatment. Dissolved oxygen (DO) concentration control is a crucial step in wastewater treatment, falling under both the wastewater treatment field and the field of intelligent control. Background Technology

[0002] With the continuous advancement of industrialization and urbanization in my country, water scarcity and water pollution have become increasingly prominent issues. The "2021 China Ecological Environment Status Bulletin" released by the Ministry of Ecology and Environment of China points out that among the surface waters nationwide, Class IV... The proportion of Class V or worse water quality reached 15%; among 210 important lakes, Class IV water quality accounted for the majority. The proportion of Class V or worse water quality reached 27%, and among the 209 lakes and reservoirs monitored for trophic status, 27% of key lakes and reservoirs were eutrophic. Water scarcity and water pollution have become major obstacles to my country's economic and social development. How to obtain clean and safe water resources and how to treat wastewater in a timely and effective manner have become urgent problems to be solved in today's society. Therefore, wastewater treatment technology has ushered in unprecedented development opportunities. Researching advanced wastewater treatment control technologies to achieve energy-saving standards in wastewater treatment while ensuring precise control is an inevitable development trend for wastewater treatment plants in the future.

[0003] Dissolved oxygen (DO) concentration is one of the most critical factors in wastewater treatment. It is closely related to the microorganisms in activated sludge. Excessive DO concentration accelerates microbial metabolism, leading to poor settling of suspended solids in the sludge. Conversely, insufficient DO concentration inhibits the growth of heterotrophic and autotrophic microorganisms, while filamentous bacteria proliferate, eventually causing sludge bulking and treatment failure. Furthermore, DO concentration directly impacts the effluent quality and energy consumption of wastewater treatment plants. Therefore, precise control of DO concentration is crucial in wastewater treatment.

[0004] Traditional PID control is widely used in wastewater treatment plants due to its simple principle, high reliability, and ease of implementation. However, in wastewater treatment systems, the controlled object exhibits nonlinear, time-varying, and large time lag characteristics, and is significantly affected by external factors such as ambient temperature, making it difficult to establish an accurate mathematical model and thus determine the optimal controller parameters. In such cases, traditional PID control suffers from low control accuracy, lack of self-learning ability, and poor adaptive capability, thus limiting its application to dissolved oxygen (DO) concentration control. Therefore, finding an intelligent method to address the problems of applying traditional PID control to DO concentration control has become an urgent issue. Summary of the Invention

[0005] The purpose of this invention is to provide an adaptive PID control method for dissolved oxygen concentration based on deep reinforcement learning. This method utilizes the powerful learning and decision-making capabilities of deep reinforcement learning to achieve real-time dynamic adjustment of PID controller parameters, thereby solving the problems of low control accuracy and poor adaptability of PID controllers when applied to dissolved oxygen concentration control.

[0006] To achieve the above objectives, the present invention adopts the following technical solution and implementation steps:

[0007] Considering the powerful learning and decision-making capabilities of deep reinforcement learning, it is combined with traditional PID control. Deep learning is used to dynamically adjust the parameters of the PID controller in real time to compensate for the shortcomings of traditional PID control. The adaptive PID control method based on deep reinforcement learning can achieve precise online control of dissolved oxygen (DO) concentration in wastewater treatment processes, while also ensuring the efficient and stable operation of the wastewater treatment process. Specifically, it includes the following steps:

[0008] Step 1: Based on the simulation baseline 1 (BSM1) model of the activated sludge wastewater treatment process, build a wastewater treatment process system WWTPs to serve as the PID control environment for dissolved oxygen concentration in the wastewater treatment process;

[0009] The BSM1 model consists of a biological reaction tank and a secondary sedimentation tank, with the biological reaction tank further divided into five units. The PID controller is an incremental PID controller used to control the aeration rate of the fifth unit of the biological reaction tank to achieve tracking control of the dissolved oxygen concentration in the fifth unit. Specifically, it is designed using the following equations:

[0010]

[0011] in, The aeration rate at time k , Let k be the aeration rate at time k-1. The aeration increment at time k; , , These are the proportional, integral, and differential coefficients, respectively. The actual measured value of dissolved oxygen concentration at time k The expected dissolved oxygen concentration at time k difference, This represents the difference between the error at the current time and the error at the previous time. Defined as:

[0012]

[0013] in, , These are the differences between the actual measured values ​​and the expected dissolved oxygen concentrations at times k-1 and k-2, respectively.

[0014] Step 2: Determine the state space, action space, and reward function, specifically as follows:

[0015] Step 2-1: Determine the state space based on the control error of dissolved oxygen concentration. ;

[0016] Among them, the state space It includes three elements: dissolved oxygen concentration control error , as well as ,Right now ;

[0017] Step 2-2: Determine the action space based on the parameters of the PID controller to be optimized in Step 1. ;

[0018] Among them, action space The three parameters of a PID controller are: proportional coefficient. Integral coefficient and differential coefficients ,Right now ;

[0019] Steps 2-3: Determine the reward based on the difference between the actual measured dissolved oxygen concentration and the desired dissolved oxygen concentration. ;

[0020] Among them, the reward at time k The calculation is as follows:

[0021]

[0022] Step 3: Based on Steps 1 and 2, construct a Deep Reinforcement Learning (DRL) agent model using the DDPG algorithm. The DRL agent model mainly includes three modules: an evaluation network module, an action network module, and an experience replay module. The construction steps for each module are as follows:

[0023] Step 3-1: Construct an evaluation network module using a deep neural network;

[0024] The evaluation network is used to fit the Q function. The network parameters are The input is the state determined in step 2. and actions The output is the value of the Q function, i.e., Q = The evaluation network module includes the actual evaluation network. and target evaluation network Two networks with identical structures, each containing four fully connected layers: one input layer, two hidden layers, and one output layer; both hidden layers use the ReLU function as the activation function, and both have the same number of hidden layer nodes.

[0025] Step 3-2: Construct an action network module using a deep neural network;

[0026] The action network is used to fit the deterministic policy function. The network parameters are The input is the state determined in step 2. The output is a deterministic policy function. The value, which is also the action determined in step 2. ,Right now The action network module includes the actual action network. and target action network Two networks with identical structures, each containing four fully connected layers: one input layer, two hidden layers, and one output layer. The hidden layers in both networks use the ReLU function as the activation function, and the number of nodes in the hidden layers is the same. The output layers in both networks use the Tanh function as the activation function.

[0027] Step 3-3: Construct an experience replay module to store the experience data generated during training of the DRL agent in the environment built in Step 1.

[0028] Step 4: Train the deep reinforcement learning (DRL) agent in the environment built in Step 1, which includes the following sub-steps:

[0029] Step 4-1: Initialize the deep reinforcement learning (DRL) agent constructed in Step 3, specifically as follows:

[0030] Step 4-1-1: Randomly initialize the actual evaluation network and actual action network parameters , The target evaluation network is initialized using these two network parameters. and target action network parameters and Initialize the learning rates of the four network models and select the Adam gradient descent algorithm to update the network parameters;

[0031] Step 4-1-2: Initialize an empty space R as the space for storing experience data in the experience playback module;

[0032] Step 4-2: Set the total number of training rounds as N, the training duration per round as D days, the current total number of training rounds n = 0, and at the same time, use the first 7 days of influent data in the 14-day influent data under dry weather provided by BSM1 as the influent data of WWTPs during training;

[0033] Step 4-3: Initialize the environment built in Step 1, specifically including:

[0034] Step 4-3-1: Initialize the biochemical reaction tank, secondary sedimentation tank, and influent, and set the influent to change every 15 minutes;

[0035] Step 4-3-2: Initialize the incremental PID controller, specifically:

[0036] Initialize the controller parameters and the error calculation module. The error calculation module samples and memorizes the errors at the current moment, the previous moment, and the moment before the previous moment, and calculates the error corresponding to the controller parameters through calculation 、 and ;

[0037] Step 4-4: Let k = 1, the sampling time interval timestep = 15 / (60*24*20) days, the initial expected value of dissolved oxygen concentration = 2 mg / L, the update time of the expected value of dissolved oxygen concentration is L hours, and the total usage time h of the current expected value of dissolved oxygen concentration = 1;

[0038] Step 4-5: If h*timestep >= L, then h = 1, and select one from the expected values of dissolved oxygen concentration 1.8, 1.9, 2.0, 2.1, 2.2 as the next expected value of dissolved oxygen concentration ; if h*timestep < L, then h + 1;

[0039] Step 4-6: Subtract the sampled value of the dissolved oxygen concentration at time k from the current expected value of dissolved oxygen concentration to obtain the error value of the dissolved oxygen concentration at time k, and then input into the error calculation module and calculate it with the error values at times k-1 and k-2 to obtain a set of output quantities ;

[0040] Step 4-7: Input the obtained in Step 4-6 into the actual action network in the DRL agent to obtain the action output by the agent, and then output the new parameters of the PID controller according to Equation (4) ;

[0041]

[0042] where is a Gaussian perturbation , for adjusting the magnitudes of three parameters;

[0043] Step 4 - 8: Input the obtained in Step 4 - 6 and the new parameters of the PID controller obtained in Step 4 - 7 into the PID controller, and calculate the aeration volume of the fifth unit of the biochemical reaction tank according to Equation (1) ;

[0044] Step 4 - 9: Set the aeration volume of the fifth unit of the biochemical reaction tank as , then repeat Step 4 - 6 at the next sampling moment to obtain the output of the error calculation module , and calculate the reward according to and Equation (3);

[0045] Step 4 - 10: Store the tuple into the experience replay module;

[0046] Step 4 - 11: Set the batch size Batch_Size = b during network model training, and the number of tuples in the current experience replay module is m. If b < m, then enter Step 4 - 12 to update the parameters of the four networks in the DRL agent, otherwise enter Step 4 - 13;

[0047] Step 4 - 12: Randomly select b tuples from the experience replay module as training data, and then train and update the parameters of the four networks in the DRL agent. The update method of the parameters of the four networks is as follows:

[0048] Step 4 - 12 - 1: Take Equation (5) as the loss function of the actual evaluation network , and use the gradient descent algorithm to update the parameters of the actual evaluation network:

[0049]

[0050] where , , are respectively and , is the discount factor;

[0051] Step 4-12-2: After updating the parameters of the actual evaluation network, calculate the policy gradient according to Equation (6), and use the gradient descent algorithm to update the parameters of the actual action network of :

[0052]

[0053] Step 4-12-3: Use the updated parameters of the actual evaluation network and the actual action network and to update the parameters of the target evaluation network and the target action network and : [[ID=二十]]

[0054] [[ID=二十一]] [[ID=二十二]] [[ID=二十三]]

[0055] where is the update rate;

[0056] Step 4-13: If (k * timestep) < D, then k+1, enter Step 4-5, and continue training within this round; if (k * timestep) >= D and n < N-1, then the current total number of training rounds n+1, enter Step 4-3, and start the next round of training; if (k * timestep) >= D and n >= N-1, then the training is completed, and the trained deep reinforcement learning agent is exported.

[0057] Step 5: Combine the trained DRL agent with the PID controller to implement an adaptive PID controller, and apply this controller to the BSM1 benchmark simulation platform to achieve precise control of the dissolved oxygen concentration in the fifth unit of the biochemical reaction pool. The specific control process is as follows:

[0058] Step 5-1: Subtract the sampling value of the dissolved oxygen concentration at the current sampling moment from the current expected value of the dissolved oxygen concentration to obtain the error value of the dissolved oxygen concentration at the current sampling moment , and then input into the error calculation module, and calculate it with the error values at the previous sampling moment and the sampling moment before the previous one to obtain a set of output quantities ;

[0059] Step 5-2: Input the obtained in Step 5-1 into the actual action network in the DRL agent to obtain the action output by the agent, and then output the new parameters of the adaptive PID controller according to Equation (8);

[0060]

[0061] Step 5-3: Take the result from step 5-1 and the new parameters of the adaptive PID controller obtained in step 5-2 The input is fed into the adaptive PID controller, and the output of the adaptive PID controller is obtained according to equation (1). ;

[0062] Step 5-4: Based on the output of the adaptive PID controller Adjusting the aeration rate of the fifth unit of the biochemical reaction tank Then, the above steps are repeated at the next sampling time to achieve precise control of dissolved oxygen concentration.

[0063] The inventiveness of this invention is mainly reflected in:

[0064] (i) This invention addresses the characteristics of wastewater treatment processes, such as nonlinearity, time-varying nature, and large time lag, by introducing deep reinforcement learning into the PID control of dissolved oxygen concentration in the wastewater treatment process. Deep reinforcement learning can interact well with the environment, has self-learning capabilities, and can adapt to the dynamic characteristics of uncertain systems. Therefore, it can adapt to the complex and ever-changing characteristics of wastewater treatment processes and improve the adaptive capability of PID controllers.

[0065] (II) This invention employs an adaptive PID control method based on deep reinforcement learning to control the dissolved oxygen (DO) concentration online in the wastewater treatment process. This method effectively solves the problem of nonlinear systems being difficult to control, achieving precise online control of the DO concentration. It also addresses the issues of low control accuracy and poor stability when traditional PID controllers are applied to control the DO concentration in wastewater treatment processes. Attached Figure Description

[0066] Figure 1 This is a system block diagram of an adaptive PID control method for dissolved oxygen concentration based on deep reinforcement learning;

[0067] Figure 2 This is a schematic diagram of a PID control environment WWTPs for dissolved oxygen concentration in a wastewater treatment process;

[0068] Figure 3 This is a flowchart illustrating the structure and update process of a deep reinforcement learning DRL agent based on the DDPG algorithm.

[0069] Figure 4 It is an action network structure diagram;

[0070] Figure 5 It evaluates the network structure diagram;

[0071] Figure 6This is a dissolved oxygen concentration tracking control diagram of the DRL-based adaptive PID controller in the example described in this invention;

[0072] Figure 7 This is a diagram showing the dissolved oxygen concentration tracking error of the adaptive PID controller based on DRL in the example described in this invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and key points of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0074] This invention provides an adaptive PID control method for dissolved oxygen concentration based on deep reinforcement learning. This adaptive PID controller is based on deep reinforcement learning, using a deep reinforcement learning DRL agent to achieve real-time dynamic adjustment of the PID controller parameters, thereby solving the problems of low control accuracy and poor adaptability when PID controllers are applied to dissolved oxygen concentration control. The system block diagram of this method is shown below. Figure 1 As shown.

[0075] The method is implemented as follows:

[0076] Step 1: Build a simulation baseline 1 (BSM1) model based on the activated sludge wastewater treatment process. Figure 2 The wastewater treatment process system WWTPs shown is used as the PID control environment for dissolved oxygen concentration in the wastewater treatment process. The specific steps are as follows:

[0077] Step 1-1: Construct the biochemical reaction tank and secondary sedimentation tank according to the BSM1 model;

[0078] The biological reactor comprises five units. The first two units are anoxic zones, primarily conducting denitrification: in anoxic conditions, denitrifying bacteria use organic carbon and nitrates as donors to reduce nitrate nitrogen to nitrogen gas. The latter three units are aerobic zones, primarily conducting nitrification: in aerobic conditions, nitrifying bacteria oxidize ammonia nitrogen to nitrates. The secondary sedimentation tank is divided into 10 layers, mainly for sludge-water separation. The supernatant after wastewater treatment is discharged into the receiving water body through the upper layer of the secondary sedimentation tank, while the sludge is discharged through the lower layer. Part of it is treated as excess sludge, and the other part is returned to the biological reactor as a reaction carrier to continue participating in the biological treatment process of wastewater.

[0079] Step 1-2: Construct a PID controller as the dissolved oxygen concentration controller for the fifth unit of the biochemical reaction tank;

[0080] The PID controller used here is an incremental PID controller, specifically designed using the following equations:

[0081]

[0082] in, The aeration rate at time k , Let k be the aeration rate at time k-1. The aeration increment at time k; , , These are the proportional, integral, and differential coefficients, respectively. The actual measured value of dissolved oxygen concentration at time k The expected dissolved oxygen concentration at time k difference, This represents the difference between the error at the current time and the error at the previous time. Defined as:

[0083]

[0084] in, , These are the differences between the actual measured values ​​and the expected dissolved oxygen concentrations at times k-1 and k-2, respectively.

[0085] Step 2: Determine the state space, action space, and reward function, specifically as follows:

[0086] Step 2-1: Determine the state space based on the control error of dissolved oxygen concentration. ;

[0087] Among them, the state space It includes three elements: dissolved oxygen concentration control error , as well as ,Right now ;

[0088] Step 2-2: Determine the action space based on the parameters of the PID controller to be optimized in Step 1. ;

[0089] Among them, action space The three parameters of a PID controller are: proportional coefficient. Integral coefficient and differential coefficients ,Right now ;

[0090] Steps 2-3: Determine the reward based on the difference between the actual measured dissolved oxygen concentration and the desired dissolved oxygen concentration. ;

[0091] Among them, the reward at time k The calculation is as follows:

[0092]

[0093] Step 3: Based on steps 1 and 2, construct according to the DDPG algorithm Figure 3 The Deep Reinforcement Learning (DRL) agent model shown here mainly consists of three modules: an evaluation network module, an action network module, and an experience replay module. The construction steps of each module are as follows:

[0094] Step 3-1: Construct an evaluation network module using a deep neural network;

[0095] The structure of the evaluation network is as follows: Figure 5 As shown, the input is the state determined in step 2. and actions The output is the value of the Q function, i.e., Q = The evaluation network module includes the actual evaluation network. and target evaluation network Two networks with identical structures, each containing four fully connected layers: one input layer, two hidden layers, and one output layer; both hidden layers use the ReLU function as the activation function, and both have 256 hidden layer nodes.

[0096] Step 3-2: Construct an action network module using a deep neural network;

[0097] The structure of the action network is as follows: Figure 4 As shown, the input is the state determined in step 2. The output is the action determined in step 2. ,Right now The action network module includes the actual action network. and target action network Two networks with identical structures, each containing four fully connected layers: one input layer, two hidden layers, and one output layer. The hidden layers in both networks use the ReLU function as the activation function, and each network has 256 hidden layer nodes. The output layers in both networks use the Tanh function as the activation function.

[0098] Step 3-3: Construct an experience replay module to store the experience data generated during training of the DRL agent in the environment built in Step 1.

[0099] Step 4: Train the deep reinforcement learning (DRL) agent in the environment built in Step 1, which includes the following sub-steps:

[0100] Step 4-1: Initialize the deep reinforcement learning (DRL) agent constructed in Step 3, specifically as follows:

[0101] Step 4-1-1: Randomly initialize the actual evaluation network and actual action network parameters , The target evaluation network is initialized using these two network parameters. and target action network parameters and Initialize the learning rates of the four network models, with the learning rate of the action network being [missing value]. The learning rate of the network is evaluated. And the Adam gradient descent algorithm is selected to update the network parameters;

[0102] Step 4-1-2: Initialize an empty space with a capacity of [size missing]. The space R serves as the space for storing experience data in the experience playback module;

[0103] Step 4-2: Set the total number of training rounds to N=180, the training duration of each round to D=1 day, the current total number of training rounds n=0, and use the first 7 days of water ingress data from the 14 days of water ingress data under dry weather provided by BSM1 as the water ingress data for WWTPs during training.

[0104] Step 4-3: Initialize the environment built in Step 1, specifically including:

[0105] Step 4-3-1: Initialize the biochemical reaction tank, secondary sedimentation tank, and influent, and set the influent to be changed every 15 minutes;

[0106] Step 4-3-2: Initialize the incremental PID controller, specifically as follows:

[0107] Initialize controller parameters: , , Initialize the error calculation module: the errors at the current time, the previous time, and the time before that are all 0.

[0108] Step 4-4: Let k=1, sampling time interval timestep=15 / (60*24*20) days, initial expected value of dissolved oxygen concentration. =2 mg / L, expected dissolved oxygen concentration The update time is L=4 hours, and the total usage time of the current expected dissolved oxygen concentration is h=1.

[0109] Step 4-5: If h * timestep >= L, then h = 1, and select one from the expected values of dissolved oxygen concentrations 1.8, 1.9, 2.0, 2.1, and 2.2 as the expected value of the dissolved oxygen concentration for the next step. ; If h * timestep < L, then h + 1;

[0110] Step 4-6: Subtract the sampled value of the dissolved oxygen concentration at time k from the current expected value of the dissolved oxygen concentration to obtain the error value of the dissolved oxygen concentration at time k , and then input into the error calculation module to calculate with the error values at times k-1 and k-2, obtaining a set of output quantities ;

[0111] Step 4-7: Input the result obtained in Step 4-6 into the actual action network in the DRL agent to obtain the action output by the agent, and then output the new parameters of the PID controller according to Equation (12);

[0112]

[0113] where is used to adjust the magnitudes of the three parameters;

[0114] Step 4-8: Input the result obtained in Step 4-6 and the new parameters of the PID controller obtained in Step 4-7 into the PID controller, and calculate the aeration volume of the fifth unit of the biochemical reaction tank according to Equation (9) ;

[0115] Step 4-9: Set the aeration volume of the fifth unit of the biochemical reaction tank to , and then repeat Step 4-6 at the next sampling moment to obtain the output quantity of the error calculation module, and calculate the reward according to and Equation (11);

[0116] Step 4-10: Store the tuple in the experience replay module;

[0117] Step 4-11: Set the batch size b = 256 during network model training, and the number of tuples in the current experience replay module is m. If b < m, then enter Step 4-12 to update the parameters of the four networks in the DRL agent, otherwise enter Step 4-13;

[0118] Step 4-12: Randomly select b tuples from the experience replay module as training data, and then train the four networks in the DRL agent and update their parameters. The update methods for the parameters of the four networks are as follows:

[0119] Step 4-12-1: Take Equation (13) as the loss function of the actual evaluation network and use the gradient descent algorithm to update the parameters of the actual evaluation network :

[0120]

[0121] where , , are respectively and , is the discount factor;

[0122] Step 4-12-2: After updating the parameters of the actual evaluation network, calculate the policy gradient according to Equation (14) and use the gradient descent algorithm to update the parameters of the actual action network :

[0123]

[0124] Step 4-12-3: Use the updated parameters and of the actual evaluation network and the actual action network to and update the parameters of the target evaluation network and the target action network:

[0125]

[0126] where is the update rate;

[0127] Step 4-13: If (k * timestep) < D, then k + 1, enter Step 4-5, and continue training within this round; if (k * timestep) >= D and n < N - 1, then the current total number of training rounds n + 1, enter Step 4-3, and start the next round of training; if (k * timestep) >= D and n >= N - 1, then complete the training and export the trained deep reinforcement learning agent.

[0128] Step 5: Combine the trained DRL agent with a PID controller to implement an adaptive PID controller, and apply this controller to the BSM1 benchmark simulation platform to achieve precise control of the dissolved oxygen concentration in the fifth unit of the biochemical reactor. The specific control process is as follows:

[0129] Step 5-1: Sample the dissolved oxygen concentration at the current sampling time. Compared with the current expected dissolved oxygen concentration The difference is calculated to obtain the error value of the dissolved oxygen concentration at the current sampling time. Then The input is fed into the error calculation module, where the error values ​​compared to the previous sampling time and the time before that are calculated to obtain a set of output values. ;

[0130] Step 5-2: Take the result from step 5-1 The actual action network input into the DRL agent The action output by the intelligent agent is obtained. Then, according to equation (16), the new parameters of the adaptive PID controller are output. ;

[0131]

[0132] Step 5-3: Take the result from step 5-1 and the new parameters of the adaptive PID controller obtained in step 5-2 The input is fed into the adaptive PID controller, and the output of the adaptive PID controller is obtained according to equation (9). ;

[0133] Step 5-4: Based on the output of the adaptive PID controller Adjusting the aeration rate of the fifth unit of the biochemical reaction tank Then, the above steps are repeated at the next sampling time to achieve precise control of dissolved oxygen concentration. Figure 6 and Figure 7 The graph shows the control effect of a deep reinforcement learning-based adaptive PID controller on dissolved oxygen concentration. Figure 6 It can be seen that the controller can track the dissolved oxygen concentration well, and even if the expected dissolved oxygen concentration changes, it can quickly stabilize the dissolved oxygen concentration around the new expected value. Figure 7 It can be seen that the control error of the controller in terms of dissolved oxygen concentration is mainly located in mg / L. Based on this result, it can be seen that the adaptive PID controller based on deep reinforcement learning not only has better adaptive ability than the traditional PID controller, but also can quickly and accurately control the concentration of dissolved oxygen.

Claims

1. A dissolved oxygen concentration adaptive PID control method based on deep reinforcement learning, characterized in that, It includes the following steps: Step 1: Build a sewage treatment process system WWTPs based on the simulation benchmark No. 1 (BSM1) model in the activated sludge process sewage treatment to serve as the dissolved oxygen concentration PID control environment for the sewage treatment process; Step 2: Determine the state space, action space, and reward function; Step 3: Based on Steps 1 and 2, construct a deep reinforcement learning DRL agent model according to the DDPG algorithm. The DRL agent model mainly includes three modules: an evaluation network module, an action network module, and an experience replay module; Step 4: Train the deep reinforcement learning DRL agent in the environment built in Step 1; The specific steps of Step 4 are as follows: Step 4-1: Initialize the deep reinforcement learning DRL agent constructed in Step 3. Specifically: Step 4-1-1: Randomly initialize the actual evaluation network and actual action network parameters , The target evaluation network is initialized using these two network parameters. and target action network parameters and Initialize the learning rates of the four network models and select the Adam gradient descent algorithm to update the network parameters; Step 4-1-2: Initialize an empty space R as the space for the experience replay module to store experience data; Step 4-2: Set the total number of training rounds to N, the training duration per round to D days, the current total number of training rounds n = 0. At the same time, use the first 7 days of influent data in the 14-day influent data provided by BSM1 under dry weather as the influent data for WWTPs during training; Step 4-3: Initialize the environment built in Step 1. Specifically: Step 4-3-1: Initialize the biochemical reaction pool, secondary sedimentation tank, and influent, and set the influent to change every 15 minutes; Step 4-3-2: Initialize the incremental PID controller. Specifically: The controller parameters and error calculation module are initialized. The error calculation module samples and memorizes the errors at the current time, the previous time, and the time before that, and calculates the error corresponding to the controller parameters. , as well as ; Step 4-4: Let the initial time k=1, the sampling time interval timestep=15 / (60*24*20) days, and the initial expected value of dissolved oxygen concentration. =2 mg / L, expected dissolved oxygen concentration The update time is L hours, and the total usage time of the current expected dissolved oxygen concentration is h=1. Step 4-5: If h * timestep >= L, then h = 1, and select one from the expected values of dissolved oxygen concentration 1.8, 1.9, 2.0, 2.1, 2.2 as the next expected value of dissolved oxygen concentration ; If h * timestep < L, then h + 1; Steps 4-6: Sample the dissolved oxygen concentration at time k. Compared with the current expected dissolved oxygen concentration The difference is calculated to obtain the error value of the dissolved oxygen concentration at time k. Then The input is fed into the error calculation module, where it is compared with the error values ​​at times k-1 and k-2 to calculate a set of output values. ; Step 4-7: Obtain the results from Step 4-6 The actual action network input into the DRL agent The action output by the intelligent agent is obtained. Then, according to equation (4), the new parameters of the PID controller are output. ; ； wherein is a Gaussian perturbation, , for adjusting the size of the three parameters; Step 4-8: Obtain the results from step 4-6 and the new parameters of the PID controller obtained in steps 4-7 The input is fed into the PID controller, and the aeration rate of the fifth unit of the biochemical reaction tank is calculated according to equation (1). ; Steps 4-9: Set the aeration rate of the fifth unit of the biochemical reaction tank to... Then, at the next sampling time, steps 4-6 are repeated to obtain the output of the error calculation module. and according to Formula (3) calculates the reward ; Step 4-10: Store the tuple into the experience replay module; Step 4-11: Set the batch size Batch_Size = b during network model training, and the number of tuples in the current experience replay module is m. If b < m, enter Step 4-12 to update the parameters of the four networks in the DRL agent. Otherwise, enter Step 4-13; Step 4-12: Randomly select b tuples from the experience replay module as training data, and then train the four networks in the DRL agent and update their parameters. The update method for the parameters of the four networks is as follows: Step 4-12-1: Use equation (5) as the actual evaluation network The loss function is used to update the parameters of the actual evaluation network using the gradient descent algorithm. : ； wherein , , are and , is a discount factor; Step 4-12-2: After updating the parameters of the actual evaluation network, the strategy gradient is calculated according to formula (6), and the actual action network is updated by using the gradient descent algorithm parameters of the actual action network ​ ； Step 4-12-3: Utilize the updated parameters of the actual evaluation network and the actual action network. and Parameters of the target evaluation network and the target action network and Update: ； wherein, is the update rate; Step 4-13: If (k * timestep) < D, then k + 1, enter Step 4-5, and continue training within this round; if (k * timestep) >= D and n < N - 1, then the current total number of training rounds n + 1, enter Step 4-3, and start the next round of training; if (k * timestep) >= D and n >= N - 1, then the training is completed, and the trained deep reinforcement learning agent is exported; Step 5: Combine the trained DRL agent with the PID controller to implement an adaptive PID controller, and apply this controller to the BSM1 benchmark simulation platform to achieve precise control of the dissolved oxygen concentration in the fifth unit of the biochemical reaction pool.

2. The dissolved oxygen concentration self-adaptive PID control method based on deep reinforcement learning according to claim 1, characterized in that, The specific steps of Step 1 are as follows: Step 1-1: Build a biochemical reaction pool and a secondary sedimentation tank according to the BSM1 model. The biochemical reaction pool contains five units; Step 1-2: Build a PID controller as the dissolved oxygen concentration controller for the fifth unit of the biochemical reaction pool. The PID controller uses an incremental PID controller, which is specifically designed by the following equation: ； in, The aeration rate at time k , Let k be the aeration rate at time k-1. The aeration increment at time k; , , These are the proportional, integral, and differential coefficients, respectively. The actual measured value of dissolved oxygen concentration at time k The expected dissolved oxygen concentration at time k difference, This represents the difference between the error at the current time and the error at the previous time. Defined as: ； wherein, , respectively the difference between the actual measured value of the dissolved oxygen concentration and the desired dissolved oxygen concentration at the time k-1, k-2.

3. The dissolved oxygen concentration self-adaptive PID control method based on deep reinforcement learning according to claim 1, characterized in that, Step 2 specifically includes the following steps: Step 2-1: Determining state space from control error of dissolved oxygen concentration ; Among them, the state space It includes three elements: dissolved oxygen concentration control error , as well as ,Right now ; Step 2-2: Determine the action space based on the parameters of the PID controller to be optimized in Step 1 ; Wherein, the action space contains three parameters of the PID controller: the proportional coefficient , the integral coefficient and the differential coefficient , that is ; Step 2-3: Determine the reward based on the difference between the actual measured value of the dissolved oxygen concentration and the desired dissolved oxygen concentration ; Among them, the reward at time k The calculation is as follows: 。 4. The dissolved oxygen concentration self-adaptive PID control method based on deep reinforcement learning according to claim 1, characterized in that, The construction steps for each module in step 3 are as follows: Step 3-1: Construct an evaluation network module using a deep neural network; The evaluation network is used to fit the Q function. The network parameters are The input is the state determined in step 2. and actions The output is the value of the Q function, i.e., Q = The evaluation network module includes the actual evaluation network. and target evaluation network Two networks with identical structures, each containing four fully connected layers: one input layer, two hidden layers, and one output layer; both hidden layers use the ReLU function as the activation function, and both have the same number of hidden layer nodes. Step 3-2: Construct an action network module using a deep neural network; The action network is used to fit the deterministic policy function. The network parameters are The input is the state determined in step 2. The output is a deterministic policy function. The value, which is also the action determined in step 2. ,Right now The action network module includes the actual action network. and target action network Two networks with identical structures, each containing four fully connected layers: one input layer, two hidden layers, and one output layer. The hidden layers in both networks use the ReLU function as the activation function, and the number of nodes in the hidden layers is the same. The output layers in both networks use the Tanh function as the activation function. Step 3-3: Construct an experience replay module to store the experience data generated during training of the DRL agent in the environment built in Step 1.

5. The dissolved oxygen concentration self-adaptive PID control method based on deep reinforcement learning according to claim 1, characterized in that, The specific control process in step 5 is as follows: Step 5-1: Sample the dissolved oxygen concentration at the current sampling time. Compared with the current expected dissolved oxygen concentration The difference is calculated to obtain the error value of the dissolved oxygen concentration at the current sampling time. Then The input is fed into the error calculation module, where the error values ​​compared to the previous sampling time and the time before that are calculated to obtain a set of output values. ; Step 5-2: Take the result from step 5-1 The actual action network input into the DRL agent The action output by the intelligent agent is obtained. Then, according to equation (8), the new parameters of the adaptive PID controller are output. ; ； Step 5-3: Take the result from step 5-1 and the new parameters of the adaptive PID controller obtained in step 5-2 The input is fed into the adaptive PID controller, and the output of the adaptive PID controller is obtained according to equation (1). ; Step 5-4: Based on the output of the adaptive PID controller Adjusting the aeration rate of the fifth unit of the biochemical reaction tank Then, the above steps are repeated at the next sampling time to achieve precise control of dissolved oxygen concentration.

Images