Neural network optimal control method for multiple devices in sewage treatment process based on reinforcement learning

Abstract

The optimal control method of multi-device neural network in sewage treatment process based on reinforcement learning involves the field of artificial intelligence. The dissolved oxygen concentration controlled by the blower and the nitrate nitrogen concentration controlled by the return pump are the main indicators affecting the effluent quality. Therefore, how to achieve stable tracking control of the blower and the return pump is crucial to the sewage treatment process. To address this problem, an optimal control method of multi-device neural network based on reinforcement learning is proposed. First, the judgment-execution network learning structure is incorporated into the control design. The judgment network is used to minimize the cost function consisting of the control error and the control variable, and the network parameters are updated based on the gradient descent algorithm. Subsequently, considering the unknown dynamic characteristics in the sewage treatment process, the execution network is used to approximate the unknown dynamics, thereby providing accurate information for the controller. Finally, the optimal controller is designed based on the data information provided by the network to reduce energy consumption while ensuring control accuracy.

Description

Technical Field

[0001] The present invention is applied to the process of urban sewage treatment, based on the reinforcement learning method, taking into account the accuracy and energy consumption of the controller at the same time, and realizing the optimal regulation of multiple devices. Background Art

[0002] In recent years, with the rapid development of the global economy, the amount of sewage has increased rapidly. The role of sewage treatment technology in alleviating water shortages has received increasing attention. It is worth noting that sewage treatment involves: (1) numerous physical and biochemical reactions associated with each treatment unit. (2) High nonlinearity. (3) Uncertainty in the concentration of each pollutant in the influent. (4) Strong disturbances in the influent flow rate and pollutant load. Therefore, there is an urgent need to develop reliable and efficient control technologies to ensure its stable operation and comply with strict environmental regulations.

[0003] Fuzzy neural network is a computational model that combines fuzzy logic and neural network. It inherits the advantages of neural network for nonlinear mapping, and introduces the concept of fuzzy logic to deal with uncertainty and ambiguity. It performs well in solving the problem of imprecise information in sewage treatment process. In addition, the control mechanism based on instantaneous tracking error update needs to be improved in sewage treatment process. If long-term tracking error is considered, the designed controller can obtain more stable control performance. At present, the optimal control method based on reinforcement learning with feedback, reward or punishment as the core performs well in solving complex system control problems. And the learning algorithm based on the judgment-execution network structure can solve the problem of a lot of time investment in training the model. Therefore, these methods are applied to the optimal control of multiple equipment in sewage treatment process, while ensuring control accuracy, reducing energy consumption. Summary of the invention

[0004] This paper designs a multi-device optimal controller based on reinforcement learning to achieve tracking control of dissolved oxygen concentration and nitrate nitrogen concentration. First, the judgment-execution network learning structure is constructed with reference to the fuzzy neural network. Then, the judgment network is designed to calculate the long-term cumulative error in the sewage treatment process, and the execution network based on gradient descent is designed to approximate the uncertain dynamics in the sewage treatment process. Finally, the optimal controller is designed to accurately control the key variable K L a5(t) and Q c (t), and ensure that the tracking error approaches zero.

[0005] The specific technical solution of the present invention is:

[0006] Step 2.1: Determine the kinetic model for dissolved oxygen and nitrate concentrations:

[0007]

[0008] Where k1=0.33 and k2=0.12 are two constants, and:

[0009] β1(t)=Q1(t)S NO,1 (t)-Q2(t)S NO,2 (t) (2)

[0010] β2(t)=Q4(t)S O,4 (t)-Q5(t)S O,5 (t) (3)

[0011] S O (t) = S O,S -S O,5 (t) (4)

[0012] in

[0013]

[0014] Where S O,m (t), m = 4, 5 is the dissolved oxygen concentration in the mth area of ​​the biochemical reaction pool, S NO,n (t), n = 1, 2, 5 is the nitrate nitrogen concentration in the nth area of ​​the biochemical reaction pool, X B,H (t) is the heterotrophic biomass concentration, Y H =0.67 represents the heterotrophic biomass yield coefficient, μ(t) is the growth respiration coefficient, K L a5(t) is the oxygen transfer coefficient, Q c (t) is the internal recirculation rate, Qq(t), q=1,2,4,5 is the flow rate of the ith pool, V5=1333m 3 is the volume of the fifth zone biochemical reaction pool, V2 = 1333m 3 is the volume of the second zone biochemical reaction pool, S O,S =8mg / L is the saturation value of dissolved oxygen concentration, μ H =4 is the maximum specific growth rate of heterotrophic bacteria, η g =0.8 is the correction factor for heterotrophic growth, k O =0.2 is the saturation coefficient of aerobic respiration of heterotrophic bacteria, k NO =0.5 is the nitrate nitrogen saturation coefficient of heterotrophic bacteria.

[0015] Step 2.2: Determine the fuzzy neural network:

[0016] Fuzzy neural network uses neural network to realize fuzzy logic. Specifically, it can use the self-learning ability of neural network to learn and optimize system parameters to realize online learning of fuzzy system. It has the advantages of fuzzy reasoning and neural network. The network consists of input layer, RBF layer, normalization layer and output layer. The network output result Q(t) is defined as follows:

[0017]

[0018] where x i (t) = [e(t) Δe(t)], where e(t) is the error value of dissolved oxygen concentration or nitrate nitrogen concentration obtained by the sensor, Δe(t) is the difference between the current error value and the error value at the previous moment, c ij (t) and σ ij (t) are the center and width values ​​of the RBF layer, which are randomly generated in (-1 1), Θ(t) is the weight value between the normalization layer and the output layer, N(t) represents the output value of the normalization layer, k = 2 represents the number of neurons in the input layer, and l = 6 represents the number of neurons in the RBF layer or the normalization layer.

[0019] Step 2.3: Design the optimal controller:

[0020] 1. Design of blower controller

[0021] (1) Define the tracking error of the blower controller:

[0022] e O (t) = S O,5 (t)-S O,set (t) (7)

[0023] Where S O,5 (t) is the actual concentration of dissolved oxygen, S O,set (t) is the set value of dissolved oxygen concentration. The recommended setting range is [1.8 2.2].

[0024] (2) Derivative of the tracking error of the dissolved oxygen controller Defined as:

[0025]

[0026] in is the derivative of the set point value of dissolved oxygen concentration.

[0027] (3) Design the judgement network: During the iterative learning phase, by providing rewards or penalties to the judgement network, the current control strategy can be adjusted according to the error between the expected and actual cost functions. The long-term cost function is applied as follows:

[0028]

[0029] Where ζ1>0 is a constant for calculating the future loss function, and η1 is an integral term, which takes a value in the range of [t ∞] as time accumulates. The instantaneous cost function is expressed as:

[0030]

[0031] At this time, a fuzzy neural network is selected to approximate the long-term cost function:

[0032]

[0033] in is the optimal output weight, N c1 (t) is the output of the normalization layer, ε c1 (t) is the network approximation error, which is obtained by subtracting the approximation value of the judgment network from the current actual long-term cost function value. The current output of the judgment network is: in is the weight of the current evaluation network. Then the function estimation error is defined as:

[0034]

[0035] As time accumulates, t goes from 0→∞, then the constant ζ1 of the future loss function will go from 0→∞. Therefore, at time t, the above formula is approximately:

[0036]

[0037] in for e O The gradient of (t). Then the cost function of the evaluation network is defined as:

[0038]

[0039] Based on the gradient update law, Designed for:

[0040]

[0041] Where 0<l c1 <1 is the learning rate.

[0042] (4) Design an adaptive optimal controller based on the execution network to identify uncertain dynamics: formulate a control strategy by gradually accumulating control system experience. Define the following Lyapunov function:

[0043]

[0044] The derivative of the Lyapunov function is defined as:

[0045]

[0046] Since there is unknown dynamic information in the sewage treatment process, the execution network composed of the fuzzy neural network of formula (6) is introduced to approximate the uncertainty of the system.

[0047]

[0048] in is the optimal network weight, N a1 (t) is the output of the network normalization layer, ε a1 (t) is the approximate error of the execution network of the blower controller, which is obtained by subtracting the output value of the execution network from the current actual uncertainty state value of the system.

[0049] The desired controller input is then designed as

[0050]

[0051] Where λ1 is the control coefficient, the change of weight is the current output weight. Therefore, the actual controller input is designed as:

[0052]

[0053] The current network estimation error is defined as:

[0054]

[0055] Then, the error of executing the network is defined as:

[0056]

[0057] So the loss function of the execution network is:

[0058]

[0059] The adaptive law of the designed execution network is:

[0060]

[0061] Where 0<l a1 <1 is the learning rate. Substituting formula (23) into formula (24) further obtains:

[0062]

[0063] 2. Design of reflux pump controller

[0064] (1) Define the tracking error of the reflux pump controller:

[0065] e NO (t) = S NO,2 (t)-S NO,set (t) (26)

[0066] Where S NO,2 (t) is the actual concentration of nitrate nitrogen, S NO,set(t) is the set value of nitrate nitrogen concentration. The recommended setting range is [0.8 1.2].

[0067] (2) Derivative of controller tracking error for:

[0068]

[0069] in is the derivative of the set value of nitrate-nitrogen concentration.

[0070] (3) Design judgment network: The long-term cost function is defined as follows:

[0071]

[0072] Where ζ2>0 is a constant for calculating the future loss function, and η2 is an integral term, which takes a value in the range [t ∞] as time accumulates. The instantaneous cost function is expressed as:

[0073]

[0074] The long-term cost function is approximated by a fuzzy neural network:

[0075]

[0076] in is the optimal output weight, N c2 (t) is the output of the network normalization layer, ε c2 (t) is the network approximation error, which is obtained by subtracting the approximation value of the judgment network from the current actual long-term cost function value. The current output of the judgment network is: in is the weight of the current evaluation network. Then the function estimation error is defined as:

[0077]

[0078] As time accumulates, t changes from 0→∞, and the constant ζ2 of the future loss function will change from 0→∞. for e NO (t). Therefore, the weight update law for judging the network at time t is:

[0079]

[0080] Where 0<l c2 <1 is the learning rate.

[0081] (4) Design an adaptive optimal controller based on the execution network to identify uncertain dynamics: Define the following Lyapunov function:

[0082]

[0083] The derivative of the Lyapunov function is defined as:

[0084]

[0085] Based on the uncertainty of the execution network approximating the system, the desired controller input is:

[0086]

[0087] Where λ2 is the control coefficient, is the optimal weight of the network, N a2 (t) is the output of the network normalization layer, ε a2 (t) is the network approximation error of the reflux pump controller, which is obtained by subtracting the judgment network approximation value from the current actual long-term cost function value. is the weight change, is the current fuzzy neural network weight. Therefore, the actual controller input is designed as:

[0088]

[0089] The error of executing the network is defined as:

[0090]

[0091] in is the current estimation error. The loss function of the execution network is:

[0092]

[0093] Then the adaptive law of the designed execution network is:

[0094]

[0095] Where 0<l a2 <1 is the learning rate.

[0096] 3. Control performance evaluation:

[0097] In order to fairly evaluate the control performance, the integrated square error (ISE), integrated absolute error (IAE) and maximum absolute error (Dev max ) as the evaluation index.

[0098]

[0099]

[0100] Dev max =max{|y(t)-y d(t)|} (42)

[0101] Where y(t) is the actual concentration of dissolved oxygen or nitrate nitrogen, which can be measured by sensors during sewage treatment. d (t) is the expected dissolved oxygen or nitrate nitrogen concentration, and L is the total number of samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 Schematic diagram of RFOC used in wastewater treatment process

[0103] Figure 2 Control results for fixed set point concentration

[0104] Figure 3 Fixed set point tracking error

[0105] Figure 4 Oxygen transfer coefficient and internal recirculation coefficient

[0106] Figure 5 Control results for dynamic set point concentration

[0107] Figure 6 Tracking error of dynamic set point

[0108] Figure 7 Oxygen transfer coefficient and internal recirculation coefficient DETAILED DESCRIPTION

[0109] (1) Case 1: Fixed setpoint tracking control

[0110] The sewage treatment plant was tracked and controlled with sunny weather as the background. The fixed set value of dissolved oxygen concentration was 2mg / L, and the fixed set value of nitrate nitrogen concentration was 1mg / L.

[0111] In order to evaluate the control performance of RFOC, it was compared with the self-organizing fuzzy neural network (SOFNN) controller, proportional differential integral (PID) controller and adaptive dynamic programming (ADP) controller. Figures (2A) and (2B) plot the change trajectories of DO and NO3-N under different control methods. The control accuracy of RFOC is significantly higher than that of FNN, PID and ADP. The tracking control errors of dissolved oxygen concentration and nitrate nitrogen concentration are shown in Figures (3A) and (3B). It can be seen that the error of the RFOC method is smaller than that of other methods. Figures (4A) and (4B) plot the evolution curves of oxygen transfer coefficient and internal recirculation coefficient.

[0112] (2) Case 2: Dynamic setpoint tracking control

[0113] With heavy rain as the background, the sewage treatment plant was dynamically set to track and control the set point. The dynamic set point of dissolved oxygen concentration was initially set to 2 mg / L, increased to 2.1 mg / L on the 4th day, decreased to 1.95 mg / L on the 8th day, and increased to 2 mg / L on the 12th day. The dynamic set point of nitrate nitrogen concentration was initially set to 1 mg / L, increased to 1.2 mg / L on the 4th day, decreased to 0.85 mg / L on the 8th day, and increased to 1 mg / L on the 12th day.

[0114] The comparison results are as follows Figure 5-Figure 7 As shown. The control tracking trajectories of dissolved oxygen concentration and nitrate nitrogen concentration are shown in Figure (5A) and Figure (5B). It can be clearly seen that RFOC not only has a smaller turning point than FNN at the inflection point, but also has better control accuracy than ADP and PID. The tracking error trajectory is shown in Figure (6A) and Figure (6B), which verifies that the proposed control method can obtain the minimum convergence range. Figure (7A) and Figure (7B) plot the evolution curves of oxygen transfer coefficient and internal recirculation coefficient. Obviously, RFOC has faster adaptive ability than FNN and higher control accuracy than ADP and PID.

Claims

1. A multi-device neural network optimal control method for sewage treatment process based on reinforcement learning, characterized in that: Here are the steps: (1) Kinetic model for determining dissolved oxygen and nitrate nitrogen concentrations: Where k1=0.33 and k2=0.12 are two constants, and: β1(t)=Q1(t)S NO,1 (t)-Q2(t)S NO,2 (t) (2) β2(t)=Q4(t)S O,4 (t)-Q5(t)S O,5 (t) (3) S O (t)=S O,S -S O,5 (t) (4) in Where S O,m (t), m = 4, 5 is the dissolved oxygen concentration in the mth area of ​​the biochemical reaction pool, S NO,n (t), n = 1, 2, 5 is the nitrate nitrogen concentration in the nth area of ​​the biochemical reaction pool, X B,H (t) is the heterotrophic biomass concentration, Y H =0.67 represents the heterotrophic biomass yield coefficient, μ(t) is the growth respiration coefficient, K L a5(t) is the oxygen transfer coefficient, Q c (t) is the internal reflux flow, Q q (t), q = 1, 2, 4, 5 is the flow rate of the ith pool, V5 = 1333 m 3 is the volume of the fifth zone biochemical reaction pool, V2 = 1333m 3 is the volume of the second zone biochemical reaction pool, S O,S =8mg / L is the saturation value of dissolved oxygen concentration, μ H =4 is the maximum specific growth rate of heterotrophic bacteria, η g =0.8 is the correction factor for heterotrophic growth, k O =0.2 is the saturation coefficient of aerobic respiration of heterotrophic bacteria, k NO =0.5 is the saturation coefficient of heterotrophic nitrate nitrogen; (2) Determine the fuzzy neural network: The network consists of an input layer, an RBF layer, a normalization layer, and an output layer; the network output result Q(t) is defined as follows: where x i (t) = [e(t)△e(t)], e(t) is the error value of dissolved oxygen concentration or nitrate nitrogen concentration obtained by the sensor, △e(t) is the difference between the current error value and the error value at the previous moment, c ij (t) and σ ij (t) are the center and width values ​​of the RBF layer, which are randomly generated in (-1 1), Θ(t) is the weight value between the normalization layer and the output layer, N(t) represents the output value of the normalization layer, k = 2 represents the number of neurons in the input layer, and l = 6 represents the number of neurons in the RBF layer or the normalization layer; (3) Design of blower controller: ①Define the tracking error of the blower controller: e O (t)=S O,5 (t)-S O,set (t) (7) Where S O,5 (t) is the actual concentration of dissolved oxygen, S O,set (t) is the set value of dissolved oxygen concentration. The set value setting range is recommended to be selected in [1.8 2.2]; ②Derivative of dissolved oxygen controller tracking error Defined as: in is the derivative of the set value of dissolved oxygen concentration; ③ Design the judgement network: During the iterative learning phase, by providing rewards or penalties to the judgement network, the current control strategy can be adjusted according to the error between the expected and actual cost functions; the long-term cost function is applied as follows: Where ζ1>0 is a constant for calculating the future loss function, η1 is an integral term with a value range of [t ∞); the instantaneous cost function is expressed as: At this time, a fuzzy neural network is selected to approximate the long-term cost function: in is the optimal output weight, N c1 (t) is the output of the normalization layer, ε c1 (t) is the network approximation error; the current output of the evaluation network is: in is the weight of the current evaluation network; then the function estimation error is defined as: As time accumulates, t changes from 0 to ∞, then the constant ζ1 of the future loss function will change from 0 to ∞; therefore, at time t, the above formula is approximately: in for e O The gradient of (t); then the cost function of the evaluation network is defined as: Based on the gradient update law, Designed for: Where 0 <l c1 <1 is the learning rate; ④ Design an adaptive optimal controller based on the execution network to identify uncertain dynamics: Lyapunov function: The derivative of the Lyapunov function is defined as: Since there is unknown dynamic information in the sewage treatment process, the execution network composed of the fuzzy neural network of formula (6) is introduced to approximate the uncertainty of the system; in is the optimal network weight, N a1 (t) is the output of the network normalization layer, ε a1 (t) is the approximation error of the execution network of the blower controller; The desired controller input is then designed as Where λ1 is the control coefficient, the change of weight is the current output weight; therefore, the actual controller input is designed to be: The current network estimation error is defined as: Then, the error of executing the network is defined as: So the loss function of the execution network is: The adaptive law of the designed execution network is: Where 0 <l a1 <1 is the learning rate; Substituting formula (23) into formula (24) further yields: (4) Design of reflux pump controller: ① Define the tracking error of the reflux pump controller: e NO (t)=S NO,2 (t)-S NO,set (t) (26) Where S NO,2 (t) is the actual concentration of nitrate nitrogen, S NO ,set(t) is the set value of nitrate nitrogen concentration, and the set value setting range is recommended to be selected in [0.8 1.2]; ② The derivative of the controller tracking error for: in is the derivative of the set value of nitrate nitrogen concentration; ③ Design judgment network: The long-term cost function is defined as follows: Where ζ2>0 is a constant for calculating the future loss function, η2 is an integral term with a value range of [t ∞); the instantaneous cost function is expressed as: The long-term cost function is approximated by a fuzzy neural network: in is the optimal output weight, N c2 (t) is the output of the network normalization layer, ε c2 (t) is the network approximation error; the current output of the evaluation network is: in is the weight of the current evaluation network; then the function estimation error is defined as: As time accumulates, t changes from 0 to ∞, and the constant ζ2 of the future loss function changes from 0 to ∞; for e NO (t); therefore, the weight update law for judging the network at time t is: Where 0 <l c2 <1 is the learning rate; ④ Design an adaptive optimal controller based on the execution network to identify uncertain dynamics: Consider the following Lyapunov function: The derivative of the Lyapunov function is defined as: Based on the uncertainty of the execution network approximating the system, the desired controller input is: Where λ2 is the control coefficient, is the optimal weight of the network, N a2 (t) is the output of the network normalization layer, ε a2 (t) is the network approximation error of the reflux pump controller, is the weight change, is the current fuzzy neural network weight; therefore, the actual controller input is designed as: The error of executing the network is defined as: in is the current estimated error; the loss function of the execution network is: Then the adaptive law of the designed execution network is: Where 0 <l a2 <1 is the learning rate; Parameter settings (1) Blower controller parameter setting: The parameter values ​​of the judgment network are set as follows: learning rate l c1 =0.1, initial value of weight Width value σ=[4.2 3.1 5.3 4.3 6.5 4.2;-9.4 -5.6 -7.4 -3.6 -4.0-8.0],center value c=[-0.1 -0.2 -0.3 -0.4 -0.6 -0.7;-0.6 -0.4 -0.3 -0.2 -0.5 -0.4]; The parameter values ​​of the execution network are set as follows: learning rate l a1 =0.1, initial value of weight Width value σ=[4.2 3.1 5.3 4.3 6.5 4.2;-9.4 -5.6 -7.4 -3.6 -4.0 -8.0],center value c=[-0.1 -0.2 -0.3 -0.4 -0.6 -0.7;-0.6 -0.4 -0.3 -0.2 -0.5 -0.4]; The controller parameters were set as follows: λ1 = 325; (2) Parameter setting of reflux pump controller: The parameter values ​​of the judgment network are set as follows: learning rate l c2 =0.1, initial value of weight Width value σ=[-4.6 -5.6 -8.7 -4.9 -5.6 -4.6;7.6 4.5 3.1 4.58.1 9.2],center value c=[-0.4 -0.6 -0.7 -0.4 -0.9 -0.5;-0.4 -0.8 -0.6 -0.7 -0.4 -0.9]; The parameter values ​​of the execution network are set as follows: learning rate l a2 =0.1, initial value of weight Width value σ=[-4.6 -5.6 -8.7 -4.9 -5.6 -4.6;7.6 4.5 3.14.5 8.1 9.2],center value c=[-0.4 -0.6 -0.7 -0.4 -0.9 -0.5;-0.4 -0.8 -0.6 -0.7 -0.4-0.9]; The controller parameters are set to: λ2=274.

Images