A sewage treatment process adaptive evaluation control method fusing knowledge transfer

Abstract

The application provides a sewage treatment process adaptive evaluation control method fusing knowledge transfer, and is used for realizing accurate tracking of a dissolved oxygen concentration set value in a sewage treatment process.In the implementation of the optimization control of a sewage treatment system with an unknown model, a direct control strategy in a traditional adaptive evaluation control (ACC) algorithm is often sensitive to system disturbance, which may cause sharp decline of control performance.The application simultaneously considers knowledge transfer and incremental control, optimizes the control strategy of a conventional ACC algorithm into an incremental type, and effectively improves the anti-interference performance.Secondly, with the aid of historical operation data of the system, knowledge transfer between different control methods is implemented, and the trial and error cost of early training of the application is successfully reduced.Thirdly, a novel utility function with a soft strategy constraint mechanism is designed, and the stability in the online optimization process is enhanced.Finally, the effectiveness of the application is verified through experiments.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of sewage treatment. BACKGROUND

[0002] With the acceleration of urbanization and the continuous increase of population density, the amount of urban sewage is rapidly increasing, which poses a serious challenge to urban water environment. As a key link for protecting the environment and promoting water resource recycling, sewage treatment is self-evident. Effective sewage treatment not only protects the ecological balance, but also provides a reliable way for water resource recycling in cities, which has far-reaching significance for promoting sustainable development and building an ecological civilization society. Therefore, optimizing the sewage treatment process and improving the treatment efficiency is an important issue to be solved in the current environmental protection field.

[0003] Accurate regulation of dissolved oxygen concentration is crucial to ensure the efficient and stable operation of the sewage treatment process. Some traditional control methods, such as proportional-integral-derivative (PID) control, on-off control, and feedforward control, can achieve the regulation of dissolved oxygen concentration to some extent, but they are difficult to dynamically adapt to the uncertainty changes in actual operation when facing complex and variable working conditions in the sewage treatment process, such as fluctuation of influent flow, change of water temperature, and difference of sludge load, etc. This leads to unstable control effect and high energy consumption. In addition, traditional control methods do not have optimization ability, which makes it difficult to meet the increasingly stringent effluent water quality standards and energy saving and emission reduction requirements.

[0004] Therefore, exploring more intelligent and efficient control strategies has become a hot topic in the field of sewage treatment. Researchers have begun to design controllers for sewage treatment processes based on intelligent control methods such as neural network control and model predictive control. However, neural network control only adjusts weights based on the tracking error at the current time, which limits its optimization ability. Model predictive control has good optimization performance because it considers long-term tracking errors in the future, but it is strongly dependent on accurate system models. The adaptive critic control (ACC) algorithm is widely used in control optimization problems of unknown nonlinear systems due to its strong optimization and adaptive performance. ACC algorithm combines the ideas of dynamic programming and reinforcement learning, and continuously optimizes the control strategy through iterative learning. It can automatically adjust parameters to adapt to changes in system state without knowing the system model, thereby achieving better control effect. Applying ACC algorithm in sewage treatment process is expected to improve sewage treatment efficiency and reduce operating costs, which has important significance for promoting the intelligent upgrading of sewage treatment technology.

[0005] Although ACC algorithm shows great potential in theory, it still faces a series of challenges when applied to wastewater treatment process. First, the control strategy of conventional ACC algorithm is only related to the current system state, and it will produce large fluctuations when disturbed. Second, the training data of online ACC algorithm needs to be obtained through gradual interaction with the system, which often faces the problem of lack of training data, resulting in slow convergence speed and poor learning stability of the algorithm. In order to solve this problem, scholars usually introduce the experience replay mechanism to store historical experience data to stabilize the learning process. However, this approach significantly increases the burden of data storage and calculation, which is not conducive to the widespread application of ACC algorithm in practical engineering. In summary, although ACC algorithm provides a new idea for dissolved oxygen concentration control in urban wastewater treatment process, there are still many challenges in its direct application. How to maintain the advantages of ACC algorithm optimization and self-adaptation while overcoming its sensitivity to disturbance, poor learning stability and heavy storage and calculation burden has become the key to current research. SUMMARY

[0006] Compared with the classic ACC algorithm, the control strategy of the present application is improved from direct to incremental, effectively improving the anti-interference performance of the algorithm. At the same time, the knowledge transfer technology and the expert experience in the traditional incremental PID control algorithm are used to construct the initial control strategy of the ACC algorithm, overcoming the slow convergence problem of online ACC algorithm. To avoid the data storage pressure caused by the experience replay mechanism, the present application innovatively designs a new utility function with a soft strategy constraint mechanism, which can improve the stability of the online optimization process of the algorithm while avoiding the increase of data storage burden. Finally, the simulation results show that the present application has good control performance for dissolved oxygen concentration in wastewater treatment process.

[0007] The structure diagram of the wastewater treatment system applying the present application is as shown in Figure 1 , but is not limited thereto. Among them, 1.1 represents a biochemical reaction tank, 1.2 represents an aerobic zone composed of unit one 1.4 and unit two 1.5 of the biochemical reaction tank, 1.3 represents an anaerobic zone composed of unit three 1.6, unit four 1.7 and unit five 1.8 of the biochemical reaction tank, 1.9 represents a secondary sedimentation tank, 2.1 represents a gas flow meter, 2.2 represents a dissolved oxygen concentration sensor, 3.1 represents a programmable logic controller, 3.2 represents a frequency converter, and 4.1 represents a blower.

[0008] In the wastewater treatment system (such as Figure 1The wastewater treatment system is shown in the figure, mainly covering two key parts of biochemical reaction tank 1.1 and secondary sedimentation tank 1.9. The first two units of the biochemical reaction tank are anaerobic zone 1.2, and the last three units are aerobic zone 1.3. The municipal wastewater flows through the anaerobic zone 1.2 and the aerobic zone 1.3 in turn, and during this period, through nitrification and denitrification, the purification of organic pollutants, nitrogen and phosphorus elements in wastewater is realized. Subsequently, the wastewater flows into the secondary sedimentation tank 1.9 for sedimentation operation, further separating the supernatant and sludge. During the entire biochemical reaction process, the dissolved oxygen concentration in unit five is one of the key factors affecting microbial activity, organic matter degradation efficiency and operating energy consumption. If the dissolved oxygen concentration is too low, the respiration of microorganisms will be limited, resulting in a significant reduction in wastewater treatment efficiency; on the contrary, too high dissolved oxygen concentration will cause energy waste, and even may lead to the growth of harmful microorganisms. Therefore, accurate adjustment of the dissolved oxygen concentration is crucial to ensure the efficient and stable operation of the wastewater treatment process. In the wastewater treatment system of the application, data is collected by gas flow meter 2.1 and dissolved oxygen concentration sensor 2.2 and transmitted to programmable logic controller 3.1, programmable logic controller 3.1 calculates and outputs control signals to frequency converter 3.2, and then frequency converter 3.2 adjusts the motor speed of air blower 4.1, finally changes the oxygen transfer coefficient, realizes the control of dissolved oxygen concentration.

[0009] Figure 2 The overall structure of the application is shown in the figure, and the overall implementation is as follows: step 1, establish a tracking control optimization problem for the dissolved oxygen concentration in the wastewater treatment system; step 2, initialize the control strategy using knowledge transfer technology; repeat step 3 to achieve accurate control of the dissolved oxygen concentration in the wastewater treatment system and online optimization of the control strategy. Next, the application gives the specific implementation process of each step.

[0010] Step 1: Establish a tracking control optimization problem for the dissolved oxygen concentration in the wastewater treatment process. The wastewater treatment system can be represented as follows:

[0011] x k+1 =H(x k ,u k ),k=0,1,2,... (1)

[0012] where the system state represents the dissolved oxygen concentration in unit five at time k, the control input represents the oxygen transfer coefficient at time k, H(·,·) is an unknown system function, represents the set of all positive real numbers. The set value d k is represented as

[0013] d k =δ(k) (2)

[0014] Where δ(·) is the set value function. Based on engineering experience, the set value of dissolved oxygen concentration is usually selected as 2 mg / L. The tracking error between the dissolved oxygen concentration and its set value is e k Defined as

[0015] e k =x k -d k (3)

[0016] In the conventional ACC algorithm, the calculated control strategy is a direct control strategy π(e k ), so that the control input u k =π(e k In order to enhance the anti-interference ability of the algorithm, the present invention designs an incremental control strategy η(e k ), so that the control input satisfies the following formula:

[0017]

[0018] In addition, in order to enhance the stability of the learning process, a utility function is proposed as follows:

[0019]

[0020] Among them, α1, α2 and α3 are constant weights greater than 0. In the present invention, α1 = 0.1, α2 = 0.01, and α3 = 0.005 are taken. k ,Δu k ), It is mainly used to reflect the cost caused by tracking error and to directly ensure control accuracy; It focuses on reflecting the cost of the change in control input, which is used to avoid excessive fluctuations in control input and ensure the stability of the control process. k ,Δu k ) is a soft policy constraint function innovatively designed by the present invention, which plays a key role in the adaptive soft policy constraint in the control strategy optimization process. It is expressed as

[0021]

[0022] Among them, θ(e k )=-121e k +102e k-1 -e k-2 This is an incremental PID control strategy that can achieve stable control of dissolved oxygen concentration. The value is less than or equal to 1×10 -3 ,S(e k,Δu k ) This item will automatically relax the restrictions on the strategy optimization process, giving the control strategy greater adjustment freedom to give full play to its optimization performance; on the contrary, if the optimization effect is not good and the control accuracy is low, then The value is greater than or equal to 1×10 -2 , which will strengthen the constraints on the strategy optimization process and guide the control strategy η(e k ) to incremental PID control strategy θ(e k ) are close to each other, which enables the present invention to optimize the control accuracy while taking into account the stability, thereby ensuring the reliable operation of the entire control system. Then, the value function V(e k ,Δu k ) is defined as

[0023]

[0024] Where γ∈(0,1) is a discount factor used to ensure that the value function is bounded. Here, γ=0.95 is selected. The optimal value function V * (e k ,Δu k ) and the optimal incremental control strategy η * (e k ) are defined as

[0025]

[0026] and

[0027]

[0028] Since Equation (8) belongs to the Hamilton-Jacobi-Bellman equation, it is difficult to directly obtain its analytical solution. Therefore, the present invention is based on the execution-judgment online optimization mechanism and uses the neural network function approximation tool to continuously approximate the optimal value function V * (e k ,Δu k ) and the optimal incremental control strategy η * (e k ).

[0029] Step 2: Use knowledge transfer technology to complete the initialization of the control strategy. The present invention consists of an execution network and a judgment network. The execution network is used to approximate the optimal incremental control strategy η * (e k ), the evaluation network is used to approximate the optimal value function V * (e k ,Δu kIt should be noted that both the execution network and the evaluation network are back-propagation neural networks consisting of an input layer, a hidden layer, and an output layer. Therefore, the control strategy can be initialized by pre-training the execution network.

[0030] In order to complete the initialization of the control strategy using the knowledge transfer technology, the present invention first uses the historical operation data of the sewage treatment system under the incremental PID control algorithm to establish a data set D = {(e′ j ,Δu′ j )|j=1,2,...,n}, where e j ′ and Δu′ j They represent the jth tracking error sample and incremental control input sample under the PID control strategy, respectively. n is the total number of samples in the dataset D. In the implementation, n = 2687 is selected, but it is not limited to this. Then, the pre-training of the execution network is completed based on the dataset D. The output of the execution network during the pre-training process can be expressed as

[0031]

[0032] in, and is the weight vector of the execution network. a =10 represents the number of neurons in the hidden layer of the execution network. In addition, the activation function φ(·) is set to the hyperbolic tangent function. The approximation error of the execution network during pre-training can be expressed as

[0033]

[0034] The performance index function of the network during pre-training can be expressed as

[0035]

[0036] Where, e′=[e1′,e2′,...,e n ′] is the input of the performance indicator function. According to the gradient descent principle, the update rule of the network weights in the pre-training process can be expressed as

[0037]

[0038] Among them, := represents the assignment operation, β a =0.005 is the learning rate of the execution network during pre-training. Repeat (10)-(13) until Ψ(e′)<10 -5 At this point, the present invention successfully completes the knowledge transfer from the PID control algorithm to the present invention based on the data set D, improving the control performance of the dissolved oxygen concentration in the initial stage.

[0039] Step 3: Online optimization of control policy based on execution-evaluation framework. To ensure the adaptive and optimization performance of the algorithm, the present application updates the value function and improves the policy according to the real-time tracking error e k The output of the evaluation network during online training can be represented as

[0040]

[0041] wherein, and

[0042] is the weight vector of the evaluation network. h c =12 represents the number of hidden layer neurons of the evaluation network. The approximation error of the evaluation network during online training can be represented as

[0043]

[0044] The performance index of the evaluation network can be represented as

[0045]

[0046] According to the gradient descent principle, the update rule of the weight vector w c2 can be represented as

[0047]

[0048] wherein, l c =0.01 is the learning rate during online training of the evaluation network. It is worth noting that, in order to reduce the computational burden of online training, the present application only updates the outer weight w c2 , and the inner weight w c1 is randomly initialized and remains unchanged. Repeat (14)-(17) until E c (e k )<10 -5 . At this time, the update of the evaluation network at time k is completed. Next, according to the evaluation results of the control policy by the evaluation network, the policy improvement is completed. The output of the execution network during online training can be represented as

[0049]

[0050] The approximation error of the execution network during online training is defined as

[0051]

[0052] wherein, U d is the ideal training target, usually set U d =0. The performance index of the execution network during online training is defined as

[0053]

[0054] According to the gradient descent principle, the weight vector w a2 The update rule can be expressed as

[0055]

[0056] Repeat (18)-(21) until At this point, the optimization of the execution network at time k is completed. Finally, the trained execution network is used to calculate the control input according to the current tracking error.

[0057]

[0058] Apply control input to the wastewater treatment system to track dissolved oxygen concentration against a set point. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Sewage treatment system structure diagram implemented by the present invention

[0060] Figure 2 Framework diagram of the present invention

[0061] Figure 3 Test results of network approximation error after pre-training

[0062] Figure 4 Changes in dissolved oxygen concentration under different control algorithms

[0063] Figure 5 Change process of the oxygen transfer coefficient of the present invention DETAILED DESCRIPTION

[0064] Benchmark simulation model no. 1 (BSM1) was jointly developed by the European Union Organization for Scientific and Technological Cooperation and the International Water Association. It is based on the activated sludge model and the double exponential sedimentation velocity function, and simulates the biochemical reactions occurring in the biochemical reaction tank and the clarification process in the secondary clarification container respectively. In view of the fact that BSM1 has sufficient reduction degree to the actual sewage treatment process, it has become a benchmark test platform widely used by researchers in the field of sewage treatment, and can fairly reflect the performance of various control algorithms. Therefore, the present invention uses BSM1 to simulate a 14-day sewage treatment process to evaluate the control performance of the present invention on dissolved oxygen concentration. Note that the following results are all based on water inlet data under sunny weather. According to the experience of algorithm debugging, the parameter values ​​of the algorithm proposed in the present invention are selected as follows:

[0065] (1) Setting value d k= 2, constant weight a1 = 0.1, a2 = 0.01, a3 = 0.005, discount factor γ = 0.95

[0066] (2) Total number of data set samples n = 2687, pre-training learning rate β a = 0.005, pre-training stop criterion Ψ (e') < 10 -5 , number of hidden layer neuron nodes h a = 10, h c = 12

[0067] (3) Online training learning rate l c = 0.01, l a = 0.01, online training stop criterion E c (e k ) < 10 -5 ,

[0068]

[0069] After completing the pre-training stage of the network, 600 groups of data are used to test the approximation accuracy of the PID control strategy. The test results are shown in Figure 3 , the approximation error is controlled within 0.025, showing good approximation accuracy. To intuitively show the superior performance of the present application, it is compared with the traditional PID control algorithm and the conventional ACC algorithm in terms of dissolved oxygen concentration control effectiveness. The experimental results are shown in Figure 4 , compared with the other two control algorithms, the present application has higher control accuracy and smaller error fluctuation for dissolved oxygen concentration. In addition, the change process of the oxygen transfer coefficient as the control input in the present application is shown in Figure 5 .

[0070] In order to more accurately evaluate the control performance of the algorithm, the present application further adopts two industry-recognized evaluation indexes, integral of squared error (ISE) and maximal deviation from set point (Dev max ), to quantitatively analyze the control performance of the algorithm. The definitions of the two evaluation indexes are as follows:

[0071]

[0072] Wherein, k1 and k2 respectively represent the start time and end time of tracking error between statistical dissolved oxygen concentration and set value. According to industry experience, data of the last 7 days in 14 days are usually counted, that is, k1=1345 represents the time starting from the 8th day, and k2=2687 is the end time of the 14th day. ISE focuses on reflecting the average control accuracy of dissolved oxygen concentration by the algorithm at a long time scale, and the smaller the value is, the better the average control effect is; Dev max focuses on reflecting the fluctuation degree of dissolved oxygen concentration control by the algorithm, and the smaller the value is, the higher the control stability is. Under the action of the PID control algorithm, ISE=5.64*10 -4 , Dev max =0.1164; under the action of the conventional ACC control algorithm, ISE=1.90*10 -5 , Dev max =0.0223; and under the action of the present application, ISE=3.19*10 -6 , Dev max =0.0084. It can be seen from the comparison of these data that the present application has obvious improvement in average control accuracy and stable control performance, which powerfully verifies the application value and innovation advantage of the present application in the field of dissolved oxygen concentration control in the sewage treatment process, and provides a new idea for the innovation of intelligent control technology of sewage treatment.

[0073] The present application improves the control strategy to be incremental, enhances the anti-interference ability, and makes it better cope with complex working conditions. The initial strategy is constructed by using knowledge transfer technology and traditional incremental PID control experience, which effectively reduces the training cost. A new type of utility function is designed, which does not cause data storage burden and effectively guarantees the stability of online optimization. Compared with the traditional PID control algorithm and the general ACC algorithm, the present application has obvious improvement in control accuracy, can more accurately make the dissolved oxygen concentration track the set value, effectively reduces the error fluctuation, and ensures the efficient and stable operation of the sewage treatment process.

Claims

1. An adaptive judgment control method for sewage treatment process integrating knowledge transfer, characterized by The following steps are involved: Step 1: Establish the tracking and control optimization problem of dissolved oxygen concentration in the sewage treatment process; The sewage treatment system is represented as a nonlinear system as follows: x k+1 =H(x k ,u k ),k=0,1,2,... (1) Among them, the system status represents the dissolved oxygen concentration in the anaerobic zone unit 5 of the biochemical reaction tank at time k, and the control input represents the oxygen transfer coefficient at time k, H(·,·) is an unknown system function, Represents the set of all positive real numbers; the set value of dissolved oxygen concentration d k Expressed as d k =δ(k) (2) Where δ(·) is the set value function; the set value of dissolved oxygen concentration is selected as 2 mg / L; the tracking error between dissolved oxygen concentration and its set value is e k Defined as e k =x k -d k (3) To design an incremental control strategy η(e k ), so that the control input satisfies the following formula: A utility function is proposed as follows: Among them, α1, α2 and α3 are constant weights greater than 0; among them: in, Then, the value function V(e k ,Δu k ) is defined as Where γ∈(0,1) is the discount factor; Optimal value function V * (e k ,Δu k ) and the optimal incremental control strategy η * (e k ) are defined as and Based on the execution-judgment online optimization mechanism and using the neural network function approximation tool, it continuously approaches the optimal value function V * (e k ,Δu k ) and the optimal incremental control strategy η * (e k ); Step 2: Use knowledge transfer technology to complete the initialization of the control strategy; The execution network is used to approximate the optimal incremental control strategy η * (e k ), the evaluation network is used to approximate the optimal value function V * (e k ,Δu k ); Both the execution network and the evaluation network are back-propagation neural networks consisting of an input layer, a hidden layer, and an output layer; The data set D = {(e′ j ,Δu′ j )|j=1,2,...,n}, where e′ j and Δu′ j They represent the jth tracking error sample and incremental control input sample under the PID control strategy, respectively, and n is the total number of samples in the data set D. Then, the pre-training of the execution network is completed based on the data set D. The output of the execution network during the pre-training process is expressed as in, and is the weight vector of the execution network; h a =10 represents the number of neurons in the hidden layer of the execution network; in addition, the activation function φ(·) is set to the hyperbolic tangent function; the approximation error of the execution network during pre-training is expressed as The performance index function of the network during pre-training is expressed as Where, e′=[e1′,e2′,...,e n ′] is the input of the performance indicator function; the update rule of the execution network weights in the pre-training process is expressed as Among them, := represents the assignment operation, β a =0.005 is the learning rate of the execution network during the pre-training process; repeat (10)-(13) until Ψ(e′)<10 -5 ; Step 3: Online optimization of control strategy based on execution-criteria framework; The output of the judgment network during online training is expressed as in, and is the weight vector of the evaluation network; h c =12 represents the number of hidden layer neurons in the evaluation network; the approximation error of the evaluation network during online training can be expressed as The performance index of the network can be expressed as According to the gradient descent principle, the weight vector w c2 The update rule can be expressed as Among them, l c =0.01 is the learning rate during the online training of the network; only the outer weight w is updated c2 , inner layer weight w c1 After random initialization, it remains unchanged; repeat (14)-(17) until E c (e k )<10 -5 ; At this point, the update of the evaluation network at time k is completed; Next, the strategy is improved based on the evaluation results of the control strategy by the judgment network. The output of the execution network during the online training process is expressed as The approximation error of the execution network during online training is defined as Among them, U d is the ideal training goal, set U d =0; define the performance index of the network during online training as Weight vector w a2 The update rule is expressed as Repeat (18)-(21) until At this point, the optimization of the execution network at time k is completed; finally, the trained execution network is used to calculate the control input based on the current tracking error to achieve the tracking of the dissolved oxygen concentration to the set value;

Images