Urban sewage treatment process dissolved oxygen concentration safety control method based on inverse reinforcement learning

Abstract

The invention provides a municipal sewage treatment process dissolved oxygen concentration safety control method based on inverse reinforcement learning so as to realize stable regulation and control of dissolved oxygen concentration under false data invasion and guarantee stable operation of a municipal sewage treatment process. Firstly, an urban sewage treatment process dissolved oxygen concentration control system with false data injection attacks is constructed; secondly, constructing a cost function of an error dynamic system; finally, a safety controller based on inverse reinforcement learning is designed, and safe and stable control over the dissolved oxygen concentration of the urban sewage treatment system is achieved. Experimental results show that the method can realize accurate control after the urban sewage treatment system is invaded by false data, and ensures safe and stable operation of the urban sewage treatment system.

Description

Technical Field

[0001] This invention proposes a method for safe control of dissolved oxygen concentration in urban wastewater treatment processes based on inverse reinforcement learning, so as to achieve stable regulation of dissolved oxygen concentration under the intrusion of false data and ensure the stable operation of urban wastewater treatment processes. This invention belongs to both the field of water treatment and the field of intelligent control. Background Technology

[0002] Urban wastewater treatment plays a vital role in protecting residents' health and maintaining the ecological environment. Efficient and stable process control in urban wastewater treatment not only effectively reduces water pollution but also enables the reuse of water resources, holding an irreplaceable position in promoting sustainable urban development. Achieving stable control of dissolved oxygen concentration is particularly crucial in urban wastewater treatment process control. This concentration directly affects the efficiency of nitrification, thus impacting whether the effluent quality meets standards. Therefore, stable control of dissolved oxygen concentration has become one of the core tasks of wastewater treatment control systems.

[0003] With the continuous improvement of informatization, urban wastewater treatment process control systems are increasingly revealing security vulnerabilities. False data injection attacks are a typical and dangerous form of attack. Attackers forge sensor data, misleading controllers into making incorrect decisions, leading to loss of control of the treatment process and threatening urban public safety. Given the highly concealed, random, and uncertain nature of false data injection attacks, traditional control methods relying on accurate system models and preset cost functions often exhibit limitations in dealing with such attacks. Therefore, there is an urgent need to develop a safe control method for dissolved oxygen concentration in urban wastewater treatment processes to improve the robustness and adaptability of urban wastewater treatment systems under false data attacks, ensuring stable control of dissolved oxygen concentration in urban wastewater treatment process control systems when subjected to false data intrusion.

[0004] This invention proposes a security control strategy based on inverse reinforcement learning. First, a dissolved oxygen concentration control system for urban wastewater treatment processes is constructed to prevent false data injection attacks. Second, a cost function for the error dynamic system is constructed. Finally, a security controller based on inverse reinforcement learning is designed to achieve safe and stable control of dissolved oxygen concentration in the urban wastewater treatment system. Summary of the Invention

[0005] A method for safe control of dissolved oxygen concentration in urban wastewater treatment processes based on inverse reinforcement learning is characterized by: constructing a dissolved oxygen concentration control system for urban wastewater treatment processes susceptible to spoofed data injection attacks; constructing a cost function for the error dynamic system; designing a safety controller based on inverse reinforcement learning; and achieving safe and stable control of dissolved oxygen concentration in the urban wastewater treatment process control system. The method includes the following steps:

[0006] (1) Construct a dissolved oxygen concentration control system for urban wastewater treatment processes that is vulnerable to false data injection attacks.

[0007] The kinetic model of the dissolved oxygen concentration control system in the urban wastewater treatment process is as follows:

[0008] x(k+1)=f(x(k),u(k)) (1)

[0009] Where x(k+1) represents the dissolved oxygen concentration at time k+1 during the wastewater treatment process, x(k) represents the dissolved oxygen concentration at time k during the wastewater treatment process, u(k) represents the oxygen transfer coefficient at time k, and f(x(k),u(k)) represents the unknown nonlinear function at time k, that is, the relationship between the dissolved oxygen concentration at time k and x(k) under the action of u(k) and the dissolved oxygen concentration at time k+1.

[0010] The dissolved oxygen concentration after a false data injection attack on the urban wastewater treatment process control system is expressed as follows:

[0011]

[0012] in, x represents the dissolved oxygen concentration at time k after the injection of spurious data. a (k) represents the spurious data injected at time k;

[0013] Combining formulas (1) and (2), the dissolved oxygen concentration control model for the fifth partition under a false data injection attack is as follows:

[0014]

[0015] (2) Constructing the cost function of the error dynamic system

[0016] To achieve tracking and control of dissolved oxygen concentration, the dissolved oxygen concentration error e(k) at time k in the fifth partition is defined as:

[0017] e(k) = x r (k)-x(k) (4)

[0018] Where, x r (k) represents the set value of dissolved oxygen concentration at time k;

[0019] Design G(e(k),u(k)) as a utility function, expressed as follows:

[0020] G(e(k),u(k))=e 2 (k)α1+u 2 (k)α2+R(e(k),u(k)) (5)

[0021] Where α1, α2 < 10-3 Let R(e(k),u(k)) be the weight parameters of the utility function, and let R(e(k),u(k)) be the reward function, used to balance error control, energy consumption control, and action smoothing control under false data intrusion, as follows:

[0022] R(e(k),u(k))=r1(k)+r2(k)+r3(k) (6)

[0023] Where r1(k) is the tracking error reward term at time k, designed as r1(k) = w1(k)e -3e(k) w1(k) represents the trainable weights at time k, and r2(k) is the energy penalty term at time k, designed as follows: w2(k) is the trainable weight at time k, and r3(k) is the action smoothing penalty term at time k. The design is r3(k) = w3(k)|u(k)-u(k-1)|, where w3(k) is the trainable weight at time k, with initial values ​​w1(0) = 3.0, w2(0) = 0.3, and w3(0) = 0.1.

[0024] Based on the utility function (5), the following cost function is designed:

[0025]

[0026] Where γ∈(0,1] is the discount factor;

[0027] (3) Design a security controller based on inverse reinforcement learning

[0028] ① Solving for the optimal control law based on iterative methods

[0029] To address the issue of spoofed data injection attacks during wastewater treatment, the control objective is to find a suitable and permissible optimal control law. * (k) minimizes the cost function (7): According to the Bellman optimality principle, the optimal cost function Q * (e(k),u(k)) satisfies the following discrete-time HJB equation

[0030]

[0031] The optimal control law can be solved using the following formula:

[0032]

[0033] Solving the optimal control law u in (9) using an iterative method * Approximate control law of (k) Given the admissible control law u0(k) at time k, calculate the precision δ and the maximum allowed number of iterations j. max In each iteration, j = 0, 1, 2, ... j max Update as follows

[0034] Q j (e(k),u j (k))=G(e(k),u j (k))+γQ j (e(k+1),u j (k+1)) (10)

[0035] Based on (10), the update control law is as follows:

[0036]

[0037] Where γ∈(0,1] is the discount factor, if ||Q j+1 (e(k),u j+1 (k))-Q j (e(k),u j (k))||<δ(δ=10 -3 (or the number of iterations reaches j) max If the value is 20, then the iteration process stops, and the optimal cost function is obtained. Approximate optimal control input function Otherwise, let j = j + 1 and continue with formula (10);

[0038] ② Constructing the evaluation network and execution network

[0039] First, a fully connected neural network is constructed, consisting of three layers: an input layer, hidden layers, and an output layer, represented as follows:

[0040]

[0041] Where the input of the j-th iteration at time k is y j (k), where z is the output of the j-th iteration at time k. j (k), W j (k) is the weight matrix from the input layer to the hidden layer at time k, in the j-th iteration, V j (k) is the weight matrix from the hidden layer to the output in the j-th iteration at time k, and T represents the transpose of the matrix. For the activation function, the tanh function can be selected.

[0042] The parameters of a fully connected neural network model are updated as follows:

[0043]

[0044] Among them, E j (k) represents the loss function of the neural network in the j-th iteration at time k. The sign for partial derivatives;

[0045] To achieve the solution of the control law in step ①, an evaluation network is first constructed based on a fully connected neural network (12) to approximate the cost function Q in equation (10). j (e(k),u j (k)), and then construct an execution network to approximate the control law u of equation (11). j+1 (k), described in detail below:

[0046] The network input is evaluated as y j (k)=[e(k),u j [k], the hidden layer consists of 15 neurons, and the output cost function Q is obtained by substituting into formula (12). j (e(k),u j The neural network approximation of (k))

[0047] To estimate the accuracy of the approximate cost function, the prediction error e in the j-th iteration at time k is defined. c(j) (k) is:

[0048]

[0049] Wherein G(e(k),u j (k)) is the utility function for the j-th iteration at time k, expressed as:

[0050]

[0051] Wherein, R(e(k),u j Let (k) be the reward function for the j-th iteration at time k, expressed as follows:

[0052] R(e(k),u j (k))=r1(k)+r 2j (k)+r 3j (k) (17)

[0053] Where r1(k)=w1(k)e -3|e(k) , r 3j (k)=w3(k)|u j (k)-u(k-1)|, w1(k), w2(k), w3(k) represent the weights at time k, and their update method is shown in step ③.

[0054] Construct the following loss function E c(j) (k):

[0055]

[0056] The loss function E j (k)=E c(j)(k) Substitute into formulas (13)-(14) to solve for the weight V of the (j+1)th iteration at time k. j+1 (k), W j+1 (k);

[0057] Based on the evaluation network, an approximate control law for the execution network is constructed according to (12), with the input being the control error y. j (k) = e(k), the hidden layer includes 12 neurons, substituting into formula (12) outputs u j+1 (k) neural network approximation

[0058] Evaluate network output As the loss function of the execution network, i.e. Substituting into formula (13)-(14), the weight V of the (j+1)th iteration at time k j+1 (k), W j+1 (k);

[0059] if Or the number of iterations reaches j max Then stop the iteration process. The approximate cost function obtained in the last iteration can be regarded as the optimal cost function. The approximate control law obtained in the last iteration is the optimal control input function. Otherwise, let j = j + 1 and continue with formula (10);

[0060] ③ Update the reward function based on inverse reinforcement learning method

[0061] Based on the inverse reinforcement learning method, the reward function R(e(k),u) in (16) is applied. j (k) is updated, and the process is as follows:

[0062] The reward function weight vector w(k) = [w1(k), w2(k), w3(k)] is updated based on the dissolved oxygen concentration error at time k. The loss function for the multi-step discounted dissolved oxygen concentration tracking error in the j-th iteration at time k is designed as follows:

[0063]

[0064] Where H = 3; discount factor γ irl ∈(0,1] is used to balance instantaneous error and long-term error;

[0065] Since the dissolved oxygen concentration error at step H=3 cannot be directly obtained, a fully connected neural network (12) is used as a reference.

[0066] The hidden layer contains 15 neurons and is designed to predict the dissolved oxygen concentration tracking error for the next H=3 steps.

[0067] Let the model input be y(k) = [u(k-1), e(k)], where e(k) is the tracking error of the dissolved oxygen concentration predicted at time k, and the network output be... This represents the predicted tracking error of dissolved oxygen concentration at time k+1.

[0068] Let the model input be Network output is Indicates that in k

[0069] The predicted tracking error of dissolved oxygen concentration at time k+2;

[0070] Let the model input be Network output is Indicates that in k

[0071] The predicted tracking error of dissolved oxygen concentration at time k+3;

[0072] Calculate L at time k IRL The gradient of (w(k)) with respect to the weight w(k) is calculated using the following formula:

[0073]

[0074] Where g(k) = [g1(k), g2(k), g3(k)] is the gradient matrix at time k.

[0075] The first-order moment estimate m(k) at time k is expressed as:

[0076] m(k)=0.99m(k-1)+0.01g(k) (21)

[0077] Where m(k) = [m1(k), m2(k), m3(k)], m1(k), m2(k), m3(k) are the three elements of the first-order moment estimation matrix m(k), m(k-1) is the first-order moment estimation matrix at time k-1, and m(0) = [0,0,0].

[0078] The second-order moment estimation matrix v(k) at time k is represented as:

[0079] v(k) = 0.999v(k-1) + 0.001[g1] 2 (k),g2 2 (k),g3 2 (k)] (22)

[0080] Where, v(k) = [v1(k), v2(k), v3(k)], v1(k), v2(k), v3(k) are the three elements of the second-order moment estimation matrix v(k), v(k-1) is the second-order moment estimation matrix at time k-1, and v(0) = [0,0,0].

[0081] The updated weights can be represented as:

[0082]

[0083] Where w(k+1) is the weight matrix at time k+1, and ε is a constant to prevent the denominator from being zero, ε = 10. -8 ;

[0084] In summary, we obtain G(e(k),u j Substituting (k) into (15), we obtain the prediction error e for the j-th iteration at time k. c(j) (k);

[0085] (4) The iteration condition ||Q is satisfied in step ②. j+1 (e(k),u j+1 (k))-Q j (e(k),u j The control law obtained by (k))||<δ As the oxygen transfer coefficient of the urban sewage treatment system subjected to a false data injection attack at time k, the programmable logic controller controls the speed of the blower to control the aeration rate based on the calculated oxygen transfer coefficient, and finally achieves safe control of the urban sewage treatment process by regulating the dissolved oxygen concentration.

[0086] The inventiveness of this invention is mainly reflected in:

[0087] (1) This invention addresses the problem of unstable control of dissolved oxygen concentration caused by false data injection attacks in urban sewage treatment. By constructing a neural network to approximate an unknown nonlinear function, minimizing the cost function, and solving for the control law, it provides a basic control strategy for safe control under false data injection attacks.

[0088] (2) This invention addresses the problem of unstable dissolved oxygen concentration control caused by false data injection attacks. It designs a security control method based on inverse reinforcement learning, combining the tracking and control effect of dissolved oxygen concentration, to improve the tracking and control effect of dissolved oxygen concentration and achieve stable control of dissolved oxygen concentration under false data injection attacks. Attached Figure Description

[0089] Figure 1 This is a diagram showing the dissolved oxygen concentration control results during the urban wastewater treatment process considered in this invention after being subjected to time-varying intrusion.

[0090] Figure 2This is a diagram showing the dissolved oxygen concentration error during the safety control of urban wastewater treatment processes subject to time-varying intrusion, as considered in this invention.

[0091] Figure 3 This is a diagram showing the dissolved oxygen concentration control results after constant intrusion during the urban wastewater treatment process considered in this invention.

[0092] Figure 4 This is a diagram showing the dissolved oxygen concentration error during the safety control of urban wastewater treatment processes under constant intrusion, as considered in this invention. Detailed Implementation

[0093] A method for safe control of dissolved oxygen concentration in urban wastewater treatment processes based on inverse reinforcement learning is characterized by: constructing a dissolved oxygen concentration control system for urban wastewater treatment processes susceptible to spoofed data injection attacks; constructing a cost function for the error dynamic system; designing a safety controller based on inverse reinforcement learning; and achieving safe and stable control of dissolved oxygen concentration in the urban wastewater treatment process control system. The method includes the following steps:

[0094] (1) Construct a dissolved oxygen concentration control system for urban wastewater treatment processes that is vulnerable to false data injection attacks.

[0095] The kinetic model of the dissolved oxygen concentration control system in the urban wastewater treatment process is as follows:

[0096] x(k+1)=f(x(k),u(k)) (1)

[0097] Where x(k+1) represents the dissolved oxygen concentration at time k+1 during the wastewater treatment process, x(k) represents the dissolved oxygen concentration at time k during the wastewater treatment process, u(k) represents the oxygen transfer coefficient at time k, and f(x(k),u(k)) represents the unknown nonlinear function at time k, that is, the relationship between the dissolved oxygen concentration at time k and x(k) under the action of u(k) and the dissolved oxygen concentration at time k+1.

[0098] The dissolved oxygen concentration after a false data injection attack on the urban wastewater treatment process control system is expressed as follows:

[0099]

[0100] in, x represents the dissolved oxygen concentration at time k after the injection of spurious data. a (k) represents the spurious data injected at time k;

[0101] Combining formulas (1) and (2), the dissolved oxygen concentration control model for the fifth partition under a false data injection attack is as follows:

[0102]

[0103] (2) Constructing the cost function of the error dynamic system

[0104] To achieve tracking and control of dissolved oxygen concentration, the dissolved oxygen concentration error e(k) at time k in the fifth partition is defined as:

[0105] e(k) = x r (k)-x(k) (4)

[0106] Where, x r (k) represents the set value of dissolved oxygen concentration at time k;

[0107] Design G(e(k),u(k)) as a utility function, expressed as follows:

[0108] G(e(k),u(k))=e 2 (k)α1+u 2 (k)α2+R(e(k),u(k)) (5)

[0109] Where α1, α2 < 10 -3 Let R(e(k),u(k)) be the weight parameters of the utility function, and let R(e(k),u(k)) be the reward function, used to balance error control, energy consumption control, and action smoothing control under false data intrusion, as follows:

[0110] R(e(k),u(k))=r1(k)+r2(k)+r3(k) (6)

[0111] Where r1(k) is the tracking error reward term at time k, designed as r1(k) = w1(k)e -3e(k) w1(k) represents the trainable weights at time k, and r2(k) is the energy penalty term at time k, designed as follows: w2(k) is the trainable weight at time k, and r3(k) is the action smoothing penalty term at time k. The design is r3(k) = w3(k)|u(k)-u(k-1)|, where w3(k) is the trainable weight at time k, with initial values ​​w1(0) = 3.0, w2(0) = 0.3, and w3(0) = 0.1.

[0112] Based on the utility function (5), the following cost function is designed:

[0113]

[0114] Where γ∈(0,1] is the discount factor;

[0115] (3) Design a security controller based on inverse reinforcement learning

[0116] ① Solving for the optimal control law based on iterative methods

[0117] To address the issue of spoofed data injection attacks during wastewater treatment, the control objective is to find a suitable and permissible optimal control law. * (k) minimizes the cost function (7): According to the Bellman optimality principle, the optimal cost function Q * (e(k),u(k)) satisfies the following discrete-time HJB equation

[0118]

[0119] The optimal control law can be solved using the following formula:

[0120]

[0121] Solving the optimal control law u in (9) using an iterative method * Approximate control law of (k) Given the admissible control law u0(k) at time k, calculate the precision δ and the maximum allowed number of iterations j. max In each iteration, j = 0, 1, 2, ... j max Update as follows

[0122] Q j (e(k),u j (k))=G(e(k),u j (k))+γQ j (e(k+1),u j (k+1)) (10)

[0123] Based on (10), the update control law is as follows:

[0124]

[0125] Where γ∈(0,1] is the discount factor, if ||Q j+1 (e(k),u j+1 (k))-Q j (e(k),u j (k))||<δ(δ=10 -3 (or the number of iterations reaches j) max If the value is 20, then the iteration process stops, and the optimal cost function is obtained. Approximate optimal control input function Otherwise, let j = j + 1 and continue with formula (10);

[0126] ② Constructing the evaluation network and execution network

[0127] First, a fully connected neural network is constructed, consisting of three layers: an input layer, hidden layers, and an output layer, represented as follows:

[0128]

[0129] Where the input of the j-th iteration at time k is y j (k), where z is the output of the j-th iteration at time k. j (k), W j (k) is the weight matrix from the input layer to the hidden layer at time k, in the j-th iteration, V j (k) is the weight matrix from the hidden layer to the output in the j-th iteration at time k, and T represents the transpose of the matrix. For the activation function, the tanh function can be selected.

[0130] The parameters of a fully connected neural network model are updated as follows:

[0131]

[0132] Among them, E j (k) represents the loss function of the neural network in the j-th iteration at time k. The sign for partial derivatives;

[0133] To achieve the solution of the control law in step ①, an evaluation network is first constructed based on a fully connected neural network (12) to approximate the cost function Q in equation (10). j (e(k),u j (k)), and then construct an execution network to approximate the control law u of equation (11). j+1 (k), described in detail below:

[0134] The network input is evaluated as y j (k)=[e(k),u j [k], the hidden layer consists of 15 neurons, and the output cost function Q is obtained by substituting into formula (12). j (e(k),u j The neural network approximation of (k))

[0135] To estimate the accuracy of the approximate cost function, the prediction error e in the j-th iteration at time k is defined. c(j) (k) is:

[0136]

[0137] Wherein G(e(k),u j (k)) is the utility function for the j-th iteration at time k, expressed as:

[0138]

[0139] Wherein, R(e(k),u j Let (k) be the reward function for the j-th iteration at time k, expressed as follows:

[0140] R(e(k),u j (k))=r1(k)+r 2j (k)+r 3j (k) (17)

[0141] Where r1(k)=w1(k)e -3|e(k) , r 3j (k)=w3(k)|u j (k)-u(k-1)|, w1(k), w2(k), w3(k) represent the weights at time k, and their update method is shown in step ③.

[0142] Construct the following loss function E c(j) (k):

[0143]

[0144] The loss function E j (k)=E c(j) (k) Substitute into formulas (13)-(14) to solve for the weight V of the (j+1)th iteration at time k. j+1 (k), W j+1 (k);

[0145] Based on the evaluation network, an approximate control law for the execution network is constructed according to (12), with the input being the control error y. j (k) = e(k), the hidden layer includes 12 neurons, substituting into formula (12) outputs u j+1 (k) neural network approximation

[0146] Evaluate network output As the loss function of the execution network, i.e. Substituting into formula (13)-(14), the weight V of the (j+1)th iteration at time k j+1 (k), W j+1 (k);

[0147] if Or the number of iterations reaches j max Then stop the iteration process. The approximate cost function obtained in the last iteration can be regarded as the optimal cost function. The approximate control law obtained in the last iteration is the optimal control input function. Otherwise, let j = j + 1 and continue with formula (10);

[0148] ③ Update the reward function based on inverse reinforcement learning method

[0149] Based on the inverse reinforcement learning method, the reward function R(e(k),u) in (16) is applied. j (k) is updated, and the process is as follows:

[0150] The reward function weight vector w(k) = [w1(k), w2(k), w3(k)] is updated based on the dissolved oxygen concentration error at time k. The loss function for the multi-step discounted dissolved oxygen concentration tracking error in the j-th iteration at time k is designed as follows:

[0151]

[0152] Where H = 3; discount factor γ irl ∈(0,1] is used to balance instantaneous error and long-term error;

[0153] Since the dissolved oxygen concentration error for the next H=3 steps cannot be directly obtained, the tracking error of the dissolved oxygen concentration for the next H=3 steps is predicted based on a fully connected neural network (12), with the hidden layer consisting of 15 neurons.

[0154] Let the model input be y(k) = [u(k-1), e(k)], where e(k) is the tracking error of the dissolved oxygen concentration predicted at time k, and the network output be... This represents the predicted tracking error of dissolved oxygen concentration at time k+1.

[0155] Let the model input be Network output is This represents the predicted tracking error of dissolved oxygen concentration at time k+2.

[0156] Let the model input be Network output is This represents the predicted tracking error of dissolved oxygen concentration at time k+3.

[0157] Calculate L at time k IRL The gradient of (w(k)) with respect to the weight w(k) is calculated using the following formula:

[0158]

[0159] Where g(k) = [g1(k), g2(k), g3(k)] is the gradient matrix at time k.

[0160] The first-order moment estimate m(k) at time k is expressed as:

[0161] m(k)=0.99m(k-1)+0.01g(k) (21)

[0162] Where m(k) = [m1(k), m2(k), m3(k)], m1(k), m2(k), m3(k) are the three elements of the first-order moment estimation matrix m(k), m(k-1) is the first-order moment estimation matrix at time k-1, and m(0) = [0,0,0].

[0163] The second-order moment estimation matrix v(k) at time k is represented as:

[0164] v(k) = 0.999v(k-1) + 0.001[g1] 2 (k),g2 2 (k),g3 2 (k)] (22)

[0165] Where, v(k) = [v1(k), v2(k), v3(k)], v1(k), v2(k), v3(k) are the three elements of the second-order moment estimation matrix v(k), v(k-1) is the second-order moment estimation matrix at time k-1, and v(0) = [0,0,0].

[0166] The updated weights can be represented as:

[0167]

[0168] Where w(k+1) is the weight matrix at time k+1, and ε is a constant to prevent the denominator from being zero, ε = 10. -8 ;

[0169] In summary, we obtain G(e(k),u j Substituting (k) into (15), we obtain the prediction error e for the j-th iteration at time k. c(j) (k);

[0170] (4) The iteration condition ||Q is satisfied in step ②. j+1 (e(k),u j+1 (k))-Q j (e(k),u j The control law obtained by (k))||<δ As the oxygen transfer coefficient of the urban sewage treatment system subjected to a false data injection attack at time k, the programmable logic controller controls the speed of the blower to control the aeration rate based on the calculated oxygen transfer coefficient, and finally achieves safe control of the urban sewage treatment process by regulating the dissolved oxygen concentration. Figure 1 This is a control effect diagram of the inverse reinforcement learning method under time-varying attack from the fifth to the sixth day. X-axis: time, unit is days; Y-axis: dissolved oxygen concentration value, unit is milligrams / liter. Figure 2 This is a control error graph under time-varying attack. X-axis: time, in days; Y-axis: dissolved oxygen concentration error value, in milligrams per liter. Figure 3This is a control effect graph of the inverse reinforcement learning method under constant attack from the fifth to the sixth day. X-axis: time, unit is days; Y-axis: dissolved oxygen concentration value, unit is milligrams / liter. Figure 4 This is a control error plot under constant attack, with the X-axis representing time in days and the Y-axis representing dissolved oxygen concentration error in milligrams per liter; the results demonstrate the effectiveness of the method.

Claims

1. A method for safe control of dissolved oxygen concentration in urban wastewater treatment processes based on inverse reinforcement learning, characterized in that: A dissolved oxygen concentration control system for an urban wastewater treatment process susceptible to spoofed data injection attacks is constructed. A cost function for the error dynamic system is built, and a safety controller based on inverse reinforcement learning is designed to achieve safe and stable control of dissolved oxygen concentration in the urban wastewater treatment process control system. The process includes the following steps: (1) Construct a dissolved oxygen concentration control system for urban wastewater treatment processes that is vulnerable to false data injection attacks. The kinetic model of the dissolved oxygen concentration control system in the urban wastewater treatment process is as follows: x(k+1)=f(x(k),u(k)) (1) Where x(k+1) represents the dissolved oxygen concentration at time k+1 during the wastewater treatment process, x(k) represents the dissolved oxygen concentration at time k during the wastewater treatment process, u(k) represents the oxygen transfer coefficient at time k, and f(x(k),u(k)) represents the unknown nonlinear function at time k, that is, the relationship between the dissolved oxygen concentration at time k and x(k) under the action of u(k) and the dissolved oxygen concentration at time k+1. The dissolved oxygen concentration after a false data injection attack on the urban wastewater treatment process control system is expressed as follows: in, x represents the dissolved oxygen concentration at time k after the injection of spurious data. a (k) represents the spurious data injected at time k; Combining formulas (1) and (2), the dissolved oxygen concentration control model for the fifth partition under a false data injection attack is as follows: (2) Constructing the cost function of the error dynamic system To achieve tracking and control of dissolved oxygen concentration, the dissolved oxygen concentration error e(k) at time k in the fifth partition is defined as: e(k)=x r (k)-x(k) (4) Where, x r (k) represents the set value of dissolved oxygen concentration at time k; Design G(e(k),u(k)) as a utility function, expressed as follows: G(e(k),u(k))=e 2 (k)α1+u 2 (k)α2+R(e(k),u(k)) (5) Where α1, α2 < 10 -3 Let R(e(k),u(k)) be the weight parameters of the utility function, and let R(e(k),u(k)) be the reward function, used to balance error control, energy consumption control, and action smoothing control under false data intrusion, as follows: R(e(k),u(k))=r1(k)+r2(k)+r3(k) (6) Where r1(k) is the tracking error reward term at time k, designed as r1(k) = w1(k)e -3e(k) w1(k) represents the trainable weights at time k, and r2(k) is the energy penalty term at time k, designed as follows: w2(k) is the trainable weight at time k, and r3(k) is the action smoothing penalty term at time k. The design is r3(k) = w3(k)|u(k)-u(k-1)|, where w3(k) is the trainable weight at time k, with initial values ​​w1(0) = 3.0, w2(0) = 0.3, and w3(0) = 0.

1. Based on the utility function (5), the following cost function is designed: Where γ∈(0,1] is the discount factor; (3) Design a security controller based on inverse reinforcement learning ① Solving for the optimal control law based on iterative methods To address the issue of spoofed data injection attacks during wastewater treatment, the control objective is to find a suitable and permissible optimal control law. * (k) minimizes the cost function (7): According to the Bellman optimality principle, the optimal cost function Q * (e(k),u(k)) satisfies the following discrete-time HJB equation The optimal control law can be solved using the following formula: Solving the optimal control law u in (9) using an iterative method * Approximate control law of (k) Given the admissible control law u0(k) at time k, calculate the precision δ and the maximum allowed number of iterations j. max In each iteration, j = 0, 1, 2, ... j max Update as follows Q j (e(k),u j (k))=G(e(k),u j (k))+γQ j (e(k+1),u j (k+1)) (10) Based on (10), the update control law is as follows: Where γ∈(0,1] is the discount factor, if ||Q j+1 (e(k),u j+1 (k))-Q j (e(k),u j (k))||<δ(δ=10 -3 (or the number of iterations reaches j) max If the value is 20, then the iteration process stops, and the optimal cost function is obtained. Approximate optimal control input function Otherwise, let j = j + 1 and continue with formula (10); ② Constructing the evaluation network and execution network First, a fully connected neural network is constructed, consisting of three layers: an input layer, hidden layers, and an output layer, represented as follows: Where the input of the j-th iteration at time k is y j (k), where z is the output of the j-th iteration at time k. j (k), W j (k) is the weight matrix from the input layer to the hidden layer at time k, in the j-th iteration, V j (k) is the weight matrix from the hidden layer to the output in the j-th iteration at time k, and T represents the transpose of the matrix. For the activation function, the tanh function can be selected. The parameters of a fully connected neural network model are updated as follows: Among them, E j (k) represents the loss function of the neural network in the j-th iteration at time k. The sign for partial derivatives; To achieve the solution of the control law in step ①, an evaluation network is first constructed based on a fully connected neural network (12) to approximate the cost function Q in equation (10). j (e(k),u j (k)), and then construct an execution network to approximate the control law u of equation (11). j+1 (k), described in detail below: The network input is evaluated as y j (k)=[e(k),u j [k], the hidden layer consists of 15 neurons, and the output cost function Q is obtained by substituting into formula (12). j (e(k),u j The neural network approximation of (k)) To estimate the accuracy of the approximate cost function, the prediction error e in the j-th iteration at time k is defined. c(j) (k) is: Wherein G(e(k),u j (k)) is the utility function for the j-th iteration at time k, expressed as: Wherein, R(e(k),u j Let (k) be the reward function for the j-th iteration at time k, expressed as follows: R(e(k),u j (k))=r1(k)+r 2j (k)+r 3j (k) (17) Where r1(k)=w1(k)e -3|e(k)| , r 3j (k)=w3(k)|u j (k)-u(k-1)|, w1(k), w2(k), w3(k) represent the weights at time k, and their update method is shown in step ③. Construct the following loss function E c(j) (k): The loss function E j (k)=E c(j) (k) Substitute into formulas (13)-(14) to solve for the weight V of the (j+1)th iteration at time k. j+1 (k), W j+1 (k); Based on the evaluation network, an approximate control law for the execution network is constructed according to (12), with the input being the control error y. j (k) = e(k), the hidden layer includes 12 neurons, substituting into formula (12) outputs u j+1 (k) neural network approximation Evaluate network output As the loss function of the execution network, i.e. Substituting into formula (13)-(14), the weight V of the (j+1)th iteration at time k j+1 (k), W j+1 (k); if Or the number of iterations reaches j max Then stop the iteration process. The approximate cost function obtained in the last iteration can be regarded as the optimal cost function. The approximate control law obtained in the last iteration is the optimal control input function. Otherwise, let j = j + 1 and continue with formula (10); ③ Update the reward function based on inverse reinforcement learning method Based on the inverse reinforcement learning method, the reward function R(e(k),u) in (16) is applied. j (k) is updated, and the process is as follows: The reward function weight vector w(k) = [w1(k), w2(k), w3(k)] is updated based on the dissolved oxygen concentration error at time k. The loss function for the multi-step discounted dissolved oxygen concentration tracking error in the j-th iteration at time k is designed as follows: Where H = 3; discount factor γ irl ∈(0,1] is used to balance instantaneous error and long-term error; Since the dissolved oxygen concentration error for the next H=3 steps cannot be directly obtained, the tracking error of the dissolved oxygen concentration for the next H=3 steps is predicted based on a fully connected neural network (12), with the hidden layer consisting of 15 neurons. Let the model input be y(k) = [u(k-1), e(k)], where e(k) is the tracking error of the dissolved oxygen concentration predicted at time k, and the network output be... This represents the predicted tracking error of dissolved oxygen concentration at time k+1. Let the model input be Network output is This represents the predicted tracking error of dissolved oxygen concentration at time k+2. Let the model input be Network output is This represents the predicted tracking error of dissolved oxygen concentration at time k+3. Calculate L at time k IRL The gradient of (w(k)) with respect to the weight w(k) is calculated using the following formula: Where g(k) = [g1(k), g2(k), g3(k)] is the gradient matrix at time k. The first-order moment estimate m(k) at time k is expressed as: m(k)=0.99m(k-1)+0.01g(k) (21) Where m(k) = [m1(k), m2(k), m3(k)], m1(k), m2(k), m3(k) are the three elements of the first-order moment estimation matrix m(k), m(k-1) is the first-order moment estimation matrix at time k-1, and m(0) = [0,0,0]. The second-order moment estimation matrix v(k) at time k is represented as: v(k)=0.999v(k-1)+0.001[g1 2 (k),g2 2 (k),g3 2 (k)] (22) Where, v(k) = [v1(k), v2(k), v3(k)], v1(k), v2(k), v3(k) are the three elements of the second-order moment estimation matrix v(k), v(k-1) is the second-order moment estimation matrix at time k-1, and v(0) = [0,0,0]. The updated weights can be represented as: Where w(k+1) is the weight matrix at time k+1, and ε is a constant to prevent the denominator from being zero, ε = 10. -8 ; In summary, we obtain G(e(k),u j Substituting (k) into (15), we obtain the prediction error e for the j-th iteration at time k. c(j) (k); (4) The iteration condition ||Q is satisfied in step ②. j+1 (e(k),u j+1 (k))-Q j (e(k),u j The control law obtained by (k))||<δ As the oxygen transfer coefficient of the urban sewage treatment system subjected to a false data injection attack at time k, the programmable logic controller controls the speed of the blower to control the aeration rate based on the calculated oxygen transfer coefficient, and finally achieves safe control of the urban sewage treatment process by regulating the dissolved oxygen concentration.

Images