An event-triggered model-free optimal control method for steam-water heat exchanger system based on Q-learning

Abstract

The present invention discloses an event-triggered model-free optimization control method for a steam-water heat exchanger system based on Q learning. The method first obtains a compact data model of the steam-water heat exchanger system, introduces a Q function indicator based on the model to evaluate the system performance; constructs a dynamically updated penalty factor embedded in the Q function indicator to obtain the optimal controller; minimizes the Q function indicator to obtain the optimal controller to ensure the final uniform boundedness of the closed-loop error system; uses the Q function value and tracking error to propose a trigger mechanism that can balance performance and network resource usage. Compared with the control method based on time-triggered data drive, this method is a new triggering method. When applied to a network control system, this method reduces the communication transmission frequency, effectively saves network resources, and has certain engineering application value.

Description

Technical Field

[0001] The present invention relates to a control method for a steam-water heat exchanger system, and in particular to an event-triggered model-free optimization control method for a steam-water heat exchanger system based on Q learning. Background Art

[0002] Steam-water heat exchanger systems are widely used in nuclear power plants, thermal power plants, chemical industry and other important industries. However, due to the inherent unmodeled dynamic characteristics, it is difficult or even impossible to obtain an accurate system model. How to design a controller using only input and output data is a very meaningful work. Based on this idea, researchers have carried out a lot of research and achieved a series of results.

[0003] However, the above research works mainly focus on the periodic data transmission situation, and do not consider the negative problems caused by network control. In order to deal with these problems, it is necessary to study event-triggered control methods. Summary of the invention

[0004] The purpose of the present invention is to propose an event-triggered model-free optimization control method for a steam-water heat exchanger system based on Q learning, which can effectively solve the problem of limited bandwidth of a steam-water heat exchanger system with an unknown model in a network control environment.

[0005] The specific technical solution of the present invention is as follows: A Q-learning-based event-triggered model-free optimization control method for a steam-water heat exchanger system comprises the following steps:

[0006] Introduce the Q function indicator to evaluate system performance;

[0007] Construct a dynamically updated penalty factor embedded in the Q function indicator to obtain the optimal controller;

[0008] Minimize the Q function index to obtain the optimal controller to ensure the final uniform boundedness of the closed-loop error system

[0009] Using the Q-function value and tracking error, a triggering mechanism is proposed to balance performance and network resource usage.

[0010] Further, the discrete mathematical expression of the steam-water heat exchanger system is as follows: y(t+1)=f(y(t),y(t-1),…,y(tc y ),u(t),u(t-1),…,u(tc u ))

[0011] Where y(t)∈R represents the output, u(t)∈R represents the input, t∈{0, 1, ...T} represents the sampling time, f(·) is an unknown nonlinear function, cy and c u is an unknown positive integer.

[0012] Furthermore, using the Q-function value and tracking error, a trigger mechanism that can balance performance and network resource usage is proposed. The specific steps include: obtaining a compact data model of the steam-water heat exchanger system, introducing a Q-function indicator based on the model to evaluate the system performance; constructing a dynamically updated penalty factor embedded in the Q-function indicator to obtain the optimal controller; minimizing the Q-function indicator to obtain the optimal controller to ensure the eventual consistent boundedness of the closed-loop error system; using the Q-function value and tracking error, a trigger mechanism that can balance performance and network resource usage is proposed.

[0013] Furthermore, using the Q function value and tracking error, a trigger mechanism that can balance performance and network resource usage is proposed. The specific steps include: obtaining a compact data model of the steam-water heat exchanger system, introducing a Q function indicator based on the model to evaluate system performance; constructing a dynamically updated penalty factor embedded in the Q function indicator to obtain the optimal controller; minimizing the Q function indicator to obtain the optimal controller to ensure that the closed-loop error system is eventually uniformly bounded; using the Q function value and tracking error, a trigger mechanism that can balance performance and network resource usage is proposed, specifically including:

[0014] Using compact dynamic linearization technology, the compact data model of the steam-water heat exchanger system is obtained as follows: y(t+1)=y(t)+Φ(t)Δu(t)

[0015] Where Φ(t)∈R is the unknown pseudo-partial derivative satisfying Δy(t+1)=y(t+1)-y(t), Δu(t)=u(t)-u(t-1);

[0016] Define an event trigger sequence {t l}, l = 1, 2, ... N;

[0017] The Q function is introduced to evaluate the system performance as follows: Q[e(t)]=J[e(t)]+γ(t)Q[e(t+1)]

[0018] Solve the equation The optimal control rate u(t) under event-triggered environment is obtained as follows:

[0019] Where, 0<ρ<1, λ>0, e(t)=y d (t)-y(t), y d (t) represents the expected output, γ(t) and γ(t l ) is the penalty factor to be designed;

[0020] Design the dynamic update rate of the penalty factor as follows:

[0021] In the formula, e l (t) = y d (t)-y l (t);

[0022] Modify the existing Φ(t) estimation algorithm so that it is updated under the triggering environment as follows:

[0023] In the formula, 0<η<2, μ>0, the algorithm can ensure The estimation error is bounded, and the proof process is as follows:

[0024] Since during the trigger interval, So we only need to prove It is bounded at the moment of triggering;

[0025] Define the estimation error of Φ(t) Subtract Φ(t) from both sides of the above equation to obtain the following equation:

[0026] Since |Φ(t)-Φ(t+1)|≤2b, scaling

[0025] gives the following formula:

[0027] Since 0<η<2, μ>0, there exists d1 such that the following equation holds:

[0028] Substituting

[0027] into

[0026] , we obtain the following formula:

[0029] The above formula can be further recursively derived into:

[0030] As time goes on, that is, t→∞, the above formula shows that Converge to End of proof;

[0031] Use neural network to approximate J[e(k)] and Q[e(t)], and the loss function Applying the gradient descent algorithm, we get the update rate of the neural network weights as follows:

[0032] Where, 0<α<1, At this weight update rate, the approximation errors of J[e(k)] and Q[e(t)] are ultimately uniformly bounded. The proof process is as follows:

[0033] because and The form is the same, so we only prove The approximation error of is bounded;

[0034] During the trigger interval, the output at the last trigger time is used to update So we only need to prove that at the triggering moment The approximation error is bounded;

[0035] make Substituting the weight update rate into it, we get the following formula:

[0036] In the formula,

[0037] By transforming the equation, we can get the following formula:

[0038] Substituting

[0036] into

[0035] , we obtain the following formula:

[0039] Select the following Lyapunov function:

[0040] Performing a differential operation on

[0039] , we obtain the following formula:

[0041] Substituting

[0038] into

[0040] , we obtain the following formula:

[0042] Because the system output is continuously excited, the following formula can be obtained:

[0043] Substituting

[0042] into

[0041] , we obtain the following formula:

[0044] Select 0<α<1, and get the following formula:

[0045] Substituting

[0042] into

[0044] , we obtain the following formula:

[0045] show Converge to this boundary Inside, the proof ends; Using the Q function value and tracking error, a trigger mechanism is proposed to balance performance and network resource usage as follows: Where χ>0. Under this trigger mechanism, the designed controller can ensure that the tracking error is bounded. The proof process is as follows: The proof is divided into two steps. First, it is proved that the output tracking error of the steam-water heat exchanger closed-loop system is bounded at the triggering moment, and then it is proved that the output tracking error of the system is also bounded during the triggering interval. At the triggering moment, t = t l , we get the following formula: e(t+1)=y d (t+1)-y(t+1) =y d (t+1)-y(t)-Φ(t)Δu(t) Substituting the control rate at the trigger time into

[0050] , we get the following formula: The equation and Substituting into

[0051] , we get the following formula: Performing absolute value operation on

[0052] , we get the following formula: Since -1<tanh(·)<1, we get the following formula: Substituting γ(t) and

[0054] into

[0053] , we obtain the following equation: Since Y is a constant matrix, we have Also because and The following formula can be obtained: Depending on the range of parameters ρ, λ and τ, there exist positive constants d2 and d3 such that: Substituting

[0057] into

[0056] , we obtain the following formula: |e(t+1)|≤(d2+d3)|e(t)|+d2|Δy d (t+1)| Δy d (t+1) represents the rate of change of the input, assuming Then the following formula holds: Because (d2+d3)<0.2<1, e(t) converges to this boundary at the triggering time. Inside; In the trigger interval, t l <t<t l+1 , since the optimal Q[e(t)] is non-negative, use Q * [e(t)] means, choose Q * [e(t)] is the Lyapunov function, which has: γ(t)Q * [e(t+1)]-γ(t-1)Q * [e(t)] = Q * [e(t)]-J * [e(t)]-γ(t-1)Q * [e(t)] =(1-γ(t-1))Q * [e(t)]-J * [e(t)] ≤Q * [e(t)]-J * [e(t)] because Where ∈1(t) and ∈2(t) represent J * [e(t)] and Q * The approximation error of [e(t)]; Substituting

[0063] into

[0062] yields Substituting the trigger mechanism into

[0064] , we obtain the following equation: Where ∈ represents the upper bound of |∈2(t)-∈1(t)|; According to

[0066] , e(t) is bounded; Unifying the proofs of the above two cases, we can obtain that e(t) is eventually uniformly bounded, and the proof ends. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 is a flow chart of a method according to an embodiment of the present invention; Figure 2 Output tracking diagram of the method proposed by the present invention is adopted in the embodiment; Figure 3 This is a diagram of the trigger interval of relaxation using the method proposed by the present invention in an embodiment; DETAILED DESCRIPTION The present invention is further explained below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application. like Figure 1 As shown, a Q-learning-based event-triggered model-free optimization control method for a steam-water heat exchanger system comprises the following steps: Step 1: Set the initial value of the parameter; Step 2: Update pseudo partial derivatives Step 3: Approximate indicator function and Step 4: Use the updated and Information, update control input u(t); Step 5: The control input u(t) is passed into the steam-water heat exchanger system to generate the actual output y(t+1), which is then passed into the trigger mechanism; Step 6: Determine whether the trigger condition is met. If so, the actual output y(t+1) of the system is transmitted to the control terminal through the network. Otherwise, it is not transmitted. Step 7: Repeat step 2 to enter the next cycle. An embodiment of the present invention is described below: Consider the output tracking control problem of the steam-water heat exchanger system, which corresponds to the Hammerstein space model: The expected tracking trajectory is: Figure 1 is a flow chart of a method according to an embodiment of the present invention; applying the proposed method, the output tracking curve is as follows Figure 2As shown, it can be seen that the output can track the expected trajectory; Figure 3 The trigger interval diagram showing the relaxation of the proposed event triggering method. From these three figures, it can be seen that the proposed method not only effectively reduces the communication frequency, but also obtains a satisfactory tracking performance. References [1] N. Lin, R. Chi, B. Huang. Event-triggered model-free adaptive control. IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol.51, pp.3358-3369, 2019. [2]Z.Hou, S.Jin.Data-driven model-free adaptive control for a class ofMIMO nonlinear discrete-time systems.IEEE Transactions on Neural Networks, vol.22, pp.2173-2188, 2011.

Claims

1. A Q-learning-based event-triggered model-free optimization control method for a steam-water heat exchanger system, characterized in that: The following steps are involved: Introduce the Q function indicator to evaluate system performance; Construct a dynamically updated penalty factor embedded in the Q function indicator to obtain the optimal controller; Minimize the Q function index to obtain the optimal controller to ensure the final uniform boundedness of the closed-loop error system; Using the Q function value and tracking error, a trigger mechanism is proposed to balance performance and network resource usage, including: (1) Using the compact dynamic linearization technology, the compact data model of the steam-water heat exchanger system is obtained as follows: y(t+1)=y(t)+Φ(t)Δu(t) Where y(t)∈R is the system output, Φ(t)∈R is the unknown pseudo partial derivative satisfying |Φ(t)|≤b, u(t)∈R is the system input, Δu(t)=u(t)-u(t-1); (2) Define an event trigger sequence {t l }, l = 1, 2, ..., N; (3) Introduce the Q function to evaluate the system performance, as shown below: Q[e(t)]=J[e(t)]+γ(t)Q[e(t+1)] Where λ>0, e(t)=y d (t)-y(t),y d (t) represents the expected output, γ(t) is the penalty factor to be designed; Solve the equation The optimal control rate u(t) under event-triggered environment is obtained as follows: Where, 0<ρ<1, γ(t l ) is the penalty factor to be designed; (4) Design the dynamic update rate of the penalty factor as follows: In the formula, e l (t)=y d (t)-y(t l ) ; (5) Modify the existing Φ(t) estimation algorithm so that it can be updated under the triggering environment as follows: In the formula, 0<η<2, μ>0, the algorithm can ensure The estimation error is bounded, and the proof process is as follows: A001: Due to the trigger interval, So we only need to prove It is bounded at the moment of triggering; A002: Define the estimation error of Φ(t) Subtract Φ(t) from both sides of the above equation to obtain the following equation: A003: Since |Φ(t)-Φ(t+1)|≤2b, scaling A002 gives the following formula: A004: Because 0<η<2,μ>0, there exists d1 such that the following equation holds: A005: Substitute A004 into A003 to obtain the following formula: A006: The above formula can be further recursively derived into: A007: As time goes on, i.e., t→∞, the above formula shows that Converge to End of proof; (6) Use neural network to approximate J[e(t)] and Q[e(t)], and the loss function and Applying the gradient descent algorithm, we get the update rate of the neural network weights as follows: Where, 0<α<1, At this weight update rate, the approximate errors of J[e(t)] and Q[e(t)] are uniformly bounded, as shown in the following proof: A008: Due to and The form is the same, so we only prove The approximation error of is bounded; A009: During the trigger interval, the output at the last trigger time is used to update So we only need to prove that at the trigger time The approximation error is bounded; A010: Order Substituting the weight update rate into it, we get the following formula: In the formula, A011: By transforming the equation, we can get the following equation: A012: Substitute A011 into A010 to obtain the following formula: A013: Select the following Lyapunov function: A014: Perform a differential operation on A013 and obtain the following formula: A015: Substitute A012 into A014 to obtain the following formula A016: Because the system output is continuously excited, the following formula can be obtained: A017: Substitute A016 into A015 to obtain the following formula: A018: Select 0<α<1, and get the following formula: A019: Substitute A016 into A018 to obtain the following formula: A020:A019 indicates Converge to this boundary Inside, the proof ends; (7) Using the Q function value and tracking error, a trigger mechanism is proposed to balance performance and network resource usage, as shown below: Where χ>0. Under this trigger mechanism, the designed controller can ensure that the tracking error is bounded. The proof process is as follows: A021: The proof is divided into two steps. First, it is proved that the output tracking error of the steam-water heat exchanger closed-loop system is bounded at the trigger time, and then it is proved that the output tracking error of the system is also bounded during the trigger interval: A022: At the triggering moment, i.e., t = t l , we get the following formula: e(t+1)=y d (t+1)-y(t+1) =y d (t+1)-y(t)-Φ(t)Δu(t) A023: Substitute the control rate at the triggering moment into A022 to obtain the following formula; A024: Transform the equation Substituting into A023, we get the following formula: A025: Perform absolute value operation on A024 and obtain the following formula: A026: Since -1<tanh(·)<1, we get the following formula: A027: Substituting γ(t) and A026 into A025, we obtain the following equation: A028: Since Y is a constant matrix, we have Also because and The following formula can be obtained: A029: According to the selection range of parameters ρ, λ, τ, there exist positive constants d2 and d3 such that the following equation holds: A030: Substitute A029 into A028 to obtain the following formula: |e(t+1)|≤(d2+d3)|e(t)|+d2|Δy d (t+1)| A031: |Δy d (t+1)| represents the rate of change of the input, assuming Then the following formula holds: A032: Because (d2+d3)<0.2<1, |e(t)| converges to this boundary at the triggering time. Inside; A033: In the trigger interval, t l <t<t l+1 , since the optimal Q[e(t)] is non-negative, use Q * [e(t)] means, choose Q * [e(t)] is the Lyapunov function, which has: γ(t)Q * [e(t+1)]-γ(t-1)Q * [e(t)]=Q * [e(t)]-J * [e(t)]-γ(t-1)Q * [e(t)] =(1-γ(t-1))Q * [e(t)]-J * [e(t)] ≤Q * [e(t)]-J * [e(t)] A034: Because Where ε1(t) and ε2(t) represent J * [e(t)] and Q * The approximation error of [e(t)]; A035: Substitute A034 into A033 to obtain: A036: Substitute the trigger mechanism into A035 to obtain the following formula: Where ε represents the upper bound of |ε2(t)-ε1(t)|; A037: According to A036, e(t) is bounded; A038: Unifying the proofs of the above two cases, we can obtain that e(t) is ultimately uniformly bounded, and the proof ends.

Images