Tracking control method based on measurement data under denial of service attack

Abstract

The invention discloses a tracking and control method based on measurement data under denial of service attack, which relates to the field of information-physical system. The invention solves the problem that the existing tracking and control method of information-physical system under denial of service attack needs to rely on system dynamics during the tracking process, but the system dynamics are difficult to obtain and it is not convenient to effectively track the reference system dynamics. The invention uses input and output data to augment the system state ξ k and the quadratic valued function V(x k ,r k ) is reconstructed, and on this basis, the Bellman equation based on the measurement data is established to obtain the safety tracking control strategy u based on the input and output data k The control strategy u is solved by online iterative reinforcement method based on measurement data. k The optimal intermediate variable kernel matrix #imgabs0# in the control strategy u k Optimal, achieving dynamic r of the reference trajectory k The present invention is mainly used for tracking with reference to trajectory dynamics.

Description

Technical Field

[0001] The present invention relates to the field of information-physical systems, and in particular to implementing secure tracking and control of information-physical systems based on input and output data under denial of service attacks. Background Art

[0002] Cyber-physical systems integrate computing resources, communication networks, and physical systems, and are a new generation of intelligent control systems. This system has been applied in various fields, such as smart grids, smart transportation, power systems, etc. Cyber-physical systems involve interactions between the network layer and the physical layer, which makes the system very vulnerable to attacks by malicious adversaries. By executing malicious attacks, the performance of cyber-physical systems may be degraded or even destroyed, or the adversary may illegally benefit from the malicious attack. There have been many cases of attacks in recent years, such as multiple power outages in Brazil, the Stuxnet computer worm attack, the attack on German steel plants, and so on. Therefore, it is of great significance to propose a security control algorithm for cyber-physical systems against malicious attacks. The block diagram of the existing cyber-physical system under denial of service attack, see for details. Figure 1 .

[0003] Most of the existing research results on the security control of cyber-physical systems under malicious attacks need to rely on system dynamics k and system model to realize the reference trajectory state r k Tracking, thereby designing a tracking controller. However, in practice, it is usually difficult to obtain an accurate mathematical model of the system to design a safe controller. Reinforcement learning / adaptive dynamic programming technology can complete the control task without the need for accurate knowledge of the system dynamics, and can ensure the stability of the system. Therefore, this method is introduced to study the safety control problem of information-physical systems. Although some existing research results have used reinforcement learning / adaptive dynamic programming methods for the safety control of information-physical systems and do not require full knowledge of the system dynamics, these research results require the use of all state information of the controlled system for state feedback. From the perspective of control system engineering, this requirement is difficult to meet because all state information may not be available or difficult to obtain. That is: whether it is an online reinforcement learning algorithm or an offline reinforcement learning algorithm, it is necessary to rely on accurate controlled system dynamics or controlled system states to realize the design of the controller. The unmeasurable dynamics of the controlled system and the difficulty in modeling the controlled system model all lead to technical defects that the tracking controller is difficult to generate control quantities. Therefore, the above problems need to be solved urgently. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the existing tracking and control methods of information-physical systems under denial of service attacks rely on system dynamics during the tracking process, and the system dynamics are difficult to obtain, making it inconvenient to effectively track the reference system dynamics; the present invention provides a tracking and control method based on measurement data under denial of service attacks.

[0005] A tracking control method based on measurement data under a denial of service attack includes the following steps:

[0006] S1, according to the reference trajectory dynamic r of the controlled system input, the controlled system output and the reference trajectory system output at time k in the time interval [kM, k] k , reconstruct the augmented system state ξ at time k under a denial of service attack k ; k and M are both integers;

[0007] S2, according to the augmented system state ξ k , reconstruct the value function V(x k ,r k );

[0008] S3, according to the value function V(x k ,r k ), obtain the corresponding initial Bellman equation, and then combine the initial Bellman equation and the augmented system state ξ k , obtain the corresponding Bellman equation based on the measured data;

[0009] S4, according to the augmented system state ξ k and the Bellman equation based on the measured data to solve the control strategy u of the tracking controller k The composition of; Among them, the control strategy u k As the input of the controlled system at time k;

[0010] S5, online iterative reinforcement learning method based on measurement data, control strategy u k The kernel matrix of the intermediate variables in Iterate to get the optimal intermediate variable kernel matrix In this way, the optimal control strategy is obtained; the optimal control strategy is used to realize the dynamic r of the reference trajectory k tracking.

[0011] Preferably, in S1, the reference trajectory dynamic r is calculated based on the input of the controlled system, the output of the controlled system and the output of the reference trajectory system at time k in the time interval [kM, k]. k , reconstruct the augmented system state ξ at time k under a denial of service attack k The implementation methods include:

[0012]

[0013] in,

[0014]

[0015]

[0016] is the intermediate variable matrix, is the vector composed of the input signal of the controlled system, the output signal of the controlled system and the dynamics of the reference trajectory in the time interval [kM, k-1], x k is the state of the controlled system at the kth moment, δ k is the attack rate at the kth moment, Γ M is the controllability matrix, A is the constant matrix of the first intermediate variable of the controlled system, T M is the observable matrix, is the observable matrix T M The pseudo-inverse of E M is the matrix variable of the intermediate calculation, G is the system matrix of the reference trajectory system, is the matrix of the input signal of the controlled system in the time interval [kM, k-1], u k-i is the input signal of the controlled system at time ki, i=1,2……M, is the matrix of the output signal of the controlled system in the time interval [kM, k-1], y k-i is the output signal of the controlled system at time ki, r k-M is the reference trajectory dynamics at the kMth moment.

[0017] Priority is given to the augmented system state ξ in S2. k , reconstruct the value function V(x k ,r k ) include:

[0018]

[0019] in, P is a symmetric matrix, and P>0; To achieve expectations.

[0020] Preferably, in S3, according to the value function V(x k ,r k ), obtain the corresponding initial Bellman equation, and then combine the initial Bellman equation and the augmented system state ξ k , the corresponding Bellman equation based on measured data is obtained as follows:

[0021] S31. According to the value function The corresponding initial Bellman equation is obtained, specifically:

[0022]

[0023] Where, Q1 = [C -I] T Q[C -I],

[0024] Q is the first weight matrix, Q ≥ 0, C is the output matrix of the controlled system, I is the unit matrix, Q1 is the intermediate variable matrix, and δ is δ k =1, is the probability, SER is the symbol error rate, R is the second weight matrix, R>0, λ is the discount factor, Take expectation, P is a symmetric matrix, P>0;

[0025] S32, substituting the expression of Formula 1 into Formula 3 to obtain the Bellman equation based on the measured data, specifically:

[0026]

[0027] in, is a vector consisting of the controlled system input, the controlled system output and the reference trajectory dynamics in the time interval [kM,k].

[0028] Preferably, in S4, according to the augmented system state ξ k and the Bellman equation based on the measured data to solve the control strategy u of the tracking controller k The implementation of the composition is:

[0029] S41. Transpose the Bellman equation based on the measured data to obtain the differential error

[0030]

[0031] S42, the differential error In This breaks down into:

[0032]

[0033] in, is the matrix of the input signal of the controlled system in the time interval [k-M+1,k-1], is the matrix of the output signal of the controlled system in the time interval [k-M+1,k], r k-M+1 is the reference trajectory dynamics at the k-M+1th moment; is the intermediate variable kernel matrix The value of the first row and first column element in , is the intermediate variable kernel matrix The value of the first row and second column element in is the intermediate variable kernel matrix The value of the first row and third column element in is the intermediate variable kernel matrix The value of the first row and fourth column of is the intermediate variable kernel matrix The value of the element in the second row and first column of is the intermediate variable kernel matrix The value of the element in the third row and first column of Intermediate variable kernel matrix The value of the element in the fourth row and first column of is the intermediate variable kernel matrix The second row and second column element in is the intermediate variable kernel matrix The second row and third column element in is the intermediate variable kernel matrix The second row and fourth column element in is the intermediate variable kernel matrix The third row and second column element in is the intermediate variable kernel matrix The third row and third column element in is the intermediate variable kernel matrix The third row and fourth column element in is the intermediate variable kernel matrix The fourth row and second column element in is the intermediate variable kernel matrix The fourth row and third column element in is the intermediate variable kernel matrix The fourth row and fourth column element in ;

[0034] S43, combine formula 6 and formula 4, and calculate u k Taking the derivative and setting the derivative to 0, we get u k The expression is:

[0035]

[0036] in, K is the gain of the tracking controller.

[0037] Preferably, in S5, the online iterative reinforcement learning method based on the measurement data is implemented by using a policy iterative reinforcement learning algorithm, and the policy iterative reinforcement learning algorithm is used to obtain the optimal intermediate variable kernel matrix The optimal control strategy is achieved as follows:

[0038] S5-11, let the initial value of the iteration number j be 0, and give a preset constant value ε;

[0039] S5-12. Given an initial stable control strategy

[0040] S5-13, Strategy evaluation: Calculate the j+1th iteration using formula 8 The column matrix Specifically,

[0041]

[0042] Where, Q1 = [C -I] T Q[C -I],

[0043] is the Kronecker product, Q1 is the intermediate variable matrix, Q is the first weight matrix, C is the output constant matrix of the controlled system, I is the unit matrix, and δ is δ k =1, is the probability, SER is the symbol error rate, R is the second weight matrix, λ is the discount factor, Take expectations, is the control strategy for the jth iteration, is the intermediate variable kernel matrix for the jth iteration The column matrix of

[0044] S5-14, Yes Perform matrix transformation to obtain the intermediate variable kernel matrix at the j+1th iteration

[0045] S5-15, Strategy Improvement: Passed Calculate the control strategy for the j+1th iteration Specifically:

[0046]

[0047] S5-16, Judgment Is it true? ||·|| is the matrix norm. If the result is yes, then output and control strategies and will As the optimal intermediate variable kernel matrix The control strategy As the optimal control strategy; the result is no, j=j+1, and step S5-13 is executed.

[0048] Principle analysis: Dynamic reference trajectory in practical applications kDuring the tracking process, the system state is not always available. To solve this problem, the present invention proposes a tracking control method based on measurement data under a denial of service attack. The present invention uses input and output data to augment the system state ξ k and the quadratic valued function V(x k ,r k ) is reconstructed. On this basis, the Bellman equation based on input and output data (that is, the Bellman equation based on measurement data) is established to obtain the safety tracking control strategy u based on input and output data. k The composition of the control strategy u is constructed to construct a security tracking strategy iteration and value iteration reinforcement learning algorithm based on input and output data under denial of service attacks, and solve the control strategy u k The optimal intermediate variable kernel matrix in So that the control strategy u k Best.

[0049] This invention deeply analyzes how to establish a reinforcement learning security control algorithm based on the system state, then studies the relationship between the system state and the input and output data, and then studies how to design a reinforcement learning security control algorithm based on the input and output data under malicious attacks. The Bellman equation is the Bellman equation.

[0050] The beneficial effects brought by the present invention are:

[0051] The present invention does not need to rely on the system state, but only needs to know the reference trajectory dynamics r of the controlled system input, the controlled system output and the reference trajectory system output. k , reconstructing the augmented system state under denial of service attackξ k , augmented system state ξ k The construction does not depend on the system state, which overcomes the difficulty in obtaining the system state in the prior art and is not convenient for dynamic reference trajectory r k The problem of effective tracking.

[0052] In order to achieve accurate tracking, the present invention proposes a reinforcement learning algorithm that only requires system input and output data for security control of information-physical systems under malicious attacks. The present invention uses input and output data to augment the system state ξ k and the quadratic valued function V(x k ,r k ) is reconstructed. On this basis, the Bellman equation based on input and output data (that is, the Bellman equation based on measurement data) is established to obtain the safety tracking control strategy u based on input and output data. k The control strategy u is solved by online iterative reinforcement method based on measurement data. k The optimal intermediate variable kernel matrix in So that the control strategy u k Optimal, achieving dynamic r of the reference trajectory k Accurate tracking.

[0053] The controlled systems mainly used by the present invention are DC motor systems, aircraft autopilot systems, balance beam systems, load frequency controlled power systems in the field of power systems, and other linear discrete time systems are described as: controlled system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a block diagram of the information-physical system under the denial of service attack of the present invention; wherein, u k is the control strategy for tracking the controller output, r k is the reference trajectory dynamics, y k is the output of the controlled system, x k is the state of the controlled system, is the control input received by the actuator after the attack; δ k is the attack rate at the kth moment;

[0055] Figure 2 is a flow chart of the tracking control method based on measurement data under a denial of service attack according to the present invention;

[0056] Figure 3 It is to verify the tracking controller gain K obtained in the case of known controlled system matrix in test 1. opt =[0.0493 2.3252 -0.000044 -2.3246] The system output y with and without denial of service attack under the tracking controller k And the reference trajectory dynamics r k Change curve relationship diagram; among them, Figure 3 a is the system output y without attack k Dynamics of the reference trajectory r k Tracking curve graph; Figure 3 b is the system output y under attack k Dynamics of the reference trajectory r k Tracking curve graph;

[0057] Figure 4 To verify the tracking controller gain K obtained in Experiment 1 opt Tracking error curves with and without denial of service attacks under the formed tracking controller;

[0058] Figure 5 is the intermediate variable kernel matrix calculated by the method of the present invention in verification experiment 1 Schematic diagram of convergence of the tracking controller gain K; Figure 5a is verification test 1 The change curve diagram of Figure 5 b is verification test 1 || KK opt ||'s changing curve chart;

[0059] Figure 6 To verify the strategy iteration reinforcement learning algorithm described in the present invention in the experiment 1, the system output y during the learning process k and the reference trajectory dynamics r k Response curve diagram of

[0060] Figure 7 To verify the tracking controller gain K obtained in Experiment 2 when the controlled system matrix is ​​known opt =[0.5330 0.9775 -0.2231 -0.7534], the system output y with and without denial of service attack under the tracking controller formed k and r k Response curve diagram of

[0061] Figure 8 To verify the test 2, Change curve chart;

[0062] Fig. 9 To verify the value iteration reinforcement learning algorithm described in the present invention in test 2, the system output y during the learning process k and the reference trajectory dynamics r k Response curve diagram. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0065] Embodiment 1:

[0066] See also Figure 2 To illustrate the present embodiment 1, the tracking control method based on measurement data under a denial of service attack described in the present embodiment 1 includes the following process:

[0067] S1, according to the reference trajectory dynamic r of the controlled system input, the controlled system output and the reference trajectory system output at time k in the time interval [kM, k] k, reconstruct the augmented system state ξ at time k under a denial of service attack k ; k and M are both integers;

[0068] S2, according to the augmented system state ξ k , reconstruct the value function V(x k ,r k );

[0069] S3, according to the value function V(x k ,r k ), obtain the corresponding initial Bellman equation, and then combine the initial Bellman equation and the augmented system state ξ k , obtain the corresponding Bellman equation based on the measured data;

[0070] S4, according to the augmented system state ξ k and the Bellman equation based on the measured data to solve the control strategy u of the tracking controller k The composition of; Among them, the control strategy u k As the input of the controlled system at time k;

[0071] S5, online iterative reinforcement learning method based on measurement data, control strategy u k The kernel matrix of the intermediate variables in Iterate to get the optimal intermediate variable kernel matrix In this way, the optimal control strategy is obtained; the optimal control strategy is used to realize the dynamic r of the reference trajectory k tracking.

[0072] The present invention does not need to rely on the system state, but only needs to know the reference trajectory dynamics r of the controlled system input, the controlled system output and the reference trajectory system output. k , reconstructing the augmented system state under denial of service attackξ k , augmented system state ξ k The construction does not depend on the system state, which overcomes the difficulty in obtaining the system state in the prior art and is not convenient for dynamic reference trajectory r k The problem of effective tracking.

[0073] In order to achieve accurate tracking, the present invention proposes a reinforcement learning algorithm that only requires system input and output data for the safety control of information-physical systems under malicious attacks. Through online iterative reinforcement learning based on measurement data, the present invention uses input and output data to augment the system state ξ k and the quadratic value function V(x k ,r k) is reconstructed. On this basis, the Bellman equation based on input and output data (that is, the Bellman equation based on measurement data) is established to obtain the safety tracking control strategy u based on input and output data. k The control strategy u is solved by online iterative reinforcement method based on measurement data. k The optimal intermediate variable kernel matrix in So that the control strategy u k Optimal, achieving dynamic r of the reference trajectory k Accurate tracking.

[0074] Specifically, in S1, according to the reference trajectory dynamic r output by the controlled system input, the controlled system output and the reference trajectory system output at time k in the time interval [kM, k] k , reconstruct the augmented system state ξ at time k under a denial of service attack k The implementation methods include:

[0075]

[0076] in,

[0077]

[0078]

[0079]

[0080] Γ M =[B AB A 2 B…A M-1 B];

[0081] Among them, C is the output constant matrix of the controlled system, and B is the second intermediate variable constant matrix of the controlled system; is the intermediate variable matrix, is the vector composed of the input signal of the controlled system, the output signal of the controlled system and the dynamics of the reference trajectory in the time interval [kM, k-1], x k is the state of the controlled system at the kth moment, δ k is the attack rate at the kth moment, Γ M is the controllability matrix, A is the constant matrix of the first intermediate variable of the controlled system, T M is the observable matrix, is the observable matrix T M The pseudo-inverse of E M is the matrix variable of the intermediate calculation, G is the system matrix of the reference trajectory system, is the matrix of the input signal of the controlled system in the time interval [kM, k-1], u k-iis the input signal of the controlled system at time ki, i=1,2……M, is the matrix of the output signal of the controlled system in the time interval [kM, k-1], y k-i is the output signal of the controlled system at time ki, r k-M is the reference trajectory dynamics at the kMth moment.

[0082] Specifically, there exists a positive integer M0 such that when M<M0, Rank(T M )<n and when M≥M0, Rank(T M )=n holds, and M0 satisfies M0p′≥n; Rank(T M ) is the matrix T M rank; p′ is the dimension of the output of the controlled system, and n is the dimension of the state of the controlled system.

[0083] Specifically, in S2, according to the augmented system state ξ k , reconstruct the value function V(x k ,r k ) include:

[0084]

[0085] in, P is a symmetric matrix, and P>0; To achieve expectations.

[0086] Specifically, in S3, according to the value function V(x k ,r k ), obtain the corresponding initial Bellman equation, and then combine the initial Bellman equation and the augmented system state ξ k , the corresponding Bellman equation based on measured data is obtained as follows:

[0087] S31. According to the value function The corresponding initial Bellman equation is obtained, specifically:

[0088]

[0089] Where, Q1 = [C -I] T Q[C -I],

[0090] Q is the first weight matrix, Q ≥ 0, C is the output matrix of the controlled system, I is the unit matrix, Q1 is the intermediate variable matrix, and δ is δ k =1, is the probability, SER is the symbol error rate, R is the second weight matrix, R>0, λ is the discount factor, Take expectation, P is a symmetric matrix, P>0;

[0091] S32, substituting the expression of Formula 1 into Formula 3 to obtain the Bellman equation based on the measured data, specifically:

[0092]

[0093] in, is a vector consisting of the controlled system input, the controlled system output and the reference trajectory dynamics in the time interval [kM,k].

[0094] Specifically, in S4, according to the augmented system state ξ k and the Bellman equation based on the measured data to solve the control strategy u of the tracking controller k The implementation of the composition is:

[0095] S41. Transpose the Bellman equation based on the measured data to obtain the differential error

[0096]

[0097] S42, the differential error In This breaks down into:

[0098]

[0099] in, is the matrix of the input signal of the controlled system in the time interval [k-M+1,k-1], is the matrix of the output signal of the controlled system in the time interval [k-M+1,k], r k-M+1 is the reference trajectory dynamics at the k-M+1th moment; is the intermediate variable kernel matrix The value of the first row and first column element in , is the intermediate variable kernel matrix The value of the first row and second column element in is the intermediate variable kernel matrix The value of the first row and third column element in is the intermediate variable kernel matrix The value of the first row and fourth column of is the intermediate variable kernel matrix The value of the element in the second row and first column of is the intermediate variable kernel matrix The value of the element in the third row and first column of Intermediate variable kernel matrix The value of the element in the fourth row and first column of is the intermediate variable kernel matrix The second row and second column element in is the intermediate variable kernel matrix The second row and third column element in is the intermediate variable kernel matrix The second row and fourth column element in is the intermediate variable kernel matrix The third row and second column element in is the intermediate variable kernel matrix The third row and third column element in is the intermediate variable kernel matrix The third row and fourth column element in is the intermediate variable kernel matrix The fourth row and second column element in is the intermediate variable kernel matrix The fourth row and third column element in is the intermediate variable kernel matrix The fourth row and fourth column element in ;

[0100] S43, combine formula 6 and formula 4, and calculate u k Taking the derivative and setting the derivative to 0, we get u k The expression is:

[0101]

[0102] in, K is the gain of the tracking controller.

[0103] Specifically, in S5, the online iterative reinforcement learning method based on measurement data is implemented using a policy iterative reinforcement learning algorithm, and the policy iterative reinforcement learning algorithm is used to obtain the optimal intermediate variable kernel matrix The optimal control strategy is achieved as follows:

[0104] S5-11, let the initial value of the iteration number j be 0, and give a preset constant value ε;

[0105] S5-12. Given an initial stable control strategy

[0106] S5-13, Strategy evaluation: Calculate the j+1th iteration using formula 8 The column matrix Specifically,

[0107]

[0108] Where, Q1 = [C -I] TQ[C -I],

[0109] is the Kronecker product, Q1 is the intermediate variable matrix, Q is the first weight matrix, C is the output constant matrix of the controlled system, I is the unit matrix, and δ is δ k =1, is the probability, SER is the symbol error rate, R is the second weight matrix, λ is the discount factor, Take expectations, is the control strategy for the jth iteration, is the intermediate variable kernel matrix for the jth iteration The column matrix of

[0110] S5-14, Yes Perform matrix transformation to obtain the intermediate variable kernel matrix at the j+1th iteration

[0111] S5-15, Strategy Improvement: Passed Calculate the control strategy for the j+1th iteration Specifically:

[0112]

[0113] S5-16, Judgment Is it true? ||·|| is the matrix norm. If the result is yes, then output and control strategies and will As the optimal intermediate variable kernel matrix The control strategy As the optimal control strategy; the result is no, j=j+1, and execute step S5-13.

[0114] Specifically, in S5, the online iterative reinforcement learning method based on measurement data is implemented using the value iteration reinforcement learning algorithm, and the value iteration reinforcement learning algorithm is used to obtain the optimal intermediate variable kernel matrix The optimal control strategy is achieved as follows:

[0115] S5-21, let the initial value of the iteration number j be 0, and give a preset constant value ε;

[0116] S5-22. Given an arbitrary initial control strategy

[0117] S5-23, Strategy evaluation: Calculate the j+1th iteration using Formula 8 The column matrix Specifically,

[0118]

[0119] Where, Q1 = [C -I] T Q[C -I], is the Kronecker product, Q is the first weight matrix, C is the output constant matrix of the controlled system, I is the unit matrix, and δ is the attack rate δ at the kth moment k =1, is the probability, SER is the symbol error rate, R is the second weight matrix, λ is the discount factor, Take expectations, is the control strategy for the jth iteration, is the intermediate variable kernel matrix for the jth iteration The column matrix of

[0120] S5-24, Yes Perform matrix transformation to obtain the intermediate variable kernel matrix at the j+1th iteration

[0121] S5-25, Strategy Improvement: Passed Calculate the control strategy for the j+1th iteration Specifically:

[0122]

[0123] S5-26, Judgment Is it true? ||·|| is the matrix norm. If the result is yes, then output and control strategies and will As the optimal intermediate variable kernel matrix The control strategy As the optimal control strategy; the result is no, j=j+1, and execute step S5-23.

[0124] Furthermore, Among them, ΔΞ k =Ξ k -λΞ k+1 ;

[0125]

[0126]

[0127]

[0128] Ξ k Reason to The expectation of the column vector of the Kronecker product, Ξ k+1 for to The expectation of the column vector of the Kronecker product, ΔΞ k For k and k+1 The expected difference between k is the intermediate variable matrix, μ is an integer.

[0129] The following is a description of the origin of the invention concept of the present application in conjunction with the following process, specifically:

[0130] (1) Description of the physical controlled object and reference trajectory dynamics

[0131] The physical controlled object is described by the following linear discrete-time system

[0132]

[0133] in, is the system status, is an n-dimensional space domain, is the control input received by the actuator, is an m-dimensional space domain, is the measured output, i.e.: the system output; is a p′-dimensional space domain; A, B, C are constant matrices with appropriate dimensions. The matrix pair (A, B) is controllable, and the matrix pair (C, A) is observable.

[0134] The reference trajectory dynamics are as follows:

[0135] r k+1 =Gr k (2);

[0136] It is worth noting that the matrix G in formula (2) is not required to be Hurwitz. Next, by combining formula (1) and formula (2), the following augmented system including the original system and the reference trajectory system can be obtained.

[0137]

[0138] in,

[0139] Design the tracking controller as u k =K x x k +K r r k , where K x and K r are the gains of the tracking controller. The subsequent content will complete the K x and K r design.

[0140] (2) Establishment of Denial of Service Attack Model

[0141] Due to the vulnerability of wireless shared networks, attackers may interfere with communication networks through denial of service attacks. In order to model this situation and describe the interaction between attackers and defenders, a communication model of signal interference plus noise ratio is introduced, so there is

[0142]

[0143] Where SER is the symbol error rate, SNR is the signal-to-noise ratio, ω is a positive constant, and κ is the integral variable.

[0144] When a denial of service attack occurs, the signal-to-noise ratio can be restated as

[0145]

[0146] Among them, k To track the transfer power of the controller, is the power of additive white Gaussian noise, π k is the attacker’s attack power. Control signal u k Whether the transmission is successful depends on the attack rate εk k , let's introduce a binary variable

[0147]

[0148] Therefore, we can get is the control input received by the actuator after the attack;

[0149]

[0150] According to the above discussion, we can get

[0151]

[0152] Where l is the length of the transmitted data, δ is the attack rate δ at the kth moment k =1, For probability.

[0153] Therefore, the system of formula (1) can be described as

[0154] x k+1 =Ax k +δ k Bu k (8);

[0155] At this time, according to the reference trajectory dynamics shown in formula (2) and the original system formula (8) under attack, the obtained augmented system becomes

[0156] ξ k+1 =Sξ k +δ k B1u k (9);

[0157] in,

[0158]

[0159] S and B1 are both intermediate variable matrices, ξ k is the augmented system at time k;

[0160] (3) Control objectives

[0161] The goal of the infinite time domain linear quadratic tracking problem is to design an optimal tracking controller for the system of formula (8) to ensure that the controlled system output y k Can track the reference signal r k , see Figure 1 It is worth noting that the standard solution to the linear quadratic tracking problem requires that the system described by reference equation (2) is asymptotically stable. To relax this constraint, the following performance metric J(x) with a discount factor λ∈(0,1] is defined: k ,r k );

[0162]

[0163] in, C is the output matrix of the system. Q≥0, Q is the first weight matrix, R>0, R is the second weight matrix. The expression of Q1 is as follows:

[0164] Q1=[C -I] T Q[C -I] (12);

[0165] Lemma 1 is given below to show that the performance index J(x k ,r k ) can be written as k The secondary form of .

[0166] Lemma 1 If the reference trajectory dynamics is given by formula (2), the tracking control strategy is designed as Then the performance index formula (11) can be written as in is a symmetric matrix, P 11 is the first row and first column element in the matrix P, P 12 is the first row and second column element in the matrix P, P 21 is the element in the second row and first column of the matrix P, P 22 is the element in the second row and first column of the matrix P.

[0167] Based on formula (11) and Lemma 1, we can get the following value function:

[0168]

[0169] Therefore, the Bellman equation is as follows:

[0170]

[0171] Then the linear quadratic tracking Hamiltonian under denial of service attack can be defined as

[0172]

[0173] Lemma 2 is given below to design the optimal tracking controller, that is, the tracking control that generates the optimal control strategy.

[0174] Lemma 2 Given the reference trajectory dynamics as in Equation (2), using Lemma 1, the optimal tracking controller for linear quadratic tracking under denial of service attacks can be designed as

[0175]

[0176] Among them, in formula 16 λ is the discount factor, R is the weight matrix, K opt is the gain of the optimal tracking controller, P represents a symmetric matrix, and the specific composition of P is obtained by formula (17). P is the solution of the following augmented linear quadratic tracking algebraic Riccati equation (17). Formula (17) is specifically:

[0177]

[0178] It can be seen that the algebraic Riccati equation contains the parameter δ; the algebraic Riccati equation, that is, formula (17), the solution P of the algebraic Riccati equation depends on the parameter δ, so it is very important to reveal the existence and uniqueness of the solution of the parameter δ and formula (17). It is difficult to directly prove that the solution of (17) exists and is unique. Therefore, we define:

[0179] Where k>0 is an integer, represents a k-order composite function. Based on the above definition, we know is equivalent to the algebraic Riccati equation. Therefore, if we can prove that the equation If there is a solution to P, then the existence and uniqueness of P can be guaranteed. Lemma 3 is given below to illustrate the existence of P.

[0180] Lemma 3 If there exists a semi-positive definite matrix Make If it holds, then for any initial condition Ω0≥0, F(Ω)=Ω, there is a unique solution Ω * ,and To ensure that P exists, the inequality should be established, where is the spectral radius of the matrix S.

[0181] In summary, the solution of the algebraic Riccati equation exists and is unique. Lemma 4 is given below to ensure the stability of the tracking error system and the optimality of the solution of the algebraic Riccati equation.

[0182] Lemma 4 Given the system as formula (9), the performance indicator function formula (11) and the tracking error e k =Cx k -r k .definition Then the optimal tracking controller, that is, formula (16), can be asymptotically stable Further, u k It can minimize the performance index function formula (11). Since we want to obtain the optimal tracking controller, we need to find the controller that minimizes formula (11).

[0183] (4) Design tracking control algorithm using system state

[0184] Given a tracking controller gain K opt , the tracking controller at this time is u k =-K opt ξ k , and then substitute it into the combined formula (9), and then substitute the result into the formula (14), and we can get the following linear quadratic tracking Lyapunov equation:

[0185]

[0186] By combining the Lyapunov equation (18) and the equation (16), an offline policy iteration algorithm and an offline value iteration algorithm can be obtained, and both offline algorithms can converge to the solution of equation (17). If the Lyapunov equation (18) is replaced by the Bellman equation (14), that is, by combining equation (14) and equation (16), an online policy iteration algorithm and an online value iteration algorithm can be obtained. Therefore, it can be seen that both online and offline algorithms require accurate system dynamics or system states. In order to get rid of these limitations, the following will propose a reinforcement learning algorithm that uses input and output and reference trajectory data to realize the design of a safe tracking controller, that is, the design of the tracking controller of the present invention, and equations (1) to (18) are all elaborations of the prior art.

[0187] (5) Reconstruction of the state and value function of the augmented system

[0188] The present invention will show how to use the input, output and reference trajectory data of the controlled system to reconstruct the value function and augment the system state. Under a denial of service attack, on the time interval [kM, k], the augmented system shown in formula (9) can be re-expressed as an extended state equation

[0189]

[0190] Among them, Γ M =[B AB A 2 B…A M-1 B] is the controllability matrix, ξ k-M is the augmented system state at time kM under a denial of service attack, is the matrix composed of the input signal of the controlled system in the time interval [kM,k-1], u k-i is the input signal of the controlled system at time ki, i=1,2…M.

[0191] Formula (19) can be equivalently written as

[0192]

[0193] Based on Γ M and The definition of and formula (1) yields

[0194]

[0195] Therefore, the sequence of the controlled system output in the time interval [kM, k-1] can be defined as:

[0196]

[0197] in,

[0198]

[0199] T M is the observable matrix, is the matrix of the output signal of the controlled system in the time interval [kM, k-1], y k-i is the output signal of the controlled system at time ki, i=1,2……M, It is the matrix composed of the input signal of the controlled system in the time interval [kM,k-1].

[0200] Since the matrix pair (C, A) is observable, there must be a positive integer M0 such that when M<M0, Rank(T M )<n and when M≥M0, Rank(TM )=n holds, and M0 satisfies M0p′≥n.

[0201] From formula (22), we can get Substituting this formula into formula (20), we can obtain:

[0202]

[0203] in,

[0204] Performing matrix operations on formula (23), we can obtain

[0205]

[0206] Define the following matrix:

[0207]

[0208]

[0209] According to and Equation 24 can be rewritten as follows:

[0210]

[0211] Combining Lemma 1 and formula (25), we can get

[0212]

[0213] From formula (25), we can know that the augmented system state ξk k has been written in the form of input and output data. According to formula (26), the value function V(x k ,r k ) is reconstructed into a quadratic form with respect to the input and output data. It is worth noting that depends on A, B, C, and G. Next, we will show how to obtain the matrix by designing a reinforcement learning algorithm that does not require precise knowledge of the system and the dynamics of the reference trajectory

[0214] (6) Establishment of online reinforcement learning algorithm based on measurement data

[0215] Combining Bellman equation (14) and formula (25), we can obtain the following Bellman equation based on measured data:

[0216]

[0217] Formula (27) is necessary to establish a model-free reinforcement learning algorithm. Based on formula (27), by shifting the terms in formula (27), we can get the following differential error:

[0218]

[0219] Using differential error Instead of augmenting the system state, measurement data can be used to implement policy evaluation. Decompose into

[0220]

[0221] Combining equations (27) and (29), we can get u in equation (27): k Taking the derivative and setting it to 0 gives u k as follows

[0222]

[0223] From formula (30), we can see that the control strategy u k It only contains input, output and reference trajectory data, and there is no need to augment the system state. K is the gain of the tracking controller. The following will propose an online reinforcement learning algorithm based on measurement data.

[0224] Use the Kronecker product to process formula 27 and transform the matrix Convert to column matrix Available

[0225]

[0226] The following is an online policy iteration and value iteration reinforcement learning tracking control algorithm based on measurement data, as shown in Algorithm 1 and Algorithm 2.

[0227] It is worth noting that the least squares method can be used to solve equation (31) online without using the system state, only the data collected online. and In fact, formula (31) is a scalar equation, and represents a (Mm+Mp′+p′)×(Mm+Mp′+p′+1)-dimensional real number, is a matrix of (Mm+Mp′+p′)×(Mm+Mp′+p′+1) dimensions, where m is an integer. contains μ0 independent elements, and Therefore, in order to solve equation (31), at least μ0 data need to be collected.

[0228] Assume that μ (μ ≥ μ00) data have been collected and define Ξk and θ k as follows

[0229]

[0230]

[0231]

[0232]

[0233]

[0234] Algorithm 1 and Algorithm 2 are solving In the process, according to k and θ k The expression of can transform formula (32) into formula (33);

[0235]

[0236] Among them, ΔΞ k =Ξ k -λΞ k+1 .

[0237] Therefore, we can get formula (32): The least squares solution of is as follows:

[0238]

[0239] In summary, the origin of the invention concept of this application can be summarized as follows:

[0240] Step 1: Construct a physical controlled object model and a reference trajectory model, and augment the two models to obtain an augmented system that includes the original system and the reference system.

[0241] Step 2: Use the signal interference plus noise ratio communication model to model the denial of service attack, and then construct an augmented system under the denial of service attack.

[0242] Step 3: For the augmented system under denial of service attack, the control objective is given, that is, a performance index function with a discount factor is given, and then the index function is written as a quadratic form with respect to the state of the augmented system, and the Bellman equation is further given. Based on the optimal control theory, the optimal tracking controller under denial of service attack is obtained, and the algebraic Riccati equation under denial of service attack is derived. Since there is a parameter δ in the algebraic Riccati equation, it is necessary to analyze the influence of this parameter on the existence and uniqueness of the solution of the algebraic Riccati equation.

[0243] Step 4: The above three steps achieve the security control of the cyber-physical system under denial of service attacks, but it is necessary to know the precise dynamics of the augmented system. In order to avoid dependence on the precise dynamics of the system, a reinforcement learning algorithm based on the system state is proposed in the fourth step. Specifically, the linear quadratic tracking Lyapunov equation is first derived based on the Bellman equation, and then the offline and online reinforcement learning security tracking control algorithms based on the augmented system state are established based on the Lyapunov equation and the Bellman equation.

[0244] Step 5: The fourth step requires the use of the system state, and the reference trajectory dynamics r in actual application k During the tracking process, the system state is not always available. In order to solve this problem, the tracking control method based on measurement data under denial of service attack described in the present invention is introduced. The tracking method proposes to use the reinforcement learning algorithm of the system input and output data to obtain the optimal intermediate variable kernel matrix In the specific process, the present invention uses input and output data to augment the system state ξk k and the quadratic valued function V(x k ,r k ) is reconstructed. On this basis, the Bellman equation based on input and output data (that is, the Bellman equation based on measurement data) is established to obtain the safety tracking control strategy u based on input and output data. k The composition of the control strategy u k The structure of the kernel matrix contains the intermediate variables In order to obtain the optimal tracking controller, it is necessary to obtain the optimal intermediate variable kernel matrix Therefore, it is necessary to construct a security tracking strategy iteration and value iteration reinforcement learning algorithm based on input and output data under denial of service attacks to solve the control strategy u k The optimal intermediate variable kernel matrix in So that the control strategy u k Best.

[0245] Verification test:

[0246] In order to verify and demonstrate the effectiveness of the reinforcement learning control algorithm proposed in the present invention, the present invention will use a DC motor system as the controlled system and a numerical example to illustrate the effectiveness of the tracking control method based on measurement data under a denial of service attack described in the present invention.

[0247] (1) Verification Test 1: Verify the effectiveness of the tracking controller designed using the policy iteration reinforcement learning algorithm based on measurement data

[0248] First, the parameters of the controlled system matrices A and B are given as follows:

[0249]

[0250] In a DC motor system (i.e., the controlled system), the control input is voltage and the system output is angular position. The system output can be expressed as y k =[1 0]x k . Assume that the reference trajectory is a constant signal r k =5rad, and set parameters Q = 1000, R = 10, δ = 0.6, and discount factor λ = 0.12. Then, by combining formula (16) and formula (17) in Lemma 2 and Lemma 3, we can get the tracking controller gain K obtained by the prior art method: opt as follows:

[0251] K opt =[0.0493 2.3252 -0.000044 -2.3246]

[0252] Figure 3 and Figure 4 They are shown separately, using K opt Under the formed tracking controller, the system output, reference trajectory and tracking error changes under the presence or absence of denial of service attacks are used as a standard to verify the tracking controller obtained using the "strategy iteration reinforcement learning algorithm" of the present invention.

[0253] Under the given parameters Q = 1000, R = 10, δ = 0.6, λ = 0.12, the tracking controller is designed using the "policy iteration reinforcement learning algorithm" of the present invention. It is worth noting that the matrix S and matrix B1 do not need to be used to execute the "policy iteration reinforcement learning algorithm" of the present invention. In order to ensure T M is full rank, and should satisfy M≥2. Here we choose M=2. The tracking control gain calculated by the policy iteration reinforcement learning algorithm of the present invention is:

[0254] K=[0.0365 2.3112 -0.0071 -2.3035];

[0255] Figure 5 Shown and||KK opt ||'s changing curve; from Figure 5 It can be seen from a that the intermediate variable kernel matrix is ​​obtained by the method of the present invention It can well follow the intermediate variable kernel matrix obtained by the prior art method from Figure 5 It can be seen from b that the tracking controller gain K obtained by the method of the present invention can well follow the tracking controller gain K obtained by the prior art method optTherefore, it can be concluded that the accuracy of the controller designed by the present invention is independent of the controlled system matrices A and B, and the method of the present invention can design the controller without relying on the controlled system matrices A and B.

[0256] Figure 6 Shows the system output y in the learning process using the policy iteration reinforcement learning algorithm k And the reference trajectory dynamics r k Change curve. Obviously, the system output y k Very close to the reference trajectory dynamics r k The effectiveness of the tracking method of the present invention implemented based on the policy iteration reinforcement learning algorithm is further verified.

[0257] (2) Verification Test 2: Verify the effectiveness of the tracking controller designed using the value iteration reinforcement learning algorithm based on measurement data

[0258] First, the controlled system matrix is ​​given as follows:

[0259]

[0260] The reference trajectory is a piecewise constant signal, that is, Assume Q = 200, R = 25, λ = 0.3, δ = 0.8. Under the premise of satisfying Lemma 3, solve the augmented algebraic Riccati equation (17) to obtain the control gain as follows:

[0261] K opt =[0.5330 0.9775 -0.2231 -0.7534].

[0262] Figure 7 Demonstrated the use of K opt The tracking controller formed under the denial of service attack y k and r k The change curve of . And use this as a standard to verify the effectiveness of the tracking controller obtained using the "value iteration reinforcement learning algorithm" of the present invention.

[0263] Under the given parameters Q = 200, R = 25, λ = 0.3, δ = 0.8, the tracking controller is designed using the value iteration reinforcement learning algorithm of the present invention. The tracking control gain calculated by the strategy iteration reinforcement learning algorithm of the present invention is:

[0264] K=[0.5290 0.9362 -0.2288 -0.7065];

[0265] By comparing K and K optThe value of can prove that the tracking controller gain K obtained by the method of the present invention can well follow the tracking controller gain K obtained by the prior art method. opt , the control gain K finally converges to K = [0.5290 0.9362 -0.2288 -0.7065].

[0266] Figure 8 Shows the value iteration reinforcement learning algorithm learned Can follow well

[0267] Fig. 9 Shows the system output y in the learning process using the value iteration reinforcement learning algorithm k and the reference signal y k Change curve. Obviously, the output y k Successfully tracked the reference signal r k The effectiveness of the tracking method of the present invention implemented based on the value iteration reinforcement learning algorithm is further verified.

[0268] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A tracking control method based on measurement data under a denial of service attack, characterized in that: The method includes the following steps: S1, according to the reference trajectory dynamic r of the controlled system input, the controlled system output and the reference trajectory system output at time k in the time interval [kM, k] k , reconstruct the augmented system state ξ at time k under a denial of service attack k ; k and M are both integers; S2, according to the augmented system state ξ k , reconstruct the value function V(x k ,r k ); S3, according to the value function V(x k ,r k ), obtain the corresponding initial Bellman equation, and then combine the initial Bellman equation and the augmented system state ξ k , obtain the corresponding Bellman equation based on the measured data; S4, according to the augmented system state ξ k and the Bellman equation based on the measured data to solve the control strategy u of the tracking controller k The composition of; Among them, the control strategy u k As the input of the controlled system at time k; S5, online iterative reinforcement learning method based on measurement data, control strategy u k The kernel matrix of the intermediate variables in Iterate to get the optimal intermediate variable kernel matrix In this way, the optimal control strategy is obtained; the optimal control strategy is used to realize the dynamic r of the reference trajectory k tracking.

2. The tracking control method based on measurement data under a denial of service attack according to claim 1 is characterized in that: In S1, the reference trajectory dynamics r is calculated based on the input of the controlled system, the output of the controlled system, and the output of the reference trajectory system at time k in the time interval [kM, k]. k , reconstruct the augmented system state ξ at time k under a denial of service attack k The implementation methods include: in, is the intermediate variable matrix, is the vector composed of the input signal of the controlled system, the output signal of the controlled system and the dynamics of the reference trajectory in the time interval [kM, k-1], x k is the state of the controlled system at the kth moment, δ k is the attack rate at the kth moment, Γ M is the controllability matrix, A is the constant matrix of the first intermediate variable of the controlled system, T M is the observable matrix, is the observable matrix T M The pseudo-inverse of E M is the matrix variable of the intermediate calculation, G is the system matrix of the reference trajectory system, is the matrix of the input signal of the controlled system in the time interval [kM, k-1], u k-i is the input signal of the controlled system at time ki, i=1,2……M, is the matrix of the output signal of the controlled system in the time interval [kM, k-1], y k-i is the output signal of the controlled system at time ki, r k-M is the reference trajectory dynamics at the kMth moment.

3. The tracking control method based on measurement data under a denial of service attack according to claim 2 is characterized in that: There exists a positive integer M0 such that when M<M0, Rank(T M )<n and when M≥M0, Rank(T M )=n holds, and M0 satisfies M0p′≥n; Rank(T M ) is the matrix T M rank; p′ is the dimension of the output of the controlled system, and n is the dimension of the state of the controlled system.

4. The tracking control method based on measurement data under a denial of service attack according to claim 2 is characterized in that: In S2, according to the augmented system state ξ k , reconstruct the value function V(x k ,r k ) include: in, P is a symmetric matrix, and P>0; To achieve expectations.

5. The tracking control method based on measurement data under a denial of service attack according to claim 2 is characterized in that: In S3, according to the value function V(x k ,r k ), obtain the corresponding initial Bellman equation, and then combine the initial Bellman equation and the augmented system state ξ k , the corresponding Bellman equation based on measured data is obtained as follows: S31. According to the value function The corresponding initial Bellman equation is obtained, specifically: Where Q1 = [CI] T Q[CI], Q is the first weight matrix, Q ≥ 0, C is the output matrix of the controlled system, I is the unit matrix, Q1 is the intermediate variable matrix, and δ is δ k =1, is the probability, R is the second weight matrix, R>0, λ is the discount factor, Take expectation, P is a symmetric matrix, P>0; S32, substituting the expression of Formula 1 into Formula 3 to obtain the Bellman equation based on the measured data, specifically: in, is a vector consisting of the controlled system input, the controlled system output and the reference trajectory dynamics in the time interval [kM,k].

6. The tracking control method based on measurement data under a denial of service attack according to claim 5 is characterized in that: In S4, according to the augmented system state ξ k and the Bellman equation based on the measured data to solve the control strategy u of the tracking controller k The implementation of the composition is: S41. Transpose the Bellman equation based on the measured data to obtain the differential error S42, the differential error In This breaks down into: in, is the matrix of the input signal of the controlled system in the time interval [k-M+1,k-1], is the matrix of the output signal of the controlled system in the time interval [k-M+1,k], r k-M+1 is the reference trajectory dynamics at the k-M+1th moment; is the intermediate variable kernel matrix The value of the first row and first column element in , is the intermediate variable kernel matrix The value of the first row and second column element in is the intermediate variable kernel matrix The value of the first row and third column element in is the intermediate variable kernel matrix The value of the first row and fourth column of is the intermediate variable kernel matrix The value of the element in the second row and first column of is the intermediate variable kernel matrix The value of the element in the third row and first column of Intermediate variable kernel matrix The value of the element in the fourth row and first column of is the intermediate variable kernel matrix The second row and second column element in is the intermediate variable kernel matrix The second row and third column element in is the intermediate variable kernel matrix The second row and fourth column element in is the intermediate variable kernel matrix The third row and second column element in is the intermediate variable kernel matrix The third row and third column element in is the intermediate variable kernel matrix The third row and fourth column element in is the intermediate variable kernel matrix The fourth row and second column element in is the intermediate variable kernel matrix The fourth row and third column element in is the intermediate variable kernel matrix The fourth row and fourth column element in ; S43, combine formula 6 and formula 4, and calculate u k Taking the derivative and setting the derivative to 0, we get u k The expression is: in, K is the gain of the tracking controller.

7. The tracking control method based on measurement data under a denial of service attack according to claim 2 is characterized in that: In S5, the online iterative reinforcement learning method based on measurement data is implemented using a policy iterative reinforcement learning algorithm, and the policy iterative reinforcement learning algorithm is used to obtain the optimal intermediate variable kernel matrix The optimal control strategy is achieved as follows: S5-11, let the initial value of the iteration number j be 0, and give a preset constant value ε; S5-12. Given an initial stable control strategy S5-13, Strategy evaluation: Calculate the j+1th iteration using formula 8 The column matrix Specifically, Where Q1 = [CI] T Q[CI], is the Kronecker product, Q1 is the intermediate variable matrix, Q is the first weight matrix, C is the output constant matrix of the controlled system, I is the unit matrix, and δ is δ k =1, is the probability, R is the second weight matrix, λ is the discount factor, Take expectations, is the control strategy for the jth iteration, is the intermediate variable kernel matrix for the jth iteration The column matrix of S5-14, Yes Perform matrix transformation to obtain the intermediate variable kernel matrix at the j+1th iteration S5-15, Strategy Improvement: Passed Calculate the control strategy for the j+1th iteration Specifically: S5-16, Judgment Is it true? ||·|| is the matrix norm. If the result is yes, then output and control strategies and will As the optimal intermediate variable kernel matrix The control strategy As the optimal control strategy; the result is no, j=j+1, and step S5-13 is executed.

8. The tracking control method based on measurement data under a denial of service attack according to claim 2 is characterized in that: In S5, the online iterative reinforcement learning method based on measurement data is implemented using the value iteration reinforcement learning algorithm, and the value iteration reinforcement learning algorithm is used to obtain the optimal intermediate variable kernel matrix The optimal control strategy is achieved as follows: S5-21, let the initial value of the iteration number j be 0, and give a preset constant value ε; S5-22. Given an arbitrary initial control strategy S5-23, Strategy evaluation: Calculate the j+1th iteration using Formula 8 The column matrix Specifically, Where, Q1 = [C -I] T Q[C -I], is the Kronecker product, Q is the first weight matrix, C is the output constant matrix of the controlled system, I is the unit matrix, and δ is the attack rate δ at the kth moment k =1, is the probability, R is the second weight matrix, λ is the discount factor, Take expectations, is the control strategy for the jth iteration, is the intermediate variable kernel matrix for the jth iteration The column matrix of S5-24, Yes Perform matrix transformation to obtain the intermediate variable kernel matrix at the j+1th iteration S5-25, Strategy Improvement: Passed Calculate the control strategy for the j+1th iteration Specifically: S5-26, Judgment Is it true? ||·|| is the matrix norm. If the result is yes, then output and control strategies and will As the optimal intermediate variable kernel matrix The control strategy As the optimal control strategy; the result is no, j=j+1, and execute step S5-23.

9. The tracking control method based on measurement data under a denial of service attack according to claim 2, characterized in that: Γ M =[B AB A 2 B … A M-1 B]; Among them, C is the output constant matrix of the controlled system, and B is the second intermediate variable constant matrix of the controlled system.

10. The tracking control method based on measurement data under a denial of service attack according to claim 7 or 8, characterized in that: Among them,DX k =Ξ k -λΞ k+1 ; Ξ k Reason to The expectation of the column vector of the Kronecker product, Ξ k+1 for to The expectation of the column vector of the Kronecker product, ΔΞ k For k and k+1 The expected difference between k is the intermediate variable matrix, μ is an integer.

Images