Safety detection and control method for CPS to resist FDIA

Abstract

The invention discloses a safety detection and control method for a CPS to resist FDIA, and relates to the technical field of deep reinforcement learning and safety control, and the method comprises the steps: initializing a CPS environment; acquiring a CPS state; calculating a local difference by using the local measurement value and the local estimation value; performing threshold analysis through deep reinforcement learning based on an improved PPO algorithm, and comparing a dynamic threshold with the calculated local difference to determine whether the CPS is attacked; and an event-triggered model prediction control method is adopted for intervention so as to resist attack of the FDIA on the CPS. According to the invention, the detection strategy can be automatically adjusted according to the environment change and historical data, so as to adapt to various different attack modes; the robustness of the DRL algorithm is enhanced, and the safety and reliability of the system are improved; and the system can be quickly started after FDIA is detected, and the system is predicted and adjusted, so that the stability of the system is quickly recovered.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning and security control technology, and in particular to a security detection and control method for a CPS to resist FDIA. Background Art

[0002] Since the early 2000s, cyber-physical systems (CPS) have been valued for their ability to integrate physical and computing components to perform complex tasks and have been widely used in fields such as smart grids, transportation systems, and healthcare. However, the interconnectedness of the physical components of CPS makes them vulnerable to malicious cyber attacks in the perception, communication, and control stages, which may cause unpredictable economic and material losses. Numerous cybersecurity incidents in the past have highlighted the importance of CPS security issues. The threats faced by CPS can be divided into three categories: data availability interruption, data integrity compromise, and data confidentiality leakage. In order to address these security issues, attack detection and control measures for false data injection attacks (FDIA) have been designed. At present, a variety of FDIA detection methods have been proposed, but traditional mechanisms are difficult to deal with new attacks due to fixed thresholds. Many studies have introduced reinforcement learning (RL) and deep reinforcement learning (DRL) technologies, using neural networks and learning algorithms to improve the system's adaptability to dynamic attack environments and its ability to identify unknown attack types.

[0003] The paper “Detection of faults and attacks including false data injection attack in smart grid using kalman filter” (K.Manandhar and X.Cao.IEEETransactions on Control of Network Systems, 1(4):370-379, 2014) designed a Euclidean detector to identify FDIA in smart grid. The paper “A novel false data injection attack detection model of the cyber-physical power system” (J.Cao, D Wang, and Qu.IEEEAccess, 8:95109-95125, 2020) proposed a novel FDIA detection model based on ensemble learning in smart grid. The paper “Distributed attack detection and secure estimation of networked cyber-physical systems against false data injection attacks and jamming attacks.” (Y:Guan and X.Ge.IEEE Transactions on Signal and Information Processing overNetworks, 4(1):48-59, 2017) independently developed a distributed attack detector and distributed security estimator for networked CPS. Nevertheless, current research has not fully considered the robustness of deep reinforcement learning algorithms in FDIA scenarios.

[0004] Security defense of CPS is also a key aspect, including strategies such as data encryption, vulnerability protection, and error correction. Among them, security control strategies stand out for their ability to respond quickly and keep the system running normally during attacks, dynamically adjusting control inputs through rule-based mechanisms to effectively respond to known and unknown attacks. Model predictive control (MPC) is a security control strategy suitable for CPS that can handle multiple-input and multiple-output systems. However, the high computational requirements for solving constrained optimal control problems have become an obstacle for CPS. As the system dimension and prediction range increase, the computational burden is further increased. Summary of the invention

[0005] In order to solve the above-mentioned limitations of the prior art, the purpose of the present invention is to provide a method for security detection and control of CPS against FDIA, which is used to solve the problems that the traditional mechanism of the current FDIA detection method is difficult to cope with new attacks due to the fixed threshold, and the deep reinforcement learning algorithm does not improve its robustness in the FDIA scenario.

[0006] The present invention solves the above problems through the following technical solutions:

[0007] A method for CPS to resist FDIA security detection and control, including:

[0008] Step A: CPS environment initialization;

[0009] Step B: Get CPS status;

[0010] Get the current CPS state and calculate the local measurement value y(k);

[0011] Collect the calculated local measurements y(k) and calculate the local estimate through the extended Kalman filter Calculate the local difference δ(k) using the local measurement value and the local estimate value;

[0012] Step C: Perform threshold analysis through deep reinforcement learning based on the improved PPO algorithm, and compare the dynamic threshold λ with the local difference δ(k) calculated in step B to determine whether the CPS is attacked;

[0013] Improve the PPO algorithm by introducing adversarial samples into the state input of the actor layer;

[0014] Step D: Use event-triggered model predictive control method to intervene to resist FDIA attack on CPS.

[0015] As a further improvement, in step B, the CPS state includes perception data, communication state and control state. The CPS state is expressed as follows:

[0016]

[0017] Among them, x(k)∈R represents the state value at time k; y(k)∈R represents the local measurement value at time k; u(k) represents the control input at time k; f(·) represents the nonlinear system dynamics; h(·) represents the nonlinear measurement function; ω(·) represents the process noise; v(·) represents the measurement noise.

[0018] As a further improvement, in step B, the calculation process of the extended Kalman filter is as follows:

[0019]

[0020] in, represents the estimated state at time k; represents the estimated measurement output at time k; P(k) represents the state estimation error covariance matrix at time k; x(k+1|k) represents the prior state estimate of time based on the available information at time k; P(k+1|k) represents the prior estimation error covariance at time k; K(k+1) represents the Kalman gain at time k+1, weighting the difference between the actual measurement value and its expected value; F k The Jacobian matrix representing the state x of the function f evaluated with respect to x(k); Indicates F k The transposed matrix of k+1 The Jacobian matrix representing the function h with respect to the state x evaluated on x(k+1|k); Indicates H k The transposed matrix of ; Q represents process noise; R represents measurement noise;

[0021] The local difference

[0022] As a further improvement, in step C, the PPO algorithm is improved by introducing adversarial samples into the state input of the actor layer. The specific steps are as follows:

[0023] C301. Define the target distribution P(i), which represents the distribution of the desired model output, where P(i) = 1 / a s ,a s Indicates the size of the space;

[0024] C302, perform a single forward propagation on the actor network, input the current CPS state, and obtain the model output Q(i), which represents the probability distribution of the action;

[0025] C303. Calculate the KL divergence between the target distribution P(i) and the model output Q(i):

[0026]

[0027] Among them, D KL (P||Q) is the KL divergence between the target distribution P(i) and the model output Q(i);

[0028] C304, taking KL divergence as loss, calculating the gradient of loss for model parameters through back propagation, and combining the calculated gradient with the original sample to generate adversarial perturbations:

[0029]

[0030] in, is the estimated value of the adversarial sample; s k is an adversarial sample; ε is a small positive number that controls the size of the perturbation; ▽D KL (P||Q) is the gradient of the KL divergence;

[0031] The estimated values ​​of these generated adversarial examples Merge into the actor network’s training dataset so that the model can learn to distinguish between normal and adversarial samples;

[0032] C305. Calculate the advantage function in the critic network of the PPO algorithm;

[0033] The advantage function indicates the action a in the current state k The advantage compared to the baseline provided by the state value estimated by the critic network; in the improved PPO algorithm, the advantage function is used to calculate the loss of updating the participant network strategy to guide the strategy improvement process.

[0034] As a further improvement, the formula of the advantage function is expressed as:

[0035]

[0036] Among them, R k is the cumulative reward starting from the current time step k, calculated by discounting future rewards; V(s k ) is the estimate of the current state obtained from the critic network.

[0037] As a further improvement, the step C further includes: in the actor network, in the current state, by comparing the probability with the difference δ(k) calculated in step B, determining whether the CPS is attacked, and deciding the currently executed action a k ;

[0038] The specific comparison judgment is:

[0039]

[0040] Among them, π(1|s k ) and π(0|s k ) respectively represent the state s k The probability of selecting actions 1 and 0 is the dynamic threshold λ; a k Indicates the action in the current state.

[0041] As a further improvement, in step D, an event-triggered model predictive control method is used to intervene to resist the attack of FDIA on CPS. The specific method is:

[0042] If the improved PPO algorithm is used in step C to select action a k , the action space is {0, 1}. If 1 is selected, the model predictive control method will be started to intervene. The purpose of the intervention optimization process is to make the cost function J MPC minimize.

[0043] As a further improvement, in the step D, during the specific execution process, J MPC The loss term including system status and control command is recorded as:

[0044]

[0045] stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p

[0046] x min ≤x(k+m)≤x max ,1≤m≤p

[0047] u min ≤u(k+m)≤u max ,0≤m≤p-1

[0048] 0≤m≤p-1 (Formula 7);

[0049] Among them, U k ={u(k),u(k+1),...,u(k+p-1)} is the optimal control sequence; X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence; loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m; p is the prediction level.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] (1) The present invention uses deep reinforcement learning (DRL) to introduce a self-learning FDIA detection method that can automatically adjust the detection strategy according to environmental changes and historical data to adapt to various attack modes. Compared with traditional methods, the present invention does not require a predefined threshold, which greatly improves the flexibility and accuracy of detection.

[0052] (2) The present invention proposes an improved PPO algorithm, which enhances the robustness of the DRL algorithm by introducing adversarial samples during the training process. Specifically, the improved PPO algorithm can better cope with the uncertainty and interference in the environment, reduce the occurrence of false positives and false negatives, and improve the security and reliability of the system.

[0053] (3) The present invention introduces an event-triggered model predictive control (MPC) strategy. This strategy can be quickly activated after detecting FDIA, predict and adjust the system, and thus quickly restore the stability of the system. The event-triggered MPC strategy can not only respond to attacks in a timely manner, but also prevent potential threats to a certain extent, ensuring the continuous and stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the single-track vehicle path tracking problem;

[0055] Figure 2 The CPS flow chart under FDI attack;

[0056] Figure 3 is a trace plot of the estimated and measured values;

[0057] Figure 4 This is an experimental comparison diagram of the detector based on the improved PPO algorithm;

[0058] Figure 5 This is an experimental comparison chart of detectors of different deep reinforcement learning algorithms;

[0059] Figure 6 Comparison chart of event triggering frequency under different algorithms, where (a) shows that the system cannot meet the threshold condition for event triggering within a period of time after the completion of FDIA, thus stopping the triggering event; (b) shows that the method based on the improved PPO algorithm keeps triggering events throughout the entire time range. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below in conjunction with the drawings in the preferred embodiments of the present application. In the drawings, the same or similar reference numerals throughout represent the same or similar parts or parts with the same or similar functions. The described embodiments are part of the embodiments of the present application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limitations on the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0061] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0062] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, or it can be an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0063] In the description of the present application, it should be understood that the terms "upper", "lower", "front", "back", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are orientations or positional relationships based on the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0064] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or display that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed but may include other steps or elements not explicitly listed or inherent to such process, method, product, or display.

[0065] The following will be combined Figure 1-6 , a method for security detection and control of a CPS against FDIA involved in an embodiment of the present application is described in detail. It is worth noting that the following embodiments are only used to explain the present application and do not constitute a limitation on the present application.

[0066] Embodiment 1:

[0067] A method for security detection and control of a CPS against FDIA in this embodiment includes the following steps:

[0068] Step 1: CPS environment initialization;

[0069] 1.1. Prepare the physical components required for the CPS environment, including sensors, actuators and other devices.

[0070] 1.2. Initialize the physical components to ensure they are in normal working condition.

[0071] 1.3. Prepare the control system, including control algorithm and execution logic.

[0072] 1.4. Initialize the control system to ensure it is in standby mode and can receive and execute commands.

[0073] 1.5. Ensure that the communication between various physical components in the CPS environment is normal to facilitate subsequent status acquisition and action execution.

[0074] Step 2: Get CPS status;

[0075] 2.1. Obtain the current CPS state and calculate the local measurement value y(k). The CPS state includes the perception data, communication state, and control state. The CPS state is expressed as follows:

[0076]

[0077] Among them, x(k)∈R represents the state value at time k; y(k)∈R represents the local measurement value at time k; u(k) represents the control input at time k; f(·) represents the nonlinear system dynamics; h(·) represents the nonlinear measurement function; ω(·) represents the process noise; v(·) represents the measurement noise.

[0078] 2.2. Collect the local measurement values ​​y(k) calculated in 2.1 and calculate the local estimate through the extended Kalman filter (EKF) Using local measurements y(k) and local estimates Calculate the local difference

[0079] Among them, the calculation process of the extended Kalman filter is as follows:

[0080]

[0081] in, represents the estimated state at time k; represents the estimated measurement output at time k; P(k) represents the state estimation error covariance matrix at time k; x(k+1|k) represents the prior state estimate of time based on the available information at time k; P(k+1|k) represents the prior estimation error covariance at time k; K(k+1) represents the Kalman gain at time k+1, weighting the difference between the actual measurement value and its expected value; F k F represents the Jacobian matrix of the state x evaluated by the function f with respect to x(k); k T Indicates F k The transposed matrix of k+1 The Jacobian matrix representing the function h with respect to the state x evaluated on x(k+1|k); Indicates H k The transposed matrix of ; Q represents process noise; R represents measurement noise;

[0082] Step 3: Perform threshold analysis through deep reinforcement learning based on the improved PPO algorithm, and compare the dynamic threshold λ with the local difference δ(k) calculated in step 2 to determine whether the CPS is attacked.

[0083] Among them, during the training phase of the PPO algorithm, it is sensitive to uncertainty and interference in the environment, and requires stronger robustness. Therefore, the PPO algorithm is improved by introducing adversarial samples in the state input of the actor layer to enhance the robustness of the PPO algorithm. The specific steps are as follows:

[0084] 3.1. Define the target distribution P(i), which represents the distribution of the desired model output, where P(i) = 1 / a s ,a s Indicates the size of the space;

[0085] 3.2. Perform a single forward propagation on the actor network, input the current CPS state, and obtain the model output Q(i), which represents the probability distribution of the action;

[0086] 3.3. Calculate the KL divergence between the target distribution P(i) and the model output Q(i):

[0087]

[0088] Among them, D KL (P||Q) is the KL divergence between the target distribution P(i) and the model output Q(i).

[0089] 3.4. Take KL divergence as loss, calculate the gradient of loss for model parameters through back propagation, and combine the calculated gradient with the original sample to generate adversarial perturbations:

[0090]

[0091] in, is the estimated value of the adversarial sample; s k is an adversarial sample; ε is a small positive number that controls the size of the perturbation; is the gradient of the KL divergence.

[0092] The estimated values ​​of these generated adversarial examples Incorporated into the actor network’s training dataset, the model is able to learn to distinguish between normal and adversarial samples.

[0093] 3.5. Calculate the advantage function in the critic network of the PPO algorithm.

[0094] The advantage function is a measure of the superiority of the current state compared to the baseline, usually represented by the critic network's estimate of the current state value. The formula for the advantage function is:

[0095]

[0096] Among them, R k is the cumulative reward starting from the current time step k, calculated by discounting future rewards; V(s k ) is the estimate of the current state obtained from the critic network.

[0097] The advantage function indicates the action a in the current state k The advantage compared to the baseline provided by the state value estimated by the critic network. A positive advantage means that the action is more advantageous relative to the baseline, while a negative advantage indicates that the action is less advantageous relative to the baseline. In the improved PPO algorithm, the advantage function is used to calculate the loss of updating the participant network policy to guide the policy improvement process.

[0098] 3.6. To determine whether the CPS is under attack, the following comparisons are required:

[0099]

[0100] Among them, π(1|s k ) and π(0|s k ) respectively represent the state s k The probability of selecting actions 1 and 0 is the dynamic threshold λ; a k Indicates the action in the current state.

[0101] In the actor network, the probability value in the current state is compared with the difference δ(k) calculated in step B to determine whether the CPS is attacked and decide the currently executed action a k .

[0102] Step 4: Use event-triggered model predictive control (MPC) method to intervene to resist FDIA attack on CPS. That is, in step C, the improved PPO algorithm is used to select action a k , the action space is {0, 1}, and if 1 is selected, MPC will be activated. In this case, MPC is used as a security control tool to predict threats, impose constraints, and guide the system to defend against FDIA using real-time system data and predictive models.

[0103] In the specific execution process, the controller performs an optimization process to calculate the optimal control command based on the system state and the current time step. The purpose of this optimization process is to make the cost function J MPC Minimize, JMPC The loss term including system status and control command is recorded as:

[0104]

[0105] stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p

[0106] x min ≤x(k+m)≤x max ,1≤m≤p

[0107] u min ≤u(k+m)≤u max ,0≤m≤p-1

[0108] 0≤m≤p-1 (Formula 7);

[0109] Among them, U k ={u(k),u(k+1),...,u(k+p-1)} is the optimal control sequence; X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence; loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m; p is the prediction level.

[0110] Embodiment 2:

[0111] Combined with Figure 1-6 As shown, a CPS security detection and control system to resist FDIA,

[0112] System parameter setting: The given system is a path tracking problem of a single-track vehicle model, such as Figure 1 As shown, represents the rotation angle of the vehicle along the longitudinal axis in the global inertial frame, s represents the lateral position of the center of gravity of the vehicle, y represents the longitudinal position of the center of gravity of the vehicle, r represents the yaw rate, a represents the distance from the center of gravity (CG) to the center of the front axle, and b represents the distance from the center of gravity to the center of the rear axle; F s,i represents the tire forces in the vehicle frame, including F sr and F sf , the formula is: F y,i represents the tire forces in the vehicle frame, including F yr and F yf , the formula is: Among them, the discrete space of i is {f, r}, which represents the front wheel or the rear wheel; v represents the speed, v s represents the velocity at the lateral position of the vehicle's center of gravity; vy represents the velocity at the longitudinal position of the vehicle's center of gravity; δ i Indicates the deviation angle of the front or rear wheels, including α r and α y .

[0113] In the single-track vehicle model, the equations governing the center of gravity and wheel dynamics can be expressed as follows:

[0114]

[0115] Where m is the mass of the vehicle and I is the moment of inertia of the vehicle in the yaw direction.

[0116] Tire pressure in frame F s,i , F y,i Modeling:

[0117]

[0118] Where R is the tire radius, C i represents the tire angular stiffness, μ i Indicates the road surface parameter value, F z,i represents the normal force, T i represents the propulsion / braking torque along the axle, α i represents the slip angle, β i Indicates the offset angle of the front or rear wheels.

[0119] Normal vector F z, i This can be modeled by static load transfer:

[0120]

[0121] Among them, L x,i Indicates the distance from the front or rear wheels.

[0122] Considering the forward steering angle as the control input, and assuming that the longitudinal control is performed by a separate controller, when a single-track vehicle travels along a sinusoidal trajectory, the longitudinal and lateral positions of the vehicle's center of gravity are as follows:

[0123]

[0124] Consider placing this problem in CPS, such as Figure 2 As shown, the system status is The control input is u=[T f ,β f ], the other parameters of the initial state parameters are Q = [0.010.010.010.010.010.01], R = [0.10.1];

[0125] Consider the system attack setting: the attacker conducts FDIA attack on the system, and the attack probability is set to 0.2. FDIA introduces predefined false attacks to the control input sequence, and the attack constant value is set to 5.0.

[0126] Then, the system state is estimated by the extended Kalman filter estimator, as Figure 3 As shown, the estimated state accurately tracks the evolution of the system state over time.

[0127] The loss function of the system is defined as follows:

[0128]

[0129] Among them, J k Indicates the loss value during system operation, J k The larger the value of , the greater the loss of the system. t+k (3) represents the third element of the state vector at time t+k, u t+k Indicates the control command at time t+k.

[0130] When the system detects FDIA, MPC will be triggered. During the optimization process, MPC calculates the optimal control command for the current time step based on the system state and the control command transmitted from the network. The optimization process aims to minimize the cost function J k .

[0131] The system for security detection and control of CPS against FDIA includes: a detector of CPS connected to a sensor and an estimator, the detector, a model predictive controller, an actuator, CPS, the sensor and the estimator form a loop, the detector is also connected to the actuator, the CPS obtains the CPS state, the sensor provides local measurement values ​​of false data injection attacks to the detector, and the estimator provides an estimation value to the detector. The detector is combined with deep reinforcement learning for detection, and the FDIA is controlled by starting the model predictive controller connected to the detector and the actuator to implement the method for security detection and control of CPS against FDIA as described above.

[0132] In the kth time interval, the safety detection and control process can be divided into the following four steps:

[0133] S1: Collect local measurements y(k) and estimates through the Extended Kalman Filter (EKF)

[0134] S2: Through the received local measurement value y(k) and the estimated value Calculate the local difference δ(k) as follows:

[0135] S3: Then compare the threshold λ and the local difference δ(k) to determine whether the CPS is attacked.

[0136] If δ(k)>λ, the model predictive control is activated;

[0137] If δ(k)≤λ, continue monitoring;

[0138] S4: Take control measures to intervene to protect the system from FDIA.

[0139] Among them, S1 and S2 are obtained from the CPS state, S3 is trained using the improved PPO algorithm, the discount factor is γ = 0.99, and the shear parameter is The drop factor of the advantage function is λ = 0.95. The policy network update frequency is set to 1, which means that the network is updated at every time step. We set the total number of events to 500, the step size to 0.2 seconds, and the reward function parameter to ρ = 0.1, which is the prediction time horizon of p = 10.

[0140] Figure 4 This is a comparative experiment of the detector proposed in the present invention. It verifies that the detector proposed in the present invention has improved the performance of CPS. Figure 5 As shown in Figure 2, it is obvious that the detector based on the PPO algorithm, the detector based on the A2C algorithm, and the detector based on the improved PPO algorithm all show better performance than the scene without detection. Figure 5 In the above figure, the detector based on the improved PPO algorithm obviously consumes less cost than the other two detectors, which verifies the effectiveness of the method proposed in the present invention.

[0141] Preferably, in S4, after obtaining the detection result in S3, it is evaluated whether the trigger condition is met. If the action selection is 1, the model predictive controller connected to the detector and the actuator is started, the actuator is connected to the CPS, and the CPS is connected to the sensor and the estimator in turn.

[0142] In the specific execution process, the model predictive controller performs an optimization process to calculate the optimal control command for the current time step based on the system state and the control command delivered from the network. The purpose of this optimization process is to make the cost function J MPC Minimize the loss term including system state and control command, denoted as:

[0143]

[0144] stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p

[0145] x min ≤x(k+m)≤x max,1≤m≤p

[0146] u min ≤u(k+m)≤u max ,0≤m≤p-1

[0147] 0≤m≤p-1 (Formula 13);

[0148] Among them, U k = {u(k),u(k+1),...,u(k+p-1)} is the optimal control sequence, X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence, loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m, and p is the prediction level.

[0149] In practical implementations of MPC, there is often actuation delay, which may be due to communication delay or computational overhead. To address this problem, a buffer is added in front of the actuator. The buffer stores the calculated MPC control sequence and ensures that the control input is applied to the CPS smoothly and in a timely manner. Denoting the buffer as B, it can be expressed as:

[0150] B={u(k),u(k+1),...,u(k+p-1)}.

[0151] Table 1 Average cost function values:

[0152]

[0153] The effect of the present invention can be represented by the average cost loss, as shown in Table 1. The method based on the improved PPO algorithm greatly reduces the computational overhead. Figure 6 In Figure 2, we can observe the frequencies of events triggered by two different methods, namely the traditional threshold-based event triggering method and the improved PPO-based algorithm. Figure 6 As shown in (a) in Figure 1, the system cannot meet the threshold conditions for event triggering for a period of time after the FDIA is completed, thus stopping the event triggering. This is because the threshold-based method is statically set and therefore cannot automatically adjust to dynamic changes in the system. Figure 6 As shown in (b), the method based on the improved PPO algorithm maintains the triggering events in the entire time range to ensure the control performance of the system.

[0154] Although the present invention is described herein with reference to the illustrative embodiments of the present invention, the above embodiments are only preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for security detection and control of CPS against FDIA, characterized in that: include: Step A: CPS environment initialization; Step B: Get CPS status; Get the current CPS state and calculate the local measurement value y(k); Collect the calculated local measurements y(k) and calculate the local estimate through the extended Kalman filter Calculate the local difference δ(k) using the local measurement value and the local estimate value; Step C: Perform threshold analysis through deep reinforcement learning based on the improved PPO algorithm, and compare the dynamic threshold λ with the local difference δ(k) calculated in step B to determine whether the CPS is attacked; Improve the PPO algorithm by introducing adversarial samples into the state input of the actor layer; Step D: Use event-triggered model predictive control method to intervene to resist FDIA attack on CPS.

2. A method for security detection and control of CPS against FDIA according to claim 1, characterized in that: In step B, the CPS state includes perception data, communication state and control state. The CPS state is expressed as follows: Among them, x(k)∈R represents the state value at time k; y(k)∈R represents the local measurement value at time k; u(k) represents the control input at time k; f(·) represents the nonlinear system dynamics; h(·) represents the nonlinear measurement function; ω(·) represents the process noise; v(·) represents the measurement noise.

3. The method for security detection and control of CPS against FDIA according to claim 1, characterized in that: In step B, the calculation process of the extended Kalman filter is as follows: in, represents the estimated state at time k; represents the estimated measurement output at time k; P(k) represents the state estimation error covariance matrix at time k; x(k+1|k) represents the prior state estimate of time based on the available information at time k; P(k+1|k) represents the prior estimation error covariance at time k; K(k+1) represents the Kalman gain at time k+1, weighting the difference between the actual measurement value and its expected value; F k The Jacobian matrix representing the state x of the function f evaluated with respect to x(k); Indicates F k The transposed matrix of k+1 The Jacobian matrix representing the function h with respect to the state x evaluated on x(k+1|k); Indicates H k The transposed matrix of ; Q represents process noise; R represents measurement noise; The local difference 4. The method for security detection and control of CPS against FDIA according to claim 1, characterized in that: In step C, the PPO algorithm is improved by introducing adversarial samples into the state input of the actor layer. The specific steps are as follows: C301. Define the target distribution P(i), which represents the distribution of the desired model output, where P(i) = 1 / a s ,a s Indicates the size of the space; C302, perform a single forward propagation on the actor network, input the current CPS state, and obtain the model output Q(i), which represents the probability distribution of the action; C303. Calculate the KL divergence between the target distribution P(i) and the model output Q(i): Among them, D KL (P||Q) is the KL divergence between the target distribution P(i) and the model output Q(i); C304, taking KL divergence as loss, calculating the gradient of loss for model parameters through back propagation, and combining the calculated gradient with the original sample to generate adversarial perturbations: in, is the estimated value of the adversarial sample; s k is an adversarial sample; ε is a small positive number that controls the size of the perturbation; is the gradient of the KL divergence; The estimated values ​​of these generated adversarial examples Merge into the actor network’s training dataset so that the model can learn to distinguish between normal and adversarial samples; C305. Calculate the advantage function in the critic network of the PPO algorithm; The advantage function indicates the action a in the current state k The advantage compared to the baseline provided by the state value estimated by the critic network; in the improved PPO algorithm, the advantage function is used to calculate the loss of updating the participant network strategy to guide the strategy improvement process.

5. A method for security detection and control of CPS against FDIA according to claim 4, characterized in that: The formula of the advantage function is expressed as: Among them, R k is the cumulative reward starting from the current time step k, calculated by discounting future rewards; V(s k ) is the estimate of the current state obtained from the critic network.

6. A method for security detection and control of CPS against FDIA according to claim 4, characterized in that: The step C also includes: in the actor network, in the current state, by comparing the probability with the difference δ(k) calculated in step B, determining whether the CPS is under attack, and deciding the currently executed action a k ; The specific comparison judgment is: Among them, π(1|s k ) and π(0|s k ) respectively represent the state s k The probability of selecting actions 1 and 0 is the dynamic threshold λ; a k Indicates the action in the current state.

7. A method for security detection and control of CPS against FDIA according to claim 6, characterized in that: In step D, an event-triggered model predictive control method is used to intervene to resist the attack of FDIA on CPS. The specific method is: If the improved PPO algorithm is used in step C to select action a k , the action space is {0, 1}. If 1 is selected, the model predictive control method will be started to intervene. The purpose of the intervention optimization process is to make the cost function J MPC minimize.

8. A method for security detection and control of CPS against FDIA according to claim 7, characterized in that: In the step D, during the specific execution process, J MPC The loss term including system status and control command is recorded as: stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p x min ≤x(k+m)≤x max ,1≤m≤p u min ≤u(k+m)≤u max ,0≤m≤p-1 0≤m≤p-1 (Formula 7); Among them, U k = {u(k),u(k+1),…,u(k+p-1)} is the optimal control sequence; X k ={x(k+1),x(k+2),…,x(k+p)} is the optimal system state sequence; loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m; p is the prediction level.

Images