A security detection and control method for CPS to resist FDIA

Abstract

The present invention discloses a method for security detection and control of a CPS against FDIA, which relates to the fields of deep reinforcement learning and security control technology. The method includes: initializing the CPS environment; acquiring the CPS state; calculating local differences using local measurements and local estimates; performing threshold analysis through deep reinforcement learning based on an improved PPO algorithm, and comparing the dynamic threshold with the calculated local differences to determine whether the CPS is under attack; and intervening using an event-triggered model predictive control method to resist FDIA attacks on the CPS. The present invention can automatically adjust the detection strategy based on environmental changes and historical data to adapt to various attack modes; and enhance the robustness of the DRL algorithm, improving the security and reliability of the system. It can also quickly start after detecting FDIA, predict and adjust the system, and quickly restore the stability of the system.

Description

Technical Field

[0001] The present invention relates to the field of deep reinforcement learning and security control technology, and more specifically, to a security detection and control method for a CPS to resist FDIA. Background Art

[0002] Since the early 2000s, cyber-physical systems (CPSs) have garnered attention for their ability to integrate physical and computational components to perform complex tasks and have been widely applied in fields such as smart grids, transportation systems, and healthcare. However, the interconnectedness of CPS physical components makes them vulnerable to malicious cyberattacks in the perception, communication, and control stages, potentially causing unpredictable economic and material losses. Numerous past cybersecurity incidents have highlighted the importance of CPS security. The threats facing CPSs can be categorized into three main categories: data availability disruption, data integrity compromise, and data confidentiality leakage. To address these security issues, attack detection and control measures against false data injection attacks (FDIAs) have been designed. Various FDIA detection methods have been proposed, but traditional mechanisms, due to their fixed thresholds, struggle to address novel attacks. Numerous studies have introduced reinforcement learning (RL) and deep reinforcement learning (DRL) techniques, leveraging neural networks and learning algorithms to improve the system's adaptability to dynamic attack environments and its ability to identify unknown attack types.

[0003] The paper “Detection of faults and attacks including false data injection attack in smart grid using kalman filter” (K.Manandhar and X.Cao.IEEE Transactions on Control of Network Systems, 1(4):370-379, 2014) designed a Euclidean detector to identify FDIA in smart grids. The paper “A novel false data injection attack detection model of the cyber-physical power system” (J.Cao, D Wang, and Qu.IEEE Access, 8:95109-95125, 2020) proposed a novel FDIA detection model based on ensemble learning in smart grids. The paper “Distributed attack detection and secure estimation of networked cyber-physical systems against false data injection attacks and jamming attacks.” (Y:Guan and X.Ge.IEEE Transactions on Signal and Information Processing over Networks, 4(1):48-59, 2017) independently developed a distributed attack detector and distributed security estimator for networked CPS. Despite this, current research has not fully considered the robustness of deep reinforcement learning algorithms in FDIA scenarios.

[0004] Security defenses for CPSs are also a key aspect, encompassing strategies such as data encryption, vulnerability protection, and error correction. Among these, security control strategies stand out for their ability to rapidly respond and maintain system operation during attacks. They dynamically adjust control inputs through rule-based mechanisms to effectively address both known and unknown attacks. Model predictive control (MPC) is a security control strategy suitable for CPSs that can handle multiple-input, multiple-output systems. However, the high computational requirements for solving constrained optimal control problems present a hurdle for CPSs. The computational burden increases further with increasing system dimensionality and prediction horizon. Summary of the Invention

[0005] In order to address the above-mentioned limitations of the prior art, the purpose of the present invention is to provide a method for security detection and control of CPS against FDIA, which is used to solve the problems that the traditional mechanism of the current FDIA detection method is difficult to cope with new attacks due to the fixed threshold, and the deep reinforcement learning algorithm does not improve its robustness in the FDIA scenario.

[0006] The present invention solves the above problems through the following technical solutions:

[0007] A security detection and control method for CPS to resist FDIA, including:

[0008] Step A: Initialize the CPS environment;

[0009] Step B: Get CPS status;

[0010] Get the current CPS state and calculate the local measurement value y(k);

[0011] Collect the calculated local measurements y(k) and calculate the local estimate through the extended Kalman filter Calculate the local difference δ(k) using the local measurement value and the local estimate value;

[0012] Step C: Perform threshold analysis through deep reinforcement learning based on the improved PPO algorithm and compare the dynamic threshold λ with the local difference δ(k) calculated in step B to determine whether the CPS is under attack;

[0013] Improve the PPO algorithm by introducing adversarial samples into the state input of the actor layer;

[0014] Step D: Use event-triggered model predictive control method to intervene to defend against FDIA attacks on CPS.

[0015] As a further improvement, in step B, the CPS state includes perception data, communication state, and control state. The CPS state is expressed as follows:

[0016]

[0017] where x(k)∈R represents the state value at time k; y(k)∈R represents the local measurement value at time k; u(k) represents the control input at time k; f(·) represents the nonlinear system dynamics; h(·) represents the nonlinear measurement function; ω(·) represents the process noise; and v(·) represents the measurement noise.

[0018] As a further improvement, in step B, the calculation process of the extended Kalman filter is as follows:

[0019]

[0020] in, represents the estimated state at time k; represents the estimated measurement output at time k; P(k) represents the state estimation error covariance matrix at time k; x(k+1|k) represents the prior state estimate based on the time of the available information at time k; P(k+1|k) represents the prior estimation error covariance at time k; K(k+1) represents the Kalman gain at time k+1, which weights the difference between the actual measurement value and its expected value; F k The Jacobian matrix representing the state x evaluated by the function f with respect to x(k); Indicates F k The transposed matrix of k+1 The Jacobian matrix representing the function h with respect to the state x evaluated on x(k+1|k); Indicates H k The transposed matrix of ; Q represents process noise; R represents measurement noise;

[0021] The local differences

[0022] As a further improvement, in step C, the PPO algorithm is improved by introducing adversarial samples into the state input of the actor layer. The specific steps are as follows:

[0023] C301. Define the target distribution P(i), which represents the distribution of the desired model output, where P(i) = 1 / a s ,a s Indicates the size of the space;

[0024] C302, perform a single forward propagation on the actor network, input the current CPS state, and obtain the model output Q(i), which represents the probability distribution over the actions;

[0025] C303. Calculate the KL divergence between the target distribution P(i) and the model output Q(i):

[0026]

[0027] Among them, D KL (P||Q) is the KL divergence between the target distribution P(i) and the model output Q(i);

[0028] C304, taking KL divergence as loss, calculate the gradient of loss to model parameters through back propagation, and combine the calculated gradient with the original sample to generate adversarial perturbations:

[0029]

[0030] in, is the estimated value of the adversarial example; s k is an adversarial sample; ε is a small positive number that controls the size of the perturbation; ▽D KL (P||Q) is the gradient of the KL divergence;

[0031] The estimated values ​​of these generated adversarial samples Incorporated into the actor network’s training dataset, enabling the model to learn to distinguish between normal and adversarial samples;

[0032] C305. Calculate the advantage function in the critic network of the PPO algorithm;

[0033] The advantage function indicates the action a in the current state k The advantage compared to the baseline provided by the state value estimated by the critic network; in the improved PPO algorithm, the advantage function is used to calculate the loss of updating the participant network policy to guide the policy improvement process.

[0034] As a further improvement, the advantage function is expressed as follows:

[0035]

[0036] Among them, R k is the cumulative reward starting from the current time step k, calculated by discounting future rewards; V(s k ) is the estimate of the current state obtained from the critic network.

[0037] As a further improvement, the step C further includes: in the actor network, in the current state, by comparing the probability with the difference δ(k) calculated in step B, determining whether the CPS is attacked, and deciding the currently executed action a k ;

[0038] The specific comparison judgment is:

[0039]

[0040] Among them, π(1|s k ) and π(0|s k ) respectively represent the state s k The probability of selecting action 1 and 0, that is, the dynamic threshold λ; a k Indicates the action in the current state.

[0041] As a further improvement, in step D, an event-triggered model predictive control method is used to intervene to resist the FDIA attack on the CPS. The specific method is as follows:

[0042] If the improved PPO algorithm is used in step C to select action a k , the action space is {0, 1}, if 1 is selected, the model predictive control method will be started to intervene, and the purpose of the intervention optimization process is to make the cost function J MPC minimize.

[0043] As a further improvement, in the step D, during the specific execution process, J MPC The loss term including system status and control command is recorded as:

[0044]

[0045] stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p

[0046] x min ≤x(k+m)≤x max ,1≤m≤p

[0047] u min ≤u(k+m)≤u max ,0≤m≤p-1

[0048] 0≤m≤p-1 (Formula 7);

[0049] Among them, U k = {u(k),u(k+1),...,u(k+p-1)} is the optimal control sequence; X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence; loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m; and p is the prediction level.

[0050] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0051] (1) This paper introduces a self-learning FDIA detection method using deep reinforcement learning (DRL). It can automatically adjust detection strategies based on environmental changes and historical data to adapt to various attack modes. Compared with traditional methods, this paper does not require predefined thresholds, greatly improving the flexibility and accuracy of detection.

[0052] (2) This paper proposes an improved PPO algorithm that enhances the robustness of the DRL algorithm by introducing adversarial samples during the training process. Specifically, the improved PPO algorithm can better cope with uncertainty and interference in the environment, reduce the occurrence of false positives and false negatives, and improve the security and reliability of the system.

[0053] (3) This invention introduces an event-triggered model predictive control (MPC) strategy. This strategy can quickly initiate after detecting an FDIA, predict and adjust the system, and thus quickly restore system stability. The event-triggered MPC strategy not only responds to attacks promptly but also prevents potential threats to a certain extent, ensuring the continued stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the single-track vehicle path tracking problem;

[0055] Figure 2 Flowchart of CPS under FDI attack;

[0056] Figure 3 It is a trace graph of estimated and measured values;

[0057] Figure 4 This is an experimental comparison chart of the detector based on the improved PPO algorithm;

[0058] Figure 5 This is an experimental comparison chart of detectors for different deep reinforcement learning algorithms;

[0059] Figure 6 Comparison chart of event triggering frequency under different algorithms, where (a) shows that the system fails to meet the threshold conditions for event triggering within a period of time after the completion of FDIA, thus stopping the triggering of events; (b) shows that the method based on the improved PPO algorithm keeps triggering events throughout the entire time range. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described in more detail below in conjunction with the drawings in the preferred embodiments of the present application. In the drawings, the same or similar reference numerals throughout represent the same or similar parts or parts with the same or similar functions. The described embodiments are part of the embodiments of the present application, not all of the embodiments. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain the present application, and should not be understood as limitations on the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0061] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0062] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to a fixed connection, an indirect connection via an intermediate medium, internal communication between two components, or an interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0063] In the description of this application, it should be understood that the terms "upper", "lower", "front", "back", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are orientations or positional relationships based on the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on this application.

[0064] In addition, the terms "comprises" and "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or display that comprises a series of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, product, or display.

[0065] The following will be combined Figure 1-6 , a method for security detection and control of a CPS against FDIA involved in an embodiment of the present application is described in detail. It is worth noting that the following embodiments are only used to explain the present application and do not constitute a limitation of the present application.

[0066] Example 1:

[0067] A method for security detection and control of a CPS against FDIA in this embodiment includes the following steps:

[0068] Step 1: CPS environment initialization;

[0069] 1.1. Prepare the physical components required for the CPS environment, including sensors, actuators, and other devices.

[0070] 1.2. Initialize the physical components to ensure they are in normal working condition.

[0071] 1.3. Prepare the control system, including control algorithms and execution logic.

[0072] 1.4. Initialize the control system to ensure it is in standby mode and can receive and execute commands.

[0073] 1.5. Ensure that the communication between various physical components in the CPS environment is normal to facilitate subsequent state acquisition and action execution.

[0074] Step 2: Get CPS status;

[0075] 2.1. Obtain the current CPS state and calculate the local measurement value y(k). The CPS state includes the perception data, communication state, and control state. The CPS state is expressed as follows:

[0076]

[0077] where x(k)∈R represents the state value at time k; y(k)∈R represents the local measurement value at time k; u(k) represents the control input at time k; f(·) represents the nonlinear system dynamics; h(·) represents the nonlinear measurement function; ω(·) represents the process noise; and v(·) represents the measurement noise.

[0078] 2.2. Collect the local measurement values ​​y(k) calculated in 2.1 and calculate the local estimate value through the extended Kalman filter (EKF) Using local measurements y(k) and local estimates Calculate local differences

[0079] Among them, the calculation process of the extended Kalman filter is as follows:

[0080]

[0081] in, represents the estimated state at time k; represents the estimated measurement output at time k; P(k) represents the state estimation error covariance matrix at time k; x(k+1|k) represents the prior state estimate based on the time of the available information at time k; P(k+1|k) represents the prior estimation error covariance at time k; K(k+1) represents the Kalman gain at time k+1, which weights the difference between the actual measurement value and its expected value; F k F represents the Jacobian matrix of the state x evaluated by the function f with respect to x(k); k T Indicates F k The transposed matrix of k+1 The Jacobian matrix representing the function h with respect to the state x evaluated on x(k+1|k); Indicates H k The transposed matrix of ; Q represents process noise; R represents measurement noise;

[0082] Step 3: Perform threshold analysis through deep reinforcement learning based on the improved PPO algorithm, and compare the dynamic threshold λ with the local difference δ(k) calculated in step 2 to determine whether the CPS is attacked.

[0083] During the training phase of the PPO algorithm, it demonstrates sensitivity to environmental uncertainty and interference, requiring stronger robustness. Therefore, the PPO algorithm is improved by introducing adversarial samples into the state input of the actor layer to enhance its robustness. The specific steps are as follows:

[0084] 3.1. Define the target distribution P(i), which represents the distribution of the desired model output, where P(i) = 1 / a s ,a s Indicates the size of the space;

[0085] 3.2. Perform a single forward propagation on the actor network, input the current CPS state, and obtain the model output Q(i), which represents the probability distribution over the actions;

[0086] 3.3. Calculate the KL divergence between the target distribution P(i) and the model output Q(i):

[0087]

[0088] Among them, D KL (P||Q) is the KL divergence between the target distribution P(i) and the model output Q(i).

[0089] 3.4. Take KL divergence as loss, calculate the gradient of loss to model parameters through back propagation, and combine the calculated gradient with the original sample to generate adversarial perturbations:

[0090]

[0091] in, is the estimated value of the adversarial example; s k is an adversarial sample; ε is a small positive number that controls the size of the perturbation; is the gradient of the KL divergence.

[0092] The estimated values ​​of these generated adversarial samples Incorporated into the training dataset of the actor network, the model can learn to distinguish between normal samples and adversarial samples.

[0093] 3.5. Calculate the advantage function in the critic network of the PPO algorithm.

[0094] The advantage function is a measure of the superiority of the current state compared to the baseline, usually represented by the critic network's estimate of the current state value. The formula of the advantage function is expressed as:

[0095]

[0096] Among them, R k is the cumulative reward starting from the current time step k, calculated by discounting future rewards; V(s k ) is the estimate of the current state obtained from the critic network.

[0097] The advantage function indicates the action a in the current state k Advantage compared to the baseline provided by the state value estimated by the critic network. A positive advantage means that the action is more advantageous relative to the baseline, while a negative advantage indicates that the action is less advantageous relative to the baseline. In the improved PPO algorithm, the advantage function is used to calculate the loss for updating the participant network policy, guiding the policy improvement process.

[0098] 3.6. To determine whether the CPS is under attack, the following comparisons are required:

[0099]

[0100] Among them, π(1|s k ) and π(0|s k ) respectively represent the state s k The probability of selecting action 1 and 0, that is, the dynamic threshold λ; a k Indicates the action in the current state.

[0101] In the actor network, the current state is determined by comparing the probability value with the difference δ(k) calculated in step B to determine whether the CPS is attacked and decide the currently executed action a k .

[0102] Step 4: Use event-triggered model predictive control (MPC) method to intervene to resist the attack of FDIA on CPS. That is, in step C, the improved PPO algorithm is used to select action a k , the action space is {0, 1}, and if 1 is selected, MPC will be activated. In this case, MPC acts as a security control tool that uses real-time system data and predictive models to predict threats, impose constraints, and guide the system to defend against FDIA.

[0103] In the specific execution process, the controller performs an optimization process to calculate the optimal control command based on the system state and the current time step. The purpose of this optimization process is to make the cost function J MPC Minimize, JMPC The loss term including system status and control command is recorded as:

[0104]

[0105] stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p

[0106] x min ≤x(k+m)≤x max ,1≤m≤p

[0107] u min ≤u(k+m)≤u max ,0≤m≤p-1

[0108] 0≤m≤p-1 (Formula 7);

[0109] Among them, U k = {u(k),u(k+1),...,u(k+p-1)} is the optimal control sequence; X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence; loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m; and p is the prediction level.

[0110] Example 2:

[0111] Combined with attachment Figure 1-6 As shown, a CPS security detection and control system for resisting FDIA,

[0112] System parameter setting: The given system is a path tracking problem of a single-track vehicle model, such as Figure 1 As shown, represents the rotation angle of the vehicle along the longitudinal axis in the global inertial frame, s represents the lateral position of the vehicle's center of gravity, y represents the longitudinal position of the vehicle's center of gravity, r represents the yaw rate, a represents the distance from the center of gravity (CG) to the center of the front axle, and b represents the distance from the center of gravity to the center of the rear axle; F s,i Represents the tire forces in the vehicle frame, including F sr and F sf , the formula is: F y,i Represents the tire forces in the vehicle frame, including F yr and F yf , the formula is: Among them, the discrete space of i is {f, r}, which represents the front wheel or the rear wheel; v represents the speed, v s represents the velocity at the lateral position of the vehicle's center of gravity; vy represents the velocity at the longitudinal position of the vehicle's center of gravity; δ i Indicates the deviation angle of the front or rear wheels, including α r and α y .

[0113] In the single-track vehicle model, the equations governing the center of gravity and wheel dynamics can be expressed as follows:

[0114]

[0115] Where m is the vehicle mass and I is the vehicle's moment of inertia in the yaw direction.

[0116] Tire pressure F in the frame s,i , F y,i Modeling:

[0117]

[0118] Where R represents the tire radius, C i represents the tire angular stiffness, μ i Indicates the road surface parameter value, F z,i represents the normal force, T i represents the propulsion / braking torque along the axle, α i represents the slip angle, β i Indicates the offset angle of the front or rear wheels.

[0119] Normal vector F z, i This can be modeled by static load transfer:

[0120]

[0121] Among them, L x,i Indicates the distance from the front or rear wheels.

[0122] Considering the forward steering angle as the control input and assuming that the longitudinal control is performed by a separate controller, when a single-track vehicle travels along a sinusoidal trajectory, the longitudinal and lateral positions of the vehicle's center of gravity are as follows:

[0123]

[0124] Consider placing this problem in CPS, such as Figure 2 As shown, the system status is The control input is u=[T f ,β f ], the other parameters of the initial state parameters are Q = [0.010.010.010.010.010.01], R = [0.10.1];

[0125] Consider the system attack setting: the attacker conducts FDIA attack on the system with the attack probability set to 0.2. FDIA introduces predefined false attacks to the control input sequence with the attack constant value set to 5.0.

[0126] Then, the system state is estimated by the extended Kalman filter estimator, as Figure 3 As shown, the estimated state accurately tracks the evolution of the system state over time.

[0127] The loss function of the system is defined as follows:

[0128]

[0129] Among them, J k Indicates the loss value during system operation, J k The larger the value of , the greater the loss of the system. t+k (3) represents the third element of the state vector at time t+k, u t+k Indicates the control command at time t+k.

[0130] When the system detects FDIA, it will trigger MPC. During the optimization process, MPC calculates the optimal control command for the current time step based on the system state and the control command transmitted from the network. The optimization process aims to minimize the cost function J k .

[0131] The system for security detection and control of a CPS against FDIA includes: a detector of the CPS connected to a sensor and an estimator, the detector, a model predictive controller, an actuator, the CPS, the sensor and the estimator forming a loop, the detector also connected to the actuator, the CPS obtaining the CPS state, the sensor providing the detector with local measurement values ​​of false data injection attacks, the estimator providing the detector with estimation values, the detector is combined with deep reinforcement learning for detection, and the FDIA is controlled by starting the model predictive controller connected to the detector and the actuator, so as to realize the method for security detection and control of a CPS against FDIA as described above.

[0132] In the kth time interval, the security detection and control process can be divided into the following four steps:

[0133] S1: Collect local measurements y(k) and estimates through the Extended Kalman Filter (EKF)

[0134] S2: By receiving the local measurement value y(k) and the estimated value Calculate the local difference δ(k) using the following formula:

[0135] S3: Then compare the threshold λ and the local difference δ(k) to determine whether the CPS is attacked.

[0136] If δ(k)>λ, then the model predictive control is activated;

[0137] If δ(k)≤λ, then continue monitoring;

[0138] S4: Take control measures to intervene to protect the system from FDIA.

[0139] Among them, S1 and S2 are obtained from the CPS state, and S3 is trained using the improved PPO algorithm with a discount factor of γ = 0.99 and a shear parameter of The advantage function is reduced by a factor of λ = 0.95. The policy network update frequency is set to 1, indicating that the network is updated at every time step. We set the total number of events to 500, the step size to 0.2 seconds, and the reward function parameter to ρ = 0.1, which is the prediction time horizon of p = 10.

[0140] Figure 4 This is a comparative experiment of the detector proposed in this invention. It verifies that the performance of the detector proposed in this invention is improved for CPS. Figure 5 As shown in Figure 2, it is obvious that the detector based on the PPO algorithm, the detector based on the A2C algorithm and the detector based on the improved PPO algorithm all show better performance than the scene without detection. Figure 5 In the results, the detector based on the improved PPO algorithm consumes significantly less cost than the other two detectors, which verifies the effectiveness of the method proposed in this paper.

[0141] Preferably, in S4, after obtaining the detection result in S3, it is evaluated whether the trigger condition is met. If the action selection is 1, the model predictive controller connected to the detector and the actuator is started, the actuator is connected to the CPS, and the CPS is connected to the sensor and the estimator in turn.

[0142] In the specific execution process, the model predictive controller performs an optimization process to calculate the optimal control command for the current time step based on the system state and the control command passed from the network. The purpose of this optimization process is to make the cost function J MPC Minimize the loss term including system state and control command, denoted as:

[0143]

[0144] stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p

[0145] x min ≤x(k+m)≤x max,1≤m≤p

[0146] u min ≤u(k+m)≤u max ,0≤m≤p-1

[0147] 0≤m≤p-1 (Formula 13);

[0148] Among them, U k = {u(k),u(k+1),...,u(k+p-1)} is the optimal control sequence, X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence, loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m, and p is the prediction level.

[0149] In practical implementations of MPC, there is often actuation delay, which can be due to communication delays or computational overhead. To address this issue, a buffer is added before the actuator. This buffer stores the calculated MPC control sequence and ensures that the control input is applied to the CPS smoothly and promptly. Denoting the buffer as B, it can be expressed as:

[0150] B={u(k),u(k+1),...,u(k+p-1)}.

[0151] Table 1 Average cost function values:

[0152]

[0153] The effect of the present invention can be represented by the average cost loss, as shown in Table 1. The method based on the improved PPO algorithm greatly reduces the computational overhead. Figure 6 In the figure, we can observe the frequencies of events triggered by two different methods, namely the traditional threshold-based event triggering method and the improved PPO-based algorithm. Figure 6 As shown in (a) in Figure 1, the system cannot meet the threshold conditions for event triggering for a period of time after the FDIA is completed, thus stopping the triggering of events. This is because the threshold-based method is statically set and therefore cannot automatically adjust to dynamic changes in the system. Figure 6 As shown in (b), the method based on the improved PPO algorithm maintains the triggering events in the entire time range to ensure the control performance of the system.

[0154] Although the present invention is described herein with reference to illustrative embodiments of the present invention, the above embodiments are merely preferred embodiments of the present invention, and the embodiments of the present invention are not limited to the above embodiments. It should be understood that those skilled in the art can design many other modifications and implementations, which will fall within the scope and spirit of the principles disclosed in this application.

Claims

1. A method for security detection and control of CPS against FDIA, characterized in that: include: Step A: Initialize the CPS environment; Step B: Get CPS status; Get the current CPS state and calculate the local measurement value y(k); Collect the calculated local measurements y(k) and calculate the local estimate through the extended Kalman filter Calculate the local difference δ(k) using the local measurement value and the local estimate value; Step C: Perform threshold analysis through deep reinforcement learning based on the improved PPO algorithm and compare the dynamic threshold λ with the local difference δ(k) calculated in step B to determine whether the CPS is under attack; In step C, the PPO algorithm is improved by introducing adversarial samples into the state input of the actor layer. The specific steps are as follows: C301. Define the target distribution P(i), which represents the distribution of the desired model output, where P(i) = 1 / a s ,a s Indicates the size of the space; C302. Perform a single forward propagation on the actor network, input the current CPS state, and obtain the model output Q(i), which represents the probability distribution over the actions; also includes: in the actor network, in the current state, by comparing the probability with the difference δ(k) calculated in step B, determining whether the CPS is under attack and deciding the current action a k ; C303. Calculate the KL divergence between the target distribution P(i) and the model output Q(i): Among them, D KL (P||Q) is the KL divergence between the target distribution P(i) and the model output Q(i); C304, taking KL divergence as loss, calculate the gradient of loss to model parameters through back propagation, and combine the calculated gradient with the original sample to generate adversarial perturbations: in, is the estimated value of the adversarial example; s k is an adversarial sample; ε is a small positive number that controls the size of the perturbation; is the gradient of the KL divergence; The estimated values ​​of these generated adversarial samples Incorporated into the actor network’s training dataset, enabling the model to learn to distinguish between normal and adversarial samples; C305. Calculate the advantage function in the critic network of the PPO algorithm; The advantage function indicates the action a in the current state k The advantage compared to the baseline provided by the state value estimated by the critic network; in the improved PPO algorithm, the advantage function is used to calculate the loss of updating the participant network policy and guide the policy improvement process; Step D: Use event-triggered model predictive control method to intervene to defend against FDIA attacks on CPS.

2. A method for security detection and control of CPS against FDIA according to claim 1, characterized in that: In step B, the CPS state includes the perception data, communication state, and control state. The CPS state is expressed as follows: where x(k)∈R represents the state value at time k; y(k)∈R represents the local measurement value at time k; u(k) represents the control input at time k; f(·) represents the nonlinear system dynamics; h(·) represents the nonlinear measurement function; ω(·) represents the process noise; and v(·) represents the measurement noise.

3. A method for security detection and control of CPS against FDIA according to claim 1 or 2, characterized in that: In step B, the calculation process of the extended Kalman filter is as follows: Where x(k)∈R represents the state value at time k; u(k) represents the control input at time k; represents the estimated state at time k; represents the estimated measurement output at time k; P(k) represents the state estimation error covariance matrix at time k; x(k+1|k) represents the prior state estimate based on the time of the available information at time k; P(k+1|k) represents the prior estimation error covariance at time k; K(k+1) represents the Kalman gain at time k+1, which weights the difference between the actual measurement value and its expected value; F k The Jacobian matrix representing the state x evaluated by the function f with respect to x(k); Indicates F k The transposed matrix of k+1 The Jacobian matrix representing the function h with respect to the state x evaluated on x(k+1|k); Indicates H k The transposed matrix of ; Q represents process noise; R represents measurement noise; The local differences 4. A method for security detection and control of CPS against FDIA according to claim 1 or 2, characterized in that: The advantage function is expressed as follows: Among them, R k is the cumulative reward starting from the current time step k, calculated by discounting future rewards; V(s k ) is the estimate of the current state obtained from the critic network.

5. A method for security detection and control of CPS against FDIA according to claim 1 or 2, characterized in that: In the actor network, the current state is determined by comparing the probability with the difference δ(k) calculated in step B to determine whether the CPS is attacked and decide the current action a k ; The specific comparison judgment is: Among them, π(0|s k ) means in state s k The probability of selecting action 0 is the dynamic threshold λ.

6. The method for security detection and control of CPS against FDIA according to claim 5, characterized in that: In step D, an event-triggered model predictive control method is used to intervene to resist the FDIA attack on the CPS. The specific method is as follows: If the improved PPO algorithm is used in step C to select action a k , the action space is {0, 1}, if 1 is selected, the model predictive control method will be started to intervene, and the purpose of the intervention optimization process is to make the cost function J MPC minimize.

7. The method for security detection and control of CPS against FDIA according to claim 6, characterized in that: In the step D, during the specific execution process, J MPC The loss term including system status and control commands is recorded as: stx(k+m)=f(x(k+m-1),u(k+m-1)),1≤k≤p x min ≤x(k+m)≤x max ,1≤m≤p u min ≤u(k+m)≤u max ,0≤m≤p-1 0≤m≤p-1 (Formula 7); Among them, U k = {u(k),u(k+1),…,u(k+p-1)} is the optimal control sequence; X k ={x(k+1),x(k+2),...,x(k+p)} is the optimal system state sequence; loss(x(k+m),u(k+m)) is the stage loss function, where x(k+m) is the system state at time step k+m, u(k+m) is the control command at time step k+m; p is the prediction level; x(k)∈R represents the state value at time k; u(k) represents the control input at time k; and f(·) represents the nonlinear system dynamics.

Images