Industrial control intrusion response security decision-making method based on security reinforcement learning

Abstract

The invention discloses an industrial control intrusion response security decision-making method based on security reinforcement learning, and the method comprises the steps: 1, building an industrial control system simulation environment, collecting the network layer alarm data of an industrial control system and the operation data of a physical layer sensor and an actuator, synchronously recording the state transition, the reward value and the security cost triggered by a decision-making action, and carrying out the simulation of the industrial control system; and forming attack and defense interaction trajectory data. And 2, analyzing the risk level of a response strategy and learning a better intrusion response strategy according to the trajectory data collected in the step 1 by adopting a hierarchical strategy optimization algorithm, and finally obtaining a safe and effective strategy. And 3, in combination with interaction data and evaluation indexes of simulation operation, counting core indexes such as defense success rates and safety costs of different models, and comprehensively measuring performance differences of response models. Test results prove that the method can effectively resist industrial control network attacks on the premise of maintaining low security risk cost, and collaborative optimization of security cost control and attack defense efficiency is realized.

Description

Technical Field

[0001] This invention belongs to the field of industrial control system security defense technology, and relates to network attack and defense game model construction technology and network intrusion response decision technology. Specifically, it relates to an industrial control intrusion response security decision method based on security reinforcement learning. Background Technology

[0002] Industrial Control Systems (ICS) are widely used in critical infrastructure such as power grids, nuclear facilities, and chemical plants. With the growth of network connectivity, the cybersecurity risks associated with ICS are also continuously escalating. Strengthening the cybersecurity of ICS is crucial to ensuring the stable operation of critical infrastructure. Intrusion Response Systems (IRS), as a core component of ICS security, dynamically execute response strategies to block attack chains and restore the system to a steady state after an intrusion detection system triggers an alarm. However, the physical-information cross-layer coupling characteristics of ICS mean that inappropriate responses can lead to consequences more severe than the attack itself. For example, delayed disconnection of faulty lines in a power grid can cause voltage collapse, and accidental emergency shutdowns in chemical processes can lead to runaway chemical reactions. Such security incidents significantly reduce the feasibility of deploying IRS in industrial scenarios. Therefore, designing and implementing secure and effective IRS is essential for improving the cybersecurity protection capabilities of safety-critical industrial control systems.

[0003] Current research on intrusion response systems in industrial control systems mainly focuses on optimizing cross-layer intrusion response decision models, aiming to select the cross-layer response strategy with optimal defense effectiveness. Current research on intrusion response in industrial control systems primarily includes methods based on stochastic game models and deep reinforcement learning. Methods based on stochastic game models analyze the strategy game between attackers and defenders through formal modeling, but their quantification of the impact on physical processes is too one-sided. While methods based on Deep Reinforcement Learning (DRL) can integrate real-time data to enhance scenario adaptability, they mainly focus on attack blocking during optimization, lacking consideration of the security boundaries of physical processes. To address optimization problems under security risk constraints, Safe Reinforcement Learning (Safe RL) has been proposed, aiming to maximize cumulative rewards while satisfying security constraints.

[0004] However, existing security reinforcement learning optimization methods still struggle to effectively address cross-layer intrusion response decision optimization problems, specifically due to two key issues: First, they neglect cross-layer security risk factors. Existing methods primarily focus on network layer attack characteristics, lacking effective modeling of physical layer security risks such as sensor overreach, leading to incomplete system security posture assessments. Simultaneously, most methods rely on static cost or real-time feedback optimization, making them ill-suited to the dynamic characteristics of cross-layer interactions in industrial control systems. For example, when the system state experiences brief fluctuations, the agent is prone to misjudging the security status. Second, the cross-layer nature of industrial control systems necessitates that intelligent response strategies handle different levels of danger, while existing methods typically employ a uniform constraint architecture, failing to dynamically adjust according to risk changes. Overly strong constraints compress the policy space and reduce defense effectiveness, while insufficient constraints may lead to security risks. Summary of the Invention

[0005] This method adapts to the multi-level dynamic changes in risk during the intrusion response optimization process of industrial control systems through a hierarchical strategy optimization mechanism, thereby realizing an intrusion response system that combines network security defense effectiveness with security risk management capabilities.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A security decision-making method for industrial control system intrusion response based on security reinforcement learning is presented, and the specific process is as follows:

[0008] Step 1: Build an industrial control system simulation environment, collect alarm data from the network layer of the industrial control system and operating data from the physical layer sensors and actuators, and synchronously record the state transitions, reward values, and security costs triggered by decision actions to form attack and defense interaction trajectory data.

[0009] Step 2: Using a hierarchical strategy optimization algorithm, based on the trajectory data collected in Step 1, the risk level of the response strategy is analyzed and a better intrusion response strategy is learned, ultimately resulting in a safe and effective strategy.

[0010] Step 3: Combine the interactive data and evaluation indicators from the simulation operation to statistically analyze the defense success rate, security cost, and other core indicators of different models, and comprehensively measure the performance differences of each response model.

[0011] An intrusion response system for industrial control systems based on security reinforcement learning includes:

[0012] The industrial control environment modeling module based on Constrained Markov Decision Process (CMDP) is used to construct an industrial control simulation environment. It collects network layer node state information and physical layer device perception data and integrates them into state input. The model outputs response actions based on this state, and simultaneously calculates the reward value and security cost value corresponding to the actions. It provides a real-time interactive interface for the decision model to obtain attack and defense trajectory data.

[0013] The trust domain-based intrusion response security policy optimization module is used to iteratively optimize the response model of the industrial control system. First, it collects trajectory data by interacting with the industrial control environment through the policy model. Then, it calculates the expected reward and cost based on the trajectory data and updates the reward and cost network parameters. Subsequently, it determines the policy risk level through risk factors and adapts the corresponding policy network update rules. The above process is repeated until the number of training rounds reaches the target. Finally, the best-performing policy network is saved as the decision response model.

[0014] The multi-index model evaluation module is used to compare the performance of different decision-making methods. First, it calculates the security risk cost of each decision-making method's response strategy; then, it calculates its defense capabilities; subsequently, it calculates the deviation of the sensor perception values ​​corresponding to each strategy; finally, it synthesizes the above indicators to calculate the overall performance of different decision-making methods' response strategies. These indicators are used to analyze the differences in response performance between different models.

[0015] The key technical point of this invention is:

[0016] I. A modeling method for industrial control environments based on CMDP is proposed, which integrates network layer and physical layer data to construct a state space, synchronously quantifies the rewards and security costs of response actions, and realizes accurate collection of attack and defense interaction trajectories in industrial control scenarios.

[0017] Second, a hierarchical strategy optimization mechanism is proposed, which combines the trust domain algorithm and risk level adaptation rules to dynamically balance defense effectiveness and security risks during iterative training, thereby achieving risk-aware updates of response strategies.

[0018] Third, a multi-dimensional model evaluation system is proposed to quantify response performance from dimensions such as security cost, defense capability, and physical equipment impact, so as to realize the full-scenario difference analysis of different decision-making methods.

[0019] Compared with the disclosed related technologies, the method implemented using this patent has the following advantages:

[0020] I. This invention constructs a unified state space and joint action space for industrial control systems across layers based on CMDP, and introduces a security cost function to dynamically quantify risks related to sensor over-limit risks, thereby achieving comprehensive perception and accurate assessment of the security status of industrial control systems and effectively reducing the security risks brought about by intrusion response strategies.

[0021] Second, by introducing a dual-constraint framework of security cost and trust domain, this invention deeply embeds risk assessment into the strategy optimization objective, forming a hierarchical optimization mechanism, which effectively ensures that network attacks are effectively eliminated while maintaining low security risk. Attached Figure Description

[0022] Figure 1This is a schematic diagram of the security decision-making process for industrial control system intrusion response based on security reinforcement learning, as described in this invention.

[0023] Figure 2 This is a flowchart illustrating the process of constructing an industrial control simulation environment and interacting with a decision-making model in accordance with the present invention.

[0024] Figure 3 This is a schematic diagram illustrating the iterative training of a response strategy based on hierarchical risk perception in accordance with the present invention.

[0025] Figure 4 This is a schematic diagram illustrating a multi-dimensional evaluation process for implementing different decision-making methods and response strategies according to the present invention.

[0026] Figure 5 The figure shows the experimental results of comparing the effectiveness of this invention with other reinforcement learning methods in Embodiment 2 of the present invention;

[0027] Figure 6 This is an experimental result diagram comparing the effectiveness of the second embodiment of the present invention with other intrusion response modeling methods; Detailed Implementation

[0028] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0029] Example 1

[0030] like Figure 1 As shown, this invention provides a security decision-making method for industrial control system intrusion response based on security reinforcement learning. This method consists of three core steps: constructing an industrial control system simulation environment that integrates the network and physical layers, and collecting cross-layer state data and attack-defense interaction trajectories; based on the collected trajectory data, training and generating a secure and effective intrusion response strategy through a hierarchical strategy optimization algorithm; and using multi-dimensional evaluation indicators to quantitatively evaluate the strategy performance and analyze its differences from existing methods and their causes. The specific details are as follows:

[0031] Step 1: Build an industrial control system simulation environment, collect alarm data from the network layer of the industrial control system and operating data from the physical layer sensors and actuators, and synchronously record the state transitions, reward values, and security costs triggered by decision actions to form attack and defense interaction trajectory data.

[0032] Step 2: Using a hierarchical strategy optimization algorithm, based on the trajectory data collected in Step 1, the risk level of the response strategy is analyzed and a better intrusion response strategy is learned, ultimately resulting in a safe and effective strategy.

[0033] Step 3: Combine the interactive data and evaluation indicators from the simulation operation to statistically analyze the defense success rate, security cost, and other core indicators of different models, and comprehensively measure the performance differences of each response model.

[0034] like Figure 2 As shown, in step 1 above, an industrial control system simulation environment is built, and alarm data of the network layer and operation data of the physical layer sensors and actuators of the industrial control system are collected. The state transitions, reward values ​​and security costs triggered by decision actions are recorded simultaneously to form attack and defense interaction trajectory data.

[0035] Step 11: Construct the state space and action space of the response decision model based on the characteristics of the industrial control system. The state space is formed by collecting real-time intrusion status information of network nodes in the industrial control environment and simultaneously acquiring sensor and actuator perception data, fusing the two types of information to form a unified state s. Based on the cross-layer coupling characteristics of the industrial control environment, the action space is composed of the defense strategy of network nodes and the recovery strategy of physical devices. The response model outputs the corresponding response action a based on the real-time state of the industrial control environment.

[0036] Step 12: Based on the real-time state information fed back from the industrial control simulation environment, the decision model outputs the corresponding response action and simultaneously calculates the reward value and safety cost generated by this action. The model uses the real-time state information s obtained at the current time t. t Output the corresponding response action a t Next, based on this information, the reward value r for the current action is calculated. t and safety cost value c t and action a t The state information s of the next moment is obtained by feeding it back into the industrial control simulation system. t+1 .

[0037] The formulas for calculating the reward value r and the safety cost value c are as follows:

[0038] (1)

[0039] Formula (1) The reward function consists of two parts: network layer reward and physical layer reward, which are calculated from the system state of the network node and the sensor measurement value, respectively.

[0040] (2)

[0041] Formula (2) calculates the degree of danger of the system state of a network node. Calculation parameters This refers to the weight of the network layer reward. This refers to the set of alerts for damaged network nodes at the current moment. This refers to the location of the damaged network node. This refers to the severity of the alert for damaged network nodes. This indicates the weight of the network layer reward.

[0042] (3)

[0043] Formula (3) calculates the sensor's deviation value. Parameters This refers to the weight of the network layer reward. Refers to the current sensor reading. Refers to the standard baseline perceived value. Refers to the baseline standard deviation. This indicates the weight of the physical layer reward. This refers to a collection of sensors. This means that a value of 1 is set when the system crashes and stops running prematurely, and 0 is set otherwise.

[0044] (4)

[0045] Formula (4) The safety cost function consists of two parts: the penalty for violating constraints and the safety risk assessment value.

[0046] (5)

[0047] Formula (5) defines the penalty for violating the constraint as the degree to which the sensor-perceived value exceeds the safety boundary, and the penalty is given according to the degree of exceedance. This refers to the time required for strategy execution to resume. A collection of sensors. For sensor safety threshold, The value sampled by the sensor at time t. This indicates whether the sensor has exceeded the safety boundary; 1 indicates exceeding the safety boundary, and 0 indicates not exceeding the safety boundary.

[0048] (6)

[0049] Formula (6) represents the safety risk. This refers to the overshoot of the sensor.

[0050] Step 13: Record and save the current state, response action, action reward value, cost value, and next moment state as a quintuple into the trajectory data. The decision model continuously interacts with the industrial control simulation environment until sufficient trajectory data is collected, and then input into the model for training.

[0051] like Figure 3 As shown, step 2 above uses a hierarchical strategy optimization algorithm to analyze the risk level of the response strategy based on the trajectory data collected in step 1 and learn a better intrusion response strategy, ultimately obtaining a safe and effective strategy.

[0052] Step 21: Collect trajectory data through interactive training between the response strategy generated by the response model and the industrial control environment: Based on the real-time interaction between the current training strategy network (i.e., the response model under learning) and the industrial control simulation environment, interactive trajectory data is generated. Continue collecting samples until a sufficient number of samples have been accumulated.

[0053] Step 22: Calculate the expected reward and cost of each state based on the trajectory data: Extract the immediate reward and safety cost corresponding to each state from the interaction trajectory data; derive the long-term expected reward of the state using the Generalized Advantage Estimation (GAE) method. and long-term expected costs This is used for subsequent optimization and updates of the reward and cost network parameters. The specific target value calculation formula is as follows:

[0054] (6)

[0055] (7)

[0056] in Let be the reward state value function, representing the long-term cumulative reward starting from state s and following the current policy. Let S be the cost state value function, representing the long-term cumulative cost of starting from state s and following the current strategy; Let be the reward advantage function for state s. Let be the cost advantage function for state s.

[0057] Step 23: Gradient backpropagation updates the weight parameters of the reward and cost networks respectively. Using the expected reward and expected cost obtained in Step 2 as target values, mean squared error (MSE) is applied as the loss function to both the reward and cost networks. The gradient backpropagation algorithm is then used to update the weight parameters of the reward and cost networks. The specific formulas are as follows:

[0058] (8)

[0059] (9)

[0060] Formulas (8) and (9) are the reward and cost network update formulas, respectively, where This refers to the response policy learned by the current policy network.

[0061] Step 24: Calculate the cumulative reward value, cumulative cost value, and risk factor, and determine the current strategy's risk level: Calculate the cumulative reward value, cumulative cost, and risk factor of the current strategy in the trajectory data, and divide the strategy into four levels: initial stage, no risk, medium risk, and high risk. The specific formula is as follows:

[0062] (10)

[0063] (11)

[0064] (12)

[0065] Formula (10) represents the evaluation factor related to the cumulative reward value, which is used to determine whether the constraints are ignored in the early stage; The discount factor is used to weigh current and future returns. Instant reward function; The cumulative reward value under trajectory data. Formula (11) represents the evaluation factor related to the cumulative cost value, which determines whether the strategy is within the safe cost domain; State In performing the action The instantaneous cost function at that time; The cumulative reward value under trajectory data, where d refers to the upper limit threshold of safety cost. Defines the highest acceptable level of safety cost. Formula (12) is a strategy risk assessment factor, which measures whether the current strategy is simultaneously constrained by the trust domain and the safety cost domain; The policy gradient corresponding to the cost function; This refers to the second-order matrix of the policy parameters. It calculates the KL (Kullback-Leibler) divergence between the target policy and the current policy. yes The result of the inverse operation of a matrix; This represents the constraint threshold.

[0066] The risk level is further determined using the following formula, which is as follows:

[0067] a) Initial stage:

[0068] b) No risk: and and

[0069] c) Medium risk: and

[0070] d) High risk: and and

[0071] Step 25: Based on the risk level of the strategy generated by the response model, select an appropriate objective to update the policy network: If the current policy is in the initial stage or has no risk, adopt planning without security constraints to update the policy with the objective of maximizing cumulative reward; if the current policy has a medium risk, enable dual-constraint objectives of security cost and KL divergence to control security cost while avoiding policy mutation; if the current policy has a high risk, prioritize the rule of minimizing security cost to ensure the safety of physical devices. The specific formula is as follows:

[0072] (13)

[0073] (14)

[0074] (15)

[0075] in This refers to the current policy network parameters. This refers to the parameters for the next round of updates; The policy gradient corresponding to the reward function; and It is a problem of solving optimization. The Lagrange multiplier of the optimal solution obtained by enabling the dual constraints of safety cost and KL divergence.

[0076] Step 26: Save the best-performing policy network during training as the decision response model: Determine whether the current training round has reached the preset value. If not, return to step 21 and continue to interact with the industrial control environment to collect new trajectory data, repeating the above process. If the target has been reached, stop the loop and save the best-performing policy network during training as the final intrusion response strategy.

[0077] like Figure 4 As shown, step 3 above combines the interactive data and evaluation indicators from the simulation operation to statistically analyze the core indicators such as the defense success rate and security cost of different models, and comprehensively measures the performance differences of each response model.

[0078] Step 31, calculate the security risk cost of different decision-making methods and response strategies: as shown in formula (16), SC represents the cumulative response security cost of each round of testing, where This refers to the total number of tests. This metric measures the security cost of the response strategy employed; a lower SC value indicates a more secure response strategy. The specific formula is as follows:

[0079] (16)

[0080] Step 32, statistically analyze the defense capabilities of different decision-making methods and response strategies: as shown in formula (17), ACR represents the proportion of attacks successfully cleared by the system. To successfully clear the number of attacks, The ACR (Attack Response Code) represents the total number of attacks. It assesses the effectiveness of the system's response strategies, with a higher ACR value indicating that the system can withstand more attacks. (17)

[0081] Step 33: Calculate the deviation of sensor perceived values ​​for different decision-making methods and response strategies. As shown in formula (18), PDS measures the system's sensitivity to fluctuations in perceived values, i.e., the sum of perceived deviations throughout all testing processes. Specifically, perceived deviation... ,in The current perceived value, and These are the expected value and the standard deviation, respectively. This refers to the sampling time step during testing. A lower PDS value indicates a lower overall fluctuation in the sensor's performance. The specific formula is as follows:

[0082] (18)

[0083] Step 34: Statistically analyze the overall performance of different decision-making methods and response strategies: As shown in formula (19), TR represents the total reward value obtained in each round of testing. This indicator reflects the overall performance of the response strategy; a higher TR indicates a better execution effect of the response strategy. The specific formula is as follows:

[0084] (19)

[0085] Based on the above method, the technical solution of the present invention also includes an intrusion response system for industrial control systems based on security reinforcement learning, comprising:

[0086] The CMDP-based industrial control environment modeling module is used to construct an industrial control simulation environment. It collects network layer node status information and physical layer device perception data and integrates them into status input. The model outputs response actions based on this status, and simultaneously calculates the reward value and security cost value corresponding to the actions. It provides a real-time interactive interface for the decision-making model to obtain attack and defense trajectory data.

[0087] The trust domain-based intrusion response security policy optimization module is used to iteratively optimize the response model of the industrial control system. First, it collects trajectory data by interacting with the industrial control environment through the policy model. Then, it calculates the expected reward and cost based on the trajectory data and updates the reward and cost network parameters. Subsequently, it determines the policy risk level through risk factors and adapts the corresponding policy network update rules. The above process is repeated until the number of training rounds reaches the target. Finally, the best-performing policy network is saved as the decision response model.

[0088] The multi-index model evaluation module is used to compare the performance of different decision-making methods. First, it calculates the security risk cost of each decision-making method's response strategy; then, it calculates its defense capabilities; subsequently, it calculates the deviation of the sensor perception values ​​corresponding to each strategy; finally, it synthesizes the above indicators to calculate the overall performance of different decision-making methods' response strategies. These indicators are used to analyze the differences in response performance between different models.

[0089] Example 2

[0090] Example 2 verifies the invention using an Eastman Chemical (TE) chemical process control system in Tennessee. The industrial control model mainly consists of a network layer and a physical layer. The network layer comprises two subnets: an enterprise network and a supervisory control network. The enterprise network consists of two production management (PM) devices and a web server, totaling three network nodes; the supervisory control network includes a human-machine interface (HMI), an operator station (OS), and an engineer station (ES)—three network nodes. The physical layer consists of the TE process, containing key components such as actuators and sensors. The focus is on three sensors—reactor pressure, separator level, and stripper level—and four feed valves: D, E, overall, and reactor. The experiments of this invention target attacks on the physical layer.

[0091] For experimental evaluation, the following four metrics are primarily used to assess the model's performance: Security Cost (SC), Attack Removal Rate (ACR), Perceived Deviation (PDS), and Overall Average Reward (TR). The specific formulas are as follows:

[0092]

[0093]

[0094]

[0095]

[0096] To evaluate the intrusion response strategy described above, each experiment ran for 100 time steps. To reduce the impact of environmental randomness in the evaluation, 10 independent experiments were performed in each round of evaluation, and the average value was recorded as the performance indicator for that round. The entire evaluation process was repeated three times, using a retrained agent each time. Finally, the standard deviation of the experiments was calculated. The mean and standard deviation are used to reflect the defensive performance and stability of the intrusion response strategy. Furthermore, considering that attacks may be blocked at the network layer, preventing them from penetrating the physical control process, to further evaluate the effectiveness of the IRS in physical layer defense, this paper designed two attack scenarios. By adjusting the number of direct injections into the physical control process per round, scenario 1 involved 3 direct injections into the physical process per round, while scenario 2 involved 5 such injections per round. In this way, the defensive performance of the IRS under different attack scenarios was compared and analyzed.

[0097] To verify the effectiveness of the hierarchical policy optimization algorithm of this invention, we conducted comparative experiments with five mainstream reinforcement learning methods (PPO-SC, PPO-Lag, SAC, SAC-SC, and CPO) under two attack scenarios. For each method, three independent repeated runs were performed to ensure statistical reliability, and the results of four core metrics, SC, ACR, PDS, and TR, were collected. Figure 5 Quantitative results for all four indicators are provided.

[0098] SC quantifies the security risk of the system, with smaller values ​​corresponding to superior security performance. In Scenario 1, our method's security risk quantification value is close to 0, lower than all other reinforcement learning baseline methods. This result demonstrates that our method effectively constrains security risk under low-intensity attack conditions. In Scenario 2, the security risk quantification values ​​of all methods increase (consistent with the increase in attack intensity), but our method still maintains a low security risk quantification value (0.3), while the security risk quantification values ​​of other baseline methods increase significantly. The advantage of our method in terms of security performance is attributed to the hierarchical policy optimization mechanism, which improves the ability to handle security cost constraints, enabling the system to maintain secure operation under different attack intensities. In summary, by comparing the SC values ​​of different reinforcement learning methods, our method demonstrates its ability to ensure system security under various attack intensities.

[0099] ACR (Attack Reduction Ratio) measures the attack mitigation effectiveness of an algorithm; a higher value indicates stronger defense capabilities. As shown in Figure 5, our method comprehensively outperforms all baselines in terms of ACR. Specifically, the SAC method, lacking a security cost constraint mechanism and with a relatively loose action space design, achieves a high ACR but suffers from significant security shortcomings, potentially leading to serious security vulnerabilities in real-world applications. Therefore, a comparison of ACR with mainstream security reinforcement learning methods confirms that our method possesses superior attack propagation suppression capabilities.

[0100] PDS assesses the physical layer stability of the system; a smaller value indicates less interference. PDS increases with increasing attack intensity. In scenario 1, the PDS differences among methods are small, and the physical layer stability of the system is within an acceptable and controllable range. In scenario 2, although our PDS is higher than in scenario 1, it is still lower than safe RL methods such as PPO-Lag and CPO. However, compared to SAC, SAC-SC, and PPO-SC, our method shows a slight upward trend in PDS. This is because in scenarios with risks in the physical process, our method is more aggressive in its physical layer recovery strategy selection. While the response strategy ensures system security, it inevitably causes fluctuations in physical layer sensors. Combined with the SC value, since the security cost remains low, this phenomenon indicates that the fluctuation amplitude of physical parameters is still strictly controlled within the safety threshold range, and will not lead to system crashes or failure of critical functions. However, it also further illustrates that pursuing security goals can have adverse effects on the mission objectives. In summary, although our method causes a slight loss in short-term physical layer stability, it achieves a significant improvement in overall system security. In summary, compared with other secure reinforcement learning methods, the stability fluctuations of our method remain within the safe threshold, making it more suitable for the operational requirements of industrial scenarios.

[0101] TR is a core metric in reinforcement learning that measures the overall performance of algorithms. Its value integrates the optimization results of three key objectives: defense clarity, system stability, and security. Our method achieves the minimum average reward value in both scenarios, with relatively small overall fluctuations. This is due to the superior performance of our proposed method in terms of both the effectiveness of defensive measures and security.

[0102] In summary, the security cost function and hierarchical strategy optimization method proposed in this paper can achieve precise control over security costs and achieve an optimal balance between security, defense, and stability, thus providing a more robust solution for industrial control system security.

[0103] To further verify the advantages of this invention in intrusion response modeling, we set up comparative experiments with ICL on a simulation system. Through these experiments, we collected a total of twenty sets of data, covering the performance of the two models in terms of SC and ACR. Figure 6 The performance of ICL and our algorithm in these scenarios is demonstrated, and representative data points are presented to ensure readability. To quantify the performance difference between SC and ACR, an independent sampling t-test was used, with a significance threshold set at α = 0.05.

[0104] like Figure 6 As shown in (a) and (c), in scenario 1, the t-test reveals a significant difference between the two algorithms: Our algorithm achieves a significantly lower security cost compared to ICL. This advantage is maintained in scenario 2, with similarly significant differences observed. Figure 6 As shown in (b) and (d), no statistically significant difference was found between ICL and Our in both scenario 1 and scenario 2. The two algorithms performed similarly in attack cleanup capabilities.

[0105] This analysis shows that our method and ICL do not differ significantly in the ACR metric, but our method has a clear advantage in the SC metric. Specifically, traditional intrusion response modeling methods prioritize attack mitigation performance but often neglect the enforcement of security risk constraints. This oversight inevitably leads to higher SC values, thereby compromising the overall operational security of the protected system. In contrast, our algorithm not only retains the effective attack mitigation capabilities of traditional intrusion response modeling methods but also achieves a significant reduction in security costs. The fundamental reason for this performance advantage lies in the CMDP modeling design, particularly the security cost function based on the security risk design of physical devices.

Claims

1. An industrial control intrusion response security decision method based on security reinforcement learning, characterized in that, The application relates to a safe and effective intrusion response strategy training method based on security reinforcement learning. The application comprises the following steps: Step 1: building an industrial control system simulation environment, collecting industrial control system network layer alarm data and physical layer sensor and actuator operation data, synchronously recording state transitions, reward values and security costs triggered by decision actions, and forming attack and defense interaction trajectory data; Step 2: using a hierarchical strategy optimization algorithm to analyze the risk level of the response strategy according to the attack and defense interaction trajectory data collected in step 1 and learning a better intrusion response strategy, and finally obtaining a safe and effective strategy; Step 3: combining the interaction data of the simulation running and the evaluation indexes, and statistically analyzing the defense success rate and security cost core indexes of different models to comprehensively measure the performance differences of the response models.

2. The safe and effective intrusion response security decision method based on security reinforcement learning according to claim 1, wherein an industrial control system simulation environment is built, industrial control system network layer alarm data and physical layer sensor and actuator operation data are collected, state transitions, reward values and security costs triggered by decision actions are synchronously recorded, and attack and defense interaction trajectory data are formed. Step 2.1: combining the characteristics of the industrial control system to build the state space and action space of the response decision model; the state space is formed by collecting the real-time intrusion state information of the network nodes in the industrial control environment, synchronously acquiring the sensing data of the sensors and actuators, and fusing the two types of information to form a unified state s; according to the cross-layer coupling characteristics of the industrial control environment, the action space is composed of the defense strategies of the network nodes and the recovery strategies of the physical devices; the response model outputs corresponding response actions a based on the real-time state of the industrial control environment; Step 2.2: the decision model outputs corresponding response actions based on the real-time state information fed back by the industrial control simulation environment, and synchronously calculates the reward value and the security cost value generated by the action; the corresponding response action at is outputted according to the real-time state information st at the current time t; the reward value rt and the security cost value ct of the current action are calculated according to the information, and the action at is fed back to the industrial control simulation system to obtain the state information st+1 at the next time; Wherein the calculation formulas of the reward value r and the security cost value c are as follows: Formula (1): the reward function is composed of two parts, namely the network layer reward and the physical layer reward, which are calculated by the system state of the network nodes and the sensor measurement value respectively; （1）； Formula (4): the security cost function is composed of two parts, namely the penalty term for violating the constraint and the security risk evaluation value; （2）； Equation (2) calculates the risk level of the system state of the network node; the calculation parameter denotes the weight of the network layer reward, denotes the alarm set of the damaged network node at the current moment, denotes the location of the damaged network node, denotes the alarm severity of the damaged network node, denotes the weight of the network layer reward; （3）； Equation (3) calculates the deviation of the sensor; parameter a weight of the network layer reward, a current sensor perception value, a standard baseline perception value, a baseline standard deviation, a weight of the physical layer reward, a set of sensors; wherein is set to 1 when the system collapses and stops running in advance, otherwise 0; （4）； Step 2.3: the current state, response action, action reward value, cost value and next time state are recorded and saved in the trajectory data as a five-tuple; the decision model and the industrial control simulation environment continuously interact until sufficient trajectory data are collected, and then the trajectory data are inputted into the model for training. （5）； The penalty term of formula (5) is defined by the degree of the sensor perception value exceeding the safety boundary, and is punished according to the degree of exceeding; indicates the time of policy execution recovery, indicates the set of sensors, is the safety threshold value of the sensor, indicates the sampling value of the sensor at the t th moment, represents whether the sensor exceeds the safety boundary, and is 1 if exceeding the safety boundary, and 0 otherwise. （6）； Equation (6) is the safety risk; The overshoot of the sensor ​ 3. The method of claim 2, wherein the trajectory data is collected by using a hierarchical policy optimization algorithm, the risk level of the response strategy is analyzed, and a better intrusion response strategy is learned, so that a safe and effective strategy is finally obtained. Step 3.1, the response strategy generated based on the response model interacts with the industrial control environment to collect trajectory data: based on the current training strategy network, i.e., the response model in learning, interacts with the industrial control simulation environment in real time to form interactive trajectory data , continuously collect until enough samples are accumulated; Step 3.2, calculating state expected reward and cost according to trajectory data: from the trajectory data of interaction, extracting the instantaneous reward and safety cost corresponding to each state; combining the generalized advantage estimation GAE method, deriving the long-term expected reward and cost of the state for the subsequent optimization and update of reward and cost network parameters; The specific target value calculation formula is as follows: ​ （6）； （7）； wherein are the reward state value functions, respectively, representing the long-term cumulative reward from this state s, following the current policy, are the cost state value functions, respectively, representing the long-term cumulative cost from this state s, following the current policy; is the reward advantage function for state s, is the cost advantage function for state s; Step 3.3, the gradient back propagation is used to update the weight parameters of the reward and cost networks respectively: the expected reward and expected cost obtained in step 2 are used as target values, the mean square error (MSE) is used as the loss function of the reward network and the cost network respectively, and the weight parameters of the reward network and the cost network are updated by the gradient back propagation algorithm; the specific formula is as follows: （8）； （9）； Equations (8) and (9) are the reward and cost network update equations, respectively, where is the response policy learned by the current policy network; Step 3.4, calculate the cumulative reward value, cumulative cost value and risk factor, and judge the risk level of the current strategy: calculate the cumulative reward value, cumulative cost and risk factor of the current strategy in the trajectory data, and divide the strategy into four levels: initial stage, no risk, medium risk and high risk; the specific formula is as follows: （10）； （11）； （12）； wherein formula (10) represents an evaluation factor related to the cumulative reward value, used to determine whether to ignore the constraint in the initial stage; denotes a discount factor, which balances the weight of current and future benefits; the immediate reward function; the cumulative reward value under the trajectory data; formula (11) represents an evaluation factor related to the cumulative cost value, used to determine whether the strategy is in the safe cost domain; denotes a state the immediate cost function when the action is performed; the cumulative reward value under the trajectory data, and d denotes a safety cost upper threshold value; the highest acceptable level of security cost; formula (12) is a strategic risk assessment factor, which measures whether the current strategy trusts the domain constraints and the security cost domain constraints at the same time; denotes the policy gradient corresponding to the cost function; denotes the second-order matrix of the policy parameter, denotes the KL divergence value of the target policy and the current policy, denotes the inverse operation result of the matrix; denotes the constraint threshold value; The risk level is judged according to the following formula, and the specific formula is as follows: a) Initial phase: ; b) no risk: and and ; c) Medium risk: and ; d) High risk: and and ; Step 3.5, generate a strategy according to the response model to select an appropriate target update strategy network according to the risk level of the current strategy: if the risk of the current strategy is the initial stage or no risk, plan without safety constraints, and update the strategy to maximize the cumulative reward; If the risk of the current strategy is medium risk, enable the safety cost and KL divergence double constraint target to control the safety cost while avoiding strategy mutation; if the risk of the current strategy is high risk, the safety cost minimization rule is preferentially used to ensure the safety of the physical device; the specific formula is as follows: （13）； （14）； （15）； where denotes the current policy network parameters, is the parameter of the next round of updates; denotes the policy gradient corresponding to the reward function; and is the Lagrange multiplier of the optimal solution to the optimization problem obtained. Step 3.6, save the strategy network with the best performance in the training process as the decision response model: judge whether the current training round number reaches the preset value, if not, return to step 3.1, continue to interact with the industrial control environment to collect new trajectory data, and repeat steps 3.1-3.5; if it has reached, stop the cycle, save the strategy network with the best performance in the training process as the final intrusion response strategy.

4. The method of claim 1, wherein the method is based on security-reinforced learning. The interaction data and evaluation indexes of the simulation running are combined to statistically analyze the defense success rate, safety cost core index and performance difference of each response model; Step 4.1, Statistics of the security risk cost of different decision methods response strategy: as shown in formula (16), SC represents the cumulative response security cost of each round of test, wherein The total number of tests is indicated; this index measures the security cost required for the response strategy, and the lower the SC value, the higher the security of the response strategy; the specific formula is as follows: （16）； Step 4.2, Statistics of defense capability of different decision methods responding to the strategy: as shown in formula (17), ACR represents the proportion of successful removal of attacks by the system; wherein the number of successful removal of attacks, the total number of attacks, ACR evaluates the effectiveness of the response strategy adopted by the system, and the higher the ACR value indicates that the system can resist more attacks; (17); Step 4.3, Statistics of the deviation of the sensor perception value responding to the strategy of different decision methods: as shown in formula (18), PDS measures the sensitivity of the system to the fluctuation of the perception value, that is, the sum of the perception deviation in all test processes; the perception deviation wherein is the current perception value, and are the expected value and the standard deviation, respectively, indicates the sampling time step in the test; the lower the PDS value, the lower the overall fluctuation value of the sensor; the specific formula is as follows: （18）； Step 4.4, statistic the overall performance of the response strategy of different decision methods: as shown in formula (19), TR represents the total reward value obtained in each test; the specific formula is as follows: （19）。 5. An intrusion response system for industrial control system based on safety reinforcement learning, which implements the method of any one of claims 1-4, comprising: An industrial control environment modeling module based on constrained Markov decision process (CMDP) is used to build an industrial control simulation environment, collect network layer node state information and physical layer device sensing data, and integrate them into state inputs; the model outputs actions based on the state, synchronously calculates the reward value and safety cost value corresponding to the actions, and provides a real-time interaction interface for the decision model to obtain attack and defense trajectory data; The trust domain-based intrusion response security policy optimization module is used to realize the iterative optimization of the industrial control system response model. First, the trajectory data is collected by interacting the policy model with the industrial control environment. Then, the state expected reward and cost are calculated based on the trajectory data, and the reward and cost network parameters are updated. Subsequently, the risk level of the policy is determined by the risk factor, and the corresponding policy network update rule is adapted. The above process is repeatedly executed until the training round number reaches the standard. Finally, the policy network with the best performance is saved as the decision response model. The multi-index model evaluation module is used to compare the performance of different decision methods. First, the security risk cost of the response strategy of each decision method is calculated. Then, the defense capability is calculated. Subsequently, the deviation degree of the sensor perception value corresponding to the strategy is calculated. Finally, the overall performance of the response strategy of different decision methods is calculated by integrating the above indexes. The response performance differences between different models are analyzed using these indexes.

Images