A dual-constraint intrusion response method based on lexicographical order strategy optimization

The dual-constraint intrusion response method optimized by lexicographical order strategy solves the problem of consistency and coordination between security control logic and defense actions in industrial control systems, improves the defense success rate and reduces conflict costs, and ensures the security and stability of the system.

CN123069360APending Publication Date: 2026-11-10BEIJING UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202610965923.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-01
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing intrusion response methods fail to effectively coordinate the consistency between industrial process safety control logic and defense actions in industrial control systems. Furthermore, existing constraint reinforcement learning methods struggle to distinguish between hard security constraints and soft conflict constraints, potentially leading to physical process anomalies or overly conservative response strategies.

Method used

A dual-constraint intrusion response method based on lexicographical order strategy optimization is adopted. The constraint modeling module constructs security control constraints and action conflict constraints, the state acquisition module obtains system state and historical data, the lexicographical order strategy optimization module performs hierarchical optimization, the response decision module executes defensive actions, and the experience storage module enables continuous learning.

Benefits of technology

It improves the success rate of defense, reduces the cost of security control violations and action conflicts, and ensures that industrial control systems maintain physical process security and consistent response strategies when facing cyberattacks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123069360A_ABST
    Figure CN123069360A_ABST
Patent Text Reader

Abstract

This invention discloses a dual-constraint intrusion response method based on lexicographical order strategy optimization, belonging to the field of industrial control system security technology. The intrusion response decision is modeled as a constrained Markov decision process, simultaneously constructing security control constraints and action conflict constraints. The security control constraints calculate a normalized security score based on the ideal values ​​and critical thresholds of key process indicators such as temperature and pressure. The action conflict constraints calculate a conflict value based on the overlap of the vulnerability sets affected by adjacent response actions and the vulnerability severity score. In the relaxation layer, conflict regularization strategy optimization is performed within this secure feasible subspace to obtain the optimal response action and iteratively update the strategy. This invention can prioritize the security of industrial process control logic without relying on explicit industrial process dynamics models, while simultaneously reducing response action conflicts and improving intrusion response reliability.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of industrial control system security and intrusion response technology, and relates to automated defense decision-making, security reinforcement learning, constraint strategy optimization and industrial process safety control technology in industrial control systems. In particular, it relates to a dual-constraint intrusion response method and system for industrial control systems based on lexicographical order strategy optimization. Background Technology

[0002] Industrial control systems are widely used in critical infrastructure sectors such as continuous production processes in petrochemical plants, monitoring and data acquisition in power systems, automated control of water treatment facilities, and signal control in rail transit. With the deep integration of information technology and operational technology, the traditional security boundaries of industrial control systems, which relied on physical isolation, are gradually being broken down, exposing them to more complex cyberattack threats. Attackers can exploit system vulnerabilities, malware, and denial-of-service attacks to launch cyberattacks against industrial facilities, causing serious consequences such as production interruptions, equipment damage, and even personal injury. To address these threats, intrusion response systems (IRS) have been developed as an active defense framework used in conjunction with intrusion detection systems (IDS). IRS detects attacks or abnormal behavior, while IRS automatically selects and executes defensive actions after an alert is generated, thereby reducing the risk of further penetration or spread of attacks.

[0003] Among existing intrusion response methods, reinforcement learning can model the response process as a sequential decision-making problem, obtaining a sequence of defensive actions through trial and error, which has advantages in improving network layer defense effectiveness. However, existing methods often prioritize maximizing network security benefits, neglecting the impact of the response actions themselves on the industrial process control logic. For example, in industrial chemical processes or production line control, some response actions, while mitigating network attacks, may cause process variables such as pressure, temperature, flow rate, or liquid level to temporarily exceed safety thresholds, leading to process anomalies or even physical damage. Furthermore, in real-world attack and defense scenarios, different defensive actions may have overlapping functions or interfere with each other. For instance, rate-limiting actions may alter the appearance of abnormal flow, affecting subsequent blocking actions triggered by abnormal flow; when multiple response actions targeting the same vulnerability or the same attack phase are executed consecutively, redundancy, conflicts, or failures may occur. Existing intrusion response models typically treat defensive actions as independent decision-making units, lacking modeling of the consistency of response actions at adjacent time steps.

[0004] Furthermore, existing safety reinforcement learning methods typically treat multiple constraints in a homogenized and scalarized manner, making it difficult to distinguish the differences in stringency between industrial process safety control constraints and response action conflict constraints. Industrial process safety constraints are usually hard constraints, and their violation can lead to unacceptable physical risks; action conflict constraints, on the other hand, are usually soft constraints, allowing for flexible trade-offs within a certain tolerance range. Placing both in the same optimization layer for uniform punishment can easily lead to overly conservative strategies or violations of hard safety constraints during the optimization process.

[0005] Therefore, existing technologies are insufficient in ensuring the physical process security and consistency of defensive actions in industrial control systems. There is an urgent need for an intrusion response technology that can explicitly coordinate dual risk constraints, so that the response strategy can take into account both defensive effectiveness and action consistency while ensuring the security of the physical process. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a dual-constraint intrusion response method for industrial control systems based on lexicographical order strategy optimization. This method addresses the issues that existing intrusion response methods only focus on network defense benefits, lack joint modeling of conflicts between industrial process control logic and defense actions, and that existing constraint reinforcement learning methods have difficulty distinguishing between hard security constraints and soft conflict constraints.

[0007] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A dual-constraint intrusion response method for industrial control systems based on lexicographical order strategy optimization, comprising five modules: a constraint modeling module, a state acquisition module, a lexicographical order strategy optimization module, a response decision module, and an experience storage and update module. The constraint modeling module is used to construct security control constraints and action conflict constraints; the state acquisition module is used to acquire interactive data such as the state of the industrial control system, alarm information, and historical response actions; the lexicographical order strategy optimization module performs hierarchical optimization on the dual-constraint intrusion response model; the response decision module executes defensive actions based on the output of the lexicographical order strategy optimization; and the experience storage and update module stores interactive data and enables continuous learning of the strategy and security assessment network. The modules and implementation steps are as follows:

[0008] 1) Constraint Modeling Module: Models the intrusion response decision process as a constrained Markov decision process that includes security control constraints and action conflict constraints.

[0009] Step 1: Model the intrusion response decision in the industrial control system as a constrained Markov decision process, and define a reward function to evaluate the effectiveness of defensive actions. and security control constraints Action conflict constraints The security control constraints are used to prevent response strategies from violating the physical process security boundaries, and the action conflict constraints are used to prevent overlap, redundancy or mutual interference between response actions targeting the same vulnerability. Execute step 2.

[0010] Step 2: Formalize safety control constraints and determine the degree of compliance of the response strategy with physical process safety using predefined safety boundary thresholds; calculate the safety score of candidate defensive actions based on the ideal values ​​and critical thresholds of key indicators of the industrial process, wherein the key indicators include at least one of temperature, pressure, flow rate or liquid level; determine candidate actions with safety scores exceeding the preset threshold as violating safety control logic, and proceed to Step 3.

[0011] Step 3: Formalize action conflict constraints, construct vulnerability-level indicators, and comprehensively calculate the number and severity of overlapping vulnerabilities between defense actions; based on the current response action... Corresponding vulnerability set Previous action response Corresponding vulnerability set The vulnerability severity score is used to calculate the action conflict constraint value, and the action conflict constraint value is compared with a preset conflict threshold. Compare the results and proceed to step 4.

[0012] 2) Status acquisition module.

[0013] Step 4: Construct a set of defense actions A. Each defense action in the set of defense actions A corresponds to one or more vulnerabilities, control variables, or response functions to be addressed, including at least one of access management, application isolation, vulnerability exploitation protection, multi-factor authentication, network traffic filtering, network segmentation, communication authenticity verification, control value reconfiguration, manual shutdown, and non-execution actions. Then, proceed to step 5.

[0014] Step 5: Collect historical industrial control system operation data and response interaction data as training samples. The historical interaction data includes state-action pairs. The corresponding response results, including reward value, safety control constraint value, and action conflict constraint value, are used to train the safety assessment network using the mean squared error loss function, enabling it to learn to predict the safety score of a given state-action pair, and then proceed to step 6.

[0015] 3) Lexicographical order strategy optimization module: Performs hierarchical optimization on the constrained Markov decision process.

[0016] Step 6: In the strict layer of lexicographical order policy optimization, a candidate action space is generated from the current policy. The security assessment network is invoked to predict the security score of each candidate action, and candidate actions that do not meet the security control constraints are filtered out, thereby constructing a safe and feasible subspace. Proceed to step 7.

[0017] Step 7, relaxation layer optimization, in the safe and feasible subspace Optimize defense reward by implementing conflict regularization; under the conditions of satisfying action conflict constraints and policy trust region constraints, maximize defense reward and minimize conflict cost to obtain the optimal response action. Proceed to step 8.

[0018] 4) Response decision module: Executes defensive actions based on the optimized output of the lexicographical order strategy.

[0019] Step 8: Execute the optimal response action. The optimal response action is converted into an actual control command and issued for execution to deal with the network attack. Step 9 is executed.

[0020] 5) Experience storage and update module: Stores interactive data and enables continuous learning of the policy and security assessment network.

[0021] Step 9: Obtain the reward, safety control constraint value, action conflict constraint value, and system state at the next moment after the action is executed. , interactive tuple Store to the experience replay buffer and proceed to step 10.

[0022] Step 10: Update the security assessment network and policy network. Update the parameters of the security assessment network and policy network based on the sampled data in the playback buffer to achieve continuous learning and optimization.

[0023] The beneficial effects of this invention are:

[0024] I. This invention proposes a dual-constraint intrusion response model for industrial control systems. This model integrates security control logic and defensive actions into the response decision-making process. This ensures that intrusion response strategies are not solely driven by network defense benefits, thereby reducing the risk of response actions compromising control process security.

[0025] Second, this invention proposes a lexicographical policy optimization method that separates the construction of the safe and feasible subspace of the strict layer and the conflict regularization reward optimization of the relaxed layer to optimize the response policy under the constructed dual constraints. This hierarchical architecture fundamentally breaks the unified scalar policy optimization mechanism in standard safe reinforcement learning, and can maintain the defensive capability and flexibility of the response policy while ensuring the safety of the physical process.

[0026] Third, the effectiveness of the proposed method was demonstrated in the Tennessee-Eastman process benchmark test. Evaluation results for multi-stage network attacks show that our method improves the defense success rate by 30.3% compared to current intrusion response models, while reducing the costs of security control violations and action conflicts, fully demonstrating its practical application value in improving the security response of industrial control systems. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the Tennessee-Eastman process control system architecture according to Embodiment 1 of the present invention.

[0028] Figure 2 This is a schematic diagram of the multi-stage attack process in Embodiment 1 of the present invention.

[0029] Figure 3 This is a flowchart illustrating the dual-constraint intrusion response method based on lexicographical order strategy optimization described in this invention.

[0030] Figure 4 This is a graph showing the comparison results of multiple indicators between the present invention and other response models under different attack scenarios.

[0031] Figure 5 This is a comparison chart of the pressure sensor trajectory changes of the present invention and other response models when defending against the same attack.

[0032] Figure 6 This is a diagram showing the ablation experiment results of this invention. Detailed Implementation

[0033] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0034] Example 1

[0035] like Figure 1 As shown, this embodiment uses the Tennessee Eastman (TE) process control system as the target protection object. The physical layer of this system includes the reactor, gas-liquid separator, stripping tower, and multiple valves such as V1-V5, pressure transmitters PT, flow transmitters FT, and component analyzer IA; the network layer includes an enterprise network consisting of gateway G1, web server WS, etc., and a control network consisting of engineering station ES, control server CS, and various flow / pressure controllers FC / CC / PC.

[0036] like Figure 2As shown, this embodiment assumes that the system encounters a multi-stage cross-layer network attack. The attacker gains initial access through the gateway G1 or the web server WS and performs privilege escalation. Then, the attacker moves laterally to the engineering station ES or the control server CS in the control network, and finally detects and intrudes into the underlying FC / CC / PC controller, attempting to tamper with the process control logic and cause destructive effects on physical devices such as V1-V5 or PT.

[0037] like Figure 3 As shown, this embodiment presents a dual-constraint intrusion response method based on lexicographical order strategy optimization for this scenario. The method consists of 10 steps. The specific details are described below:

[0038] Step 1: Model the multi-stage intrusion response process in the TE process control system as a Constrained Markov Decision Process (CMDP). Define a reward function to maximize network defense benefits, such as successfully blocking malicious traffic during the lateral movement phase. Simultaneously define two types of constraints: security control constraints to prevent physical safety breaches such as reactor pressure PT overload due to cutting off control communication; and action conflict constraints to prevent conflicting network isolation or traffic filtering policies from being issued simultaneously on the ES and CS nodes. Execute Step 2.

[0039] Step 2, Formalize safety control constraints. Based on the ideal values ​​and critical thresholds of key TE process indicators such as reactor pressure PT and reactant flow rate FT, calculate the safety score of candidate defensive actions. The calculation formula is as follows:

[0040]

[0041] Using predefined safety boundary thresholds, determine whether cutting off communication with specific controllers such as FC_A will cause the V1-related valves to malfunction and trigger abnormal reactor pressure. If the predicted action would lead to a physical state exceeding limits, it is determined to be a violation of safety control logic. Proceed to step 3.

[0042] Step 3, Formalize Action Conflict Constraints. Construct vulnerability-level metrics. For multi-stage attack paths, such as lateral movement from WS to ES, calculate the overlap between the current set of defense actions and the set of defense actions at the previous moment, as well as the vulnerability severity score. The calculation formula is as follows:

[0043]

[0044] This is used to assess system overhead and conflict levels, and compare them with preset conflict thresholds. Compare the results. Proceed to step 4.

[0045] Step 4, construct the set of defensive actions This set of capabilities covers multi-dimensional handling of enterprise and control networks, including: initial access blocking, lateral movement isolation, access control, control value reconfiguration, and manual shutdown for severe physical threats. Proceed to step 5.

[0046] Step 5: Collect real-time operating status information such as pressure and flow rate of the TE system, as well as network alarm information. During the pre-training phase, historical state-action pairs are used to input control variables and input actions into the safety assessment network. The safety score is predicted, and the mean squared error loss is calculated. To update the network weights, the calculation formula is as follows:

[0047]

[0048] To enable it to predict whether a specific action will cause a physical hazard to the TE system, proceed to step 6.

[0049] Step 6: The strict layer constructs a secure and feasible subspace. When facing an attack, the current policy generates candidate actions across the entire action space. These actions are input into a pre-trained security evaluation network, which outputs a security score for each action. The sigmoid function is used to binary-divide actions into secure and insecure categories. The system directly filters out insecure actions, thus constructing a secure and feasible subspace. Proceed to step 7.

[0050] Step 7: Relaxation layer conflict regularization optimization. Within the aforementioned strict safe and feasible subspace, optimization is performed starting from the initial point. The optimization objective is to find the endpoint, i.e., the optimal action, while satisfying the action conflict constraint and the policy trust region constraint. Proceed to step 8.

[0051] Step 8, execute the optimal response action. The algorithm's output actions are transformed into routing interception rules for the G1 / G2 gateway, access control list updates for the ES / CS, or security control parameters sent to the FC / CC controller to completely block the multi-stage attack chain and ensure reactor security. Execute step 9.

[0052] Step 9: Obtain environmental feedback after the action is executed. Observe the TE system state at the next moment. Collect rewards and conflict constraint values Interactive tuples Store in the experience replay buffer In the middle. Execute step 10.

[0053] Step 10, from the playback buffer The system uses sampled data to continuously update the parameters of the policy network and security assessment network, enabling it to continuously learn and optimize in response to evolving advanced persistent threats (APTs).

[0054] like Figure 4 As shown in the figure, this invention compares the proposed method with the baseline model ICL-IRS under various attack scenarios across multiple evaluation metrics. The figure demonstrates that the proposed model achieves optimal performance in both defense success rate and overall reward value, effectively reducing conflict and security costs, thus proving that this method has better overall performance in intrusion response decision-making.

[0055] like Figure 5 As shown, this invention compares the dynamic change curves of pressure indicators for different defense response strategies. The comparative experimental results show that, after a cyberattack, the response strategy of this invention can recover stability the fastest and consistently keep system pressure below the dangerous threshold, effectively avoiding physical security violations that occur in other models.

[0056] like Figure 6 As shown, this invention conducted ablation experiments on safety control constraints and action conflict constraints. Experimental data demonstrate that, compared to variant models that retain only a single constraint, the complete model DC-IRS, which incorporates both constraints, achieves ablation. SC It performed best in core evaluation indicators such as defense success rate (DSR), verifying the necessity and effectiveness of joint modeling of the two constraints.

Claims

1. A dual-constraint intrusion response method based on lexicographical order strategy optimization, characterized in that, The method is used to perform response decisions on an industrial control system after an intrusion detection system generates an alarm. The method includes: Step 1: Model the intrusion response decision in the industrial control system as a constrained Markov decision process, and define a reward function to evaluate the effectiveness of defensive actions. and security control constraints Action conflict constraints The security control constraints are used to prevent response strategies from violating the physical process security boundaries, and the action conflict constraints are used to prevent overlap, redundancy or mutual interference between response actions for the same vulnerability. Execute step 2. Step 2: Formalize safety control constraints and determine the degree of compliance of the response strategy with physical process safety using predefined safety boundary thresholds; calculate the safety score of candidate defensive actions based on the ideal values ​​and critical thresholds of key indicators of the industrial process, wherein the key indicators include at least one of temperature, pressure, flow rate or liquid level; determine candidate actions with safety scores exceeding the preset threshold as violating safety control logic, and proceed to Step 3; Step 3: Formalize action conflict constraints, construct vulnerability-level indicators, and comprehensively calculate the number and severity of overlapping vulnerabilities between defense actions; based on the current response action... Corresponding vulnerability set Previous action response Corresponding vulnerability set The vulnerability severity score is used to calculate the action conflict constraint value, and the action conflict constraint value is compared with a preset conflict threshold. Compare the results and proceed to step 4; Step 4: Construct a set of defense actions A. Each defense action in the set of defense actions A corresponds to one or more vulnerabilities, control variables, or response functions to be dealt with, including at least one of access management, application isolation, vulnerability exploitation protection, multi-factor authentication, network traffic filtering, network segmentation, communication authenticity verification, control value reconfiguration, manual shutdown, and non-execution actions. Then execute step 5. Step 5: Collect historical industrial control system operation data and response interaction data as training samples. The historical interaction data includes state-action pairs. The corresponding response results, including reward value, safety control constraint value and action conflict constraint value; train the safety assessment network using the mean squared error loss function to learn to predict the safety score of a given state-action pair, and execute step 6; Step 6: In the strict layer of lexicographical order policy optimization, a candidate action space is generated from the current policy. The security assessment network is invoked to predict the security score of each candidate action, and candidate actions that do not meet the security control constraints are filtered out, thereby constructing a safe and feasible subspace. Proceed to step 7; Step 7, relaxation layer optimization, in the safe and feasible subspace Optimize defense reward by implementing conflict regularization; under the conditions of satisfying action conflict constraints and policy trust region constraints, maximize defense reward and minimize conflict cost to obtain the optimal response action. Proceed to step 8; Step 8: Execute the optimal response action. The optimal response action is converted into an actual control command and issued for execution to deal with the network attack. Step 9 is executed. Step 9: Obtain the reward, safety control constraint value, action conflict constraint value, and system state at the next moment after the action is executed. , interactive tuple Store to the experience replay buffer and proceed to step 10; Step 10: Update the security assessment network and policy network. Update the parameters of the security assessment network and policy network based on the sampled data in the playback buffer to achieve continuous learning and optimization.

2. The dual-constraint intrusion response method based on lexicographical order strategy optimization according to claim 1, characterized in that... Steps 1-4 consider the dual-constraint response paradigm that takes into account the risk of the strategy itself, specifically including: The constrained Markov decision process is represented as a quintuple. ,in This represents the state space of the attacked industrial control system. Indicates the available defensive action space. Represents the reward function, Indicates that it is subject to safety control constraints. Action conflict constraints The constrained set formed Indicates unknown dynamic characteristics or state transition relationships of an industrial control system; Safety control constraints Key performance indicators and predefined security boundaries are introduced to quantify the compliance of each response strategy with these constraints, jointly determining the specific constraint value for each response strategy, i.e., the security score; the calculation method is as follows:

3. Among them, Indicates the state Execute action The later Actual readings of key performance indicators Indicates the first Ideal values ​​for class indicators This indicates the critical threshold for the corresponding indicator; when the maximum safety score after normalization of each indicator does not exceed 1, the response action is determined to be within the safe operating range. Action Conflict Constraints Action conflicts are considered a soft constraint because violating this constraint does not directly lead to serious system failure; in order to assess the state The following actions Compared to the previous action The conflict between them is defined and calculated as follows: ; in, Indicates vulnerability Severity score, and These represent the sets of vulnerabilities affected by the current response action and the previous response action, respectively. This is the action conflict threshold.

4. The dual-constraint intrusion response method based on lexicographical order strategy optimization according to claim 1, characterized in that... The lexicographical order strategy optimization method in steps 5-7 specifically includes: The traditional scalar strategy optimization method is replaced by a hierarchical approach. By constructing the trust region update as a strict layer and a relaxed layer to coordinate the two constraints, the response strategy can be further optimized under the constructed dual constraints. Strict Layer: A physical process-guided neural network model is proposed, whose input is a state-action pair. The actions It includes three control variables representing the valve opening levels of different systems; its output is a two-dimensional prediction vector, providing safety scores for temperature and pressure; the true safety score serves as a monitoring signal, optimizing prediction accuracy by minimizing the mean square error (MSE). ; Relaxation Layer: Under the constraints of the safe and feasible subspace constructed by the strict layer, the relaxation layer performs response reward-oriented updates. The original problem is simplified to optimizing the expected reward only under the constraints of soft constraints on action conflicts and the boundary of the trust region. Based on the trajectory in the empirical replay buffer, the reward gradient, conflict constraint gradient, and Fisher information matrix H are estimated, and the policy update is transformed into a trust region quadratic programming problem. The policy parameter update direction is solved by Lagrange duality. When there is no feasible solution to the trust region quadratic programming problem, a recovery update is performed within the trust region to make the policy prioritize reducing the violation value of action conflict constraints and keep the candidate actions corresponding to the policy still within the safe and feasible subspace.