Industrial control cross-layer security decision method based on partial observable markov decision

Abstract

The application discloses an industrial control cross-layer security decision-making method based on a partially observable Markov decision, which comprises the following steps: step 1, using a monitoring system to perform state sensing on an industrial control system with partially observable state information, and collecting training data through continuous interaction between a model and the industrial control system; step 2, using a T-DRQN deep reinforcement learning method based on a POMDP decision-making framework to combine the data obtained through the interaction with historical observation information for training; and step 3, using alarm information of network layer device nodes and physical layer sensor readings to evaluate the response effect of an intrusion response strategy, and using these indexes to analyze the difference in response performance between different models. The application can effectively resist multi-stage cross-layer attacks from the cyberspace to the physical space, adopts a double-branch network structure, more obviously distinguishes state value and action advantage, and thus more accurately and efficiently guides the intrusion response decision-making, and is more suitable for processing industrial control scenes with partially observable system state information.

Description

Technical Field

[0001] The present invention belongs to the field of industrial control system security defense technology, relates to network attack and defense game model construction technology and network intrusion response decision technology, and particularly relates to a cross-layer security decision-making method for industrial control systems based on partially observable Markov decision processes (POMDPs). Background Art

[0002] Industrial control systems (ICS) are automated systems used to monitor and control industrial processes. They are widely used in critical infrastructure sectors such as energy, manufacturing, transportation, and water. In recent years, with the increasing intelligence of ICS, the integration of modern ICS with information technology has become increasingly close. The openness and interconnectedness of information networks have increased the security risks they face, making ICS security increasingly important. Once an ICS is hacked or subjected to malicious manipulation, the consequences are often catastrophic, resulting not only in economic losses but also in serious impacts on national and social security. Therefore, to address the potential risks of ICS attacks, establishing an intrusion response decision-making system is crucial for ICS security.

[0003] Traditional intrusion response decision-making methods rely on a mathematical model consisting of an objective function and constraints, using multi-objective optimization or game theory to construct the decision-making system. However, abstracting the precise system model, objective function, and constraints from complex industrial control systems is extremely difficult. Reinforcement learning, due to its model-free nature, eliminates the need to establish a system model in advance and learns optimal strategies through direct interaction between the agent and the environment. This improves the traditional method's high reliance on system models. However, in practical industrial control system applications, the following issues still exist:

[0004] Existing intrusion response decision-making methods are typically based on Markov decision processes (MDPs), which rely on accurate perception of the complete system state and are difficult to handle in partially observable industrial control system scenarios. Markov decision processes (MDPs) are a general framework for solving sequential decision problems. In MDPs, complete and accurate environmental state information is required to ensure the effectiveness of decision results. However, due to the complexity of industrial control systems and the risk of underreporting and false alarms in monitoring systems, system state information is often only partially observable. Decision-making systems often struggle to accurately and timely perceive system state changes, making it difficult to handle partially observable system environments, greatly limiting the guiding value of decision-making methods.

[0005] Existing intrusion response decision-making methods rely on prior knowledge when solving the POMDP problem. Furthermore, their single-branch neural network architecture limits the ability to distinguish between the value of the system state and the advantage of an action, making them difficult to effectively apply to dynamic, unknown system environments. Existing methods transform the POMDP problem into a Belief-MDP problem by introducing a belief state. However, the calculation and updating of belief states are highly dependent on prior knowledge. In practical applications, abstracting explicit prior knowledge from dynamic industrial control systems is extremely difficult. Furthermore, existing decision-making methods use a single-branch neural network architecture to output the value function, making it difficult to further distinguish between the value of the system state and the advantage of an action, limiting their effectiveness and stability. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a cross-layer security decision-making method for industrial control systems based on POMDP, which is used to solve the problem that existing decision-making methods are difficult to effectively guide intrusion response decisions in some considerable industrial control scenarios.

[0007] The present invention solves the above-mentioned technical problem with the following technical solution: a cross-layer security decision-making method for industrial control systems based on POMDP, comprising:

[0008] Step 1: Use the monitoring system to perceive the state of the industrial control system where the state information is partially observable. The collected alarm information is used as the state input. The decision model takes action based on the state information. The industrial control system environment changes due to the influence of the action. The training data is collected through the continuous interaction between the model and the industrial control system.

[0009] Step 2: Use the T-DRQN deep reinforcement learning method based on the POMDP decision framework to combine the interactive data with historical observation information for training. After the model reaches convergence, its output in different states is used as the intrusion response strategy, and finally applied to the intrusion response of the industrial control system.

[0010] Step 3: Use the alarm information of network layer device nodes and physical layer sensor readings to evaluate the response effect of the intrusion response strategy, and statistically analyze the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models, and use these indicators to analyze the differences in response performance between different models.

[0011] Furthermore, in step 1, the monitoring system is used to perform state perception on the industrial control system whose state information is partially observable. The collected alarm information is used as state input. The decision model takes action based on the state information. The industrial control system environment changes due to the action. The process of collecting training data through continuous interaction between the model and the industrial control system specifically includes:

[0012] Step 11: Construct some impressive industrial control scenarios based on the reality of missed reports and false reports in the monitoring system.

[0013] Step 12: Construct the state space and action space of the decision system based on the alarm information and defense actions.

[0014] In step 13, the decision model selects a response action based on the state information and obtains training data through continuous interaction with the industrial control system.

[0015] Furthermore, in step 2, the T-DRQN deep reinforcement learning method based on the POMDP decision framework is used to combine the interactive data with historical observation information for training. After the model reaches convergence through learning, its output in different states is used as the intrusion response strategy, and finally applied to the intrusion response of the industrial control system.

[0016] Step 21, read the database data into the data buffer pool;

[0017] Step 22: sampling the data in the data buffer pool;

[0018] Step 23, the recursive neural network processes the sampled information sequence;

[0019] Step 24, the dual-branch neural network architecture evaluates the Q-value function;

[0020] Step 25: Update the neural network parameters using gradient descent.

[0021] In step 26, when the training reaches a threshold and the loss function is relatively stable, the decision model with the best performance in the training process is saved.

[0022] Furthermore, in step 3, the response effect of the intrusion response strategy is evaluated using the alarm information of the network layer device node and the physical layer sensor readings, and the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models is statistically analyzed. The process of using these indicators to analyze the response performance differences between different models specifically includes:

[0023] Step 31, calculating the overall performance of different decision-making methods;

[0024] Step 32: Calculate the action execution costs of different decision-making methods;

[0025] Step 33, statistically analyzing the defense capabilities of different decision-making methods;

[0026] Step 34, counting the offset values ​​of the physical layer sensor readings of different decision methods;

[0027] Step 35: Analyze the reasons for the performance differences of different decision-making methods.

[0028] Based on the above method, the technical solution of the present invention also includes a cross-layer security decision-making system based on POMDP in an industrial control system, including:

[0029] Some impressive industrial control system modules, based on the reality that monitoring systems have missed alarms and false alarms, provide incomplete system status information to the decision-making model, and generate the data required for training through continuous interaction with the decision-making model.

[0030] The intrusion response decision module based on the POMDP decision framework uses the T-DRQN deep reinforcement learning method to combine the interactive data with historical observation information for training. After the model reaches convergence, its output in different states is used as the intrusion response strategy and applied to the industrial control system.

[0031] The model evaluation module uses corresponding evaluation indicators to evaluate the response effect of the intrusion response strategy, and statistically calculates the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models, and uses these indicators to analyze the response performance differences between different models.

[0032] Furthermore, the industrial control system module with a significant portion of status information includes:

[0033] The network layer module is used to simulate the network part of the industrial control system. Its network topology is composed of three layers of network structure: isolated area network, enterprise network, and control network.

[0034] The physical layer module is used to simulate the physical part of the industrial control system, and its physical control process is composed of the Tennessee-Eastman chemical process.

[0035] The monitoring system module is used to simulate the real-world situations of missed alarms and false alarms in industrial control systems, and provide alarm information to the decision-making model in real time.

[0036] Furthermore, the intrusion response decision module based on the POMDP decision framework includes:

[0037] The recurrent neural network module is used to extract temporal features from historical information to help the decision-making model better understand the true state of the system.

[0038] The dual-branch network architecture module is used to decompose the Q-value function into a state-value function and an action-value function, helping the decision-making model to more clearly distinguish between state value and action advantages.

[0039] Furthermore, the model evaluation module includes:

[0040] The response performance evaluation module is used to calculate the corresponding indicators to compare different decision-making methods.

[0041] The response performance explanation module is used to analyze the differences in response performance between different decision-making methods.

[0042] The beneficial effects of the present invention are:

[0043] First, this paper formulates the intrusion response decision-making problem as a partially observable Markov decision process (POMDP) ​​problem, aiming to address the challenge of incomplete access to system state information in real industrial control systems. Within this framework, a deep reinforcement learning-based intrusion response decision-making method is proposed, which effectively defends against multi-stage, cross-layer attacks from cyberspace to physical space.

[0044] Second, to address the proposed POMDP problem and reduce the decision-making approach's over-reliance on system models, this paper develops a model-free reinforcement learning algorithm based on a deep recurrent Q-network (T-DRQN). This algorithm combines LSTM and a deep Q-network (DQN) with a dual-branch network structure to more clearly distinguish between state value and action advantage, thereby guiding intrusion response decisions more accurately and efficiently.

[0045] 3. The present invention was evaluated in a simulated industrial control system. The experimental results show that compared with other existing reinforcement learning-based decision-making methods, the proposed T-DRQN intrusion response decision-making method can significantly improve the security of industrial control systems and is more suitable for industrial control scenarios where the processing system status information is partially considerable. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of a cross-layer security decision-making method for an industrial control system based on POMDP according to the present invention;

[0047] Figure 2 Schematic diagram of a flow chart of interaction between an industrial control system with partially observable state information and a decision model in accordance with the first embodiment of the present invention;

[0048] Figure 3 A schematic diagram of a process for generating a decision strategy using the T-DRQN deep reinforcement learning method based on the POMDP decision framework in Example 1 of the present invention;

[0049] Figure 4 A schematic diagram of a process for performing evaluation and analysis using relevant statistical results and different decision-making methods in Example 1 of the present invention;

[0050] Figure 5 This is a graph showing the experimental results of the total reward values ​​obtained after convergence of different decision-making methods in Example 2 of the present invention;

[0051] Figure 6 This is a graph showing the experimental results of the attacker's attack capabilities under different decision-making methods in Example 2 of the present invention;

[0052] Figure 7 This is a graph showing the experimental results of comparing the maximum position reached by an attack under different decision-making methods in Example 2 of the present invention;

[0053] Figure 8 This is a diagram showing the experimental results comparing the differences in action selection under different decision-making methods in Example 2 of the present invention.

[0054] Figure 9 This is a graph showing experimental results comparing the performance of different decision-making methods under different parameter settings in Example 2 of the present invention. DETAILED DESCRIPTION

[0055] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention but not to limit the scope of the present invention.

[0056] Example 1

[0057] like Figure 1 As shown, Example 1 is a cross-layer security decision-making method for industrial control systems based on POMDP. The method consists of three steps: constructing an industrial control system with partially observable state information, interacting with the decision-making system to generate the required training data, using the T-DRQN deep reinforcement learning method based on the POMDP decision framework to train the decision model, generating the final intrusion response strategy, and using corresponding evaluation indicators to evaluate different decision-making methods and analyze the reasons for the differences. The specific contents are as follows:

[0058] Step 1: Use the monitoring system to perceive the state of the industrial control system where the state information is partially observable. The collected alarm information is used as the state input. The decision model takes action based on the state information. The industrial control system environment changes due to the influence of the action. The training data is collected through the continuous interaction between the model and the industrial control system.

[0059] Step 2: Use the T-DRQN deep reinforcement learning method based on the POMDP decision framework to combine the interactive data with historical observation information for training. After the model reaches convergence, its output in different states is used as the intrusion response strategy, and finally applied to the intrusion response of the industrial control system.

[0060] Step 3: Use the alarm information of network layer device nodes and physical layer sensor readings to evaluate the response effect of the intrusion response strategy, and statistically analyze the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models, and use these indicators to analyze the differences in response performance between different models.

[0061] like Figure 2As shown in the figure, in step 1 above, the monitoring system is used to perceive the state of the industrial control system with partially observable state information. The collected alarm information is used as the state input. The decision model takes action based on the state information. The industrial control system environment changes due to the influence of the action. The specific content of collecting training data through the continuous interaction between the model and the industrial control system is as follows:

[0062] Step 11: Based on the reality of missed reports and false alarms in the monitoring system, construct some impressive industrial control scenarios: divide the status levels of network layer devices according to the behavioral characteristics of cross-layer attacks, and divide the status levels of physical layer devices according to the offset values ​​of sensor readings. Different levels correspond to different alarm rates and false alarm rates.

[0063] Step 12: Construct the state space and action space of the decision-making system based on the alarm information and defense actions: the alarm information provided by all monitoring systems, including industrial control equipment at the network layer and physical layer, is used as the state space, and all defense actions formulated for potential intrusion behaviors are used as the action space.

[0064] In step 13, the decision model selects a response action based on the state information and obtains training data through continuous interaction with the industrial control system. In this process, the decision model executes a response action based on the current system state. The state of the industrial control system changes under the influence of the action, and the reward value and new state information are fed back to the decision model. The current state, action, reward value, and new state are recorded as a four-tuple and saved in the database. The reward value is calculated as follows:

[0065] R(s,a)=α1·R cost +α2·R cyber +α3·R physics

[0066] (α1+α2+α3)·10=1

[0067]

[0068] The total reward is composed of three parts: the cost of the defensive action, the network layer reward, and the physical layer reward. α1, α2, and α3 are the weights of each component of the reward function. Y is the current sensor reading, M is the normal sensor reading, and σ is the standard deviation of the sensor reading under normal operation. The network layer reward is the cumulative sum of the network layer number and the node state of all hacked network nodes. The physical layer reward is the cumulative sum of the deviations of each sensor reading.

[0069] like Figure 3As shown in the above step 2, the T-DRQN deep reinforcement learning method based on the POMDP decision framework is used to combine the interactive data with historical observation information for training. After the model reaches convergence through learning, its output in different states is used as the intrusion response strategy. The specific content of the intrusion response finally applied to the industrial control system is as follows:

[0070] Step 21, reading the database data into the data buffer pool: the training data obtained by the continuous interaction between the decision model and the industrial control system is stored in the data buffer pool to speed up the data reading and processing.

[0071] Step 22, sampling the data in the data buffer pool: extracting a continuous period of data within a certain period of time from the buffer pool in batches, ensuring that the sampled data has time series correlation to maintain the consistency of the training data.

[0072] Step 23: The recursive neural network processes the sampled information sequence: the sampled data is input into the recursive neural network, the temporal features therein are extracted, and a feature sequence with greater contextual relevance is generated.

[0073] Step 24, dual-branch neural network architecture evaluates Q-value function: Use a dual-branch neural network architecture to estimate the Q value and action value in the current state respectively to improve the accuracy and stability of decision-making.

[0074] Step 25: Update the neural network parameters using gradient descent: By calculating the gradient of the loss function, the gradient descent algorithm is used to update the weights and bias parameters of the network to minimize the prediction error.

[0075] Step 26: When the training reaches the threshold and the loss function is relatively stable, save the decision model with the best performance during the training process: After the training reaches the preset conditions, save the current best performing model for subsequent deployment.

[0076] like Figure 4 As shown in the figure, in step 3 above, the alarm information of the network layer device nodes and the physical layer sensor readings are used to evaluate the response effect of the intrusion response strategy, and the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models is statistically analyzed. The specific content of using these indicators to analyze the response performance differences between different models is as follows:

[0077] Step 31: Statistical analysis of the overall performance of different decision-making methods: Calculate the total reward value accumulated by each decision-making method during the entire simulation to evaluate the overall performance; where N e Represents the total number of test rounds, n represents the time step in the test, T represents the simulation time of each round, Rn t Represents the reward value at each time step t (as shown in step 13). The specific calculation formula is as follows:

[0078]

[0079] Step 32: Calculate the action execution costs of different decision methods: Calculate the average execution cost of the defensive action selected by each decision method at each time step; where cost(a) represents the execution cost of the defensive action, and the specific calculation formula is as follows:

[0080]

[0081] Step 33: Statistical analysis of the defense capabilities of different decision-making methods: Calculate the average number of infected nodes per hour under the defense of each decision-making method to evaluate the defense effect; where n compromised Indicates the number of nodes at levels 4 and 5. The specific calculation formula is as follows:

[0082]

[0083] Step 34: Count the offset values ​​of the physical layer sensor readings under different decision methods: Calculate the cumulative offset value of the physical layer sensor readings under the defense of each decision method to evaluate the impact on the system; where Y represents the current reading of the sensor and M is the normal value reading of the sensor. The specific calculation formula is as follows:

[0084]

[0085] Step 35: Analyze the reasons for the performance differences between different decision-making methods: Through multiple rounds of experiments, the maximum number of layers reached by the attacker under the defense of each decision-making method and the frequency of defensive action selection are counted to explain the reasons for the performance differences between the decision-making methods.

[0086] Based on the above method, the technical solution of the present invention also includes a cross-layer security decision-making system based on POMDP in an industrial control system, including:

[0087] Some impressive industrial control system modules, based on the reality that monitoring systems have missed alarms and false alarms, provide incomplete system status information to the decision-making model, and generate the data required for training through continuous interaction with the decision-making model.

[0088] The intrusion response decision module based on the POMDP decision framework uses the T-DRQN deep reinforcement learning method to combine the interactive data with historical observation information for training. After the model reaches convergence, its output in different states is used as the intrusion response strategy and applied to the industrial control system.

[0089] The model evaluation module uses corresponding evaluation indicators to evaluate the response effect of the intrusion response strategy, and statistically calculates the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models, and uses these indicators to analyze the response performance differences between different models.

[0090] The industrial control system modules described above, which provide significant status information, include: a network layer module, which simulates the network portion of the industrial control system, with its network topology consisting of a three-layer network structure: an isolated zone network, an enterprise network, and a control network. A physical layer module, which simulates the physical portion of the industrial control system, with its physical control process consisting of the Tennessee-Eastman chemical process. A monitoring system module, which simulates the real-world scenarios of missed alerts and false alarms in the industrial control system, provides real-time alert information to the decision-making model.

[0091] The described intrusion response decision module based on the POMDP decision framework includes a recurrent neural network module, which is used to extract temporal features from historical information to help the decision model better understand the true state of the system. A dual-branch network architecture module is used to decompose the Q-value function into a state-value function and an action-value function, helping the decision model to more clearly distinguish between state value and action advantages.

[0092] The described model evaluation module includes: a response performance evaluation module for calculating corresponding indicators to compare different decision methods; and a response performance interpretation module for analyzing the differences in response performance between different decision methods.

[0093] Example 2

[0094] In Example 2, the proposed method was tested using the Tennessee Eastman industrial control simulation environment. This industrial control system consists of two parts: the network layer and the physical layer. The network layer consists of nine nodes and three subnets. These subnets are: a demilitarized zone (DMZ) for external networks, containing a web server, a DNS server, and an FTP server; an enterprise network for internal offices, containing three computing nodes and a database; and a control network for the operation and management of the industrial control system, including an engineer station (ES) and a human-machine interface (HMI). The physical layer consists of the Tennessee Eastman process and key components. The main control component of this invention is the valve used to adjust the reactor input (XMV3) and output rate (XMV7). The focus is on the values ​​of four sensors: reactor pressure, separator liquid level, product content in the discharge valve, and separator temperature. The experiment focuses on attacks related to these physical processes.

[0095] In terms of evaluation, the following indicators are used to evaluate the response effect of the decision-making method:

[0096]

[0097] where N e represents the total number of test rounds, n represents the total time steps in the test, T represents the simulation duration of each round, Y represents the current sensor reading, M represents the normal value of the sensor reading, Total Reward represents the overall performance of the intrusion response decision method, and Average Cost Per Step represents the average execution cost of the industrial control system's defensive actions at each time step. Average Hourly Nodes Compromised represents the average number of infected nodes (node ​​status at levels 4 and 5) per hour. This metric measures the attacker's attack capability. Average Deviation Perepisode represents the average deviation of the industrial control system's sensor readings from the normal readings in each simulation round.

[0098] This paper tested the proposed T-DRQN algorithm's ability to resist attacks by simulating a 48-hour operation timeframe for an industrial control system. Experiments with other decision-making algorithms (DQN, PPO, ACSO, and DRQN) were also conducted using the same simulation parameters for comparison. To more fully represent the attack-defense interaction process, this paper divides the continuous timeline into discrete time slices. In each time slice, after the attacker executes an attack action, the defender uses the monitoring system to read the system state, make a decision, and execute the corresponding defensive action. The interaction duration between the attacker and defender in each time slice is set to 0.2 hours.

[0099] Figure 5 The performance of the proposed method and other methods in some industrial control scenarios with significant information is described. Experimental results show that the POMDP-based T-DRQN decision-making method is more suitable for industrial control scenarios with partially significant system state information. The improved T-DRQN method is more stable than other methods and can achieve higher reward values, with improvements of 32.86% and 47.72% compared to the DRQN ​​and ACSO decision-making methods, respectively, and 62.70% and 68.03% compared to the PPO and DQN decision-making methods, respectively.

[0100] Figure 6The attack capabilities of attackers under different defense agents are demonstrated. The proposed T-DRQN defense agent demonstrates the slowest attack progress. When training completes and converges, the average number of infected host nodes per hour is only 0.41 (approaching 0), a reduction of 34.92% and 49.38% compared to the DRQN ​​and ACSO decision-making methods, respectively, and significantly lower than the PPO and DQN decision-making methods. Furthermore, the average execution cost of executing defense actions is the lowest, at only 3.532 per step. This demonstrates that the proposed defense agent can promptly infer system state changes and make appropriate intrusion response decisions at the initial stages of an attack, effectively preventing the further spread of the attack. However, due to the limitations of their MDP-based reinforcement learning algorithms, the PPO and DQN reinforcement learning algorithms can only make intrusion response decisions based on incomplete system state information provided by the current monitoring system, failing to effectively block the attack in a timely manner, resulting in higher values ​​for their related evaluation metrics.

[0101] In order to analyze the differences in the experimental results of different intrusion response decision methods, after the training reached convergence, the present invention conducted 100 rounds of attack tests on different decision methods under a simulation running time of 48 hours, and counted the maximum position reached by the attack in the industrial control system.

[0102] according to Figure 7 The results show that the proposed T-DRQN decision-making method can block the vast majority of attacks within the demilitarized zone (DMZ) (93%), with only a small portion reaching the second-layer control network (7%), thus successfully blocking attacks before they reach the physical layer of the industrial control system. The DRQN ​​decision-making method also achieved good results (85% in the DMZ and 15% in the enterprise network). The ACSO decision-making method can block attacks within the first three layers of the industrial control system network. In contrast, while the PPO and DQN decision-making methods can block most attacks within the enterprise network (70% and 66%, respectively), a small portion of attacks still reach the control network (10% and 17%) and the physical layer of the industrial control system (5% and 7%), posing a serious threat to the security of the industrial control system. Therefore, these two decision-making methods are limited in their effectiveness in blocking attacks, which is why their total reward values ​​are relatively low.

[0103] To further analyze the differences between the T-DRQN and DRQN ​​decision-making methods, we conducted 100 rounds of testing and counted the frequencies of defensive action selections, obtaining detailed information on action selection between the two decision-making methods. Figure 8 The action selection frequency distributions of the two decision-making methods after training reaches convergence are shown.

[0104] like Figure 8As shown in Figure 2, the two decision-making methods have similar trends. In particular, the execution frequency of defense actions for DMZ network devices is significantly higher, such as actions 1-3, 10-12, 19-21, 28-30, and 37-40. Figure 7 The maximum stage of the attack can be understood as the reason for this phenomenon. The intelligent agent wants to block the attack in the isolated network as much as possible to minimize the further spread of the attack and prevent it from causing serious damage to the industrial control system. Further analysis shows that, in addition to defense actions 1-3, the T-DRQN decision method is more inclined to execute defense actions 10-12. These defense actions can block the attack in the initial stage of the attack, while the DRQN ​​decision method is more inclined to execute actions 37-40. Although these defense actions can effectively block the attack, their cost overhead is relatively high. This difference in action selection is mainly due to the dual-branch structure of the T-DRQN algorithm. This structure can more clearly distinguish the value of the current state and the pros and cons of different actions through the value function V and the action advantage function A, thereby enabling the decision method to timely perceive changes in the system state in the initial stage of the attack and make appropriate intrusion response decisions. For enterprise networks, defense actions 4-7 and 13-16 are executed more frequently. In contrast, the execution frequency of defense actions for the control network and physical layer is lower. This phenomenon can be explained by the fact that the T-DRQN algorithm has a high probability of execution. Figure 7 The results of are combined to explain this, because both decision-making methods can effectively block the attack in the first two layers of the network.

[0105] In order to evaluate the performance of the decision-making method under different parameter settings, the present invention defines four groups of alarm rates corresponding to different alarm levels 1-5, namely λ1 = {0.1, 0.3, 0.4, 0.5, 0.6}, λ2 = {0.2, 0.4, 0.5, 0.6, 0.7}, λ3 = {0.3, 0.5, 0.6, 0.7, 0.8}, and λ4 = {1.0, 1.0, 1.0, 1.0, 1.0}. When the parameter is set to 4, it means that alarm information of any level can be fully perceived by the monitoring system. At this time, the partially observable POMDP scenario degenerates into a fully observable MDP scenario.

[0106] like Figure 9As shown in the figure, the experimental results show that as the alarm rate increases, the reward value obtained by the decision-making method gradually becomes higher. Among them, the changes of the MDP-based decision methods PPO and DQN are more significant. In the POMDP scenario, the reward value is significantly lower than that of the other two decision-making methods. However, when the parameter is set to 4, the difference in the reward value of the five decision-making methods in the MDP scenario is not significant. This shows that these two decision-making methods rely on the complete perception of the system environment state and are difficult to cope with some considerable industrial control scenarios. For the three decision-making methods of ACSO, DRQN ​​and T-DRQN, since they can infer the system state based on historical information, the reward value obtained is relatively high. Among them, the reward value obtained by the T-DRQN decision method proposed in this invention is higher than that of other decision-making methods under different parameter settings. Combined with previous experiments, these results clearly show that the T-DRQN method is more suitable for processing industrial control scenarios with partially considerable system state information than other decision-making methods.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cross-layer security decision-making method for industrial control based on partially observable Markov decision making, characterized by: include: Step 1: Use the monitoring system to perceive the state of an industrial control system where state information is partially available. Collected alarm information is used as state input. The decision model takes actions based on this state information. The industrial control system environment changes as a result of these actions. Training data is collected through continuous interaction between the model and the industrial control system. Step 2: Use the T-DRQN deep reinforcement learning method based on the POMDP decision framework to combine the interactive data with historical observation information for training. After the model reaches convergence, its output in different states is used as the intrusion response strategy, which is ultimately applied to the intrusion response of the industrial control system. Step 3: Use the alarm information from network-layer device nodes and the physical-layer sensor readings to evaluate the effectiveness of the intrusion response strategy. Statistically analyze the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models. Use these metrics to analyze the differences in response performance between different models. In step 2, the T-DRQN deep reinforcement learning method based on the POMDP decision framework is used to combine the interactive data with historical observation information for training. After the model reaches convergence through learning, its output in different states is used as the intrusion response strategy, which is finally applied to the intrusion response of the industrial control system. Step 21, read the database data into the data buffer pool; Step 22: sampling the data in the data buffer pool; Step 23, the recursive neural network processes the sampled information sequence; Step 24, the dual-branch neural network architecture evaluates the Q-value function; Step 25, gradient descent updates the neural network parameters; Step 26: When the training reaches the threshold and the loss function is stable, save the decision model with the best performance during the training process; Step 31: Statistical analysis of the overall performance of different decision-making methods: Calculate the total reward value accumulated by each decision-making method during the entire simulation to evaluate the overall performance; where N e Represents the total number of test rounds, n represents the time step in the test, T represents the simulation time of each round, Rn t Represents the reward value at each time step t. The specific calculation formula is as follows: Step 32: Calculate the action execution costs of different decision methods: Calculate the average execution cost of the defensive action selected by each decision method at each time step; where cost(a) represents the execution cost of the defensive action, and the specific calculation formula is as follows: Step 33: Statistical analysis of the defense capabilities of different decision-making methods: Calculate the average number of infected nodes per hour under the defense of each decision-making method to evaluate the defense effect; where n compromised Indicates the number of nodes at level 4 and 5. The specific calculation formula is as follows: Step 34: Count the offset values ​​of the physical layer sensor readings under different decision methods: Calculate the cumulative offset value of the physical layer sensor readings under the defense of each decision method to evaluate the impact on the system; where Y represents the current reading of the sensor and M is the normal value reading of the sensor. The specific calculation formula is as follows: Step 35: Analyze the reasons for the performance differences between different decision-making methods: Through multiple rounds of experiments, the maximum number of layers reached by the attacker under the defense of each decision-making method and the frequency of defensive action selection are counted to explain the reasons for the performance differences between the decision-making methods.

2. The industrial control cross-layer security decision-making method based on partially observable Markov decision according to claim 1 is characterized in that: In step 1, the monitoring system is used to perceive the state of the industrial control system with partially observable state information. The collected alarm information is used as the state input. The decision model takes action based on the state information. The industrial control system environment changes due to the action. The process of collecting training data through continuous interaction between the model and the industrial control system specifically includes: Step 11: Build some achievable industrial control scenarios based on the reality of missed reports and false reports in the monitoring system; Step 12: construct the state space and action space of the decision-making system based on the alarm information and defense actions; In step 13, the decision model selects a response action based on the state information and obtains training data through continuous interaction with the industrial control system.

3. A cross-layer security decision-making system for POMDP using the method according to any one of claims 1-2, characterized in that: include: Some impressive industrial control system modules, based on the reality that monitoring systems have missed reports and false reports, provide incomplete system status information to the decision model and generate the data required for training through continuous interaction with the decision model; The intrusion response decision module, based on the POMDP decision framework, uses the T-DRQN deep reinforcement learning method to combine interactive data with historical observation information for training. After the model reaches convergence, its output in different states is used as the intrusion response strategy for application to the industrial control system; The model evaluation module uses corresponding evaluation indicators to evaluate the response effect of the intrusion response strategy, and statistically calculates the frequency distribution of the attacker's arrival location and action selection in different intrusion response decision models, and uses these indicators to analyze the response performance differences between different models.

4. The cross-layer security decision-making system of POMDP according to claim 3, characterized in that: The industrial control system modules that can provide status information include: The network layer module is used to simulate the network part of the industrial control system. Its network topology is composed of three layers: isolated area network, enterprise network, and control network. The physical layer module is used to simulate the physical part of the industrial control system, and its physical control process is composed of the Tennessee-Eastman chemical process; The monitoring system module is used to simulate the real-world situations of missed alarms and false alarms in industrial control systems, and provide alarm information to the decision-making model in real time.

5. The cross-layer security decision-making system of POMDP according to claim 3, characterized in that: The intrusion response decision module based on the POMDP decision framework includes: A recurrent neural network module, which is used to extract temporal features from historical information to help the decision-making model better understand the true state of the system; The dual-branch network architecture module is used to decompose the Q-value function into a state-value function and an action-value function, helping the decision-making model to more clearly distinguish between state value and action advantages.

6. The cross-layer security decision-making system of POMDP according to claim 3, characterized in that: The model evaluation module includes: Response performance evaluation module, used to calculate corresponding indicators to compare different decision-making methods; The response performance explanation module is used to analyze the differences in response performance between different decision-making methods.

Images