An automatic defense decision-making method for industrial control networks with partially unknown security states

Abstract

This application belongs to the field of industrial control network technology, specifically a method for automatic defense decision-making for industrial control networks with partially unknown security states, including the steps of generating an attack graph, estimating the state of a dynamic attack graph, and making a dynamic defense decision. This application can realize the linkage between the attack graph and the real environment; it solves the problem of partially unknown security states of monitoring nodes caused by the concealment of attackers' attacks, and can estimate the node states; it uses the DQN algorithm and defines actions and reward values ​​based on industrial control networks, which can make automatic defense decisions; its practical scope is to make automatic defense decisions for industrial control networks with unknown security states of some devices in applicable scenarios of industrial control systems, providing technical support for the security of industrial control systems, and has a very broad application prospect.

Description

Technical Field

[0001] The present application belongs to the field of industrial control network technology, and more specifically, relates to an automatic defense decision-making method for an industrial control network facing a partially unknown security state. Background Art

[0002] With the development of network technology, industrial control systems (ICS) are gradually connected to the Internet to achieve remote monitoring and management, which brings more convenience and efficiency, but at the same time increases network security risks. In addition, ICS has a long life cycle. Due to real-time requirements, many system software are not easy to update, making the system vulnerable to attacks. Given the limited resources of system security managers, usually only key equipment is monitored. However, the attack behavior of some attackers is relatively covert, which makes it impossible for system security managers to clearly know whether all devices have been attacked. Once the ICS is destroyed, it will cause great losses to production and operations, so the attack needs to be dealt with in a timely manner. For the above-mentioned industrial control networks facing partially unknown security states, there is an urgent need to propose a feasible defense decision-making method. At present, there are three common technologies for analyzing industrial control network security:

[0003] Attack graphs are used to model vulnerability exploitation relationships within a network. From the attacker's perspective, they analyze the network environment and vulnerability information to enumerate all attack paths, visually demonstrating the causal relationships between different attack steps and the potential threats posed by vulnerability exploitation. For industrial control networks, where the network environment is often volatile, attack graphs can be used to analyze network vulnerabilities.

[0004] The Partially Observable Markov Decision Process (POMDP) ​​is a probabilistic finite-state Markov decision process model that can solve decision-making problems with uncertainty. It can be used to evaluate network attacks and defenses, and to implement defense strategies. For industrial control networks, the security status of some devices in the network is unknown due to the stealthiness of attackers. Therefore, the state changes of monitored devices can be modeled as POMDPs. By analyzing the attacker's state in the network, the state of the partially unknown industrial control network can be estimated, enabling better defense strategies.

[0005] Reinforcement Learning (RL) is a branch of machine learning that studies how to achieve specific goals through the interaction of intelligent agents with their environments to obtain optimal strategies. In RL, an agent observes the state of the environment, selects actions, and adjusts its strategy based on rewards to maximize cumulative rewards. Defense decisions for industrial control networks can be modeled as RL processes. System security managers need to execute defensive actions that maximize cumulative rewards based on the attacker's current activity within the industrial control network.

[0006] In summary, an attacker's attack may occur at any time, and attack graph technology is unable to handle such dynamically changing attacks, which can cause immeasurable damage to industrial control networks with high real-time requirements. Due to the concealed nature of the attacker's attack, the security status of some devices in the industrial control network is unknown. The intelligent agent can only observe partial information and cannot directly observe the complete status. RL makes decisions based on known information and previous experience and cannot handle this incomplete status information. Current patents lack a definition of defensive actions in industrial control networks and lack effective means to measure the usability of defensive actions. Furthermore, some patents combine RL and POMDP to enable RL to handle partially unknown security states, but they do not consider the ability to infer the attacker's true activity status in the industrial control network based on the attack graph.

[0007] Application Contents

[0008] To achieve the above objectives, the technical solution adopted in this application is to provide an automatic defense decision-making method for industrial control networks with partially unknown security states, including an attack graph generation step, a dynamic attack graph state estimation step, and a dynamic defense decision-making step;

[0009] The attack graph generation step generates a corresponding attack graph based on network topology information, device service information, and vulnerability information; defines the node status in the attack graph;

[0010] The dynamic attack graph state estimation step involves monitoring the device status in the network, linking the static attack graph with the dynamic attacker, observing the current attack status of the industrial control network, and obtaining observation information of the dynamic attack graph; combining POMDP modeling with recurrent neural networks to model the historical observation information and defense actions of the dynamic attack graph and estimate the state of each node in the dynamic attack graph;

[0011] The dynamic defense decision-making step adopts the RL algorithm to learn the defense actions under different states based on the state estimated by the dynamic attack graph to achieve the defense purpose.

[0012] Optionally, the step of generating an attack graph includes information collection, attack path analysis, and dynamic attack graph construction;

[0013] The information is collected in the following ways:

[0014] First, collect network topology information;

[0015] Secondly, collect device information;

[0016] Finally, collect vulnerability information.

[0017] Optionally, the attack path analysis method is as follows:

[0018] Step S101, determining the attack target, based on the collected network topology information, device information and vulnerability information;

[0019] Step S102, determining the attack path, and determining the attack path according to the attack target;

[0020] Step S103: Draw an attack graph. Draw the attack graph based on the attack target and the attack path. The attack graph is used to represent information about the attack target and the attack path.

[0021] Optionally, the dynamic attack graph is constructed to define the status of each node in the attack graph, and each node indicates whether the corresponding device is attacked;

[0022] The construction method of dynamic attack graph construction is as follows:

[0023] Step S201: Obtain network topology information, device service information, and vulnerability information to create an attack graph;

[0024] Step S202: according to the preset time step Collect network data;

[0025] Step S203, determining attack behaviors occurring in the network based on the collected network data;

[0026] Step S204: modify the attack graph obtained in step S103 according to the determined attack behavior, and display the attack process in chronological order.

[0027] Optionally, the dynamic attack graph state estimation step includes POMDP modeling and RNN hidden information acquisition;

[0028] The process of POMDP modeling is as follows:

[0029] First, define the state space S to represent all possible states; define the action space A to represent the actions that the agent can perform; define the observation space Z to represent the information the agent can obtain about the current state; define the state transition function T to represent the probability distribution from state s to state s'; define the observation function O to represent the probability distribution corresponding to the observation obtained by the agent; define the reward function R to represent the reward obtained when performing action a from state s to reach state s'; and define the initial state of the agent;

[0030] Then, a POMDP model is established to describe the changing process of the dynamic attack graph, enabling the agent to make decisions. ;

[0031] , according to the strategy , you can get the expected cumulative reward:

[0032]

[0033] in, is the discount factor, Indicates the mathematical expectation value. is the state at time step t, is the state at time step t+1, is the action with time step t, is the time step;

[0034] make Represents historical information of observations and actions, and adopts belief states based on historical information Inferring the state distribution of nodes, belief state The distribution of is:

[0035]

[0036] yes The function of , which means estimating the node status through historical information;

[0037] When learning the function When , the state estimation of the dynamic attack graph at the current moment is obtained based on the historical information of observations and actions.

[0038] Optionally, the method for obtaining RNN hidden information is as follows:

[0039] Step S301, obtaining historical information of observations and actions, including historical observation values ​​and historical action values;

[0040] Step S302: for each time step , convert historical observation values ​​and historical action values ​​into vector representations and then process them through RNN;

[0041] Step S303, at the last time step of RNN , output a vector representing the estimated state value of the dynamic attack graph at the current moment;

[0042] Step S304: pass the output vector to a state estimation function to map it to a state in the actual state set.

[0043] Optionally, in step S301, when at time step Obtaining observations of dynamic attack graphs , and perform defensive actions When the dynamic attack graph changes from the current state Transition to the next state , and generate new observations ; Record the action taken at each time step and the observation obtained by performing the action, and form a time series of actions and observations:

[0044] ,

[0045] The obtained time series information is used as the historical information of observations and actions;

[0046] In step S302, at time step , the calculation of RNN is shown in formula (1):

[0047] (1)

[0048] in, is the time step The hidden state of is the input data, and are the weight matrices of hidden state and input data respectively, is the bias term of the hidden state, is the activation function.

[0049] Optionally, the dynamic defense decision-making step includes defining actions and rewards, and training defense strategies;

[0050] The action and reward are defined, and the agent is used to perform the defense action. When the dynamic attack graph is in the time step The state is recorded as , the agent performs actions in the network Time, action Change the state in the dynamic attack graph and make the agent Reach a new state , the environment generates feedback rewards Given the agent, the agent is based on the new state and feedback rewards Perform new actions , and iteratively interacts with the environment to achieve the agent's cumulative reward G;

[0051] The cumulative reward G is .

[0052] Optionally, the status Execute an action , the system security manager receives the utility value , as shown in formula (2):

[0053] (2)

[0054] in, It is the reward for maintaining network security. Is to perform an action the costs incurred; As shown in formula (3):

[0055] (3)

[0056] R represents the basic security level of the network, which is expressed as a constant; the basic indicators in the common vulnerability assessment system are used to evaluate the network security level. 、 and Conduct assessments; Indicates that after being attacked by the attacker, the node The impact of the action on the availability of the service is defined as .

[0057] Optionally, the training defense strategy adopts a deep Q network training defense strategy, and the training method is as follows:

[0058] Step S401, determining the input dimension of the deep Q network according to the scale of the dynamic attack graph;

[0059] Step S402: establishing a deep Q network to estimate the Q value of each action taken under the current input;

[0060] Step S403: Deep Q network adopts Greedy strategy, used to balance the agent's exploration of new strategies and utilization of existing experience during the training phase;

[0061] Step S404: training the deep Q network. The training data is obtained by the interaction between the agent and the dynamic attack graph, including the state, the action taken, the next state, and the reward signal.

[0062] Step S405: The deep Q network includes a behavior network and a target network. After training for a fixed number of steps C, the parameters of the behavior network are copied to the target network to achieve a stable training process.

[0063] Step S406: Save the model until the average cumulative reward obtained from the agent training no longer changes.

[0064] This application proposes an automatic defense decision-making method for industrial control networks with partially unknown security states. This method establishes an attack graph for the industrial control network, redefines the meaning of the nodes in the attack graph, and constructs a dynamic attack graph. It then combines an RNN network with a POMDP model to estimate the true state of the dynamic attack graph. Based on the estimated true state, the DQN algorithm is used to select the optimal defense strategy for automatic defense. Specifically, this method has the following beneficial effects:

[0065] (1) Defining nodes in the attack graph can realize the linkage between the attack graph and the real environment;

[0066] (2) It solves the problem of unknown security status of monitoring nodes caused by the concealment of attackers' attacks, and can estimate the node status;

[0067] (3) Using the DQN algorithm and defining actions and reward values ​​based on the industrial control network, automatic defense decisions can be made;

[0068] (4) The scope of application is to make automatic defense decisions for industrial control networks in the context of industrial control systems where some equipment is in an unknown security state, thus providing technical support for the security of industrial control systems. The application prospects are very broad. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0070] Figure 1 This is an architectural diagram of the automatic defense decision-making method for industrial control networks with partially unknown security states in this application;

[0071] Figure 2 Attack graph after modeling the industrial control network;

[0072] Figure 3 This is a schematic diagram of a dynamic attack graph;

[0073] Figure 4 Schematic diagram of state estimation of dynamic attack graph;

[0074] Figure 5 Schematic diagram of deep Q network (DQN) training. DETAILED DESCRIPTION

[0075] In order to make the technical problems, technical solutions and beneficial effects to be solved by this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0076] The automatic defense decision-making method for industrial control networks facing partially unknown security states provided by the embodiment of the present application is now described. Figure 1 This paper proposes an automated defense decision-making method for industrial control networks with partially unknown security states. First, using the MulVAL tool, an attack graph is generated based on network topology, device service information, and vulnerability information. Node states in the attack graph are defined as either attacked or unattacked. By continuously monitoring the status of network devices, the static attack graph is linked to dynamic attackers, observing the current attack state of the industrial control network and obtaining observational information for the dynamic attack graph. The method then combines POMDP with a recurrent neural network (RNN) to model the historical observations of the dynamic attack graph and the defensive actions taken by the system security manager, estimating the true state of each node in the dynamic attack graph. Finally, based on the estimated true state, a reinforcement learning algorithm is used to continuously learn the optimal defensive actions under different states, achieving the defense objective.

[0077] Specifically, the following steps are included:

[0078] Attack graph generation step, dynamic attack graph state estimation step and dynamic defense decision step.

[0079] Among them, the steps of generating the attack graph include information collection, attack path analysis and dynamic attack graph construction.

[0080] Furthermore, the information collection method is as follows:

[0081] First, collect network topology information. By analyzing and segmenting the network structure, we can obtain the network topology, including information such as the connection between devices and network subnets. This information is the basis for constructing the attack graph and provides the foundation for subsequent information collection.

[0082] You can use scanning tools such as Nmap and Masscan to obtain network topology information, including host IP addresses, open ports, and services.

[0083] Secondly, device information is collected. In this step, it is necessary to obtain information such as the operating system, services, running applications, and configuration of each device. This information can help attackers identify weaknesses and vulnerabilities in the system.

[0084] Finally, vulnerability information is collected. Attackers need to obtain information about known vulnerabilities in the system in order to launch targeted attacks. Vulnerability information can be collected using vulnerability information and vulnerability scanning tools.

[0085] Using vulnerability scanning tools such as Metasploit, Nessus, OpenVAS, etc., you can automatically scan for vulnerabilities in the system and use the collected vulnerability information to generate attack graphs to more accurately simulate the attacker's behavior.

[0086] Furthermore, the attack path analysis method is as follows:

[0087] Step S101, determine the attack target: determine the attack target based on the collected network topology information, device information and vulnerability information. For example, a specific device, network or application;

[0088] Step S102, determining the attack path: by analyzing the weaknesses and vulnerabilities of the system and according to the attack target, determining the attack path, that is, the possible methods that the attacker may take to achieve the attack purpose.

[0089] Step S103, drawing an attack graph: drawing an attack graph based on the attack target and attack path. The attack graph is used to represent information such as the attack target and attack path.

[0090] In step S103, an attack graph can be generated using an attack graph generation tool. The attack graph generation tool graphically displays the attack process based on the collected network topology information, device information, and vulnerability information according to the existing inference rules. For example, MulVAL uses the Graphviz image generator to draw an attack graph based on the vulnerability scanning results of vulnerability scanners such as Nessus or OVAL, the configuration information of network nodes, and other relevant information. CVE vulnerability information refers to a public vulnerability information library used to identify and track publicly disclosed computer security vulnerabilities. Each vulnerability is assigned a unique identifier (CVEID), such as "CVE-2021-12345". This identifier can be searched on the CVE vulnerability information website to learn more about the vulnerability, the scope of impact, solutions, etc.

[0091] See also Figure 2This example uses an attack graph generated in an industrial control network as an example. PC1, PC2, and PC3 are host devices in the network. Historian is the historical database. SCADA1 and SCADA2 are data acquisition and monitoring devices. EWS is the engineer station, OWS is the operator station, and PLC1 and PLC2 are control devices. Based on the reachability relationships between devices and their CVE vulnerability information, each node displays only the device name. The attack graph is drawn as follows.

[0092] Furthermore, nodes in the attack graph represent attack attributes such as the attacker's privilege level and device vulnerabilities. Based on the dependencies between nodes, an attack graph is drawn from the attacker to the target device node. Because attack graphs are static and cannot dynamically analyze the network's attack status, a dynamic attack graph construction method is used to define the status of each node in the attack graph, with each node indicating whether the corresponding device is attacked.

[0093] The construction method of dynamic attack graph construction is as follows:

[0094] Step S201: Obtain network topology information, device service information, and vulnerability information to create an attack graph;

[0095] Step S202: collecting network data at regular time intervals, such as network traffic information, log information, etc.

[0096] Step S203, determining attack behaviors occurring in the network based on the collected network data;

[0097] Step S204: modify the attack graph obtained in step S103 according to the determined attack behavior, and display the attack process in chronological order;

[0098] Among them, step S202 to step S204 can be repeated in a cycle.

[0099] By linking the attack behavior in the industrial control network with the attack graph, the attacker's attack can be mapped to the attack graph, which is conducive to analyzing the attacker's subsequent actions and goals. Figure 2 and Figure 3 As shown in the figure, at time step t-1, the attacker compromises devices PC1 and PC2, and then further compromises the Historian database and SCADA1 at time steps t and t+1. The figure below details the attacker's attack route, which helps system security managers make defensive decisions.

[0100] The dynamic attack graph state estimation step includes POMDP modeling and RNN hidden information acquisition.

[0101] Because attackers' attacks are often covert, some attacks go undetected by system security managers. Therefore, the dynamic attack graph generated in the attack graph generation step may not reflect the true state of the network. To describe the discrepancy between the observed state and the true state, the state change process of the dynamic attack graph can be modeled as a POMDP.

[0102] Furthermore, the process of POMDP modeling is as follows:

[0103] In POMDP, the state is unknown and can only be obtained through observation. Define the state space S to represent all possible states; define the action space A to represent the actions that the agent can perform; define the observation space Z to represent the information that the agent can obtain about the current state; define the state transition function T to represent the probability distribution from state s to state s', these probabilities are not completely known; define the observation function O to represent the probability distribution corresponding to the observation obtained by the agent; define the reward function R to represent the reward obtained when performing action a from state s to state s'; finally, define the initial state of the agent;

[0104] Through the above steps, the POMDP model is successfully established to describe the changing process of the dynamic attack graph, so that the agent can make decisions under incomplete information. The purpose of POMDP modeling is to design strategies. , so that it can maximize the expected cumulative reward:

[0105]

[0106] in, is the discount factor, Indicates the mathematical expectation. Represents the historical information of observations and actions. Since POMDP is partially observable, it is necessary to combine historical information with the uncertainty of the unobservable aspects of the node to infer the true state distribution of the node based on the historical information, using the belief state , define the distribution of belief states as:

[0107]

[0108] yes The function of , which means estimating the true state of the node through historical information. When the real state information of the environment at the current moment is obtained directly based on the historical information of observations and actions.

[0109] Furthermore, the method of obtaining RNN hidden information is as follows:

[0110] Step S301, obtaining historical information of observations and actions, including historical observation values ​​and historical action values;

[0111] When in time step Obtaining observations of dynamic attack graphs , and perform defensive actions When the dynamic attack graph changes from the current state Transition to the next state , and generate new observations . Record the action taken at each time step and the observation obtained by performing the action, and the obtained action and observation will form a time series: , the obtained time series information is used as the historical information of observations and actions, and then the hidden information in the historical information of observations and actions is learned using RNN.

[0112] Step S302: for each time step , convert historical observation values ​​and historical action values ​​into vector representations and then process them through RNN;

[0113] RNN uses a hidden state to record and save previous information and use this information in the current time step, which can effectively capture long-term dependencies in the data. Its core is to share parameters across time steps so that the same weights are used for any position in the sequence. , the calculation formula of RNN is shown in (1):

[0114] (1)

[0115] in, is the time step The hidden state of is the input data, and are the weight matrices of hidden state and input data respectively, is the bias term of the hidden state, is the activation function. In RNN, the hidden state is The RNN network is trained by observing and taking actions in order to learn the past information. To obtain the true state of the dynamic attack graph at the current moment.

[0116] Step S303, at the last time step of RNN , output a vector representing the estimated state value of the dynamic attack graph at the current moment;

[0117] Step S304: pass the output vector to a state estimation function to map it to a state in the actual state set.

[0118] When using RNN for state estimation, the output vector cannot usually be directly interpreted as the actual state, so a state estimation function is needed to map it to a real state in the set of actual states.

[0119] by Figure 4 For example, Figure 4 The left side of the figure shows the dynamic attack graph at time step The observation state , the attacker has destroyed the device PC1 and the Historian database, but the two devices PC1 and PC2 are not directly connected. Therefore, a device has been destroyed by the attacker, but the system security manager failed to detect it. By using RNN for state estimation, we can get the following Figure 4 The actual state of the dynamic attack graph shown on the right , the attacker also compromised the device PC2.

[0120] The dynamic defense decision-making steps include defining actions and rewards, and training defense strategies.

[0121] Furthermore, the actions and rewards are defined according to the RNN to obtain the dynamic attack graph at time step The real state information on the system, the system security manager needs to take action to defend. This application uses reinforcement learning to train the intelligent agent to perform defensive actions. When the dynamic attack graph is in time step The state is recorded as , the agent performs actions in the network Time, action Changed the state in the dynamic attack graph and made the agent Reach a new state , the environment generates feedback rewards To the agent. The agent is based on the new state and feedback rewards Perform new actions , and iteratively interacts with the environment to maximize the cumulative reward G of the agent.

[0122] The cumulative reward G can be defined as .

[0123] Table 1 Defensive action table

[0124]

[0125] To train the aforementioned intelligent agent, this application defines the actions that the agent can perform and the corresponding rewards it receives. Industrial control systems require high reliability and real-time performance, so defense measures for industrial control networks must consider both real-time performance and stability to avoid impacts to the system during the defense process. Table 1 defines some examples of action names and their physical meanings.

[0126] Shutting down remote access can prevent attackers from further infiltrating the network, but may affect remote maintenance and troubleshooting. Stopping critical processes can prevent attackers from using infected devices to further attack other systems, but this may affect the availability and reliability of the system. Isolating devices can prevent attackers from continuing to control the device, but this needs to be weighed against the reliability and availability losses that isolation may bring. Empty actions have no impact on the network. In order to quantify the value of network defense actions under different states, it is necessary to assign a utility value to each pair of defense action and system state. After executing the defense action, a utility value can be obtained to quantify the cost-effectiveness of the action. The agent evaluates the executed actions based on the utility value received, and guides the agent to execute the action with the largest cumulative utility value.

[0127] In state Execute an action , the system security manager receives the utility value ,

[0128] (2)

[0129] in It is the reward for maintaining network security. Is to perform an action Costs incurred, such as isolation of equipment, etc. Defined by confidentiality, integrity, and availability impact (CIA) ,

[0130] (3)

[0131] R represents the basic security level of the network and can be expressed as a constant. The basic indicators in the Common Vulnerability Scoring System (CVSS) are used to evaluate the network security level. 、 and Conduct an assessment, Indicates that after being attacked by the attacker, the node The impact on confidentiality, integrity, and availability of the service is defined in terms of the impact of the action on the availability of the service. To express the reward for keeping the network safe, The total impact score of damaged nodes in the network is subtracted from . Therefore, The larger the value, the more secure the network.

[0132] Furthermore, Deep Q-Network (DQN) is used to train defense strategies to solve the problem of large state space of dynamic attack graphs. The specific training method is as follows:

[0133] Step S401, determining the input dimension of the deep Q network (DQN) according to the scale of the dynamic attack graph;

[0134] Step S402: establishing a deep Q network (DQN) to estimate the Q value of each action taken under the current input;

[0135] Step S404, Deep Q Network (DQN) adopts The greedy strategy is used to balance the agent's exploration of new strategies and the use of existing experience during the training phase, ensuring that the agent can discover new states and actions and preventing the agent from falling into a local optimal solution.

[0136] Step S403: training a deep Q network (DQN). The training data is obtained by the interaction between the agent and the dynamic attack graph, including the state, the action taken, the next state, and the reward signal.

[0137] In step S405, the Deep Q-Network (DQN) consists of a behavior network and a target network. After training for a fixed number of steps, C, the parameters of the behavior network are copied to the target network to ensure a stable training process. The behavior network estimates the Q-values ​​of all possible actions in the current state and selects the action with the maximum Q-value as the action for the current state. This allows the agent to learn which action yields the maximum cumulative reward in the current state. The target network, a copy of the behavior network, is used to calculate the target Q-value, reducing target volatility during the update process and ensuring the stability of the training process. Specifically, it ensures the stability of the target Q-value function and prevents the direction of the target network's gradient descent from being disturbed by real-time changes.

[0138] In order to stabilize the training process, DQN also uses a target network, which can ensure the stability of the target Q value function and prevent the direction of gradient descent from being disturbed by real-time changes.

[0139] Step S406: Save the model until the average cumulative reward obtained from the agent training no longer changes.

[0140] Figure 5 With the Deep Q Network (DQN) shown in the figure, system security managers can select the defense action with the highest Q value based on the state of the dynamic attack graph, so as to minimize the impact on the industrial control network.

[0141] This application proposes an automated defense decision-making method for industrial control networks in partially unknown security states. This method combines attack graphs, RNNs, POMDPs, and DQNs, enabling real-time defense of industrial control networks. Specifically, by establishing a dynamic attack graph for the industrial control network and linking the static attack graph with dynamic attackers, it is possible to observe the attacker's activities within the industrial control network, providing a visual perspective for system security managers to make automated defense decisions. The RNN and POMDP models are combined to estimate the true state of the dynamic attack graph. Based on this estimated true state, the optimal defense strategy is selected for automated defense. Compared to traditional defense methods, the proposed method can more comprehensively observe and evaluate the attacker's activities within the industrial control network, improving the efficiency and accuracy of system security managers' decisions. Furthermore, by fully considering the dynamic nature of attackers and the problem of incomplete information in industrial control networks, it can provide targeted automated defense for industrial control networks in partially unknown security states. This method demonstrates high innovation and practicality in the automated defense of industrial control networks, and has promising application prospects.

[0142] This application defines nodes in the attack graph, which can realize the linkage between the attack graph and the real environment; it solves the problem of some unknown security states of monitoring nodes caused by the concealment of attackers' attacks and can estimate the node states; it uses the DQN algorithm and defines actions and reward values ​​based on industrial control networks to make automatic defense decisions; its practical scope is to make automatic defense decisions for industrial control networks with unknown security states of some equipment in applicable scenarios of industrial control systems, providing technical guarantees for the security of industrial control systems, and has very broad application prospects.

[0143] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. An automatic defense decision-making method for industrial control networks facing partially unknown security states, characterized by: It includes the steps of generating attack graph, estimating the state of dynamic attack graph and dynamic defense decision-making; The attack graph generation step generates a corresponding attack graph based on network topology information, device service information, and vulnerability information; and defines node states in the attack graph; The dynamic attack graph state estimation step is to monitor the state of devices in the network, associate the static attack graph with the dynamic attacker, observe the current attack status of the industrial control network, and obtain observation information of the dynamic attack graph; Combining POMDP modeling with recurrent neural networks, we model the historical observation information and defense actions of the dynamic attack graph and estimate the state of each node in the dynamic attack graph. The dynamic defense decision-making step adopts the RL algorithm to learn the defense actions under different states based on the state estimated by the dynamic attack graph to achieve the defense purpose.

2. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state according to claim 1, characterized in that: The step of generating an attack graph includes information collection, attack path analysis and dynamic attack graph construction; The information is collected in the following ways: First, collect network topology information; Secondly, collect device information; Finally, collect vulnerability information.

3. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state as claimed in claim 2, characterized in that: The attack path analysis method is as follows: Step S101, determining the attack target, based on the collected network topology information, device information and vulnerability information; Step S102, determining the attack path, and determining the attack path according to the attack target; Step S103: Draw an attack graph. Draw the attack graph based on the attack target and the attack path. The attack graph is used to represent information about the attack target and the attack path.

4. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state according to claim 3, characterized in that: The dynamic attack graph is constructed to define the status of each node in the attack graph, and each node indicates whether the corresponding device is attacked; The construction method of dynamic attack graph construction is as follows: Step S201: Obtain network topology information, device service information, and vulnerability information to create an attack graph; Step S202: according to the preset time step Collect network data; Step S203, determining attack behaviors occurring in the network based on the collected network data; Step S204: modify the attack graph obtained in step S103 according to the determined attack behavior, and display the attack process in chronological order.

5. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state according to claim 1, characterized in that: The dynamic attack graph state estimation step includes POMDP modeling and RNN hidden information acquisition; The process of POMDP modeling is as follows: First, define the state space S to represent all possible states; define the action space A to represent the actions that the agent can perform; define the observation space Z to represent the information the agent can obtain about the current state; define the state transition function T to represent the probability distribution from state s to state s'; define the observation function O to represent the probability distribution corresponding to the observation obtained by the agent; define the reward function R to represent the reward obtained when performing action a from state s to reach state s'; and define the initial state of the agent; Then, a POMDP model is established to describe the changing process of the dynamic attack graph, enabling the agent to make decisions. ; , according to the strategy , you can get the expected cumulative reward: in, is the discount factor, Indicates the mathematical expectation value. is the state at time step t, is the state at time step t+1, is the action with time step t, is the time step; make Represents historical information of observations and actions, and adopts belief states based on historical information Inferring the state distribution of nodes, belief state The distribution of is: yes The function of , which means estimating the node status through historical information; When learning the function When , the state estimation of the dynamic attack graph at the current moment is obtained based on the historical information of observations and actions.

6. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state as claimed in claim 5, characterized in that: The method of obtaining RNN hidden information is as follows: Step S301, obtaining historical information of observations and actions, including historical observation values ​​and historical action values; Step S302: for each time step , convert historical observation values ​​and historical action values ​​into vector representations and then process them through RNN; Step S303, at the last time step of RNN , output a vector representing the estimated state value of the dynamic attack graph at the current moment; Step S304: pass the output vector to a state estimation function to map it to a state in the actual state set.

7. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state according to claim 6, characterized in that: In step S301, when the time step Obtaining observations of dynamic attack graphs , and perform defensive actions When the dynamic attack graph changes from the current state Transition to the next state , and generate new observations ; Record the actions taken at each time step and the observations obtained by performing the action, and form a time series of actions and observations: ， The obtained time series information is used as the historical information of observations and actions; In step S302, at time step , the calculation of RNN is shown in formula (1): （1） in, is the time step The hidden state of is the input data, and are the weight matrices of hidden state and input data respectively, is the bias term of the hidden state, is the activation function.

8. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state according to claim 1, characterized in that: The dynamic defense decision-making step includes defining actions and rewards, and training defense strategies; The action and reward are defined, and the agent is used to perform the defense action. When the dynamic attack graph is in the time step The state is recorded as , the agent performs actions in the network Time, action Change the state in the dynamic attack graph and make the agent Reach a new state , the environment generates feedback rewards Given the agent, the agent is based on the new state and feedback rewards Perform new actions , and iteratively interacts with the environment to achieve the agent's cumulative reward G; The cumulative reward G is .

9. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state as claimed in claim 8, characterized in that: state Execute an action , the system security manager receives the utility value , as shown in formula (2): （2） in, It is the reward for maintaining network security. Is to perform an action the costs incurred; As shown in formula (3): （3） R represents the basic security level of the network, which is expressed as a constant; the basic indicators in the common vulnerability assessment system are used to evaluate the network security level. 、 and Conduct assessments; Indicates that after being attacked by the attacker, the node The impact, defined in terms of the effect of the action on the availability of the service .

10. The method for automatic defense decision-making of an industrial control network facing a partially unknown security state according to claim 9, characterized in that: The training defense strategy adopts the deep Q network training defense strategy, and the training method is as follows: Step S401, determining the input dimension of the deep Q network according to the scale of the dynamic attack graph; Step S402: establishing a deep Q network to estimate the Q value of each action taken under the current input; Step S403: Deep Q network adopts Greedy strategy, used to balance the agent's exploration of new strategies and utilization of existing experience during the training phase; Step S404: training the deep Q network. The training data is obtained by the interaction between the agent and the dynamic attack graph, including the state, the action taken, the next state, and the reward signal. Step S405: The deep Q network includes a behavior network and a target network. After training for a fixed number of steps C, the parameters of the behavior network are copied to the target network to achieve a stable training process. Step S406: Save the model until the average cumulative reward obtained from the agent training no longer changes.

Images