Reinforcement learning decision-making method and device for network threats, and storage medium

Abstract

The invention relates to a reinforcement learning decision-making method and device for network threats and a storage medium, and relates to the field of reinforcement learning decision-making of network security. The method comprises the following steps: creating a network state based on a network relation graph and learnable context representation; local and global information transmission is carried out on a network state through an information transmission neural network to obtain an input code of a decision-making agent in reinforcement learning, and the decision-making agent comprises a first decision-making network and a second decision-making network which are trained in reinforcement learning under a threat scene introduced into different network topology structures, the first decision-making network selects a target host for dealing with the threat according to the global code; the second decision-making network predicts a target action adopted by the target host for dealing with the threat according to the hidden state of the target node and the node vector feature; the target host is controlled to execute the target action, and threat processing is carried out. The method supports decision distribution modeling from the network state to the action of the dynamic network.

Description

Technical Field

[0001] The present invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning decision-making method, device and storage medium for network threats. Background Art

[0002] With the development of power automation and network technology, the industrial control systems used in power grids are gradually connected to the Internet to achieve remote monitoring and management of power grids, which brings more convenience and efficiency. However, industrial control systems have a long life cycle, and many system software are not easy to update, making the system vulnerable to attacks and increasing network security risks. Once the industrial control system is destroyed, it will cause great losses to production and operation, so it is necessary to deal with the attack in a timely manner.

[0003] In the security field of complex networks including industrial control systems, reinforcement learning is used to make decisions on how to deal with network threats. Reinforcement learning is a branch of machine learning that studies how to obtain optimal strategies and achieve specific goals through the interaction between agents and the environment. The core of reinforcement learning is that the agent based on neural networks consists of two parts: evaluation and decision-making. The decision-making agent observes the state of the environment and selects actions. The evaluation agent evaluates the actions taken by the decision-making agent and controls the decision-making agent to adjust the strategy according to the reward to obtain the maximum cumulative reward. However, the agents in reinforcement learning are often based on a fixed network structure. The number of hosts participating in threat defense in the network is , each host can execute the threat If all the strategies are combined into a policy space, then in the network, for any threat, all the strategies of each host and the policy space will form possible processing actions. When the network structure changes, is a variable, the action range predicted by the agent actually changes, and the decision made by the agent trained based on the original network structure cannot guarantee the network security of the new network structure. In a wide range of application scenarios, the network structure is often dynamic. Therefore, a reinforcement learning decision-making method for network threats that can adapt to changes in network structure is needed. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides a reinforcement learning decision-making method for network threats.

[0005] In a first aspect, the present invention provides a reinforcement learning decision-making method for network threats, comprising: Create network states based on network relationship graphs and learnable context representations; The feature extraction agent based on the information transfer neural network performs local and global information transfer on the network state to obtain the input code of the decision agent. After the local and global information transfer of the L-layer information transfer neural network, the local code is finally obtained. and global encoding ,in, Nodes obtained for information transfer neural network The final hidden state of the global encoding The information is passed to the neural network to obtain the final context representation; The decision agent comprises: a first decision network and a second decision network, The first decision network is based on the global encoding Select the target host to respond to the threat: , in, is the target node with the highest probability and its node feature vector, is the target parameter of the first decision network; the first decision network is essentially modeled by a fully connected network from the global encoding The distribution function of the node selection probability is used to calculate the global encoding Output the probability of each node handling the threat, and take the host corresponding to the node with the highest probability as the target host; The second decision network predicts the action taken by the target host to deal with the threat based on the hidden state of the target node and the node vector feature: , in, is the target action with the highest probability, is the target parameter of the second decision network; is the final hidden state of the target node, and the second decision network is essentially a distribution function from the final hidden state of the target node and the node feature vector to the action selection probability modeled by a fully connected network, which is used to output the probability of the executable action used by the target node to handle the threat according to the final hidden state of the target node and the node feature vector, and take the executable action with the largest probability as the target action; Control the target host to execute the target action and handle the threat.

[0006] Further, the process of creating a network state includes: Obtain the network topology of the industrial control system network and the overall enterprise network; A network relationship graph is constructed based on the topological structure of the network, and the network relationship graph includes nodes and edges; a node represents any host in the network, and the switch that plays a transit role, the server in the network, the user host, the industrial control operation server and the industrial control user host are all regarded as hosts; an edge represents the connection relationship between the hosts; Constructing a node feature vector, the node feature vector includes a host category vector, an action type vector and a threat authority level vector; Initialize a global context representation that is independent of network hosts and their connections; The node feature vector, edge-based adjacency matrix and global context representation are constructed into the network state.

[0007] Furthermore, for the host involved in handling the threat, its action type comes from the action space, and the action space contains all executable actions for network threats, and the executable actions include: permission deletion actions set according to the level of host privilege escalation in the network and bait process creation actions for the threat; wherein the bait process creation action is used to create a bait process, and the bait process is a disguise of the real process of the host. When working, it receives parameters or instructions required by the process, but does not produce corresponding execution results acting on the host; when the threat extracts the primary permissions of the host, the threat's access rights to the host are deleted by remove, and when the threat extracts the advanced permissions of the host, the threat's access rights to the host are deleted by reset.

[0008] Furthermore, the information transfer neural network is for any node The local information transfer process is as follows: The information transfer neural network extracts information from the node feature vectors of all nodes through the information extraction unit: , in, Indicates The information extracted by the layer information extraction unit, Indicates The linear layer in the layer information extraction unit is of the form , is the weight of the linear layer in the information extraction unit, is the bias of the linear layer in the information extraction unit, is the nonlinear activation function in the information extraction unit; Message passing neural networks utilize extracted information, context representations, and arbitrary nodes by aggregating units. Features to update any node layer by layer The hidden state of: ; in, is the feature fusion function, and nodes The node feature vector fusion of is the normalized adjacency matrix, For the The global context representation of the layer, Indicates The fully connected layer in the layer aggregation unit, the fused features are input to the linear layer of the aggregation unit , in the form of , is the weight of the linear layer in the aggregation unit, is the bias of the linear layer in the aggregation unit, For the Any node output by the layer aggregation unit The hidden state of For the Any node output by the layer aggregation unit The hidden state of .

[0009] Furthermore, after obtaining the hidden states of all nodes, the process of global information transfer of the information transfer neural network for context representation is as follows: Based on attention, the hidden states of all nodes are fused in the form of: ; in, It is the fusion feature of all node hidden states based on attention. A linear layer that processes the node hidden state to get attention, A linear mapping layer for processing node hidden states; The context update unit uses all node hidden states to fuse features Update the context representation in the following form: ; in, is the linear layer in the context update unit, which is of the form , is the weight of the linear layer in the aggregation unit, is the bias of the linear layer in the aggregation unit.

[0010] Furthermore, the first decision network and the second decision network are trained to reach the target parameters through reinforcement learning. The training process of reinforcement learning includes: Build networks with different topologies and simulate threats invading the network; Initialize the first decision network and the second decision network ,Will As a Q-value network, where For the fully connected layer, define the first target decision network , the second target decision network ,Will As the target Q value network; Initialize the first target decision network , the second target decision network ; Randomly select a batch of experience tuples from the experience pool; each experience tuple contains the current action , Current network status , Penalty score and the next network state after executing the current action ; The first decision network and the second decision network Predict the host that responds to the threat based on the global and local encoding of the next network state and executable actions executed by the host : , ; Then the target Q value network is based on the next network state and Q-value network prediction of host responses to threats and executable actions executed by the host Calculate the target Q value : ; in, is the discount factor, and the target Q value And the Q-value network predicts the Q-value The mean square error between is used as the loss function, and the parameters of the Q value network are updated using the gradient descent algorithm. , so as to minimize the loss function; Periodically change the parameters of the Q-value network Assign to the target Q value network.

[0011] Furthermore, the first decision network is based on the global encoding of any network Predict hosts that respond to threats: , in, is the node with the highest probability and its node feature vector; The second decision network predicts the actions taken by the host to respond to the threat based on the hidden state and node vector features of the nodes selected by the first decision network: , Corresponding Node The host performs the action Get the next network state, observe the network state after the action is executed, determine the penalty score based on the network state after the action is executed, and if the threat situation remains unchanged, the penalty score is set ; If the threat is eliminated, the penalty is zero; if the threat spreads in the network, the penalty is ,in is a base greater than 1, is the diffusion ratio; The current action, the current network state, the penalty score, and the next network state after executing the current action are saved in the experience pool as an experience tuple.

[0012] Furthermore, experience pools are constructed for different network topologies, and when random batch sampling is performed from the experience pools, random batch sampling is performed uniformly from experience pools of different network topologies.

[0013] In a second aspect, the present invention provides a reinforcement learning decision-making device for network threats, comprising: at least one processing unit, the processing unit is connected to a storage unit via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the reinforcement learning decision-making method for network threats is implemented.

[0014] In a third aspect, the present invention provides a computer-readable storage medium, which stores a computer program, characterized in that when the computer program is executed by a processor, the reinforcement learning decision-making method for network threats is implemented.

[0015] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art: The reinforcement learning agent of the present application includes a feature extraction agent and a decision-making agent. In the present application, a network state is created based on a network relationship graph and a learnable context representation; a feature extraction agent based on an information transfer neural network performs local and global information transfer on the network state to obtain an input code for the decision-making agent, and the decision-making agent includes: a first decision network and a second decision network trained by reinforcement learning under threat scenarios introduced in different network topologies, the first decision network selects a target host to respond to the threat according to the global code: the second decision network predicts the target action taken by the target host to respond to the threat according to the hidden state of the target node and the node vector feature: the target host is controlled to execute the target action to handle the threat.

[0016] For the network as a whole, networks of various sizes are obtained by combining different local topology combinations. Even if the network is dynamic, the local topology combination of the network has a relatively stable local decision distribution. This local decision distribution corresponds to the selection of executable actions against threats within the fixed form of local topology combination. On the basis of this stable local decision distribution, the context representation weighs the weight of the stable local decision distribution in the global decision distribution of the network according to the interaction of different types of local topology combinations in the global network. This weight represents which local topology combinations or clusters are more important to the network as a whole in threat processing, that is, the global decision distribution corresponds to the selection of hosts that execute executable actions.

[0017] In the present application, the information transfer neural network uses a combination of network relationship graphs and context representations to construct features within the local domain of any node, and combines the features within the local neighborhood with the node feature vector, and adds them to the hidden state layer by layer, so that the hidden state can be better used to model the local decision distribution. The information transfer neural network uses the global fusion features of all hidden states to update the previous global context representation that is independent of the network host and its connection relationship, and supports the importance of each local decision from the global modeling, so that each local decision distribution can be better combined to model the overall decision distribution. The first decision network of the present application models the importance of each local decision from the global modeling based on global coding, and uses global associations to make decisions on the target host. The second decision network models the local decision distribution based on hidden features, and uses local states to accurately make target action decisions. The present application decouples the entire decision into determining the local decision distribution and the global decision distribution as above, and supports the decision distribution modeling of the network state to action of the dynamic network. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart of a reinforcement learning decision-making method for network threats provided by an embodiment of the present invention; Figure 2 A schematic diagram of a network topology; Figure 3 A schematic diagram of a network relationship diagram provided by an embodiment of the present invention; Figure 4 A schematic diagram of a feature extraction agent based on an information transfer neural network provided by an embodiment of the present invention; Figure 5 A flowchart of creating a network status provided by an embodiment of the present invention; Figure 6 A schematic diagram of a reinforcement learning decision-making device for network threats provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0023] Example 1 like Figure 1 As shown, the technology of the present invention implements a reinforcement learning decision-making method for network threats, including: Create network states based on the network relationship graph and learnable context representations.

[0024] In the specific implementation process, Figure 5 As shown, the process of creating a network state includes: Obtain the network topology of the industrial control system network and the overall enterprise network. Figure 2 As shown, Figure 2 The network topology structure in the example includes three subnets, namely, the subnet connected to the user host, the subnet connected to the server, and the subnet connected to the industrial control operation server and the industrial control user host. The subnet connected to the server allows communication with the other two subnets.

[0025] A network relationship graph is constructed based on the topological structure of the network, and the network relationship graph includes nodes and edges. A node represents any host in the network. In this application, switches that play a transfer role, servers in the network, user hosts, industrial control operation servers, and industrial control user hosts are all regarded as hosts. Edges represent the connection relationship between hosts.

[0026] Among them, for a network containing N hosts, the node set in the network relationship diagram is , Represents the nodes and their node feature vectors of the N hosts in the corresponding network; the set of edges is ,in Indicates different nodes In this application, the connection relationship between all nodes is constructed into an adjacency matrix. .

[0027] like Figure 3 As shown, Figure 3 Give the corresponding Figure 2 A network diagram of a network topology.

[0028] Construct a node feature vector, which includes a host category vector, an action type vector, and a threat permission level vector. Specifically, in order to enable the agent to adapt to network changes and have the universality of adapting to different hosts, only the host category vector is configured for the node at the category granularity, and the process state, network connection state, and network file information of the host represented by the node are mapped into an action type vector and a threat permission level vector, wherein the action type vector is obtained based on the process state mapping, and the threat permission level vector is obtained based on the network connection state and the network file information situation.

[0029] For hosts involved in handling threats, their action types come from the action space AS, which contains all executable actions for network threats, including: permission deletion actions set according to the level of host privilege escalation in the network and decoy process creation actions for threats; wherein, the decoy process creation action is used to create a decoy process, which is a disguise of the host's real process. When working, it receives parameters or instructions required by the process, but does not produce corresponding execution results acting on the host; during the threat intrusion process, it will be lured by the decoy process, and the host will be elevated using the decoy process instead of the real process. In this way, any privilege escalation scheme that can be implemented based on the real process cannot be implemented through the decoy process. When the threat extracts the primary permissions of the host, the threat's access rights to the host are deleted by remove, and when the threat extracts the advanced permissions of the host, the threat's access rights to the host are deleted by reset.

[0030] for Figure 2The network topology shown in the figure includes switches, servers in the network, user hosts, industrial control operation servers and industrial control user hosts. There are host types, and a host can only be of one type. Therefore, the host category vector requires at least 3 binary digits. Since the host can execute any action in the action space and its combination, the number of action types in the state space is set to K. Therefore, the action type vector requires at least K binary digits. For any host, the threat of privilege escalation includes: the threat of not obtaining any privileges, the threat of obtaining primary access rights, and the threat of obtaining advanced access rights. Therefore, the threat privilege level vector requires at least 2 binary digits.

[0031] Initializes a global context representation that is independent of network hosts and their connections.

[0032] The node feature vector, edge-based adjacency matrix and global context representation are constructed into the network state.

[0033] For the network as a whole, networks of various sizes are obtained by combining different local topology combinations. Even if the network is dynamic and the overall size of the adjacency matrix is ​​dynamic, the local topology combination of the network or the clustering of local connection relationships in the adjacency matrix has a relatively stable local decision distribution. This local decision distribution corresponds to the selection of executable actions for threats within a fixed form of local topology combination or cluster. On the basis of this stable local decision distribution, the context representation weighs the weight of the stable local decision distribution in the global decision distribution of the network according to the interaction of different types of local topology combinations in the global network. This weight represents which local topology combinations or clusters are more important to the network as a whole in threat processing, that is, the global decision distribution corresponds to the selection of hosts that execute executable actions. Decoupling into the above local decision distribution and global decision distribution can model the decision distribution from network state to action for dynamic networks.

[0034] To achieve the above decision distribution modeling process, in this application, the feature extraction agent based on the information transfer neural network performs local and global information transfer on the network state to obtain the input code of the decision agent in reinforcement learning. The input code includes local code and global code.

[0035] Combination Figure 4 As shown, the information transfer neural network is for any node The local information transfer process is as follows: The information transfer neural network extracts information from the node feature vectors of all nodes through the information extraction unit: , in, Indicates The information extracted by the layer information extraction unit, Indicates Linear layer processing node in the layer information extraction unit The node feature vector of , is the weight of the linear layer in the information extraction unit, is the bias of the linear layer in the information extraction unit, is a nonlinear activation function in the information extraction unit. The nonlinear activation function uses any one of the tanh activation function, the relu activation function, and the sigmoid activation function.

[0036] Message passing neural networks utilize extracted information, context representations, and arbitrary nodes by aggregating units. Features to update any node layer by layer The hidden state of: ; in, is the feature fusion function, and nodes The node feature vector fusion of is the normalized adjacency matrix, For the The global context representation of the layer, Indicates The fully connected layer in the layer aggregation unit, the fused features are input to the linear layer of the aggregation unit , in the form of , is the weight of the linear layer in the aggregation unit, is the bias of the linear layer in the aggregation unit, For the Any node output by the layer aggregation unit The hidden state of For the Any node output by the layer aggregation unit The hidden state of . The fully connected layer in the layer aggregation unit uses the adjacency matrix, information and context representation to build nodes The features in the local area are combined with the node feature vector in the local neighborhood and added to the hidden state layer by layer, so that the hidden state can be better used to model the local decision distribution.

[0037] After obtaining the hidden states of all nodes, the process of global information transfer by the information transfer neural network for context representation is as follows: Based on attention, the hidden states of all nodes are fused in the form of: ; in, It is the fusion feature of all node hidden states based on attention. A linear layer that processes the node hidden state to get attention, A linear mapping layer for processing node hidden states; The context update unit uses all node hidden states to fuse features Update the context representation in the following form: ; in, is the linear layer in the context update unit, which is of the form , is the weight of the linear layer in the aggregation unit, is the bias of the linear layer in the aggregation unit.

[0038] The previous global context representation independent of the network hosts and their connection relationships is updated using the global fusion features of all hidden states, supporting the importance of each local decision modeling from a global perspective, so that each local decision distribution can be better combined to model the overall decision distribution.

[0039] After local and global information transfer by L layers of information transfer neural network, the final local code is obtained. and global encoding ,in, Nodes obtained for information transfer neural network The final state, global encoding The information is passed to the neural network to obtain the final context representation.

[0040] The decision-making agent includes: a first decision network and a second decision network trained by reinforcement learning, and the first decision network and the second decision network are constructed through a fully connected network.

[0041] The first decision network is based on the global encoding Select the target host to respond to the threat: , in, is the target node with the highest probability and its node feature vector, is the target parameter of the first decision network; the first decision network is essentially modeled by a fully connected network from the global encoding The distribution function of the node selection probability is used to calculate the global encoding Output the probability of each node handling the threat, and take the host corresponding to the node with the highest probability as the target host.

[0042] The second decision network predicts the target action taken by the target host to respond to the threat based on the hidden state of the target node and the node vector feature: , in, is the target action with the highest probability, Determine the target parameters of the network for the first time; is the final hidden state of the target node. The second decision network is essentially a distribution function from the final hidden state of the target node and the node feature vector to the action selection probability modeled by a fully connected network. It is used to output the probability of the executable action used by the target node to handle the threat according to the final hidden state of the target node and the node feature vector, and take the executable action with the largest probability as the target action.

[0043] The first decision network of the present application models the importance of each local decision from a global perspective based on global coding, and uses global associations to make decisions on the target host. The second decision network models the local decision distribution based on hidden features, and uses local states to accurately make target action decisions.

[0044] Execute the action jointly determined by the first decision network and the second decision network, that is, control the target host to execute the target action to handle the threat.

[0045] To achieve the above objectives, reinforcement learning is required to train the first decision network and the second decision network to reach the target parameters. The training process of reinforcement learning includes: Build networks with different topologies and simulate threats invading the network; Initialize the first decision network and the second decision network ,Will As a Q-value network, where For the fully connected layer, define the first target decision network , the second target decision network ,Will As the target Q value network; The first target decision network will be initialized , the second target decision network ; The first decision network is based on the global encoding of any network Predict hosts that respond to threats: , in, is the node with the highest probability and its node feature vector; The second decision network predicts the actions taken by the host to respond to the threat based on the hidden state and node vector features of the nodes selected by the first decision network: , Corresponding Node The host performs the action Get the next network state, observe the network state after the action is executed, determine the penalty score based on the network state after the action is executed, and if the threat situation remains unchanged, the penalty score is set ; If the threat is eliminated, the penalty is zero; if the threat spreads in the network, the penalty is ,in is a base greater than 1, is the diffusion ratio; The current action, the current network state, the penalty score, and the next network state after executing the current action are saved in the experience pool as experience tuples. Each experience tuple contains the current action. , Current network status , Penalty score and the next network state after executing the current action , superscripts c and c+1 indicate the order.

[0046] During the specific implementation process, in order to ensure the universality of the model to networks with different topologies, experience pools are constructed for different network topologies. When randomly sampling in batches from the experience pools, random batch sampling is performed evenly from experience pools with different network topologies.

[0047] After sampling, the first decision network and the second decision network Predict the host that responds to the threat based on the global and local encoding of the next network state Executable actions performed by the host : , ; Then the target Q value network is based on the next network state and Q-value network prediction of host responses to threats Executable actions performed by the host Calculate the target Q value : ; in, is the discount factor, and the target Q value And the Q-value network predicts the Q-value The mean square error between is used as the loss function, and the parameters of the Q value network are updated using the gradient descent algorithm. , so as to minimize the loss function; Periodically change the parameters of the Q-value network Assign to the target Q value network.

[0048] Example 2 See also Figure 6 As shown, an embodiment of the present invention provides a reinforcement learning decision-making device for network threats, including: at least one processing unit, the processing unit is connected to a storage unit through a bus unit, the storage unit is a computer-readable storage medium, and can be used to store software programs, computer executable programs, and modules, such as the software programs, computer executable programs, and modules corresponding to a reinforcement learning decision-making method for network threats in an embodiment of the present invention. The processing unit implements the above-mentioned reinforcement learning decision-making method for network threats by running the software programs, computer executable programs, and modules stored in the storage unit, including: Create network states based on network relationship graphs and learnable context representations; The feature extraction agent based on the information transfer neural network performs local and global information transfer on the network state to obtain the input code of the decision-making agent in reinforcement learning. After the local and global information transfer of the L-layer information transfer neural network, the local code is finally obtained. and global encoding ,in, Nodes obtained for information transfer neural network The final hidden state of the global encoding The information is passed to the neural network to obtain the final context representation; The decision agent comprises: a first decision network and a second decision network, The first decision network is based on the global encoding Select the target host to respond to the threat: , in, is the target node with the highest probability and its node feature vector, is the target parameter of the first decision network; the first decision network is essentially modeled by a fully connected network from the global encoding The distribution function of the node selection probability is used to calculate the global encoding Output the probability of each node handling the threat, and take the host corresponding to the node with the highest probability as the target host; The second decision network predicts the action taken by the target host to deal with the threat based on the hidden state of the target node and the node vector feature: , in, is the target action with the highest probability, is the target parameter of the second decision network; is the final hidden state of the target node, and the second decision network is essentially a distribution function from the final hidden state of the target node and the node feature vector to the action selection probability modeled by a fully connected network, which is used to output the probability of the executable action used by the target node to handle the threat according to the final hidden state of the target node and the node feature vector, and take the executable action with the largest probability as the target action; Control the target host to execute the target action and handle the threat.

[0049] Of course, the computer program stored in the storage unit of the device for implementing the reinforcement learning decision-making method for network threats provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the reinforcement learning decision-making method for network threats provided by any embodiment of the present invention.

[0050] Example 3 An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the reinforcement learning decision-making method for network threats is implemented, including: Create network states based on network relationship graphs and learnable context representations; The feature extraction agent based on the information transfer neural network performs local and global information transfer on the network state to obtain the input code of the decision-making agent in reinforcement learning. After the local and global information transfer of the L-layer information transfer neural network, the local code is finally obtained. and global encoding ,in, Nodes obtained for information transfer neural network The final hidden state of the global encoding The information is passed to the neural network to obtain the final context representation; The decision agent comprises: a first decision network and a second decision network, The first decision network is based on the global encoding Select the target host to respond to the threat: , in, is the target node with the highest probability and its node feature vector, is the target parameter of the first decision network; the first decision network is essentially modeled by a fully connected network from the global encoding The distribution function of the node selection probability is used to calculate the global encoding Output the probability of each node handling the threat, and take the host corresponding to the node with the highest probability as the target host; The second decision network predicts the action taken by the target host to deal with the threat based on the hidden state of the target node and the node vector feature: , in, is the target action with the highest probability, is the target parameter of the second decision network; is the final hidden state of the target node, and the second decision network is essentially a distribution function from the final hidden state of the target node and the node feature vector to the action selection probability modeled by a fully connected network, which is used to output the probability of the executable action used by the target node to handle the threat according to the final hidden state of the target node and the node feature vector, and take the executable action with the largest probability as the target action; Control the target host to execute the target action and handle the threat.

[0051] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program which is not limited to the method operations described above, but can also execute related operations in a reinforcement learning decision-making method for network threats provided in any embodiment of the present invention.

[0052] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0053] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0054] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0055] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A reinforcement learning decision-making method for network threats, characterized in that: include: Create network states based on network relationship graphs and learnable context representations; The feature extraction agent based on the information transfer neural network performs local and global information transfer on the network state to obtain the input code of the decision agent. After the local and global information transfer of the L-layer information transfer neural network, the local code is finally obtained. and global encoding ,in, Nodes obtained for information transfer neural network The final hidden state of the global encoding The information is passed to the neural network to obtain the final context representation; The decision agent includes: a first decision network and a second decision network trained by reinforcement learning, wherein the first decision network is based on a global encoding Select the target host to respond to the threat: ， in, is the target node with the highest probability and its node feature vector, is the target parameter of the first decision network; the first decision network is essentially modeled by a fully connected network from the global encoding The distribution function of the node selection probability is used to calculate the global encoding Output the probability of each node handling the threat, and take the host corresponding to the node with the highest probability as the target host; The second decision network predicts the target action taken by the target host to respond to the threat based on the hidden state of the target node and the node vector feature: ， in, is the target action with the highest probability, is the target parameter of the second decision network; is the final hidden state of the target node, and the second decision network is essentially a distribution function from the final hidden state of the target node and the node feature vector to the action selection probability modeled by a fully connected network, which is used to output the probability of the executable action used by the target node to handle the threat according to the final hidden state of the target node and the node feature vector, and take the executable action with the largest probability as the target action; Control the target host to execute the target action and handle the threat.

2. The reinforcement learning decision-making method for network threats according to claim 1, characterized in that: The process of creating a network state includes: Obtain the network topology of the industrial control system network and the overall enterprise network; A network relationship diagram is constructed based on the topological structure of the network, and the network relationship diagram includes nodes and edges; a node represents any host in the network; a switch that plays a transit role, a server in the network, a user host, an industrial control operation server and an industrial control user host are all regarded as hosts; an edge represents the connection relationship between hosts; Constructing a node feature vector, the node feature vector includes a host category vector, an action type vector and a threat authority level vector; Initialize a global context representation that is independent of network hosts and their connections; The node feature vector, edge-based adjacency matrix and global context representation are constructed into the network state.

3. The reinforcement learning decision-making method for network threats according to claim 2, characterized in that: For hosts involved in handling threats, their action types come from an action space, which contains all executable actions for network threats, including: permission deletion actions set according to the level of host privilege escalation in the network and decoy process creation actions for threats; wherein the decoy process creation action is used to create a decoy process, which is a disguise of the host's real process. When working, it receives parameters or instructions required by the process, but does not produce corresponding execution results acting on the host; when the threat extracts the primary permissions of the host, the threat's access rights to the host are deleted by remove, and when the threat extracts the advanced permissions of the host, the threat's access rights to the host are deleted by reset.

4. The reinforcement learning decision-making method for network threats according to claim 1, characterized in that: The information transfer neural network is for any node The local information transfer process is as follows: The information transfer neural network extracts information from the node feature vectors of all nodes through the information extraction unit: ， in, Indicates The information extracted by the layer information extraction unit, Indicates Linear layer processing nodes in the layer information extraction unit The node feature vector of , is the weight of the linear layer in the information extraction unit, is the bias of the linear layer in the information extraction unit, is the nonlinear activation function in the information extraction unit; Message passing neural networks utilize extracted information, context representations, and arbitrary nodes by aggregating units. Features to update any node layer by layer The hidden state of: ； in, is the feature fusion function, and nodes The node feature vector fusion of is the normalized adjacency matrix, For the The global context representation of the layer, Indicates The fully connected layer in the layer aggregation unit, the fused features are input to the linear layer of the aggregation unit , in the form of , is the weight of the linear layer in the aggregation unit, is the bias of the linear layer in the aggregation unit, For the Any node output by the layer aggregation unit The hidden state of For the Any node output by the layer aggregation unit The hidden state of .

5. The reinforcement learning decision-making method for network threats according to claim 4, characterized in that: After obtaining the hidden states of all nodes, the process of global information transfer of the information transfer neural network for context representation is as follows: Based on attention, the hidden states of all nodes are fused in the form of: ； in, It is the fusion feature of all node hidden states based on attention. A linear layer that processes the node hidden state to get attention, A linear mapping layer for processing node hidden states; The context update unit uses all node hidden states to fuse features Update the context representation in the following form: ； in, is the linear layer in the context update unit, which is of the form , is the weight of the linear layer in the aggregation unit, is the bias of the linear layer in the aggregation unit.

6. The reinforcement learning decision-making method for network threats according to claim 1, characterized in that: The first decision network and the second decision network are trained to reach the target parameters through reinforcement learning. The training process of reinforcement learning includes: Build networks with different topologies and simulate threats invading the network; Initialize the first decision network and the second decision network ,Will As a Q-value network, where For the fully connected layer, define the first target decision network , the second target decision network ,Will As the target Q value network; Initialize the first target decision network , the second target decision network ; Randomly select a batch of experience tuples from the experience pool; each experience tuple contains the current action , Current network status , Penalty score and the next network state after executing the current action , superscripts c and c+1 indicate the order; The first decision network and the second decision network Predict the host that responds to the threat based on the global and local encoding of the next network state and executable actions executed by the host : ， ； Then the target Q value network is based on the next network state and Q-value network prediction of host responses to threats and executable actions executed by the host Calculate the target Q value : ； in, is the discount factor, and the target Q value And the Q-value network predicts the Q-value The mean square error between is used as the loss function, and the parameters of the Q value network are updated using the gradient descent algorithm. , so as to minimize the loss function; Periodically change the parameters of the Q-value network Assign to the target Q value network.

7. The reinforcement learning decision-making method for network threats according to claim 6, characterized in that: The first decision network is based on the global encoding of any network Predict hosts that respond to threats: ， in, is the node with the highest probability and its node feature vector; The second decision network predicts the actions taken by the host to respond to the threat based on the hidden state and node vector features of the nodes selected by the first decision network: ， Corresponding Node The host performs the action Get the next network state, observe the network state after the action is executed, determine the penalty score based on the network state after the action is executed, and if the threat situation remains unchanged, the penalty score is set ; If the threat is eliminated, the penalty is zero; if the threat spreads in the network, the penalty is ,in is a base greater than 1, is the diffusion ratio; The current action, the current network state, the penalty score, and the next network state after executing the current action are saved in the experience pool as an experience tuple.

8. The reinforcement learning decision-making method for network threats according to claim 6, characterized in that: Experience pools are constructed for different network topologies, and when random batch sampling is performed from the experience pools, random batch sampling is performed uniformly from experience pools of different network topologies.

9. A reinforcement learning decision-making device for network threats, characterized in that: include: At least one processing unit, the processing unit is connected to a storage unit via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the reinforcement learning decision-making method for network threats as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the reinforcement learning decision-making method for network threats as described in any one of claims 1 to 8 is implemented.

Images