Action selection device, learning device, action selection method, and program for autonomous security verification of network systems using reinforcement learning.

Abstract

To provide a method which properly expresses a bigger state, and action space, in security verification on a network system, and a method which can receive these as input by the same model even if the number of network topologies or hosts differs.SOLUTION: By observing a state of a network system as a graph, in security verification using reinforcement learning, and dividing and processing a host and an action type in the inside, a strategy model which outputs action which should be taken next is adopted.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This invention relates to an action selection device, a learning device, an action selection method, and a program for autonomous security verification of a network system using reinforcement learning. [Background technology]

[0002] In vulnerability testing of network systems, the state of the network system under test is often observed, and actions for vulnerability testing are selected based on that observed state. This type of vulnerability testing can be considered an example of a Markov decision process model. In this case, by associating the state space in the Markov decision process with the state of the network system and the action space with the actions for vulnerability testing, reinforcement learning can be applied to vulnerability testing. Furthermore, there is an expectation that applying reinforcement learning to vulnerability testing of network systems can lead to more efficient testing.

[0003] Furthermore, in recent years, security verification of network systems has seen the rise of Red Teaming Operations, where a group of security experts forms an attack team and conducts various realistic attacks, both physical and virtual, against the system under verification to test its security response capabilities. In such Red Teaming Operations, the experts in the attack team (Red Team) are required to possess advanced knowledge, making it difficult to completely automate the exercises. However, with the advancement of IoT and other technologies, and the increasing sophistication and networking of modern technology, security verification of all devices will become essential. There is an expectation that the introduction of AI will enable the automation of this verification process.

[0004] Non-patent document 1 discloses research on automating penetration testing, which involves launching external attacks to infiltrate a system and inspect for vulnerabilities in computers and networks. It proposes a model that enables fast and stable reinforcement learning by hierarchizing the action space and suppressing the number of actions output by each agent. However, in this model, the number of states obtained from the environment increases as the number of host machines and subnets increases. In addition, although information such as the OS and services running on the machines being inspected is important in reality, the research in Non-Patent Document 1 does not take into account the state of services running on each machine. Therefore, it is difficult to apply this model to a realistic network system.

[0005] Generally speaking, penetration testing is one method of vulnerability testing, and vulnerability testing is positioned as a specific action for verifying security. The precise definition of the term "security verification" used in this specification will be discussed later. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Application No. 2022-129996 (Filing Date: August 17, 2022; Title of Invention: State Representation Device, Learning Device, State Representation Method, and State Representation Program) [Non-patent literature]

[0007] [Non-Patent Document 1] K. Tran et al., “Deep hierarchical reinforcement agents for automated penetration testing,” Proc. International Workshop on Adaptive Cyber ​​Defense, 2021. [Overview of the project] [[Problem to be Solved by the Invention]]

[0008] In existing studies, only limited settings have been examined for the state space observed from the environment and the action space selected by the agent. Therefore, application to larger state and action spaces is an issue. In addition, existing studies can only handle environments that are identical to the learned network system. Therefore, when the network topology or the number of hosts is different, it was necessary to change the model structure. In order to solve these problems, development of a method for accurately representing larger state and action spaces, and a method that can receive these as inputs with the same model even when the network topology or the number of hosts is different, is desired.

[0009] Note that the aforementioned Patent Document 1 (earlier application) proposes an apparatus or the like for obtaining an efficient representation (graph) indicating the state of a network system, and suggests that the convergence of reinforcement learning can be improved by using such an apparatus or the like. However, it does not disclose a specific method for verifying the security of a network system.

[0010] In order to solve the aforementioned problems, an object of the present invention is to provide a method for accurately representing larger state and action spaces, and a method that can receive these as inputs with the same model even when the network topology or the number of hosts is different. To this end, in security verification using reinforcement learning, the present invention observes the state of a network system as a graph, considers local features of the graph, also considers past states of the network system, and further performs processing by internally dividing hosts and action types, thereby it is an object of the present invention to provide a model that outputs the next action to be taken. [[Means for Solving the Problem]]

[0011] The action selection device according to the present invention receives as input a graph that represents the state of a network system subject to security verification by a triple consisting of: a set of nodes corresponding to hosts in the network system, a set of edges corresponding to connection relationships between the hosts, and a set of node feature vectors corresponding to feature vectors indicating attribute information of each of the hosts. The action selection device comprises: a graph processing unit that outputs a vector representing features of the entire graph; a local feature processing unit that receives as input the vector representing features of the entire graph and the set of feature vectors of the nodes of the graph, and outputs a vector that takes into account local features of the graph; a recursive learning processing unit that receives as input the vector that takes into account local features of the graph, and outputs a vector that takes into account past states of the network system; and a host and action type selection processing unit that receives as input the vector that takes into account past states, and outputs a vector indicating the probability of an action to be taken next for security verification.

[0012] In addition to the vector that takes into account local features of the graph, the recursive learning processing unit may receive a hidden vector as input, output an updated hidden vector in addition to the vector that takes into account past states, and use the updated hidden vector as a hidden state vector in processing for the next time step.

[0013] The host and action type selection processing unit may decompose the vector that takes into account past states into a vector representing a host and a vector representing an action type, obtain a value including both the host and the action type from a matrix product of these vectors, and obtain a vector indicating the action probability based on the obtained value.

[0014] Furthermore, the learning device according to the present invention is a learning device that performs reinforcement learning to optimize a function that indicates the actions to be taken for security verification targeting a network system, wherein the function is a policy function that takes as an argument a state that takes into account the local features of a graph representing the state of the network system that is the target of security verification, and returns the probability of the next action to be taken for security verification, taking into account the past state of the network system.

[0015] Furthermore, the action selection method according to the present invention includes: a graph processing step that receives as input a graph representing the state of a network system subject to security verification as a triplicate of a set of nodes corresponding to hosts in the network system, a set of edges corresponding to the connection relationships of the hosts, and a set of node feature vectors corresponding to feature vectors representing the attribute information of each host, and outputs a vector representing the features of the entire graph; a local feature processing step that receives as input a vector representing the features of the entire graph and a set of node feature vectors of the graph, and outputs a vector that takes into account the local features of the graph; a recurrent learning processing step that receives as input a vector that takes into account the local features of the graph, and outputs a vector that takes into account the past state of the network system; and a host / action type selection processing step that receives as input a vector that takes into account the past state, and outputs a vector indicating the probability of the next action to be taken for security verification.

[0016] The recurrent learning process step may take a hidden vector as input in addition to a vector that takes into account the local features of the graph, output an updated hidden vector in addition to a vector that takes into account past states, and receive the updated hidden vector as the hidden state vector in the processing at the next time step.

[0017] The host / action type selection processing step may involve decomposing the vector considering past states into a vector representing the host and a vector representing the action type, obtaining a value that includes both the host and the action type by matrix product of these, and obtaining a vector indicating the probability of the action based on this.

[0018] Furthermore, the action selection program according to the present invention may be a program for causing a computer to function as the aforementioned action selection device. [Effects of the Invention]

[0019] According to this invention, by providing a graph processing unit, it becomes possible to represent the state of the network subject to security verification in detail. Furthermore, the local feature processing unit and the recurrent learning processing unit make it possible to stabilize complex learning, including graph processing. In addition, since the host and action type selection processing unit determines the target host and action type using different functions, the correspondence between the weights learned by each function and the actions is clarified, and learning becomes even more stable. By stabilizing learning, it becomes possible to handle larger networks that are difficult to learn with existing methods. [Brief explanation of the drawing]

[0020] [Figure 1] This is a block diagram showing an example of the functional configuration of a behavior selection device in an embodiment of the present invention. [Figure 2] This flowchart shows an example of the procedure for a behavior selection method in an embodiment of the present invention. [Figure 3] This figure shows an example of a policy model in an embodiment of the present invention. [Modes for carrying out the invention]

[0021] An example of an embodiment of the present invention will be described below. First, the following definitions of terms used to describe embodiments of the present invention are provided.

[0022] "Security verification" refers to operations that attempt to scan a system, infiltrate it, or execute commands in order to discover vulnerabilities or other defects in a system and to evaluate whether they can be used as attacks.

[0023] The "environment" refers to the network system that is the target of security verification. The environment receives actions (commands) from the agent (described later), executes those actions within the environment, and then returns to the agent the state representing the environment's internal workings and the reward at that time. The reward, for example, is a value indicating whether the attack was successful or not.

[0024] An "agent" is an entity that selects its next action (a command to the environment) based on the state and reward it receives from the environment. An agent has a function that takes a state as an argument and returns the next action to take (or its probability). Reinforcement learning algorithms are used to optimize this function. The function that takes a state as an argument and returns the probability of the next action is called a "policy function."

[0025] A "reinforcement learning algorithm" refers to an algorithm that optimizes a function (specifically, a policy function) that selects actions in a way that maximizes the expected value of the cumulative reward when a series of states and actions are taken.

[0026] The set of "actions for security verification" is formed by the Cartesian product of the hosts present in the environment and the types of actions that can be performed on each host. As a specific example, three types of hosts H a H b H c Let's assume there are two types of actions, A1 and A2. Here, a host is a host machine in the system, and an action type is an action performed on each host. More specifically, action types correspond to things like port scans and the execution of privilege escalation commands. In this example, the set of actions A is the Cartesian product of the host and the action type, resulting in a set of six elements. A={(H a ,A1),(H a ,A2),(Hb , A1), (H b , A2), (H c , A1), (H c , A2)} It is as follows.

[0027] In the present invention, in security verification using reinforcement learning, a Graph Neural Network (GNN) is used in the model to represent the current state of the network system as a graph and input the graph into the model. At this time, in order to stabilize learning, a mechanism for processing local features and performing recursive learning processing is introduced, and further, a mechanism for processing the target host of an action and the action type with different functions is provided. These processes are performed in each of the processing units (functional blocks) described later.

[0028] FIG. 1 is a block diagram showing the functional configuration of the action selection device 1 according to the present embodiment. The action selection device 1 is an information processing device (computer) including an input unit 10, a control unit 20, a storage unit 30, an output unit 40, and a communication unit (not shown) and the like.

[0029] The control unit 20 is a part that controls the entire action selection device 1, and implements each mechanism in the present embodiment by appropriately reading and executing various programs stored in the storage unit 30. The control unit 20 may be a CPU.

[0030] The storage unit 30 is a storage area for various programs and various data for causing a hardware group to function as the action selection device 1, and may be a ROM, a RAM, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0031] The control unit 20 includes a graph processing unit 21, a local feature processing unit 22, a recursive learning processing unit 23, and a host and action type selection processing unit 24. As shown in FIG. 1, each processing unit is implemented as a functional block in the control unit 20, and the processing in each processing unit is executed by the CPU reading a corresponding program.

[0032] Figure 2 is a flowchart showing the procedure for selecting an action in this embodiment. In step S1, the action selection device 1 receives a graph representing the state of the environment as input. In step S2, the action selection device 1 performs graph processing to output a vector representing the characteristics of the entire received graph.

[0033] In step S3, the action selection device 1 performs local feature processing to output a vector that takes into account the local features of the graph, based on a vector representing the features of the entire graph and a set of feature vectors of the nodes of the graph. In step S4, the action selection device 1 performs a recurrent learning process.

[0034] In step S5, the action selection device 1 performs host and action type selection processing. In step S6, the results of the host and activity type selection process are output. Specific examples of processing in each processing unit and each processing step will be explained in the proposed method within the policy model described later.

[0035] In reinforcement learning-based security verification, the function that allows the reinforcement learning algorithm to select actions for security verification through the interaction between the environment and the agent is optimized to ensure efficient verification.

[0036] This invention relates to reinforcement learning algorithms that use policy functions, and the model of the policy function is called the policy model. The action selection device 1 in Figure 1 is a concrete embodiment of the policy model as a device. The arguments of the policy model (policy function) include the state received from the environment. The state is a value or set of values ​​that represents the properties of the network system being security inspected at that time. Here, it refers to a value that quantifies and represents as a vector information about the number of host machines included in the network system, their connection relationships, and the operating systems and applications running on each host machine.

[0037] The output of the policy model is the next action to take for security verification and its probability. Specifically, it can be represented as a vector consisting of the same number of elements as the number of possible actions. In this case, the sum of the elements in the vector is 1, and each element represents the probability of taking the corresponding action. Based on the values ​​obtained from the policy model's output, the actual action to take, i.e., the next action to take for security verification, is determined. Typically, the action corresponding to the largest value among the values ​​of each element of the vector (the action with the highest probability) is selected.

[0038] The policy model consists of four types of processing units: a graph processing unit, a local feature processing unit, a recurrent learning processing unit, and a host / action type selection processing unit. Below, we will explain specific examples of processing in each processing unit, referring to Figure 3, which illustrates the proposed method in the policy model.

[0039] The graph processing unit (graph processing unit 21 in Figure 3) receives the state of the environment, which is the network system to be inspected, as a graph G=(V,E,X), and generates a vector h representing the characteristics of the entire graph. G The output is as follows: Here, V is the set of nodes in the graph, corresponding to the set of hosts in a network system; E is the set of edges in the graph, corresponding to the set of host connectivity relationships; and X is the set of node feature vectors, corresponding to the set of feature vectors representing the attribute information of each host. The attribute information includes information such as the OS and applications installed on each host. In addition, it can also include information on the progress of vulnerability testing (for example, whether a port scan has been performed, whether an intrusion has already been successful, etc.).

[0040] Furthermore, a representation (graph) showing the state of the environment, which is the network system being inspected, can be efficiently obtained, for example, by using the device proposed in the aforementioned Patent Document 1 (prior application).

[0041] The local feature processing unit (local feature processing unit 22 in Figure 3) outputs h, which represents the features of the entire graph, from the graph processing unit. GIt takes X, which represents a set of node feature vectors, as input, and generates a vector c that takes into account the local features of the graph. G It outputs the value of X. Here, by referring to the value of X, h G Values ​​that take into account the local characteristics of each node are assigned (weighted). Examples of specific local feature processing include attention mechanisms (also called attention mechanisms or attention) and processing using clustering. Figure 3 shows the case where an attention mechanism is used as local feature processing.

[0042] The recurrent learning processing unit (recurrent learning processing unit 23 in Figure 3) receives the vector c output from the local feature processing unit. G Then, it receives the hidden state vector g and the vector z that takes past states into account. G Then, the updated hidden state vector g' is output. Here, the input hidden state vector g is the hidden state vector g' output in the previous time step; in short, it receives and updates the hidden state vector from the previous step. Note that for the first step, an arbitrary value is set for the hidden state vector g. By recording past states using hidden state vectors, it is possible to acquire new representations based on previously observed states (past history). This makes it possible to output different actions even when observing the same state repeatedly.

[0043] Examples of specific processing methods used in recurrent learning processing units include LSTM (Long Short-Term Memory), GRU (Gated Recurrent Unit), and RNN (Recurrent Neural Network). Figure 3 shows an example where GRU is used as the recurrent learning process.

[0044] Note that g and g' shown in Figure 3 are not input / output with an external source, but rather hidden state vectors passed between processing at preceding and succeeding time points within the same policy model. As mentioned earlier, g shown in Figure 3 is the result of receiving the hidden state vector g' output in the processing at the previous time point, and g' shown in Figure 3 is passed to the processing at the subsequent time point to become the hidden state vector g.

[0045] The host / behavior type selection processing unit (host / behavior type selection processing unit 24 in Figure 3) uses the vector z G It receives the input and outputs a vector a that indicates the probability of the next action. Internally, vector z G The vector v representing the host h and a vector v representing the type of action a It is decomposed (divided) into these. The matrix product v h T ×v a This obtains a value that includes both the host and the type of action. Based on this, a vector 'a' representing the probability of the action is obtained.

[0046] In Figure 3, the host / behavior type selection processing unit 24 processes the vector z G vector v representing the host h and a vector v representing the type of action a We will show that it can be decomposed into these, and the matrix product v h T ×v a This shows that a vector 'a' representing the probability of an action (host and action type) has been obtained.

[0047] Furthermore, the host / action type selection processing unit, particularly the part that handles action type selection, can be configured in a hierarchical manner. A specific example is to hierarchically structure action types into "strategy" and "tactics." Here, "strategy" is the overall action plan, and specific examples in security verification include scanning and intrusion. "Tactics" are specific actions, and specific examples in security verification include executing port scans and process scans corresponding to scanning, and executing intrusion commands via SMB (Server Message Block) or RDP (Remote Desktop Protocol) services corresponding to intrusion.

[0048] Also, vector v h and v a These can each be extended to matrices. In addition, vectors (or extended matrices) v h and v a Constraints may be added to ensure that they are orthogonal. For example, v h and v a The dimensions can be forced to be the same. h and v a If the axes are orthogonal, the host and behavior type can be represented independently, potentially leading to further performance improvements.

[0049] Furthermore, this embodiment makes it possible to represent in detail the state of the network system that is the target of security verification, and by stabilizing the learning process for optimizing the policy function for selecting actions for security verification, it becomes possible to handle larger networks. Thus, it becomes possible to contribute to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation."

[0050] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments described above are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0051] For example, Figures 1 to 3 show an embodiment in which the local feature processing unit, the recurrent learning processing unit, and the host / behavior type selection processing unit are connected in series, and the processing in each processing unit is performed in the order in which they are connected, but the system is not limited to this. For example, the host / behavior type selection processing unit may be made to perform local feature processing and recurrent learning processing again, thereby further stabilizing the learning process. Furthermore, the processing unit may include functions that do not accept arguments from outside the processing unit, such as normalization or linear transformation. [Explanation of Symbols]

[0052] 1. Action Selection Device (Learning Device) 10 Input section 20 Control Unit 21 Graph Processing Unit 22 Local feature processing unit 23 Recurrent Learning Processing Unit 24 Host / Action Type Selection Processing Unit 30 Storage section

Claims

1. A graph processing unit receives as input a graph that represents the state of a network system subject to security verification, using a triplicate set of nodes corresponding to hosts in the network system, edges corresponding to the connection relationships of the hosts, and feature vectors of nodes corresponding to feature vectors representing the attribute information of each host, and outputs a vector representing the features of the entire graph. A local feature processing unit receives a vector representing the features of the entire graph and a set of feature vectors of the nodes of the graph as input, and outputs a vector that takes into account the local features of the graph. A recurrent learning processing unit receives a vector that takes into account the local features of the graph as input and outputs a vector that takes into account the past state of the network system, A host / action type selection processing unit receives a vector that takes the aforementioned past state as input and outputs a vector indicating the probability of the next action to be taken for security verification, Equipped with, The host / action type selection processing unit decomposes the vector received as input, which takes the past state into a vector representing the host and a vector representing the action type, obtains a value that includes both the host and the action type by matrix product of these vectors, and obtains an action selection device that uses this to obtain a vector indicating the probability of the action.

2. The recurrent learning processing unit receives a hidden vector as input in addition to a vector that takes into account the local features of the graph, outputs an updated hidden vector in addition to a vector that takes into account past states, and receives the updated hidden vector as the hidden state vector for processing at the next time step, as described in claim 1.

3. A graph processing step that receives as input a graph representing the state of a network system subject to security verification, which is a triplicate consisting of a set of nodes corresponding to hosts in the network system, a set of edges corresponding to the connection relationships of the hosts, and a set of feature vectors of nodes corresponding to feature vectors representing the attribute information of each host, and outputs a vector representing the features of the entire graph. A local feature processing step that takes a vector representing the features of the entire graph and a set of feature vectors of the nodes of the graph as input, and outputs a vector that takes into account the local features of the graph, A recurrent learning process step that takes a vector considering the local features of the graph as input and outputs a vector considering the past state of the network system, A host / action type selection processing step that receives a vector considering the aforementioned past state as input and outputs a vector indicating the probability of the next action to be taken for security verification, Includes, The host / action type selection processing step involves decomposing the vector received as input, which takes into account the past state, into a vector representing the host and a vector representing the action type, obtaining a value that includes both the host and the action type by matrix product of these, and obtaining a vector indicating the probability of the action based on this.

4. The action selection method according to claim 3, wherein the recurrent learning processing step receives a hidden vector as input in addition to a vector that takes into account the local features of the graph, outputs an updated hidden vector in addition to a vector that takes into account past states, and receives the updated hidden vector as the hidden state vector in the processing of the next time step.

5. An action selection program for causing a computer to function as an action selection device according to claim 1 or claim 2.

Images