Network attack path discovery method and system based on knowledge graph

Abstract

The invention relates to the technical field of network security, in particular to a network attack path discovery method and system based on a knowledge graph, and the method comprises the steps: extracting entities, attributes and relationships among the entities in network attack and defense data, and constructing the network attack and defense knowledge graph based on the extracted entities, attributes and relationships among the entities; obtaining vector representations of entities and relationships in the network attack and defense knowledge graph by using a TransE model to obtain entity vectors and relationship vectors in network attack and defense; a reinforcement learning agent is set, an entity vector and a relation vector in network attack and defense are used as an interaction environment, an attack path prediction problem is expressed as a problem of exploring a sequence decision by using the reinforcement learning agent, and an attack path is reasoned and predicted through an interaction process of the reinforcement learning agent and the interaction environment. According to the method, relation reasoning is carried out by using reinforcement learning to accurately and efficiently predict the network attack path, so that a defense party can conveniently formulate practical and effective network security defense measures according to a prediction result.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network attack path discovery method and system based on knowledge graph. Background Art

[0002] With the development and popularization of network technology, people are increasingly dependent on the network in all areas of work and life, and the importance of the network is becoming increasingly prominent. However, in recent years, network attacks have frequently occurred, and the means of network attacks have become more advanced and complex. The network security situation is becoming increasingly severe, causing serious impacts and harm to the country, society and individuals. Therefore, it is particularly urgent to strengthen the research on network attacks and formulate effective and feasible network security defense measures. However, existing defense technologies such as intrusion detection and firewalls are post-defense measures, which have a certain delay and cannot detect attacks that have occurred but have not reached the target and are about to occur. Therefore, how to achieve pre-defense, that is, to analyze the attack behavior that has occurred through alarm correlation technology, infer the attacker's subsequent attack steps, effectively predict and perceive the attack path, and provide a decision-making basis for the scheduling of defense resources, so as to assist defense personnel to take timely countermeasures to reduce the risk of network attacks and reduce losses is extremely important.

[0003] At present, network attacks have a significant feature - multi-step, which means that attackers use certain weaknesses in the network to carry out planned, multi-stage attacks to achieve their ultimate goals. The purpose of attack path prediction is to predict the attacker's next move and the overall attack path by detecting and analyzing the attack behavior that has already occurred, thereby providing an important basis for defenders to take proactive countermeasures. Attack path prediction is mainly divided into two steps: 1) Identify and analyze the attack actions that have already occurred; 2) Infer the attacker's next move and the overall attack path. Existing attack path prediction technologies often have problems such as redundant paths, reliance on expert experience, poor prediction accuracy, and high prediction costs. At the same time, due to the rapid development of the network and the explosion of network security data, attack path prediction has become increasingly difficult. Summary of the invention

[0004] To this end, the present invention provides a network attack path discovery method and system based on knowledge graph, which uses reinforcement learning to perform relational reasoning to more accurately and efficiently predict the network attack path, making it easier for the defender to formulate practical and effective network security defense measures based on the prediction results.

[0005] According to the design scheme provided by the present invention, on the one hand, a network attack path discovery method based on knowledge graph is provided, comprising:

[0006] Extract entities, attributes and relationships between entities in network attack and defense data, and build a network attack and defense knowledge graph based on the extracted entities, attributes and relationships between entities. The entities include attack and defense parties, attack and defense modes, vulnerabilities and assets. The attributes include names of attack and defense parties, numbers of attack and defense parties, vulnerability numbers and vulnerability scores. The relationships are associations between entities.

[0007] The TransE model is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph, and the entity vector and relationship vector in the network attack and defense are obtained;

[0008] A reinforcement learning agent is set up, and the entity vectors and relationship vectors in network attack and defense are used as the interactive environment. The attack path prediction problem is expressed as a problem of exploring sequential decision-making using the reinforcement learning agent. The attack path is inferred and predicted through the interaction process between the reinforcement learning agent and the interactive environment.

[0009] As a network attack path discovery method based on knowledge graph of the present invention, further, entities, attributes and relationships between entities in network attack and defense data are extracted, including:

[0010] Extract attack pattern-related knowledge information from the open source attack pattern public standard platform based on the technical means adopted by the attacker during the attack; extract vulnerability-related knowledge information from the open source public vulnerability risk platform based on the weaknesses and defects existing in the system and / or equipment; use the open source network device general enumeration platform to extract asset-related knowledge information; extract attacker-related knowledge information based on the historical attack behavior information initiated by the attacker, and use the graph database Neo4j to store the extracted entity knowledge information.

[0011] As a network attack path discovery method based on knowledge graph of the present invention, further, the TransE model is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph, including:

[0012] Perform data preprocessing on the knowledge information in the network attack and defense knowledge graph, establish associations between entities by assigning values ​​to entities and relationships, and form triples of entities, attributes, and relationships;

[0013] For each entity and relationship, randomly generate entity vectors and relationship vectors of arbitrary dimensions;

[0014] Randomly select some triplets from the triplets, use the selected triplets as correct triplets, generate incorrect triplets by replacing entities in the correct triplets, and use the correct triplets and incorrect triplets to form training data;

[0015] Using the training data and based on the loss function, the randomly generated entity vectors and relationship vectors are updated by the gradient descent method to obtain the optimal entity vectors and relationship vectors. The loss function is used to optimize the TransE model by minimizing the correct triple vector distance and maximizing the incorrect triple vector distance.

[0016] As a network attack path discovery method based on knowledge graph of the present invention, further, the attack path is inferred and predicted by strengthening the interaction process between the learning agent and the interactive environment, including:

[0017] A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions and interactive environments. The rewards are used to represent the rewards given back to the agent by the interactive environment after the agent performs an action. The states are used to represent the changes in the interactive environment after the agent performs an action. The actions are used to find actions in the path that links the most information-rich path of entities with a given relationship. The action space is set to all relationships in the network attack and defense knowledge graph.

[0018] Through the interaction between the reinforcement learning agent and the interactive environment, the agent can continuously conduct trial and error learning through the feedback of the interactive environment to adjust and optimize the state and obtain the optimal path between entity pairs with specified relationships.

[0019] As the network attack path discovery method based on knowledge graph of the present invention, further, the reward fed back to the intelligent agent by the interactive environment after the intelligent agent performs the action includes:

[0020] If the agent reaches the target node after executing a series of actions, the interactive environment will give the agent an offline positive reward. Otherwise, the interactive environment will give the agent an offline negative reward.

[0021] The path between entity pairs is set as a relation sequence, and the interactive environment gives the agent path efficiency reward based on the inverse of the length of the relation sequence;

[0022] The cosine similarity is used to calculate the similarity of the paths between entity pairs, and the interactive environment uses this similarity to feed back diversity rewards to the agent, so as to use diversity rewards to encourage the agent to find different paths between entity pairs.

[0023] As a network attack path discovery method based on knowledge graph of the present invention, further, the intelligent agent is enabled to continuously perform trial and error learning through interactive environment feedback, including:

[0024] The supervised policy network is pre-trained using a bidirectional randomized breadth-first search algorithm to initially obtain multiple paths between different head entities and tail entities;

[0025] The pre-trained supervised policy network is retrained using the reward function to perform path prediction for entity pairs with specified relationships by calling the re-trained supervised policy network.

[0026] As a network attack path discovery method based on knowledge graph of the present invention, further, a bidirectional random breadth-first search algorithm is used to pre-train the supervised strategy network, including:

[0027] A random mechanism is added to the breadth-first search algorithm. The random mechanism is used to randomly select the intermediate nodes between the head entity and the tail entity. Breadth-first search is performed between the head entity and the intermediate nodes, and between the intermediate nodes and the tail entity. The search results are spliced ​​to preliminarily obtain multiple paths between the head entity and the tail entity.

[0028] On the other hand, the present invention also provides a network attack path discovery system based on knowledge graph, comprising: a knowledge extraction module, a vector representation module and a path prediction module, wherein:

[0029] A knowledge extraction module is used to extract entities, attributes and relationships between entities in network attack and defense data, and to construct a network attack and defense knowledge graph based on the extracted entities, attributes and relationships between entities. The entities include attack and defense parties, attack and defense modes, vulnerabilities and assets. The attributes include names of attack and defense parties, numbers of attack and defense parties, vulnerability numbers and vulnerability scores. The relationships are associations between entities.

[0030] The vector representation module is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph using the TransE model, and obtain the entity vectors and relationship vectors in the network attack and defense;

[0031] The path prediction module is used to set up a reinforcement learning agent, take the entity vectors and relationship vectors in network attack and defense as the interactive environment, express the attack path prediction problem as a problem of exploring sequential decision-making using the reinforcement learning agent, and infer and predict the attack path through the interaction process between the reinforcement learning agent and the interactive environment.

[0032] Beneficial effects of the present invention:

[0033] The present invention organizes, manages and utilizes network attack and defense data by using knowledge graphs. The specific operation is to apply knowledge graph technology to integrate thousands of data such as various hosts, vulnerabilities, attack behaviors themselves and the relationships between them in the network into a knowledge graph for intuitive visualization; use knowledge vectorization representation method, use translation-based embedding method, use TransE model to map high-dimensional entities and relationships into low-dimensional vector space, convert the objects to be described into low-dimensional vectors, obtain vector representations of entities and relationships, and provide accurate semantic information for subsequent application of reinforcement learning; apply reinforcement learning methods to knowledge graph relationship reasoning, and train a strategy recommendation agent based on graph embedding representation learning model to interact with the environment designed around the knowledge graph to perform path discovery. Compared with other studies that use graph traversal methods to predict paths, which have problems such as long path finding time and redundant paths in large-scale networks, the reinforcement learning method is introduced. By setting three evaluation reward indicators in the reinforcement learning process, the relationship path can be predicted more accurately and efficiently. The defender can understand the possible attack path of the attacker, the attack method used, and the vulnerability of the host on the path, so as to take corresponding defense measures according to the defense strategy for the attack method and vulnerability, and further improve the security of the network system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the network attack path discovery process based on the knowledge graph in the embodiment;

[0035] Figure 2 This is a diagram of TransE model training in the embodiment;

[0036] Figure 3 This is a schematic diagram of the overall structure of the reinforcement learning system in the embodiment;

[0037] Figure 4 This is a schematic diagram of the knowledge graph construction process in the embodiment;

[0038] Figure 5 This is a schematic diagram of the knowledge preprocessing effect in the embodiment;

[0039] Figure 6 It is a schematic diagram of entity vectors in the embodiment;

[0040] Figure 7 It is a schematic diagram of the relationship vector in the embodiment;

[0041] Figure 8 This is a schematic diagram of the network model pre-training process in the embodiment;

[0042] Fig. 9 This is a schematic diagram of the retraining results of the network model in the embodiment;

[0043] Fig.10 This is a schematic diagram of the network topology simulation structure in the embodiment;

[0044] Fig.11 An example of an attacker and a target asset in the embodiment;

[0045] Fig.12 This is a schematic diagram of the attack path prediction result in the embodiment;

[0046] Fig.13 The attack path diagram in the embodiment is shown schematically. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0048] Network attack path prediction is of great significance for improving the security of network systems. In recent years, network attack incidents have frequently occurred, and the network security situation has become increasingly severe. Therefore, it is particularly urgent to strengthen the research on network attack path prediction and formulate effective and feasible network security defense measures based on the prediction results. The existing attack path prediction technology often has problems such as redundant paths, reliance on expert experience, poor prediction accuracy, and high prediction cost. To this end, the embodiments of the present invention, see Figure 1 As shown, a network attack path discovery method based on knowledge graph is provided, which specifically includes:

[0049] S101. Extract entities, attributes, and relationships between entities from network attack and defense data, and construct a network attack and defense knowledge graph based on the extracted entities, attributes, and relationships between entities. The entities include attack and defense parties, attack and defense modes, vulnerabilities, and assets. The attributes include names of attack and defense parties, numbers of attack and defense parties, vulnerability numbers, and vulnerability scores. The relationships are associations between entities.

[0050] The knowledge graph in the field of network attack and defense is constructed based on the triple model. The triple model can be expressed as: <entity, attribute, relationship>, which can express the entities in the knowledge graph, the attributes possessed by the entities, and the relationships between the entities. The concepts are not listed in the model, which represent the general term for a certain type of entity. The concepts in the knowledge graph constructed in the embodiment of this case include attackers, attack patterns, vulnerabilities, assets, etc.; instances are specific entities of a certain type, such as a specific attacker in the concept of attackers, sql injection attacks and denial of service attacks in the concept of attack patterns, CVE-2022-0001 and CVE-2022-0002 in the concept of vulnerabilities, and hosts and servers in the concept of assets, etc.; attributes are the collection of attribute values ​​in the instance. In the constructed knowledge graph, the attributes of the attacker include attacker ID and capability level, the attributes of the attack mode include the severity of the attack mode, the attributes of the vulnerability include vulnerability score and CWEid associated with the vulnerability, and the attributes of the asset include asset ID, version number, partition, and type. The relationship is the relationship between each instance. The relationships in the knowledge graph constructed in this study include the usage relationship between attackers and attack patterns, the utilization relationship between attack patterns and vulnerabilities, and the existence relationship between vulnerabilities and assets.

[0051] Specifically, we can extract attack pattern-related knowledge information from the open source attack pattern public standard platform based on the technical means used by attackers during attacks; extract vulnerability-related knowledge information from the open source public vulnerability risk platform based on the weaknesses and defects existing in the system and / or equipment; use the open source network device general enumeration platform to extract asset-related knowledge information; extract attacker-related knowledge information based on historical attack behavior information initiated by attackers, and use the graph database Neo4j to store the extracted entity knowledge information.

[0052] Attack patterns, also known as attack methods, refer to the techniques and methods used by attackers when attacking. The data referenced in the embodiments of this case comes from the Common Attack Pattern Evaluation (CAPEC) platform. CAPEC provides the most common attacks against software. Users can understand how enemies use weaknesses or vulnerabilities in networks, programs, and software to attack, helping developers or network security personnel understand attack behaviors and how to prevent them. Taking CAPEC-21 Exploitation of Trusted Identifiers as an example, each CAPEC entry includes the ID, description, attack severity, correlation with other attack patterns, attack scope, attack mechanism, attack process, how to mitigate attacks, and the corresponding relationship with CWE.

[0053] A vulnerability refers to a weakness or defect that may exist in a system or device. The data referenced in the embodiments of this case comes from the Common Vulnerabilities and Exposures (CVE) platform. The task of the CVE platform is to identify, describe, and classify publicly disclosed security vulnerabilities and name them, so as to help users identify, discover, and then repair security vulnerabilities in software and systems more quickly and efficiently. Taking CVE-2022-26317 as an example, each CVE entry includes the vulnerability ID, description, CVSS severity and indicators, weakness enumeration, and CPE affected by the CVE.

[0054] Assets refer to various devices on various networks. The reference data mainly comes from the Common Platform Enumeration (CPE), which uses the software, operating system, hardware and other information included in the CPE to represent assets. Each CPE entry includes the name, version number, source and product model of the asset.

[0055] The attacker dimension data mainly includes the attack modes that the attacker can use and the assets that the attacker can attack. These two relationships can be obtained by collecting and analyzing the historical attack behavior information launched by the attacker. According to the experimental conditions, the attacker dimension data can also be simulated and constructed.

[0056] Neo4j is a graph database. Unlike relational databases, it stores data in the network, making it easier to handle a series of problems that are difficult to solve with traditional relational databases. There are two types of data in the Neo4j graph database, nodes and edges. Nodes represent entities, while edges represent the relationship between two entities. The acquired knowledge data is stored in the Neo4j graph database. Python is used to import knowledge data into the neo4j graph database step by step, and the relationship between entities is established. Finally, the color and size of the entities and relationships are adjusted to facilitate distinguishing and viewing. In neo4j, attackers, attack patterns, vulnerabilities, etc. are represented as entities, attacker names, attacker numbers, vulnerability numbers, vulnerability scores, etc. are represented as attributes, and the connections between attackers, attack patterns, vulnerabilities, assets, etc. are represented as relationships with edges as the medium. The corresponding relationships between various entities after import can be shown in Table 1 below:

[0057] Table 1

[0058] entity relation entity 1 Attacker attack assets 2 Attacker exploit Vulnerabilities 3 Attacker use Attack Mode 4 Vulnerabilities exist assets 7 assets canprecede assets 8 Attack Mode childof Attack Mode

[0059] S102. Use the TransE model to obtain vector representations of entities and relationships in the network attack and defense knowledge graph, and obtain entity vectors and relationship vectors in the network attack and defense.

[0060] Among them, the TransE model is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph, which can be designed to include:

[0061] Perform data preprocessing on the knowledge information in the network attack and defense knowledge graph, establish associations between entities by assigning values ​​to entities and relationships, and form triples of entities, attributes, and relationships;

[0062] For each entity and relationship, randomly generate entity vectors and relationship vectors of arbitrary dimensions;

[0063] Randomly select some triplets from the triplets, use the selected triplets as correct triplets, generate incorrect triplets by replacing entities in the correct triplets, and use the correct triplets and incorrect triplets to form training data;

[0064] Using the training data and based on the loss function, the randomly generated entity vectors and relationship vectors are updated by the gradient descent method to obtain the optimal entity vectors and relationship vectors. The loss function is used to optimize the TransE model by minimizing the correct triple vector distance and maximizing the incorrect triple vector distance.

[0065] The knowledge used to import the knowledge graph is preprocessed to convert the knowledge into a data format that is convenient for vector representation. Since the data used to build the knowledge graph is in csv format and lacks relationships, the data needs to be converted into entity files, relationship files, and triple format files that meet the requirements of dimensionality reduction processing, that is, assigning ID values ​​to entities, relationships, etc. and establishing relationships between entities to form triple formats.

[0066] The steps of the TransE algorithm are shown in Algorithm 1:

[0067]

[0068] In Algorithm 1, Step 1-Step 3: Initialize the vectors of entities and relations. Step 6: Randomly sample a minibatch for embedding training. Step 7-Step 11: Complete the construction of the training dataset. Randomly select a certain number of correct triplets (h, r, t), replace the h in the correct triplets with h′ in other triplets, and replace the t in the correct triplets with t′ in other triplets to obtain the wrong triplets (h′, r, t) or (h, r, t′), obtain the wrong triplets, merge them together to complete the construction of the batch training dataset. Step 5, Step 12: Represents vector update, and updates the vectors of the correct triplets (h, r, t) and the wrong triplets (h′, r, t) or (h, r, t′) in the batch training dataset constructed in the previous step.

[0069] Compared with the entity relationship table given in the knowledge graph construction process, it is also necessary to define and vectorize the reverse relationships of exist, exploit, canprecede, and childof. The types of correct triples (h, r, t) are shown in Table 2:

[0070] Table 2

[0071]

[0072]

[0073] Model training process, such as Figure 2 As shown in the figure, the first step is to import the training data, obtain the number of entities and entity lists, the total number of relations and relationship lists, and the total number of triples and triple lists. Then instantiate the TransE class, put the entity list, relationship list, and triple list into it, set the spacing γ to 1, and then set the dimension of the output vector to 20. The second step is to call the initialize() method of transE for initialization. For each entity, randomly generate a vector of any dimension, then normalize the vector to ensure that it is a unit vector, and then form a list of 20 elements to represent the vector of this entity. At the same time, put this entity and its corresponding randomly generated vector into the newly created dictionary entityVectorList. The same operation is performed for relations. The third step is to train, randomly select a certain number of triplets, traverse these triplets, call the getCorruptedTriplet() method in the code to obtain the broken triplet of a certain triplet, put the broken triplet and the correct triplet together to form a new tuple, and put it into the list after deduplication. The fourth step is to update the vector, calculate the loss function, and update the previously randomly generated vector through the gradient descent method to obtain an optimal vector.

[0074] S103. Set a reinforcement learning agent, use entity vectors and relationship vectors in network attack and defense as an interactive environment, express the attack path prediction problem as a problem of exploring sequential decision-making using a reinforcement learning agent, and infer and predict the attack path through the interaction process between the reinforcement learning agent and the interactive environment.

[0075] Specifically, reasoning and predicting attack paths through the interaction process between the reinforcement learning agent and the interactive environment may include:

[0076] A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions and interactive environments. The rewards are used to represent the rewards given back to the agent by the interactive environment after the agent performs an action. The states are used to represent the changes in the interactive environment after the agent performs an action. The actions are used to find actions in the path that links the most information-rich path of entities with a given relationship. The action space is set to all relationships in the network attack and defense knowledge graph.

[0077] Through the interaction between the reinforcement learning agent and the interactive environment, the agent can continuously conduct trial and error learning through the feedback of the interactive environment to adjust and optimize the state and obtain the optimal path between entity pairs with specified relationships.

[0078] like Figure 3 As shown in Figure 1, the reinforcement learning system consists of two parts. The first part is the external environment represented by a Markov decision process. Define a tuple<S,A,P,R> To represent the Markov decision process, where S is a finite state set, A = {a 1 ,a 2 ,...,a n} is a finite set of actions, P(S t+1 =s′|S t =s,A t =a) is the transition probability matrix, and R(s,a) is the probability matrix for each (s,a) pair. The second part of the system, the agent, uses the strategy π θ (s, a) = p(a|s; θ), which maps the state vector to a random policy.

[0079] Figure 3 In the figure, the left part is the reinforcement learning environment, which gives the path process and related nodes of the initial node Attacker202 and the target node admin-font-editor to be queried. The right part is the policy-based agent, in which a fully connected neural network is used to parameterize the policy function. The neural network consists of two hidden layers, each of which is followed by a rectifier nonlinear layer, and the output layer is normalized using softmax. The line in the middle indicates that the agent takes an action in the environment, and the environment feeds back a new state and reward to the agent. The components of the reinforcement learning environment are composed of actions, states, and rewards. The reinforcement learning environment is composed of vector representations of entities and relationships.

[0080] Action: Given an entity pair (e s ,e t ), we want the agent to find the most informative path linking these entity pairs. s Initially, the agent uses the policy network to select the most promising relationship to extend its path at each step until it reaches the target entity e tTo keep the output dimensionality of the policy network consistent, the action space is defined as all relations in the knowledge graph.

[0081] State: It is impossible to directly model all the discrete atomic information in the knowledge graph. In order to capture the semantic information of these symbols, we use the translation-based embedding method, TransE, to process the entity and relationship data used to build the knowledge graph in the above steps to construct a continuous vector space. In our framework, the state can capture the position of the agent in the continuous space. After the agent performs an action, it will move from one entity to the next. The state vector of step t is as follows: s t =(e t, e target -e t) . Where e t Represents the vector of the current entity node, e target represents the vector of the target entity. In the initial state, e t =e source .

[0082] Reward: The quality of the path predicted by the agent will be affected by the reward. In order to make the path prediction more accurate and efficient, the reward function can be used to set the reward scoring criteria.

[0083] Specifically, the reward given to the agent by the interactive environment after the agent performs an action may include:

[0084] If the agent reaches the target node after executing a series of actions, the interactive environment will give the agent an offline positive reward. Otherwise, the interactive environment will give the agent an offline negative reward.

[0085] The path between entity pairs is set as a relation sequence, and the interactive environment gives the agent path efficiency reward based on the inverse of the length of the relation sequence;

[0086] The cosine similarity is used to calculate the similarity of the paths between entity pairs, and the interactive environment uses this similarity to feed back diversity rewards to the agent, so as to use diversity rewards to encourage the agent to find different paths between entity pairs.

[0087] Specifically, for global accuracy: For the environment setting, the number of actions that the agent can take is very large, the number of incorrect sequence decisions may be greater than the number of correct decisions, and the number of these incorrect sequence decisions will increase exponentially as the path becomes longer. To address this problem, the first reward function that can be added to the reinforcement learning model is the global reward: that is, if the agent reaches the goal after performing a series of actions, it will be given an offline positive reward of +1. The calculation process can be expressed as follows:

[0088]

[0089] Regarding path efficiency: In relation prediction tasks, shorter paths are often more accurate and reliable than long paths, and encouraging short paths can also limit the number of actions performed by the reinforcement learning agent, thereby improving reasoning efficiency. The path efficiency reward is defined as follows: A path p is defined as a relation sequence r 1 →r 2 →...→r n , the specific calculation process can be expressed as:

[0090]

[0091] For path diversity: The agent is trained to find paths using positive samples for each relationship. These training samples (e source , e target ) have similar state representations in the vector space. The paths found by the agent are usually similar and contain a lot of redundant information. In order to make the agent find different paths, a diversity reward function can be defined using cosine similarity: where p represents the relationship chain r 1 →r 2 →...→r n Path embedding. The specific calculation process can be expressed as:

[0092]

[0093] Specifically, the interactive environment feedback enables the agent to continuously conduct trial and error learning, which can be designed to include:

[0094] The supervised policy network is pre-trained using a bidirectional randomized breadth-first search algorithm to initially obtain multiple paths between different head entities and tail entities;

[0095] The pre-trained supervised policy network is retrained using the reward function to perform path prediction for entity pairs with specified relationships by calling the re-trained supervised policy network.

[0096] In practice, a big challenge of reasoning on knowledge graphs is that the set of relations can be very large. For a typical knowledge graph, the reinforcement learning agent often faces hundreds (thousands) of possible actions, i.e., the output layer of the policy network has a large dimension. Due to the complexity of the relation graph and the large action space, if the reinforcement learning model is directly trained by trial and error, it will show very poor convergence. After a long period of training, the agent did not find any valuable paths. To solve this problem, a supervised policy network can be trained first. By pre-training the model, multiple paths can eventually be found between two given nodes.

[0097] For each relation, a set of all positive samples is used to learn a supervision policy. For each positive sample (e souree , e target ), using the bilateral breadth-first search method to find the correct path between entities. 1 →r 2 →...→r n Path p uses the Monte Carlo policy gradient method to update the parameters θ to maximize the expected cumulative reward.

[0098]

[0099] where J(θ) is the expected total reward for each iteration. For supervised learning, a reward of +1 is given for each successful iteration. By inserting the path found by BFS, the approximate gradient used to update the policy network is as follows:

[0100]

[0101] Through supervised learning, the agent can initially find some desired paths.

[0102] The breadth-first search algorithm is different from the depth-first search algorithm. The depth-first search aims to explore all the way to the end no matter how many forks there are, and tends to find long paths, while the breadth-first search algorithm tends to search for short paths. The basic idea is: starting from a certain node, try to go through all the nodes that the node can be connected to, record each fork, and then choose a fork to enter the next node, and then record all the forks of the node again, and then return to select a fork to enter, and repeat the operation directly to the last node.

[0103] An entity node can be linked to a large number of neighbors with the same relationship links. If we simply use the breadth-first search algorithm to search from the head entity to the tail entity step by step, the number of entities passed through in the middle will grow exponentially. Therefore, in this embodiment, a bidirectional path search algorithm is applied, that is, a random mechanism is added to the breadth-first search. Suppose we want to search for the path between two nodes A and B, we randomly select an intermediate node C, and execute the breadth-first search algorithm twice between nodes A and C, and between C and B. After using this method, the number of entities passed through in the middle can be greatly reduced. The bidirectional path constraint search algorithm is as follows:

[0104]

[0105] In order to obtain the prediction path controlled by the reward function, these reward functions are used to retrain the supervised policy network obtained by the previous pre-training. Starting from the source node, the agent selects a relationship to extend its reasoning path according to the random policy function π(a|s) with all relationship possibilities. This relationship link may lead to a new entity or nothing. Wrong steps will cause the agent to receive negative rewards. In order to improve training efficiency, a maximum length is limited to each process. If the agent does not reach the end point within the maximum limit step, the event ends. After each event, the following gradient is used to update the policy network:

[0106]

[0107] The algorithm for retraining the model with the reward function is as follows:

[0108]

[0109] Furthermore, based on the above method, an embodiment of the present invention also provides a network attack path discovery system based on knowledge graph, comprising: a knowledge extraction module, a vector representation module and a path prediction module, wherein:

[0110] A knowledge extraction module is used to extract entities, attributes and relationships between entities in network attack and defense data, and to construct a network attack and defense knowledge graph based on the extracted entities, attributes and relationships between entities. The entities include attack and defense parties, attack and defense modes, vulnerabilities and assets. The attributes include names of attack and defense parties, numbers of attack and defense parties, vulnerability numbers and vulnerability scores. The relationships are associations between entities.

[0111] The vector representation module is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph using the TransE model, and obtain the entity vectors and relationship vectors in the network attack and defense;

[0112] The path prediction module is used to set up a reinforcement learning agent, take the entity vectors and relationship vectors in network attack and defense as the interactive environment, express the attack path prediction problem as a problem of exploring sequential decision-making using the reinforcement learning agent, and infer and predict the attack path through the interaction process between the reinforcement learning agent and the interactive environment.

[0113] In order to verify the effectiveness of this solution, the following is a further explanation based on experimental data:

[0114] Based on the acquired and sorted network attack and defense data, the knowledge graph is imported into the neo4j graph database to build the knowledge graph. The process of building the knowledge graph is as follows: Figure 4 After running the code to import the data into neo4j, you can see the constructed knowledge graph after opening neo4j.

[0115] Then, the data is preprocessed and the constructed TransE model is used to obtain the vector representation of entities and relationships. Since the bidirectional path search method is used when building the reinforcement learning model, a reverse relationship must be defined when defining the relationship between entities during data preprocessing. At the same time, a relationship is added between the entity in the entity file and the relationship in the relationship file to obtain a triple. The effect after preprocessing is as follows: Figure 5 After the data preprocessing in the previous step, vectorization is performed next, and entities and relationships are represented as low-dimensional vectors (20 dimensions) to obtain entity vector files and relationship vector files. The data obtained after vectorization is as follows Figure 6 and 7 shown.

[0116] Use the bidirectional random breadth-first search algorithm to pre-train the model. The training process is as follows: Figure 8 As shown, it can be seen that through the bidirectional breadth-first search algorithm, multiple paths can be initially found between different head entities and tail entities.

[0117] Based on the pre-training, the pre-trained model is re-trained using the defined reward function. The training results are as follows: Fig. 9 As shown in the figure, in the training results, you can see whether the wheel diameter search is successful, as well as the specific path found, path length, search time, etc. After the model is trained, you can directly call the model for prediction.

[0118] Based on the above data set, an experimental scenario is constructed to simulate the network topology of a department in the company. The topology is as follows Fig.10 As shown in the figure, external network attackers can access this network through the Internet. The intruders are separated from the external network by a firewall. The network consists of 3 hosts and 2 servers. The hosts can be interconnected with each other, and the servers can be interconnected with each other. The firewall here does not impose rule information restrictions.

[0119] According to the previous attack information records, the attack methods that the attacker can use and the vulnerabilities exploited by the attack methods are obtained, as shown in Table 3. The CPE information obtained in the previous steps is used to represent the hosts in the topology map. At the same time, vulnerability scans are performed on each host and server to obtain the existing vulnerability information, as shown in Table 4.

[0120] Table 3

[0121]

[0122] Table 4

[0123]

[0124] Assuming that the attack target is host 3, the network topology diagram shows that the reachable paths from the attacker to the target host are

[0125] Path 1: Attacker -> Host 1 -> Host 2 -> Host 3 -> Host 4

[0126] Path 2: Attacker -> Host 2 -> Host 3 -> Host 4

[0127] Path3: Attacker -> Host 3 -> Host 4

[0128] Then, according to the attack relationship table and vulnerability information table, we can know that Path2 and Path3 can be successfully attacked. The attack methods used and the vulnerabilities exploited during the attack are respectively

[0129] Path2: Host 2 (TCP_Connect_Scan, CVE-2020-0256) -> Host 3 (Eavesdropping, CVE-2020-0340) -> Host 4 (Malicious_Automated_Software_Update_via_Spoofing, CVE-2020-0001)

[0130] Path3: Host 3 (Eavesdropping, CVE-2020-0209) -> Host 4 (Malicious_Automated_Software_Update_via_Spoofing, CVE-2020-0001)

[0131] Next, run the program to verify the model and manually enter the attacker and target assets, such as Fig.11 After the input is completed, the system will give the attack path predicted by the model, such as Fig.12 As shown in the figure, this path is Path3 obtained after the above analysis. This path is a reachable path, and it is the shortest and uses the least attack methods and exploits. After the specific path is given, the path is imported back to neo4j for display, as shown in the figure below: Fig.13 shown.

[0132] From the above experimental data, we can see that this solution can successfully predict a reachable path, with high accuracy and taking into account path efficiency (tending to find a shorter path). Compared with the results obtained by simple traversal of the graph traversal algorithm, the path predicted by this model takes less time and has no redundant paths. After the prediction is completed, the attack path that the attacker may take can be intuitively displayed. The defender can understand the attack method used by the attacker and the vulnerabilities of the host on the path, so as to take corresponding defense measures according to the defense strategy for the attack method and vulnerability, and strengthen the important asset nodes that need to be protected.

[0133] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0134] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0135] The units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation is not considered to be beyond the scope of the present invention.

[0136] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0137] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A network attack path discovery method based on knowledge graph, characterized in that: Include: Extract entities, attributes and relationships between entities in network attack and defense data, and build a network attack and defense knowledge graph based on the extracted entities, attributes and relationships between entities. The entities include attack and defense parties, attack and defense modes, vulnerabilities and assets. The attributes include names of attack and defense parties, numbers of attack and defense parties, vulnerability numbers and vulnerability scores. The relationships are associations between entities. The TransE model is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph, and the entity vector and relationship vector in the network attack and defense are obtained; A reinforcement learning agent is set up, and the entity vectors and relationship vectors in network attack and defense are used as the interactive environment. The attack path prediction problem is expressed as a problem of exploring sequential decision-making using the reinforcement learning agent. The attack path is inferred and predicted through the interaction process between the reinforcement learning agent and the interactive environment.

2. The network attack path discovery method based on knowledge graph according to claim 1 is characterized in that: Extract entities, attributes, and relationships between entities from network attack and defense data, including: Extract the attack pattern-related knowledge information in the open source attack pattern public standard platform based on the technical means adopted by the attacker during the attack; extract the vulnerability-related knowledge information in the open source public vulnerability risk platform based on the weaknesses and defects existing in the system and / or equipment; An open source network device general enumeration platform is used to extract asset-related knowledge information. Based on the historical attack behavior information initiated by the attacker, attacker-related knowledge information is extracted, and the graph database Neo4j is used to store the extracted entity knowledge information.

3. The network attack path discovery method based on knowledge graph according to claim 1 is characterized in that: The TransE model is used to obtain vector representations of entities and relationships in the network attack and defense knowledge graph, including: Perform data preprocessing on the knowledge information in the network attack and defense knowledge graph, establish associations between entities by assigning values ​​to entities and relationships, and form triples of entities, attributes, and relationships; For each entity and relationship, randomly generate entity vectors and relationship vectors of arbitrary dimensions; Randomly select some triplets from the triplets, use the selected triplets as correct triplets, generate incorrect triplets by replacing entities in the correct triplets, and use the correct triplets and incorrect triplets to form training data; Using the training data and based on the loss function, the randomly generated entity vectors and relationship vectors are updated by the gradient descent method to obtain the optimal entity vectors and relationship vectors. The loss function is used to optimize the TransE model by minimizing the correct triple vector distance and maximizing the incorrect triple vector distance.

4. The network attack path discovery method based on knowledge graph according to claim 1 is characterized in that: Infer and predict attack paths through the interaction process between the reinforcement learning agent and the interactive environment, including: A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions and interactive environments. The rewards are used to represent the rewards given back to the agent by the interactive environment after the agent performs an action. The states are used to represent the changes in the interactive environment after the agent performs an action. The actions are used to find actions in the path that links the most information-rich path of entities with a given relationship. The action space is set to all relationships in the network attack and defense knowledge graph. Through the interaction between the reinforcement learning agent and the interactive environment, the agent can continuously conduct trial and error learning through the feedback of the interactive environment to adjust and optimize the state and obtain the optimal path between entity pairs with specified relationships.

5. The network attack path discovery method based on knowledge graph according to claim 4 is characterized in that: After the agent performs an action, the interactive environment gives the agent a reward, which includes: If the agent reaches the target node after executing a series of actions, the interactive environment will give the agent an offline positive reward. Otherwise, the interactive environment will give the agent an offline negative reward. The path between entity pairs is set as a relation sequence, and the interactive environment gives the agent path efficiency reward based on the inverse of the length of the relation sequence; The cosine similarity is used to calculate the similarity of the paths between entity pairs, and the interactive environment uses this similarity to feed back diversity rewards to the agent, so as to use diversity rewards to encourage the agent to find different paths between entity pairs.

6. The network attack path discovery method based on knowledge graph according to claim 4 or 5 is characterized in that: The interactive environment feedback enables the agent to continuously learn through trial and error, including: The supervised policy network is pre-trained using a bidirectional randomized breadth-first search algorithm to initially obtain multiple paths between different head entities and tail entities; The pre-trained supervised policy network is retrained using the reward function to perform path prediction for entity pairs with specified relationships by calling the re-trained supervised policy network.

7. The network attack path discovery method based on knowledge graph according to claim 6 is characterized in that: Pre-training the supervised policy network using a bidirectional randomized breadth-first search algorithm, including: A random mechanism is added to the breadth-first search algorithm. The random mechanism is used to randomly select the intermediate nodes between the head entity and the tail entity. Breadth-first search is performed between the head entity and the intermediate nodes, and between the intermediate nodes and the tail entity. The search results are spliced ​​to preliminarily obtain multiple paths between the head entity and the tail entity.

8. A network attack path discovery system based on knowledge graph, characterized in that: It includes: knowledge extraction module, vector representation module and path prediction module, among which, A knowledge extraction module is used to extract entities, attributes and relationships between entities in network attack and defense data, and to construct a network attack and defense knowledge graph based on the extracted entities, attributes and relationships between entities. The entities include attack and defense parties, attack and defense modes, vulnerabilities and assets. The attributes include names of attack and defense parties, numbers of attack and defense parties, vulnerability numbers and vulnerability scores. The relationships are associations between entities. The vector representation module is used to obtain the vector representation of entities and relationships in the network attack and defense knowledge graph using the TransE model, and obtain the entity vectors and relationship vectors in the network attack and defense; The path prediction module is used to set up a reinforcement learning agent, take the entity vectors and relationship vectors in network attack and defense as the interactive environment, express the attack path prediction problem as a problem of exploring sequential decision-making using the reinforcement learning agent, and infer and predict the attack path through the interaction process between the reinforcement learning agent and the interactive environment.

9. An electronic device, characterized in that: include: at least one processor, and a memory coupled to the at least one processor; The memory stores a computer program, and the computer program can be executed by the at least one processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 can be implemented.

Images