Knowledge graph-based network attack path discovery method and system

Abstract

The present application relates to the technical field of network security, in particular to a network attack path discovery method and system based on a knowledge graph, which extracts entities, attributes and relationships between entities in network attack and defense data, constructs a network attack and defense knowledge graph based on the extracted entities, attributes and relationships between entities, obtains vector representations of entities and relationships in the network attack and defense knowledge graph using a TransE model, sets a reinforcement learning agent, takes the entity vectors and relationship vectors in the network attack and defense as an interactive environment, represents the attack path prediction problem as a sequence decision problem explored by the reinforcement learning agent, and reasons and predicts the attack path through the interaction process between the reinforcement learning agent and the interactive environment. The present application uses reinforcement learning to perform relationship reasoning to accurately and efficiently predict the network attack path, which facilitates the defense party to develop effective network security defense measures based on the prediction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and particularly relates to a network attack path discovery method and system based on a knowledge graph. BACKGROUND

[0002] With the development and popularization of network technology, people rely more and more on the network in various fields of work and life, and the importance of the network is increasingly prominent. However, in recent years, network attack incidents have frequently occurred, network attack means are increasingly advanced and complex, and the network security situation is increasingly severe, which has caused serious influence and harm to the country, society and individuals, and therefore it is particularly urgent to strengthen the research on network attacks and develop practical and effective network security defense measures. However, the existing defense technologies such as intrusion detection and firewall are all post-defense measures, which have a certain delay and cannot detect attacks that have already occurred but have not reached the target and attacks that are about to occur. Therefore, it is extremely important to achieve pre-defense, that is, to analyze the attack behavior that has already occurred through alarm correlation technology, to infer the subsequent attack steps of the attacker, to effectively predict the attack path, to provide a decision basis for defense resource scheduling, and to assist defense personnel to take timely response measures to reduce the risk of network attacks and reduce losses.

[0003] At present, network attacks have a significant feature of multi-step, which means that attackers use certain weaknesses in the network to carry out planned and multi-stage attacks to achieve their ultimate goal. The purpose of attack path prediction is to detect and analyze the attack behavior that has already occurred, to predict the next action of the attacker and the overall attack path, and to provide an important basis for defense personnel to take proactive response measures. Attack path prediction mainly includes two steps: 1) identifying and analyzing the attack actions that have already occurred; and 2) inferring the next action of the attacker and the overall attack path. In the existing attack path prediction technology, there are problems such as redundant paths, reliance on expert experience, poor prediction accuracy, and high prediction cost. At the same time, due to the rapid development of the network and the explosion of network security data, it is increasingly difficult to predict attack paths. SUMMARY

[0004] Therefore, the present application provides a network attack path discovery method and system based on a knowledge graph, which uses reinforcement learning to perform relationship reasoning to accurately and efficiently predict network attack paths, so as to facilitate the defense party to develop practical and effective network security defense measures based on the prediction results.

[0005] According to the design scheme provided by the present application, on the one hand, a network attack path discovery method based on a knowledge graph is provided, which includes:

[0006] Entities, attributes, and relationships between entities are extracted from network attack and defense data. A network attack and defense knowledge graph is constructed based on the extracted entities, attributes, and relationships between entities. The entities include attacking and defending parties, attack and defense modes, vulnerabilities, and assets. The attributes include the names of the attacking and defending parties, their numbers, vulnerability numbers, and vulnerability scores. The relationships are the associations between entities.

[0007] The TransE model is used to obtain vector representations of entities and relations in the network attack and defense knowledge graph, resulting in entity vectors and relation vectors in network attack and defense.

[0008] A reinforcement learning agent is set up, and the entity vectors and relation vectors in network attack and defense are used as the interaction environment. The attack path prediction problem is represented as a problem of exploring sequence decision using the reinforcement learning agent. The attack path is inferred and predicted through the interaction process between the reinforcement learning agent and the interaction environment.

[0009] As a network attack path discovery method based on knowledge graphs in this invention, further, the method extracts entities, attributes, and relationships between entities from network attack and defense data, including:

[0010] Based on the technical means employed by the attacker, knowledge information related to attack patterns in the open-source attack pattern public standard platform is extracted; based on the vulnerabilities and defects existing in the system and / or equipment, knowledge information related to vulnerabilities in the open-source public vulnerability risk platform is extracted; asset-related knowledge information is extracted using the open-source network device general enumeration platform; and attacker-related knowledge information is extracted based on the attacker's historical attack behavior information, so as to store the extracted entity knowledge information using the graph database Neo4j.

[0011] As a network attack path discovery method based on knowledge graphs in this invention, further, the TransE model is used to obtain vector representations of entities and relations in the network attack and defense knowledge graph, including:

[0012] Data preprocessing is performed on the knowledge information in the network attack and defense knowledge graph. By assigning values ​​to entities and relations, associations are established between entities, and triples of entities, attributes, and relations are formed.

[0013] For each entity and relation, randomly generate entity vectors and relation vectors of arbitrary dimensions;

[0014] Randomly select a portion of the triplets and use them as the correct triplets. Generate incorrect triplets by replacing the entities in the correct triplets. Combine the correct triplets and incorrect triplets to form the training data.

[0015] Using training data and based on a loss function, the randomly generated entity vectors and relation vectors are updated using gradient descent to obtain the optimal entity vectors and relation vectors. The loss function is used to optimize the TransE model by minimizing the distance between correct triplet vectors and maximizing the distance between incorrect triplet vectors.

[0016] As a knowledge graph-based network attack path discovery method of the present invention, further, the attack path is inferred and predicted through the interaction process between the reinforcement learning agent and the interactive environment, including:

[0017] A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions, and interactive environments. The reward is used to represent the reward fed back to the agent by the interactive environment after the agent performs an action. The state is used to represent the change in the interactive environment after the agent performs an action. The action is used to find the action in the path that links entities with given relationships with the richest information. The action space is set as all relationships in the network attack and defense knowledge graph.

[0018] By reinforcing the interaction between the agent and the interactive environment, and through feedback from the interactive environment, the agent continuously learns through trial and error to adjust and optimize its state, thereby obtaining the optimal path between entity pairs with a specified relationship.

[0019] As part of the knowledge graph-based network attack path discovery method of this invention, the reward fed back to the agent by the interactive environment after the agent performs an action includes:

[0020] If the agent reaches the target node after performing a series of actions, the interactive environment will provide a positive offline reward to the agent; otherwise, the interactive environment will provide a negative offline reward to the agent.

[0021] The paths between entity pairs are set as relation sequences, and the interactive environment provides path efficiency rewards to the agent based on the reciprocal of the relation sequence length.

[0022] Cosine similarity is used to calculate the similarity of paths between entity pairs. The interactive environment uses this similarity to provide diversity rewards to the agent, thereby encouraging the agent to find different paths between entity pairs.

[0023] As a knowledge graph-based network attack path discovery method of the present invention, further, the agent continuously learns through trial and error by using interactive environment feedback, including:

[0024] A bidirectional random breadth-first search algorithm is used to pre-train a supervised policy network to initially obtain multiple paths between different head and tail entities.

[0025] The pre-trained supervised policy network is retrained using a reward function, so that path prediction can be performed on entity pairs with specified relationships by calling the retrained supervised policy network.

[0026] As a knowledge graph-based network attack path discovery method of the present invention, further, a bidirectional random breadth-first search algorithm is used to pre-train the supervised policy network, comprising:

[0027] A random mechanism is added to the breadth-first search algorithm to randomly select intermediate nodes between the head entity and the tail entity. Breadth-first search is then performed sequentially between the head entity and intermediate nodes, and between intermediate nodes and the tail entity. The search results are then concatenated to initially obtain multiple paths between the head entity and the tail entity.

[0028] Furthermore, this invention also provides a network attack path discovery system based on knowledge graphs, comprising: a knowledge extraction module, a vector representation module, and a path prediction module, wherein...

[0029] The knowledge extraction module is used to extract entities, attributes, and relationships between entities from network attack and defense data. Based on the extracted entities, attributes, and relationships between entities, a network attack and defense knowledge graph is constructed. The entities include attacking and defending parties, attack and defense modes, vulnerabilities, and assets. The attributes include the names of the attacking and defending parties, their numbers, vulnerability numbers, and vulnerability scores. The relationships are the associations between entities.

[0030] The vector representation module is used to obtain vector representations of entities and relations in the network attack and defense knowledge graph using the TransE model, and to obtain entity vectors and relation vectors in network attack and defense.

[0031] The path prediction module is used to set up a reinforcement learning agent, using entity vectors and relation vectors in network attack and defense as the interaction environment. It represents the attack path prediction problem as a problem of exploring sequence decision using a reinforcement learning agent, and infers and predicts the attack path through the interaction process between the reinforcement learning agent and the interaction environment.

[0032] The beneficial effects of this invention are:

[0033] This invention organizes, manages, and utilizes network attack and defense data using knowledge graphs. Specifically, it applies knowledge graph technology to fuse thousands of data points—including various hosts, vulnerabilities, attack behaviors, and their relationships—onto a single knowledge graph for intuitive visualization. Using a knowledge vectorization representation method and a translation-based embedding approach, the TransE model maps high-dimensional entities and relationships to a low-dimensional vector space, converting the objects to be described into low-dimensional vectors, thus obtaining vector representations of entities and relationships. This provides accurate semantic information for subsequent reinforcement learning applications. Reinforcement learning methods are then applied to knowledge graph relationship reasoning. A policy recommendation agent based on a graph embedding representation learning model is trained to interact with an environment designed around the knowledge graph, thereby enabling path discovery. Compared to other studies that use graph traversal methods to predict paths, which suffer from problems such as long pathfinding times and redundant paths in large-scale networks, this study introduces reinforcement learning. By setting three evaluation reward metrics during the reinforcement learning process, it can more accurately and efficiently predict relational paths. Defenders can then understand the attacker's possible attack paths, the attack methods used, and the vulnerabilities exploited on the hosts along the path. Based on this, they can take corresponding defensive measures according to the defense strategies against the attack methods and vulnerabilities, thereby further improving the security of the network system. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the network attack path discovery process based on knowledge graphs in the embodiment;

[0035] Figure 2 This is a schematic diagram of the TransE model training in the example;

[0036] Figure 3 This is a schematic diagram of the overall structure of the reinforcement learning system in the embodiment;

[0037] Figure 4 This is a schematic diagram of the knowledge graph construction process in the example.

[0038] Figure 5 This is a schematic diagram illustrating the knowledge preprocessing effect in the embodiment;

[0039] Figure 6 This is a schematic diagram of entity vectors in the embodiment;

[0040] Figure 7 This is a schematic diagram of the relation vector in the embodiment;

[0041] Figure 8 This is a schematic diagram of the network model pre-training process in the embodiment;

[0042] Figure 9 This is a schematic diagram of the network model retraining results in the embodiment;

[0043] Figure 10 This is a schematic diagram of the network topology simulation structure in the embodiment;

[0044] Figure 11 This example illustrates the attacker and the target asset in the embodiment.

[0045] Figure 12 This is a schematic diagram of the attack path prediction results in the embodiment;

[0046] Figure 13 This is a schematic diagram illustrating the attack path in the example. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described in detail below with reference to the accompanying drawings and technical solutions.

[0048] Network attack path prediction is crucial for improving network system security. With the frequent occurrence of network attacks in recent years, the network security situation is becoming increasingly severe. Therefore, strengthening research on network attack path prediction and developing practical and effective network security defense measures based on the prediction results is particularly urgent. Existing attack path prediction technologies often suffer from problems such as redundant paths, reliance on expert experience, poor prediction accuracy, and high prediction costs. Therefore, this invention provides an embodiment, see [link to embodiment]. Figure 1 As shown, a network attack path discovery method based on knowledge graphs is provided, which specifically includes:

[0049] S101. Extract entities, attributes, and relationships between entities from network attack and defense data. Construct a network attack and defense knowledge graph based on the extracted entities, attributes, and relationships between entities. The entities include attacking and defending parties, attack and defense modes, vulnerabilities, and assets. The attributes include the names of the attacking and defending parties, their numbers, vulnerability numbers, and vulnerability scores. The relationships are the associations between entities.

[0050] A knowledge graph for network attack and defense is constructed based on the triple model. The triple model can be expressed as: <entity, attribute, relation>, which can represent entities in the knowledge graph, the attributes possessed by entities, and the relationships between entities. Concepts not listed in the model represent a general term for a certain type of entity. In this embodiment, the concepts in the constructed knowledge graph include attackers, attack patterns, vulnerabilities, assets, etc.; instances are specific entities of a certain type, such as a specific attacker in the concept of attackers, SQL injection attacks and denial-of-service attacks in the concept of attack patterns, CVE-2022-0001 and CVE-2022-0002 in the concept of vulnerabilities, and hosts and servers in the concept of assets; attributes are the collection of attribute values ​​in an instance. In the constructed knowledge graph, attacker attributes include attacker ID and capability level; attack pattern attributes include attack pattern severity; vulnerability attributes include vulnerability score and the CWEid associated with the vulnerability; and asset attributes include asset ID, version number, partition, and type. Relationships are the relationships between instances. In the knowledge graph constructed in this study, the relationships include the usage relationship between attackers and attack patterns, the exploitation relationship between attack patterns and vulnerabilities, and the existence relationship between vulnerabilities and assets.

[0051] Specifically, attack pattern-related knowledge information can be extracted from the open-source attack pattern public standard platform based on the technical means used by the attacker; vulnerability-related knowledge information can be extracted from the open-source public vulnerability risk platform based on the weaknesses and defects existing in the system and / or equipment; asset-related knowledge information can be extracted using the open-source network device general enumeration platform; and attacker-related knowledge information can be extracted based on the attacker's historical attack behavior information, so as to store the extracted entity knowledge information using the graph database Neo4j.

[0052] Attack patterns, also known as attack methods, refer to the techniques and methods used by attackers. The data referenced in this embodiment comes from the Common Attack Patterns (CAPEC) platform. CAPEC provides information on the most common attacks against software, allowing users to understand how adversaries exploit weaknesses or vulnerabilities in networks, programs, and software to launch attacks. This helps developers and cybersecurity personnel understand attack behaviors and how to defend against them. For example, CAPEC-21 Exploitation of Trusted Identifiers includes the attack pattern's ID, description, attack severity, correlation with other attack patterns, attack scope, attack mechanism, attack flow, mitigation methods, and correspondence with the Common Weapon System (CWE).

[0053] A vulnerability refers to a weakness or defect that may exist in a system or device. The data referenced in this embodiment comes from the Public Vulnerability and Risk (CVE) platform. The task of the CVE platform is to identify, describe, and classify publicly disclosed security vulnerabilities and name them, thereby helping users to identify, discover, and then fix security vulnerabilities in software and systems more quickly and effectively. Taking CVE-2022-26317 as an example, each CVE entry includes the vulnerability's ID, description, CVSS severity and indicators, vulnerability enumeration, and CPEs affected by the CVE.

[0054] Assets refer to various devices on networks. The data referenced is mainly from Common Platform Enumeration (CPE), which uses information such as software, operating system, and hardware included in the CPE to represent assets. Each CPE entry includes the asset's name, version number, source, and product model.

[0055] Attacker-dimensional data primarily includes the attack patterns available to the attacker and the corresponding assets that the attacker can attack. These two relationships can be obtained by collecting and analyzing information on the attacker's historical attack behavior. Attacker-dimensional data can also be simulated and constructed depending on experimental conditions.

[0056] Neo4j is a graph database. Unlike relational databases, it stores data on the network, facilitating the handling of a range of problems that traditional relational databases struggle with. Neo4j graph databases use two data types: nodes and edges. Nodes represent entities, while edges represent relationships between entities. The acquired knowledge data is stored in the Neo4j graph database. Python is used to import the knowledge data into the Neo4j graph database step by step, establish relationships between entities, and finally adjust the color and size of entities and relationships for easier differentiation. In Neo4j, attackers, attack patterns, and vulnerabilities are represented as entities; attacker name, attacker ID, vulnerability ID, and vulnerability score are represented as attributes; and relationships between attackers, attack patterns, vulnerabilities, and assets are represented by edges. The correspondence between various entities after import is shown in Table 1 below.

[0057] Table 1

[0058] Entity Relationship Entity 1 Attacker Attack Asset 2 Attacker Exploit Vulnerability 3 Attacker Use Attack Pattern 4 Vulnerability Exist Asset 7 Asset CanPrecede Asset 8 Attack Pattern ChildOf Attack Pattern

[0059] S102. Use the TransE model to obtain the vector representations of entities and relations in the network attack and defense knowledge graph, and obtain the entity vectors and relation vectors in the network attack and defense.

[0060] Among them, the vector representation of entities and relations in the network attack and defense knowledge graph obtained by using the TransE model can be designed to include:

[0061] Data preprocessing is performed on the knowledge information in the network attack and defense knowledge graph. By assigning values ​​to entities and relations, associations are established between entities, and triples of entities, attributes, and relations are formed.

[0062] For each entity and relation, randomly generate entity vectors and relation vectors of arbitrary dimensions;

[0063] Randomly select a portion of the triplets and use them as the correct triplets. Generate incorrect triplets by replacing the entities in the correct triplets. Combine the correct triplets and incorrect triplets to form the training data.

[0064] Using training data and based on a loss function, the randomly generated entity vectors and relation vectors are updated using gradient descent to obtain the optimal entity vectors and relation vectors. The loss function is used to optimize the TransE model by minimizing the distance between correct triplet vectors and maximizing the distance between incorrect triplet vectors.

[0065] The knowledge used in the imported knowledge graph undergoes data preprocessing, transforming it into a data format that is easy to vectorize. Since the data used in constructing the knowledge graph is in CSV format with missing relationships, it needs to be converted into entity files, relation files, and triplet files that meet the requirements for dimensionality reduction processing. This involves assigning ID values ​​to entities and relations, and establishing triplet formats between entities.

[0066] The steps of the TransE algorithm are shown in Algorithm 1:

[0067]

[0068] In Algorithm 1, steps 1-3 initialize the vectors of entities and relations. Step 6 randomly samples a minibatch for training the embedding. Steps 7-11 complete the construction of the training dataset. A certain number of correct triples (h, r, t) are randomly selected. The h in the correct triples is replaced with h′ from other triples, and the t in the correct triples is replaced with t′ from other triples, resulting in incorrect triples (h′, r, t) or (h, r, t′). These incorrect triples are then merged to complete the construction of the batch training dataset. Steps 5 and 12 represent vector updates, updating the vectors of the correct triples (h, r, t) and incorrect triples (h′, r, t) or (h, r, t′) in the batch training dataset constructed in the previous step.

[0069] Compared to the entity relation table provided during the knowledge graph construction process, it is also necessary to define and vectorize the reverse relations of exist, exploit, canprecede, and childof. The types of correct triples (h,r,t) are shown in Table 2 below:

[0070] Table 2

[0071]

[0072]

[0073] Model training process, such as Figure 2 As shown, the first step is to import the training data, obtaining the number of entities and their list, the total number of relations and their list, and the total number of triples and their list. Then, the `TransE` class is instantiated, and the entity list, relation list, and triple list are added to it, with the spacing γ set to 1. The output vector is then set to a dimension of 20. The second step is to call the `initialize()` method of `transE` for initialization. For each entity, a vector of arbitrary dimension is randomly generated, then normalized to ensure it is a unit vector. A list of 20 elements is then created to represent the entity's vector. Simultaneously, the entity and its corresponding randomly generated vector are added to a newly created dictionary, `entityVectorList`. The same operation is performed for relations. The third step is training. A certain number of triples are randomly selected, and these triples are iterated through. The `getCorruptedTriplet()` method is called to obtain the broken triples of a given triple. The broken triples are combined with the correct triples to form a new tuple, which is then deduplicated and added to a list. The fourth step is to update the vector, calculate the loss function, and update the previously randomly generated vector using gradient descent to obtain an optimal vector.

[0074] S103. Set up a reinforcement learning agent, take the entity vectors and relation vectors in network attack and defense as the interaction environment, and represent the attack path prediction problem as a problem of exploring sequence decision using a reinforcement learning agent. Reason and predict the attack path through the interaction process between the reinforcement learning agent and the interaction environment.

[0075] Specifically, reasoning and predicting attack paths through the interaction process between the reinforcement learning agent and the interactive environment can include:

[0076] A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions, and interactive environments. The reward is used to represent the reward fed back to the agent by the interactive environment after the agent performs an action. The state is used to represent the change in the interactive environment after the agent performs an action. The action is used to find the action in the path that links entities with given relationships with the richest information. The action space is set as all relationships in the network attack and defense knowledge graph.

[0077] By reinforcing the interaction between the agent and the interactive environment, and through feedback from the interactive environment, the agent continuously learns through trial and error to adjust and optimize its state, thereby obtaining the optimal path between entity pairs with a specified relationship.

[0078] like Figure 3 As shown, the reinforcement learning system consists of two parts. The first part, the external environment, is represented as a Markov decision process. A tuple is defined.<S,A,P,R> Let S represent a Markov decision process, where S is a finite set of states, and A = {a1, a2, ..., a...}. n} is a finite set of actions, P(S) t+1 =s′|S t =s,A t =a) is the transition probability matrix, and R(s,a) is the probability matrix for each (s,a) pair. The second part of the system, the agent, uses policy π. θ The expression (s, a) = p(a|s; θ) represents the state vector that maps to a stochastic policy.

[0079] Figure 3 The left half of the diagram represents the reinforcement learning environment, showing the path and related nodes for querying the initial node Attacker202 and the target node admin-font-editor. The right half represents the policy-based agent, which uses a fully connected neural network to parameterize the policy function. The neural network consists of two hidden layers, each followed by a rectified nonlinear layer, and the output layer is normalized using softmax. The lines in the middle represent the agent taking actions in the environment, and the environment providing the agent with a new state and reward. The components of the reinforcement learning environment are actions, states, and rewards. The environment is represented by vectors of entities and relations.

[0080] Action: Given a pair of entities (e) with relation r. s ,e t The goal is for the agent to find the most information-rich path linking these entity pairs. From the source entity e... s Initially, the agent uses a strategy network to select the most promising relationship, extending its path at each step until it reaches the target entity e. tTo maintain consistency in the output dimension of the policy network, the action space is defined as all relations in the knowledge graph.

[0081] State: Directly modeling all discrete atomic information in a knowledge graph is impossible. To capture the semantic information of these symbols, a translation-based embedding method—TransE—is used to process the entity and relation data used in the above steps to construct the knowledge graph, creating a continuous vector space. In our framework, the state captures the agent's position in this continuous space; after performing an action, the agent transitions from one entity to the next. The state vector for step t is as follows: s t =(e t, e target -e t) Among them, e t The vector representing the current entity node, e target This represents a vector representing the target entity. In the initial state, e t =e source .

[0082] Rewards: The quality of the path predicted by the agent is affected by the reward. To make path prediction more accurate and efficient, a reward function can be used to set the reward scoring criteria.

[0083] Specifically, the reward given to the agent by the interactive environment after the agent performs an action may include:

[0084] If the agent reaches the target node after performing a series of actions, the interactive environment will provide a positive offline reward to the agent; otherwise, the interactive environment will provide a negative offline reward to the agent.

[0085] The paths between entity pairs are set as relation sequences, and the interactive environment provides path efficiency rewards to the agent based on the reciprocal of the relation sequence length.

[0086] Cosine similarity is used to calculate the similarity of paths between entity pairs. The interactive environment uses this similarity to provide diversity rewards to the agent, thereby encouraging the agent to find different paths between entity pairs.

[0087] Specifically, regarding global accuracy: given the environment, the number of actions an agent can take is very large, and the number of incorrect sequence decisions may exceed the number of correct decisions. Furthermore, the number of these incorrect sequence decisions increases exponentially with the length of the path. To address this issue, the first reward function added to the reinforcement learning model can be a global reward: that is, if the agent reaches the goal after performing a series of actions, an offline positive reward of +1 is given. The calculation process can be represented as follows:

[0088]

[0089] Regarding path efficiency: In relation prediction tasks, shorter paths are often more accurate and reliable than long paths in reasoning. Furthermore, encouraging shorter paths can limit the number of actions a reinforcement learning agent performs, thereby improving inference efficiency. The path efficiency reward is defined as follows: a path p is defined as a relation sequence r1→r2→...→r n The specific calculation process can be expressed as follows:

[0090]

[0091] To address path diversity: the agent is trained to find paths for each relation using positive samples. These training samples (e source e target In vector space, these paths have similar state representations. The paths found by the agent are often similar and therefore contain a lot of redundant information. To enable the agent to find different paths, a diversity reward function can be defined using cosine similarity: where p represents the relation chain r1→r2→...→r n Path embedding. The specific calculation process can be represented as follows:

[0092]

[0093] Specifically, by using feedback from the interactive environment to enable the agent to continuously learn through trial and error, it can be designed to include:

[0094] A bidirectional random breadth-first search algorithm is used to pre-train a supervised policy network to initially obtain multiple paths between different head and tail entities.

[0095] The pre-trained supervised policy network is retrained using a reward function, so that path prediction can be performed on entity pairs with specified relationships by calling the retrained supervised policy network.

[0096] In practice, a major challenge of knowledge graph reasoning is the potentially enormous size of the relation set. For a typical knowledge graph, reinforcement learning agents often face hundreds (or thousands) of possible operations, meaning the output layer of the policy network has a high dimensionality. Due to the complexity of the relation graph and the large action space, directly training the reinforcement learning model through trial and error will result in very poor convergence. After a long training period, the agent may fail to find any valuable paths. To address this issue, a supervised policy network can be trained first. Through pre-training, the model can eventually find multiple paths between a given two nodes.

[0097] For each relation, a supervised policy is learned using a set containing all positive samples. For each positive sample (e souree e targetA bilateral breadth-first search method is used to find the correct path between entities. For the relation sequence r1→r2→...→r n For path p, use the Monte Carlo policy gradient method to update the parameters θ to maximize the expected cumulative reward.

[0098]

[0099] Where J(θ) is the expected total reward for each iteration. For supervised learning, a reward of +1 is given for each successful iteration. The approximate gradient used to update the policy network by inserting the path found by BFS is shown below:

[0100]

[0101] Through supervised learning, the agent can initially find some desired paths.

[0102] Unlike depth-first search (DFS), breadth-first search (BFS) aims to explore all possible paths, regardless of the number of branches, and tends to find long paths. BFS, on the other hand, tends to search for short paths. Its basic idea is to start from a node, explore all the nodes that the node can connect to, record each branch, select a branch to enter the next node, record all the branches of the current node again, and then return to select a branch to enter. This process is repeated until the last node is reached.

[0103] An entity node can be linked to a large number of neighbors with the same relationships. If we simply use a breadth-first search algorithm to search from the beginning entity to the end entity, the number of intermediate entities will increase exponentially. Therefore, in this embodiment, a bidirectional path search algorithm is applied, that is, a random mechanism is added to the breadth-first search. Suppose we want to search for a path between two nodes A and B, we randomly select an intermediate node C, and perform the breadth-first search algorithm twice between nodes A and C, and C and B. After using this method, the number of intermediate entities can be greatly reduced. The bidirectional path constraint search algorithm is as follows:

[0104]

[0105] To obtain prediction paths controlled by reward functions, the previously pre-trained supervised policy network is retrained using these reward functions. Starting from the source node, the agent selects a relation to extend its inference path based on a stochastic policy function π(a|s) with a probability distribution of all relations. This relation link may lead to a new entity or nothing at all; incorrect steps will result in a negative reward for the agent. To improve training efficiency, a maximum length is imposed on each process. If the agent does not reach the endpoint within the maximum limit of steps, the event ends. After each event, the policy network is updated using the following gradients:

[0106]

[0107] The algorithm for retraining the model using the reward function is shown below:

[0108]

[0109] Furthermore, based on the above method, this embodiment of the invention also provides a network attack path discovery system based on knowledge graphs, comprising: a knowledge extraction module, a vector representation module, and a path prediction module, wherein...

[0110] The knowledge extraction module is used to extract entities, attributes, and relationships between entities from network attack and defense data. Based on the extracted entities, attributes, and relationships between entities, a network attack and defense knowledge graph is constructed. The entities include attacking and defending parties, attack and defense modes, vulnerabilities, and assets. The attributes include the names of the attacking and defending parties, their numbers, vulnerability numbers, and vulnerability scores. The relationships are the associations between entities.

[0111] The vector representation module is used to obtain vector representations of entities and relations in the network attack and defense knowledge graph using the TransE model, and to obtain entity vectors and relation vectors in network attack and defense.

[0112] The path prediction module is used to set up a reinforcement learning agent, using entity vectors and relation vectors in network attack and defense as the interaction environment. It represents the attack path prediction problem as a problem of exploring sequence decision using a reinforcement learning agent, and infers and predicts the attack path through the interaction process between the reinforcement learning agent and the interaction environment.

[0113] To verify the effectiveness of this solution, the following explanation is based on experimental data:

[0114] Based on the acquired and organized network attack and defense data, a knowledge graph is constructed by importing it into the Neo4j graph database. The knowledge graph construction process is as follows: Figure 4 As shown. After running the code to import the data into Neo4j, you can see the constructed knowledge graph after opening Neo4j.

[0115] Subsequently, data preprocessing is performed, and the constructed TransE model is used to obtain vector representations of entities and relations. Since a bidirectional path search method is used when constructing the reinforcement learning model, a reverse relation must be defined simultaneously when defining relations between entities during data preprocessing. Relationships are also added between entities in the entity file and relations in the relation file, resulting in triples. The preprocessed result is as follows: Figure 5 As shown in the image. After the previous data preprocessing step, the next step is vectorization, representing entities and relations as low-dimensional vectors (20 dimensions), resulting in entity vector files and relation vector files. The data obtained after vectorization is shown in the image. Figure 6 and 7 As shown.

[0116] The model is pre-trained using a bidirectional random breadth-first search algorithm, and the training process is as follows: Figure 8 As shown, the bidirectional breadth-first search algorithm can initially find multiple paths between different head and tail entities.

[0117] Based on the pre-training, the pre-trained model is retrained using the defined reward function, and the training result is as follows. Figure 9 As shown, the training results display information such as whether the wheel path search was successful, the specific path found, path length, and search time. Once the model is trained, it can be directly used for prediction.

[0118] Based on the above dataset, an experimental scenario was constructed to simulate the network topology of a department within a company. The topology structure is as follows: Figure 10 As shown, external network attackers can access this network via the Internet. The intruder is separated from the external network by a firewall. The network consists of 3 hosts and 2 servers. The hosts can communicate with each other, and the servers can communicate with each other. The firewall does not impose any rules or restrictions on this network.

[0119] Based on previous attack information records, the attack methods that attackers can use and the vulnerabilities exploited by these attack methods are obtained, as shown in Table 3. The CPE information obtained in the previous steps is used to represent the hosts in the topology diagram. At the same time, vulnerability scans are performed on each host and server to obtain their existing vulnerability information, as shown in Table 4.

[0120] Table 3

[0121]

[0122] Table 4

[0123]

[0124] Assuming the target is host 3, the network topology diagram shows that there are several reachable paths from the attacker to the target host.

[0125] Path 1: Attacker -> Host 1 -> Host 2 -> Host 3 -> Host 4

[0126] Path 2: Attacker -> Host 2 -> Host 3 -> Host 4

[0127] Path 3: Attacker -> Host 3 -> Host 4

[0128] Subsequently, based on the attack relationship table and vulnerability information table, it can be determined that Path2 and Path3 can be successfully attacked. The attack methods and vulnerability exploits used during the attack process are as follows:

[0129] Path 2: Host 2 (TCP_Connect_Scan, CVE-2020-0256) —> Host 3 (Eavesdropping, CVE-2020-0340) —> Host 4 (Malicious_Automated_Software_Update_via_Spoofing, CVE-2020-0001)

[0130] Path 3: Host 3 (Eavesdropping, CVE-2020-0209) -> Host 4 (Malicious_Automated_Software_Update_via_Spoofing, CVE-2020-0001)

[0131] Next, run the program to verify the model. Manually input the attacker and target assets, such as... Figure 11 As shown. After inputting the data, the system will provide a predicted attack path based on the model, such as... Figure 12 As shown, this path is Path3 obtained after the above analysis. This path is reachable, shortest, and uses the fewest attack methods and exploits. The specific path is then displayed in Neo4j. Figure 13 As shown.

[0132] The experimental data above shows that the proposed solution can successfully predict a reachable path, exhibiting both high accuracy and consideration for path efficiency (preferring to find shorter paths). Compared to the results obtained through simple graph traversal algorithms, the path predicted by this model takes less time and has no redundant paths. After prediction, the system can visually display the attack paths that attackers might take. Defenders can then understand the attack methods employed and the vulnerabilities exploited on the hosts along the path, allowing them to implement appropriate defensive measures based on the attack methods and vulnerabilities targeted, thus strengthening the protection of critical asset nodes.

[0133] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention.

[0134] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0135] The units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations are not considered to be beyond the scope of this invention.

[0136] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This invention is not limited to any particular combination of hardware and software.

[0137] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for discovering network attack paths based on knowledge graphs, characterized in that, Include: Entities, attributes, and relationships between entities are extracted from network attack and defense data. A network attack and defense knowledge graph is constructed based on the extracted entities, attributes, and relationships between entities. The entities include attacking and defending parties, attack and defense modes, vulnerabilities, and assets. The attributes include the names of the attacking and defending parties, their numbers, vulnerability numbers, and vulnerability scores. The relationships are the associations between entities. The TransE model is used to obtain vector representations of entities and relations in the network attack and defense knowledge graph, resulting in entity vectors and relation vectors in network attack and defense. Set up a reinforcement learning agent, take the entity vectors and relation vectors in network attack and defense as the interaction environment, and represent the attack path prediction problem as a problem of exploring sequence decision using the reinforcement learning agent. Reason and predict the attack path through the interaction process between the reinforcement learning agent and the interaction environment. The method of reasoning and predicting attack paths through the interaction process between the reinforcement learning agent and the interactive environment includes: A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions, and interactive environments. The reward is used to represent the reward fed back to the agent by the interactive environment after the agent performs an action. The state is used to represent the change in the interactive environment after the agent performs an action. The action is used to find the action in the path that links entities with given relationships with the richest information. The action space is set as all relationships in the network attack and defense knowledge graph. By reinforcing the interaction between the agent and the interactive environment, and through the feedback from the interactive environment, the agent can continuously learn through trial and error to adjust and optimize its state, and obtain the optimal path between entity pairs with a specified relationship. The reward that the interactive environment provides to the agent after the agent performs an action includes: If an agent reaches the target node after performing a series of actions, the interactive environment will provide a positive offline reward to the agent; otherwise, the interactive environment will provide a negative offline reward to the agent. The path between entity pairs is set as a relation sequence, and the interactive environment will provide a path efficiency reward to the agent based on the reciprocal of the length of the relation sequence. The similarity of paths between entity pairs is calculated using cosine similarity. The interactive environment uses this similarity to provide diversity rewards to the agent, thereby encouraging the agent to find different paths between entity pairs. The method of enabling the agent to continuously learn through trial and error via feedback from the interactive environment includes: A bidirectional random breadth-first search algorithm is used to pre-train a supervised policy network to initially obtain multiple paths between different head and tail entities. The pre-trained supervised policy network is retrained using a reward function, so that path prediction can be performed on entity pairs with specified relationships by calling the retrained supervised policy network. The pre-training of the supervised policy network using a bidirectional random breadth-first search algorithm includes: A random mechanism is added to the breadth-first search algorithm to randomly select intermediate nodes between the head entity and the tail entity. Breadth-first search is then performed sequentially between the head entity and intermediate nodes, and between intermediate nodes and the tail entity. The search results are then concatenated to initially obtain multiple paths between the head entity and the tail entity.

2. The network attack path discovery method based on knowledge graphs according to claim 1, characterized in that, Extracting entities, attributes, and relationships between entities from network attack and defense data, including: Based on the technical means employed by attackers, extract knowledge information related to attack patterns from open-source attack pattern public standard platforms; based on the weaknesses and defects existing in systems and / or devices, extract vulnerability-related knowledge information from open-source public vulnerability risk platforms. The system uses an open-source network device general enumeration platform to extract asset-related knowledge information; it also extracts attacker-related knowledge information based on the attacker's historical attack behavior information, and uses the graph database Neo4j to store the extracted entity knowledge information.

3. The network attack path discovery method based on knowledge graphs according to claim 1, characterized in that, The TransE model is used to obtain vector representations of entities and relations in a network attack and defense knowledge graph, including: Data preprocessing is performed on the knowledge information in the network attack and defense knowledge graph. By assigning values ​​to entities and relations, associations are established between entities, and triples of entities, attributes, and relations are formed. For each entity and relation, randomly generate entity vectors and relation vectors of arbitrary dimensions; Randomly select a portion of the triplets and use them as the correct triplets. Generate incorrect triplets by replacing the entities in the correct triplets. Combine the correct triplets and incorrect triplets to form the training data. Using training data and based on a loss function, the randomly generated entity vectors and relation vectors are updated using gradient descent to obtain the optimal entity vectors and relation vectors. The loss function is used to optimize the TransE model by minimizing the distance between correct triplet vectors and maximizing the distance between incorrect triplet vectors.

4. A network attack path discovery system based on knowledge graphs, characterized in that, It includes: a knowledge extraction module, a vector representation module, and a path prediction module, among which, The knowledge extraction module is used to extract entities, attributes, and relationships between entities from network attack and defense data. Based on the extracted entities, attributes, and relationships between entities, a network attack and defense knowledge graph is constructed. The entities include attacking and defending parties, attack and defense modes, vulnerabilities, and assets. The attributes include the names of the attacking and defending parties, their numbers, vulnerability numbers, and vulnerability scores. The relationships are the associations between entities. The vector representation module is used to obtain vector representations of entities and relations in the network attack and defense knowledge graph using the TransE model, and to obtain entity vectors and relation vectors in network attack and defense. The path prediction module is used to set up a reinforcement learning agent, take entity vectors and relation vectors in network attack and defense as the interaction environment, and represent the attack path prediction problem as a problem of exploring sequence decision using a reinforcement learning agent. The attack path is inferred and predicted through the interaction process between the reinforcement learning agent and the interaction environment. The method of reasoning and predicting attack paths through the interaction process between the reinforcement learning agent and the interactive environment includes: A reinforcement learning system is constructed based on reinforcement learning agents, states, rewards, actions, and interactive environments. The reward is used to represent the reward fed back to the agent by the interactive environment after the agent performs an action. The state is used to represent the change in the interactive environment after the agent performs an action. The action is used to find the action in the path that links entities with given relationships with the richest information. The action space is set as all relationships in the network attack and defense knowledge graph. By reinforcing the interaction between the agent and the interactive environment, and through the feedback from the interactive environment, the agent can continuously learn through trial and error to adjust and optimize its state, and obtain the optimal path between entity pairs with a specified relationship. The reward that the interactive environment provides to the agent after the agent performs an action includes: If an agent reaches the target node after performing a series of actions, the interactive environment will provide a positive offline reward to the agent; otherwise, the interactive environment will provide a negative offline reward to the agent. The path between entity pairs is set as a relation sequence, and the interactive environment will provide a path efficiency reward to the agent based on the reciprocal of the length of the relation sequence. The similarity of paths between entity pairs is calculated using cosine similarity. The interactive environment uses this similarity to provide diversity rewards to the agent, thereby encouraging the agent to find different paths between entity pairs. The method of enabling the agent to continuously learn through trial and error via feedback from the interactive environment includes: A bidirectional random breadth-first search algorithm is used to pre-train a supervised policy network to initially obtain multiple paths between different head and tail entities. The pre-trained supervised policy network is retrained using a reward function, so that path prediction can be performed on entity pairs with specified relationships by calling the retrained supervised policy network. The pre-training of the supervised policy network using a bidirectional random breadth-first search algorithm includes: A random mechanism is added to the breadth-first search algorithm to randomly select intermediate nodes between the head entity and the tail entity. Breadth-first search is then performed sequentially between the head entity and intermediate nodes, and between intermediate nodes and the tail entity. The search results are then concatenated to initially obtain multiple paths between the head entity and the tail entity.

5. An electronic device, characterized in that, include: At least one processor, and a memory coupled to said at least one processor; The memory stores a computer program that can be executed by the at least one processor to implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, enables the implementation of the method as described in any one of claims 1 to 3.

Images