A threat hunting method based on graph kernel clustering
By constructing a behavior dependency graph and using a graph kernel clustering algorithm, the problem of insufficient detection of unknown attacks in existing threat hunting methods is solved, and efficient threat hunting and abnormal behavior identification are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2022-03-25
- Publication Date
- 2026-07-21
AI Technical Summary
Existing threat hunting methods rely on threat intelligence and expert knowledge, making it difficult to effectively detect unknown attacks and camouflage behaviors. Furthermore, existing graph matching methods lack sufficient accuracy in detecting attack scenarios.
By constructing a behavior dependency graph and employing a graph kernel clustering algorithm, threat hunting is performed based on audit logs. The graph kernel clustering algorithm separates normal behavior from abnormal behavior, and unknown attacks are discovered through threat quantification assessment.
It achieves effective detection of unknown attacks, reduces reliance on expert knowledge, improves detection accuracy and efficiency, and reduces false alarm rate.
Smart Images
Figure CN114662096B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of log analysis, and in particular relates to threat hunting based on audit logs. Background Technology
[0002] Advanced persistent threats (APs) are characterized by their persistence and stealth. These threats can evade threat detection software (TDS) and remain dormant, potentially causing attacks that have already occurred but have not yet been detected within enterprise information systems. To better prevent and respond to such attacks, endpoint detection and response tools (EDRs) are widely used in enterprise security. However, these tools rely on matching low-level indicators of intrusions (IOCs), such as hashes of malware samples, suspicious IPs, or domain names. This leads to alert fatigue and fails to reveal the complete attack scenario. To overcome this challenge, recent research has explored causal analysis of audit logs to identify cyber threats. In fact, the causal relationships and contextual information in audit logs implicitly contain advanced attacker behaviors and objectives that cannot be hidden.
[0003] Threat hunting, the proactive search for potential attacks, has become a crucial component of APT attack defense. Existing work extracts attack behaviors from threat intelligence and designs matching algorithms to search for these known attacks from audit logs. To achieve this, graph-based matching methods construct audit logs as proof graphs containing rich contextual information and model threat hunting as a graph matching problem. Furthermore, query-based methods have designed the Threat Behavior Query Language (TBQL) to query audit logs stored in databases. However, these methods heavily rely on threat intelligence, which leads to several limitations. On one hand, attack activity may be missed when there is a discrepancy between threat intelligence and the facts. On the other hand, descriptions of the same APT attack event may come from different reports, and the information in these reports may differ or contradict each other. Many APT attacks have not yet been disclosed by threat intelligence, and the techniques and strategies of APT attacks are constantly evolving, making knowledge-based methods inadequate in dealing with new and unknown attacks; therefore, existing methods cannot detect these unknown attacks.
[0004] Furthermore, some attack hunting solutions, such as matching rule knowledge bases or tagging strategies, require manual involvement from domain experts. The completeness and accuracy of expert knowledge affect the analysis results, and a anticipated bottleneck is the need for manual involvement from domain experts to specify these rules, which require continuous updates. To address these issues, some work has constructed Event Frequency Databases to replace rule knowledge bases, considering that attack-related audit events are infrequent. However, to avoid detection, attackers may disguise themselves as normal behavior or use seemingly legitimate processes, such as svchost.exe, which can affect the accuracy of threat scoring methods based on matching or single event frequency.
[0005] In recent years, graph kernels have been widely applied in various fields, including chemical information, social networks, and cybersecurity, where data is often graph-based or can be converted into graph representations. A graph kernel is a kernel function that computes the inner product of a graph; intuitively, it can be understood as a function that measures the similarity between graphs. They allow kernel learning algorithms (such as support vector machines) to work directly on graphs without needing to extract features and convert them into fixed-length real-valued feature vectors. Using graph kernel methods for classification or clustering tasks has achieved excellent results in the field of graph data analysis. Summary of the Invention
[0006] This invention proposes a threat hunting method based on graph kernel clustering. By constructing audit logs into a behavior dependency graph and designing a graph kernel clustering method, it can separate normal behavior from abnormal behavior and perform threat quantification assessment on the behavior dependency graph to discover unknown attacks.
[0007] This invention provides a threat hunting method based on graph kernel clustering, comprising the following steps:
[0008] 1) Based on the entity type, extract the relevant attribute information of the entity and the relationship information between the entities from the audit log. Based on this information, construct a dependency graph, assign labels to nodes and edges, partition long-running processes, and generate a behavioral dependency graph. The behavioral dependency graph is a labeled directed graph, where nodes represent system-level entities and directed edges represent the operational relationships between entities.
[0009] 2) Based on the behavior dependency graph, a graph kernel clustering algorithm is designed. The graph kernel consists of two parts. The first part calculates the kernel value between nodes by iteratively calculating the kernel value between the neighboring nodes of a node. The second part constructs a mapping set between nodes and calculates the kernel value between the graphs based on the kernel value between the nodes, thus obtaining the graph kernel matrix. The graph kernel matrix can be regarded as a similarity matrix.
[0010] 3) Use unsupervised clustering algorithms to analyze the graph kernel matrix, cluster similar behaviors into one category, thereby separating normal behavior from abnormal behavior;
[0011] 4) Determine abnormal behavior based on the frequency of occurrence, and quantify the threat of the abnormal behavior graph from three aspects: suspicious IP, user permissions, and sensitive information. Sort the threat scores of the abnormal behavior dependency graph. When the threat score exceeds the threshold, the abnormal behavior is determined to be a threatening behavior and an alarm is issued to achieve threat hunting.
[0012] Furthermore, the behavior-dependent graph builder and graph kernel clustering algorithm include:
[0013] a) The dependency graph builder should be able to accurately extract entity information and relationships between entities from the audit log. Entity types include processes, files, IPs, and users. Entity information includes entity names and relationships between entities, such as process names and process IDs, file paths and file names, and process read and write operations on files, and assign node labels to nodes based on entity information.
[0014] b) Partition long-running processes based on the density of dependencies occurring on the timeline during their lifecycle. The density is calculated using the following formula:
[0015]
[0016] Among them, Time start It is the time when the first dependency occurs. end It is the time when the last dependency ends, T i This represents the time interval between two dependencies. It iterates through all dependencies and considers consecutive dependencies with a density higher than the average density as belonging to the same partition, thereby splitting the dependency graph and generating a behavioral dependency graph.
[0017] c) In the graph kernel clustering algorithm, the graph kernel plays the role of embedding the graph into a high-dimensional space and calculating similarity. It should be able to represent the rich label information and context information of the behavior dependency graph well. In order to facilitate calculation, the string label set is mapped to the numeric label set, where each string element corresponds to a unique value. Based on the above mapping, the string labels in the behavior dependency graph are replaced with numeric labels.
[0018] d) Based on the neighbor nodes, each node is assigned a new label:
[0019]
[0020] u represents node v i The neighboring nodes, For node v i The set of neighboring nodes, l e (v i ,u) represents the label of a directed edge, l(v) iM(v) represents the numerical label of the node. i ) is node v i The new set of labels, and the first-order kernel value between nodes can be calculated using the following formula:
[0021]
[0022] v1 and v2 are nodes in the two graphs, respectively;
[0023] e) For different types of directed edges, a kernel value calculation method is introduced to improve accuracy. The kernel value between nodes is calculated iteratively based on the similarity of the node label sets, as shown in the following formula:
[0024]
[0025] in, Let α and β represent the kernel values of the edges, where α and β are non-negative constants. Represents the k-th order kernel value between nodes;
[0026] f) Calculate the kernel value between nodes based on the kernel value between nodes, using the following formula:
[0027]
[0028] Where B(V1,V2) represents the mapping between nodes, k G The kernel values between the (G1G2) graphs ultimately yield the positive definite kernel value matrix K. N×N K i,j It is G i and G j The kernel values between them, the kernel value matrix can be considered as a similarity matrix;
[0029] g) Use unsupervised clustering algorithms to analyze the graph kernel matrix. This does not require prior declaration of the number of classes and has good robustness to outliers, enabling more accurate separation of abnormal behavior from normal behavior.
[0030] Furthermore, step 1) considers four entity types, including processes, files, IPs, and users, and defines the relationships between different types of entities, such as processes reading and writing files, thereby filtering out unimportant information, reducing the size of the generated graph, and improving analysis efficiency.
[0031] Furthermore, in step 1), the attribute information of entities is mapped to labels and assigned to nodes using rules. For example, report.doc and data.xls under the path D:\download\ are mapped to office file. The mapping process filters out information that affects similarity calculation and selects an appropriate granularity to represent the attribute information of nodes.
[0032] Furthermore, the graph kernel algorithm in step 2) constructs a mapping set when calculating the kernel value between graphs, which establishes a mapping relationship between similar nodes in two graphs, thereby improving the computational efficiency and accuracy of the graph kernel;
[0033] Furthermore, in step 4), in order to quantify the threat of the behavior dependency graph, three databases were constructed, including an abnormal IP database, a user permission database, and a sensitive information database.
[0034] The method of this invention can better detect network threats lurking in enterprises and can discover unknown threats, offering the following advantages compared to existing technologies:
[0035] 1. This invention proposes a density-based partitioning method, which alleviates the dependency explosion problem caused by time-consuming processes to a certain extent;
[0036] 2. This invention proposes a graph kernel clustering method for threat hunting, which separates threatening behaviors from normal behaviors by comparing the differences between them, greatly reducing the reliance on expert knowledge. Moreover, compared with threat hunting methods based on threat intelligence, this invention can discover undisclosed unknown attacks.
[0037] 3. This invention uses graph kernels to calculate the similarity between graphs. Taking advantage of the rich node attributes and contextual information of behavior dependency graphs, a graph kernel function for behavior dependency graphs is designed based on the message passing concept. This function can calculate the similarity between different behavior dependency graphs and generate a similarity matrix. Attached Figure Description
[0038] Figure 1 This is a structural diagram of the method framework of the present invention. It is mainly divided into three modules: behavior dependency graph constructor, graph kernel clustering, and threat assessment.
[0039] Figure 2 This is a table of entity types and relationships, used to determine the attribute and relationship information of entities from the audit logs.
[0040] Figure 3 This is a density-based partitioning method where dependencies are represented as points on the time axis.
[0041] Figure 4 For two-dimensional visualization of reachability distances, isolated points represent threat behavior dependency graphs.
[0042] Figure 5 The recall, precision, and F1-score of the method are given at different thresholds.
[0043] Figure 6 This is a case study for evaluating the method. Detailed Implementation
[0044] To make the above-mentioned features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.
[0045] The threat hunting method designed in this invention is based on graph kernel clustering and is suitable for discovering threat behaviors from audit logs. This method constructs audit logs as behavior dependency graphs and calculates the similarity between graphs to ultimately achieve threat hunting. The specific implementation process is as follows: Figure 1 As shown, its main steps include:
[0046] Step 101: Extract entity and attribute information from the audit log, including entity type, entity name and relationship between entities.
[0047] Step 102: Construct a dependency graph based on the extracted entity information. The nodes in the dependency graph represent entities, and appropriate node labels are assigned to the nodes according to the attribute information of the entities. The directed edges in the dependency graph represent the relationships between entities.
[0048] Step 103: Partition long-running processes based on the density of events occurring within the process's lifecycle to segment the dependency graph and generate a behavioral dependency graph.
[0049] Step 201: Map the string tag set to the numeric tag set, where each string element corresponds to a unique value. Based on the above mapping, replace the string tags in the behavior dependency graph with numeric tags.
[0050] Step 202: Assign a new label to each node based on the neighboring nodes, and iteratively calculate the kernel value between nodes according to the similarity of the node label sets.
[0051] Step 203: Calculate the kernel values between nodes to obtain the positive definite kernel value matrix K. N×N K i,j It is G i and G j The kernel value matrix can be considered as a similarity matrix. Unsupervised clustering methods are used to analyze the kernel value matrix to separate normal behavior from malicious behavior.
[0052] Step 301: For each cluster, count the number of behavior dependency graphs. When the number is less than the threshold, the behavior dependency graph in the cluster is judged as abnormal behavior.
[0053] Step 302: Perform threat quantification on the obtained behavior dependency graph representing abnormal behavior. Traverse each abnormal behavior dependency graph and quantify it from three aspects: suspicious IP, user permissions, and sensitive information.
[0054]
[0055] This represents the threat value of an IP address and URL, calculated using a publicly available database of malicious IP addresses and domain access rankings. This represents the quantification of user permissions; the higher the permissions a user possesses, the higher the threat level. The quantification of sensitive information is generally achieved through manual annotation or sensitive information identification tools. α, β, and γ represent weights, which can be adjusted as needed.
[0056] Step 303: Sort the threat scores of the abnormal behavior dependency graph. When the threat score exceeds the threshold, the abnormal behavior is judged as a threat behavior and an alarm is issued to achieve threat hunting.
[0057] This invention uses a simulated dataset containing malicious attacks, the DAPRA CADETS dataset, and a dataset from a real network environment to conduct experimental analyses on the feasibility, accuracy, and false positive rate of the method. The experimental results show that the method can detect threat attacks in audit logs with a low false positive rate and can also detect unknown attacks that have not been disclosed.
[0058] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A threat hunting method based on graph kernel clustering, characterized in that, include: A. Constructing a behavioral dependency graph: Extract entities and relationships between entities from audit logs to construct a behavioral dependency graph, and partition long-running processes by utilizing the density of dependencies occurring on the time axis during the process lifecycle. The behavioral dependency graph is a labeled directed graph, where nodes represent system-level entities and directed edges represent operational relationships between entities. B. Similarity Calculation and Clustering: Based on the characteristics of behavior dependency graphs, a graph kernel method is designed to embed the behavior dependency graphs into a high-dimensional space. The similarity between graphs is calculated in the high-dimensional space to obtain the graph kernel matrix, which is the similarity matrix. Then, a clustering method is used to analyze the graph kernel matrix to separate normal behavior from abnormal behavior. C. Threat Assessment: By determining the number of graphs in a cluster, we identify which clusters contain anomalous behavior. We then quantify the threat assessment of the anomalous behavior graphs, calculate their threat values, and ultimately achieve threat hunting.
2. The threat hunting method based on graph kernel clustering according to claim 1, characterized in that, Step A further includes the following steps: A1. Extract entities and their attribute information from the audit logs. The types of entities are processes, files, IPs, and users. The attribute information includes the entity name and the relationship between entities, namely the process name and process ID, file name path and file name, and the process's read and write operations on the file. A2. Construct a dependency graph based on the entity information extracted in step A1. The nodes in the dependency graph represent entities, and node labels are assigned to the nodes according to the attribute information of the entities. The directed edges in the dependency graph represent the relationships between entities. A3. Partition long-running processes by utilizing the density of dependencies occurring on the timeline throughout their lifecycle. The density calculation formula is as follows: in, It is the time when the first dependency occurs. It is the time when the last dependency occurs. This represents the time interval between two dependencies. It iterates through all dependencies and considers consecutive dependencies with a density higher than the average density as belonging to the same partition, thereby splitting the dependency graph and generating a behavioral dependency graph.
3. The threat hunting method based on graph kernel clustering according to claim 1, characterized in that, Step B further includes the following steps: B1. Map the string label set to the numeric label set, where each string element corresponds to a unique value. Based on the above mapping, replace the string labels in the behavior dependency graph with numeric labels. B2. Node-based neighbor nodes assign new labels to each node: N Represents a node The neighboring nodes, N For nodes The set of neighboring nodes, Labels representing directed edges. Numeric labels representing nodes, For nodes A new set of tags; The first-order kernel value between nodes can be calculated using the following formula: in and These are nodes in the two graphs respectively; B3. Iteratively calculate the kernel value between nodes based on the similarity of their node label sets: in , representing the kernel value of the edge. and They are all non-negative constants. Represents the k-th order kernel value between nodes; B4. Calculate the kernel values between the nodes using the kernel values obtained in step B3: Represents the mapping between nodes. The calculated kernel values between the graphs ultimately yield a positive definite kernel value matrix. , yes and The kernel values between them, the kernel value matrix is the similarity matrix; B5. Use unsupervised clustering methods to analyze the kernel value matrix to separate normal behavior from attack behavior.
4. The threat hunting method based on graph kernel clustering according to claim 1, characterized in that, Step C further includes the following steps: C1. For each cluster, count the number of behavior dependency graphs within it. When the number is less than the threshold, the behavior dependency graphs in the cluster are judged as abnormal behavior. C2. Perform threat quantification on the behavior dependency graphs representing anomalous behavior obtained in step C1. Traverse each anomalous behavior dependency graph and quantify it from three aspects: suspicious IP, user permissions, and sensitive information. This represents the threat value of an IP address and URL, calculated using a publicly available database of malicious IP addresses and domain access rankings. This represents the quantification of user permissions; the higher the permissions a user possesses, the higher the threat level. The quantification of sensitive information is achieved through manual annotation or sensitive information identification tools. This indicates the weight, which can be adjusted as needed; C3. Sort the threat scores of the abnormal behavior dependency graph. When the threat score exceeds the threshold, the abnormal behavior is judged as a threat behavior and an alarm is issued to achieve threat hunting.