A method for automatically extracting high-level behaviors from audit logs

Abstract

The application relates to the field of data security and discloses a method for automatically extracting advanced behaviors from audit logs, which comprises the following steps: parsing audit logs to be analyzed into triples, constructing a provenance graph based on the triples of the audit logs; inferring the semantics of the context of each node in the provenance graph, enumerating subgraphs for representing corresponding behavior instances from the provenance graph, and performing semantic aggregation of behaviors based on the semantics and the behavior instances to obtain the semantics of the behavior instances; clustering the behavior instances into clusters according to preset rules, and extracting the most representative behavior signatures of the behavior instances from each cluster to obtain the semantics of the advanced behaviors. Finally, the application assists analysts in analyzing the audit logs in the form of behavior clustering, greatly reducing their workload, and does not need to rely on expert knowledge.

Description

Technical Field

[0001] This invention belongs to the field of data security, and further relates to a method for automatically extracting advanced behaviors from audit logs. Background Technology

[0002] Due to the continuous security incidents that have occurred to large enterprise systems in recent years, these attacks have become increasingly large-scale and complex, causing huge losses to enterprises. In order to better prevent and respond to such attacks, enterprises have begun to widely deploy endpoint monitoring solutions such as security information and event management tools to continuously record system-level activities as audit logs. Analyzing these recorded audit logs can help detect threats early or trace the source of attacks and determine the scope of damage when security incidents occur.

[0003] System audit logs enable analysts to gain deeper insights into cyberattacks by identifying the source of the data. Each audit event records operating system-level operations (i.e., system calls), such as process execution, file creation, and network connections. Specifically, an event can be defined as a triple (Subject, Relation, Object), where Subject is the process entity, Object is the system entity (i.e., process, file, or network socket), and Relation is the system call function. System entities are associated with a set of attributes used for identification, such as labels (e.g., PID and inode) and names (e.g., file path, process path, and IP address). Furthermore, each individual event (e.g., a process writing to a file) represents the flow of information between the subject and the object. To facilitate causal analysis of attacks, the research community uses source graphs to effectively trace the flow of information in audit logs. A source graph is a general representation of the causal relationships of system behavior; the direction of the edges in the source graph indicates the data transmission patterns between system entities. In investigating specific security incidents, analysts can use source maps to search for information related to cyberattacks. Specifically, analysts can trace the incident backward to determine its root cause, or trace it forward from the initial compromise point to explore the consequences of the same attack. By examining the ancestors and descendants of a security incident through backward and forward tracing, analysts can infer how the incident occurred and the high-level behaviors responsible for it—a series of actions performed by the user around their intended purpose.

[0004] When analysts perform audit log analysis, they need to identify not only malicious behaviors such as data breaches, but also benign behaviors such as program compilation and uploading. Although source plots provide an intuitive representation to visualize causal relationships and remove irrelevant events, analysts still spend too much time investigating relevant but benign events due to the normal activities that occur every day. Therefore, abstracting behaviors from audit events is an effective strategy for analysts to browse a large number of events and focus on specific information. In essence, behavior represents an abstraction of audit data, and analyzing audit logs at the behavioral level can effectively reduce the workload of analysis.

[0005] Existing technical solutions bridge this gap by matching audit events with knowledge storage of rules describing behavior defined by experts. The technical solution closest to the objective of this invention is a tag-based strategy that identifies advanced behaviors through tag propagation. Specifically, this method first constructs a source graph from audit logs, then uses a customizable strategy framework developed for tag initialization and propagation to tag each node in the graph. The tags provide important contextual information and are categorized into three types: benign and trustworthy, benign, and unknown. These tags are used to identify the subjects, objects, and events most likely to be involved in the attack, thus addressing the priority issue of the analyzed entities. After tagging the source graph, tag-based anomaly detection is implemented. After an alarm is detected, a tag-based bidirectional analysis is performed on the alarm trigger point, namely forward analysis and backward analysis, generating an attack subgraph with semantic meaning.

[0006] The bottleneck of the aforementioned tag-based technical solution is that it relies on experts' domain knowledge when defining and assigning tags, which may be hindered in practical applications.

[0007] Therefore, how to reduce reliance on expert knowledge and decrease the amount of manual work in log analysis is a problem that needs to be solved. Summary of the Invention

[0008] To address the problems raised in the background art, this invention develops a strategy for extracting representative behaviors from audit events for analysts to investigate. By automatically abstracting high-level behaviors and clustering semantically similar behaviors, analysts only need to analyze the behavioral signatures in each cluster, which can greatly reduce the number of events that need to be investigated, while eliminating the need to rely on expert knowledge.

[0009] This invention provides the following technical solution: a method for automatically extracting advanced behaviors from audit logs, comprising:

[0010] The audit logs to be analyzed are parsed into triples, and a source graph based on the triples of the audit logs is constructed.

[0011] The semantics of the context of each node in the source graph are inferred, and subgraphs representing corresponding behavior instances are enumerated from the source graph. Then, the semantics of the behavior are aggregated based on the semantics and the behavior instances to obtain the semantics of the behavior instances.

[0012] Behavior instances are clustered into clusters according to preset rules, and the behavior signatures of the most representative behavior instances are extracted from each cluster to obtain the semantics of high-level behaviors.

[0013] Preferably, the step of parsing the audit logs to be analyzed into triples and constructing a source graph based on the triples of the audit logs includes:

[0014] The audit logs to be analyzed are parsed into triples consisting of a header, relation, and tail, thereby constructing a source graph that encodes the semantic relation of the information flow from beginning to end in the form of the triples.

[0015] Each triple corresponds to an audit event in the audit log. The header and the tail are system entities of any type and are not simultaneously files or network sockets. The relationship is the corresponding system operation. The system entity type of the triple is consistent with the type of the audit event. The system entity includes processes, files, and network sockets.

[0016] Preferably, the semantic inference of the context of each node in the source graph includes:

[0017] The TransH embedding model is used to map the triples of audit events to a vector space;

[0018] The vectors of triples are concatenated to generate a single vector, which is used to represent the semantics of the audit event.

[0019] Preferably, the step of enumerating subgraphs from the source graph to represent corresponding behavior instances to summarize the behavior instances includes:

[0020] Perform an adaptive forward depth-first search on the source graph based on data objects;

[0021] During graph traversal, one-step edges that reach system entities are further merged.

[0022] The source graph is divided into subgraphs, each describing a behavior instance, which is a sequence of audit events that operate on related data and are interconnected through information flow.

[0023] Preferably, a constraint is enforced during graph traversal, which requires that the timestamp of each subsequent edge must monotonically increase from all preceding edges.

[0024] Preferably, the semantic aggregation of behaviors based on the semantics and the behavior instances to obtain the semantics of the behavior instances includes:

[0025] Use Inverse Document Frequency (IDF) to assign weights to each audit event in the behavior instance;

[0026] The semantics of the behavior instance are obtained by aggregating the semantics of the audit events that constitute the behavior instance;

[0027] The formula for IDF is as follows:

[0028]

[0029] Where e represents an audit event, S is the number of all sessions containing event e, and Se is the number of a specific session.

[0030] Preferably, the behavior instances are clustered into clusters according to preset rules, the preset rules including:

[0031] The semantic similarity of behavioral instances is calculated using the following formula:

[0032]

[0033] Among them, F m and F n Let e ​​be a vector representation of two behavioral instances. i and e j They are vectors F m and F n For each dimension, S(Fm,Fn) represents the cosine similarity.

[0034] Preferably, the behavior instances are clustered into clusters according to preset rules, wherein the preset rules further include:

[0035] The hierarchical clustering analysis algorithm HCA is used to iteratively calculate the semantic similarity of behavioral instances and combine the two closest clusters until the maximum similarity is below the merging threshold.

[0036] Preferably, the behavior signature of the most representative behavior instance is extracted from each cluster to obtain the semantics of the high-level behavior, including:

[0037] The representativeness of each instance in the cluster is quantified by calculating the average similarity between each behavior instance and other behavior instances, and the instance with the highest average similarity is selected as the behavior signature of the cluster.

[0038] The present invention has the following beneficial effects:

[0039] This invention employs an embedding model to transform audit events into vectors to represent the semantics of audit time. Then, behavioral instances are summarized based on the information flow, and the semantics of behavioral instances are derived by aggregating the semantics of audit events. Behavioral instances with similar semantics are clustered into clusters, and a representative behavior is found from each cluster. Analysts only need to analyze the semantics of the representative behavior in each cluster to obtain the semantics of that behavior. Ultimately, this invention assists analysts in audit log analysis by using behavioral clustering, greatly reducing their workload and eliminating the need to rely on expert knowledge. Attached Figure Description

[0040] Figure 1 This is a flowchart of a method for automatically extracting advanced behaviors from audit logs according to the present invention;

[0041] Figure 2 This is an overall framework diagram of the present invention for automatically extracting advanced behaviors from audit logs;

[0042] Figure 3 This is an example. Figure 3 Tuple vector graph. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] Please see Figure 1-2 In a preferred embodiment, it is observed that high-level behaviors primarily revolving around a user's intended goal can be reflected as a series of system operations applied to data objects. Therefore, high-level behaviors are defined as the user's intended goal, and the means to achieve that goal are defined as behavior instances. Examples of such behaviors include data breaches, backdoor installations, and program compilation. Each behavior instance is a series of operations performed by the user to achieve the goal, and these operations can be further modeled as data transmissions, and behavior instances can be modeled as sequences of such data transmissions.

[0045] A method for automatically extracting advanced behaviors from audit logs includes:

[0046] The audit logs to be analyzed are parsed into triples, and a source graph based on the triples of the audit logs is constructed.

[0047] The semantics of the context of each node in the source graph are inferred, and subgraphs representing corresponding behavior instances are enumerated from the source graph. Then, semantic aggregation of behavior is performed based on the semantics and the behavior instances to obtain the semantics of the behavior instances.

[0048] Behavior instances are clustered into clusters according to preset rules, and the behavior signatures of the most representative behavior instances are extracted from each cluster to obtain the semantics of high-level behaviors.

[0049] In this embodiment, parsing the audit logs to be analyzed into triples and constructing a source graph based on the triples of the audit logs includes:

[0050] The audit logs to be analyzed are parsed into triples consisting of a head, a relation, and a tail, thereby constructing a source graph that encodes the semantic relations of the information flow from head to tail in the form of the triples.

[0051] Each triple corresponds to an audit event in the audit log. The header and the tail are system entities of any type and are not simultaneously files or network sockets. The relationship is the corresponding system operation. The system entity type of the triple is consistent with the type of the audit event. The system entity includes processes, files, and network sockets.

[0052] In this embodiment, understanding the semantics of audit events is the first step in abstracting high-level behaviors. Specifically, accurate understanding is based on appropriate representation and granularity, allowing for efficient semantic comparison. A common practice in previous work was to treat each log event as the basic unit of analysis. However, a single audit event comprises three elements (Head, Relation, and Tail), each contributing to the event's semantics. Therefore, performing semantic analysis at the element level provides a more detailed view compared to events, as the context of each element is explicit. Working at the element level allows for obtaining the semantics of audit events by merging the three constituent elements, and the semantics of behavior instances by merging the events that constitute the behavior. Although there is a trade-off between scalability and accuracy due to the different granularity of semantic analysis, choosing a computationally efficient embedding model allows us to maintain accuracy when processing a large number of events found in logs. Therefore, this invention chooses a single element, rather than an audit event, as the basic unit of semantic reasoning.

[0053] Since embedding models can learn the semantics of audit events from element-based contextual information, the next problem is to map the elements to the embedding space (i.e., the vector space). Intuitively, the goal is to transform each element into a vector, where small distances between elements represent similar semantics, while large distances represent opposite semantics. To achieve this goal, this invention proposes using the translation-based embedding model series *trans* to learn the mapping from elements to the embedding space. *TransH* has a wider range of applications than its sibling *TransE*, which is only suitable for solving one-to-one problems, while *TransH* can effectively solve many-to-one, one-to-many, or many-to-many problems. Therefore, this invention selects the *TransH* model as the embedding model.

[0054] The semantics of inferring the context of each node in the source graph include:

[0055] The TransH embedding model is used to map triples of audit events to a vector space.

[0056] The vectors of triples are concatenated to generate a single vector, which is used to represent the semantics of the audit event.

[0057] In TransH, vectors in the embedding space describe the semantic relationships between Head, Relation, and Tail. Specifically, the embedding space has the following property: given a triple (Head, Relation, Tail), the corresponding vectors are h, r, and t, respectively, and the projection matrix of relation r is denoted as w. r ,like Figure 3 As shown.

[0058] A relation r corresponds to a hyperplane, and two entities h and t in the entity space are connected through the relation matrix w. r Projecting onto this hyperplane, this projection mechanism is not a typical perpendicular projection; instead, the projection method uses matrix multiplication, depending on the corresponding entity. The projected vectors are as follows:

[0059]

[0060]

[0061] Therefore, the scoring function is defined as

[0062]

[0063] During the training phase, TransH optimizes the element embedding space by minimizing the translation distance of correct triples (training triples) and maximizing the score of incorrect triples (damaged triples). The loss function for embedding model optimization is summarized as follows:

[0064]

[0065] Where, [x] + `max(x,0)` represents the maximum value; `γ` is a hyperparameter that specifies the margin between correct and incorrect triples in the model. Finally, the embeddings of the three components (Head, Relation, and Tail) of the audit event are concatenated using the TransH embedding model to generate a vector representing the semantics of the audit event.

[0066] In this embodiment, the second step of behavior abstraction is to identify behavior instances from the source graph. This invention defines a behavior instance as a sequence of audit events that operate on related data and are interconnected through information flow. Therefore, summarizing a single behavior instance can be simplified to extracting a causal connection subgraph rooted at a data object. This invention partitions the source graph based on the subgraph to represent behavior instances. To extract the subgraph summarizing behavior instances, this invention performs an adaptive forward depth-first search on the data object-based source graph. Enforcing constraints during graph traversal—that the timestamp of each subsequent edge must monotonically increase from all preceding edges—prevents erroneous dependencies due to information flowing from future events to past events. Furthermore, the ancestors of system entities often contain key behavioral contexts, but such ancestors are lost in ordinary forward depth-first search because they are backward dependencies.

[0067] The step of enumerating subgraphs representing corresponding behavior instances from the source graph to summarize the behavior instances includes:

[0068] Perform an adaptive forward depth-first search on the source graph based on data objects.

[0069] During graph traversal, one-step edges that reach system entities are further merged.

[0070] The source graph is divided into subgraphs, each describing a behavior instance, which is a sequence of audit events that operate on related data and are interconnected through information flow.

[0071] In this embodiment, a constraint is enforced during graph traversal, which requires that the timestamp of each subsequent edge must monotonically increase from all preceding edges.

[0072] In this embodiment, after summarizing the behavioral instances, the next step is to derive the semantics of the behavioral instances. As mentioned earlier, each behavioral instance consists of audit events, and the semantics of the audit events have been transformed into vector representations. However, the semantics of a behavioral instance cannot be obtained by simply adding the vectors of its constituent events, because this method is only effective under the assumption that all constituent events contribute equally to the semantics of the behavioral instance. However, in reality, different events have different relative importance to the semantics of a behavioral instance. Therefore, this invention first assigns a corresponding weight to each constituent event of the behavioral instance to represent its importance to the behavior.

[0073] Weighting: Based on observational analysis, events unrelated to behavior are more prevalent in sessions because they recur across different behaviors, while events actually related to behavior occur less frequently. Based on this observation, this invention uses event frequency as a measure of its importance. More specifically, this invention uses Inverse Document Frequency (IDF) to determine the importance of a specific event to the overall behavior. IDF, as a term weighting technique, is widely used in information retrieval. Its principle is to give less common terms in documents more discriminative power. In this specific scenario, each audit event and user session is treated as a term and a document.

[0074] Based on the semantics and the behavior instance, semantic aggregation of the behavior is performed to obtain the semantics of the behavior instance, including:

[0075] Weights are assigned to each audit event in the behavior instance using Inverse Document Frequency (IDF).

[0076] The semantics of the behavior instance are obtained by aggregating the semantics of the audit events that constitute the behavior instance.

[0077] The formula for IDF is as follows:

[0078]

[0079] Where e represents an audit event, S is the number of all sessions containing event e, and Se is the number of specific sessions. After using IDF-weighted events, the semantic representation of a behavior instance can be obtained by aggregating the semantics of the audit events that make up the behavior instance. This invention adopts a pooling approach to derive the vector representation of the behavior instance, because simply summing the semantics of the events has effectively integrated the semantic information of the events.

[0080] In this embodiment, a behavior instance is a variant of an advanced behavior. A behavior can be considered as a group of semantically similar instances, and the behavior signature is the most representative instance in the cluster. Therefore, we cluster semantically similar behavior instances into clusters and then extract the behavior signature from each cluster. Each cluster corresponds to an advanced behavior, and analysts only need to analyze the behavior signature of each cluster to obtain the semantics of that behavior.

[0081] Behavioral instances are clustered into groups according to preset rules, which include:

[0082] The semantic similarity of behavioral instances is calculated using the following formula:

[0083]

[0084] Among them, F m and F n Let e ​​be a vector representation of two behavioral instances. i and e j They are vectors F m and F n For each dimension, S(F) m ,F n The cosine similarity score (S) represents the cosine similarity score. A higher cosine similarity score (S) indicates higher Fm and F. n The more similar the semantics of the behaviors, the more effective the addition of event pool embeddings in behavior instances becomes. Since Fi and Fj are the sums of their respective constituent events, cosine similarity is essentially comparing the similarity of individual events in two instances.

[0085] In this embodiment, behavior instances are clustered into clusters according to preset rules, which further include:

[0086] The Hierarchical Cluster Analysis (HCA) algorithm is used to iteratively calculate the semantic similarity of behavioral instances and combine the two closest clusters until the maximum similarity is below a merging threshold. Initially, each behavioral instance belongs to its own cluster. Then, HCA iteratively calculates the cosine similarity between clusters and combines the two closest clusters until the maximum similarity is below the merging threshold. We choose centroid connectivity as the criterion for judging cluster similarity. That is, cluster similarity estimation depends on the centroids (arithmetic mean positions) in the clusters.

[0087] In this embodiment, the behavior signature of the most representative behavior instance is extracted from each cluster to obtain the semantics of the high-level behavior, including:

[0088] The representativeness of each instance in a cluster is quantified by calculating the average similarity between each behavior instance and other behavior instances, and the instance with the highest average similarity is selected as the behavior signature for that cluster. By distinguishing behavior signatures, we expect to see a significant reduction in analytical workload, as semantically similar behaviors are clustered before manual analysis.

[0089] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for automatically extracting advanced behaviors from audit logs, characterized in that, include: The audit logs to be analyzed are parsed into triples, and a source graph based on the triples of the audit logs is constructed, including: The audit logs to be analyzed are parsed into triples consisting of a header, relation, and tail, thereby constructing a source graph that encodes the semantic relation of the information flow from beginning to end in the form of the triples. Inferring the semantics of the context of each node in the source graph includes: The TransH embedding model is used to map the triples of audit events to a vector space; The vectors of triples are concatenated to generate a single vector, which is used to represent the semantics of the audit event. And from the source graph, enumerate the subgraphs used to represent the corresponding behavior instances, including: Perform an adaptive forward depth-first search on the source graph based on data objects; During graph traversal, one-step edges that reach system entities are further merged. The source map is divided into sub-graphs, each sub-graph describing a behavior instance, which is a sequence of audit events that operate on related data and are interconnected through information flow; During graph traversal, a constraint is enforced that the timestamp of each subsequent edge must monotonically increase from all preceding edges. Therefore, semantic aggregation of behaviors is performed based on the semantics and the behavior instances to obtain the semantics of the behavior instances; Behavior instances are clustered into clusters according to preset rules, and the behavior signatures of the most representative behavior instances are extracted from each cluster to obtain the semantics of high-level behaviors.

2. The method for automatically extracting advanced behaviors from audit logs according to claim 1, characterized in that, The step of parsing the audit logs to be analyzed into triples and constructing a source graph based on the triples of the audit logs includes: Each triple corresponds to an audit event in the audit log. The header and the tail are system entities of any type and are not both files or network sockets. The relationship is the corresponding system operation. The type of the system entity in the triple is consistent with the type of the audit event. The system entity includes processes, files, and network sockets.

3. The method for automatically extracting advanced behaviors from audit logs according to claim 1, characterized in that, Based on the semantics and the behavior instance, semantic aggregation of the behavior is performed to obtain the semantics of the behavior instance, including: Use Inverse Document Frequency (IDF) to assign weights to each audit event in the behavior instance; The semantics of the behavior instance are obtained by aggregating the semantics of the audit events that constitute the behavior instance.

4. The method for automatically extracting advanced behaviors from audit logs according to claim 3, characterized in that, Behavioral instances are clustered into groups according to preset rules, which include: The semantic similarity of behavioral instances is calculated using the following formula: Among them, F m and F n For the vector representation of two behavioral instances, and They are vectors F m and F n For each dimension, S(F) m , F n ) represents cosine similarity.

5. The method for automatically extracting advanced behaviors from audit logs according to claim 3, characterized in that, The behavior instances are clustered into clusters according to preset rules, which further include: The hierarchical clustering analysis algorithm HCA is used to iteratively calculate the semantic similarity of behavioral instances and combine the two closest clusters until the maximum similarity is below the merging threshold.

6. The method for automatically extracting advanced behaviors from audit logs according to claim 1, characterized in that, The behavior signatures of the most representative behavior instances are extracted from each cluster to obtain the semantics of the high-level behaviors, including: The representativeness of each instance in the cluster is quantified by calculating the average similarity between each behavior instance and other behavior instances, and the instance with the highest average similarity is selected as the behavior signature of the cluster.

Images