A method and device for constructing a provenance graph based on system audit logs

Abstract

The application discloses a kind of based on system audit log's traceability graph construction method and device, the method includes: based on system audit log design data mode, the data mode includes the attribute information content and format specification required in the node of traceability graph, edge;The node corresponds to entity in system audit log, the edge corresponds to event in system audit log;After data mode design, the node, the edge record or the attribute information corresponding to node, edge missing is reasoned and improved;Data compression is carried out to the data after reasoning and improving;Based on the data after compression, traceability graph is constructed, and the traceability graph of system audit log constructed is visualized.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a method and apparatus for constructing a source graph based on system audit logs. Background Technology

[0002] In recent years, the rapid development of computer technology has brought us great convenience. However, the resulting cyberattacks, leading to information leaks and damage to critical infrastructure, have caused significant disruption to our lives. Passive defense technologies only respond when losses occur, while threat hunting, as a proactive defense technology, discovers known and unknown threats by actively and iteratively searching for traces of compromise within the system. Traditional security tools such as IDS and SIEM struggle to perform correlation analysis and discovery of complex attacks like APTs, while attribution mapping, a technique that can model complex relationships between system objects, is widely used in the field of threat discovery.

[0003] Modeling system audit logs into a source graph, which contains rich information about system objects and their interactions, effectively reflects the attack's occurrence. Therefore, using source graphs for threat hunting is a reliable approach for hunting complex attacks.

[0004] Current research lacks a detailed description of the source graph construction process based on system logs, and there is a lack of a complete scheme for recording the entire source graph construction process. Most current research focuses on using source graphs for threat discovery, without much description of the construction process, yet source graph construction is fundamental to threat discovery using source graph technology. Therefore, there is an urgent need for a complete scheme that can describe the source graph construction process to guide source graph construction. Summary of the Invention

[0005] This invention addresses the lack of detailed descriptions of the source graph construction process in related works by proposing a source graph construction method and apparatus based on system audit logs.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention proposes a method for constructing a source graph based on system audit logs, comprising:

[0008] The data pattern is designed based on the system audit log. The data pattern includes the attribute information and format specifications required for the nodes and edges in the traceability graph. The nodes correspond to entities in the system audit log, and the edges correspond to events in the system audit log.

[0009] Reasoning and improvement are carried out on missing node and edge records or corresponding attribute information after the data model design.

[0010] Compress the data after the reasoning has been refined.

[0011] A source graph is constructed based on the compressed data, and the source graph of the constructed system audit logs is visualized.

[0012] Furthermore, the data pattern designed based on the analyzed and processed system audit logs includes:

[0013] Based on general data specifications, analyzed and processed system audit logs, and source map construction requirements, the essential attributes of nodes for source map-based source analysis are determined.

[0014] Analyze the event types in the system audit log data, classify them according to source node and destination node types, and summarize the interaction types between different types of nodes;

[0015] Based on general data specifications, analyzed and processed system audit logs, and source graph construction requirements, the essential attributes of edges for source graph source analysis are determined.

[0016] Construct the common attribute information of nodes and edges in the source graph.

[0017] Furthermore, the step of reasoning and improving the missing node and edge records or the attribute information corresponding to nodes and edges after the data pattern design includes:

[0018] For missing key attribute values ​​of nodes and edges, the path information of all entity objects in the event is first counted. After the count is completed, all path information of the entity is preprocessed and then stored in a set collection for deduplication. After deduplication, the set is converted into a list collection. If more than one path information remains, abnormal path analysis is performed, and the final path information of the entity is determined. If only one path information exists, the path information is stored as the path information of the object.

[0019] For missing node records, complete the node records as follows:

[0020] Retrieve information about node Y involved in the event from the known entity data. If node Y is not found, treat node Y as an unknown node. Based on the system call type, infer the type information of the unknown node Y involved and determine the value of the type X of node Y. If the type of node Y cannot be inferred based on the system call type, set the type X of Y to an unknown type.

[0021] If a parent process N exists in process node M, but no record of a fork relationship between N and M can be found in the log, then it is considered that a fork event between N and M is missing, and a new fork record needs to be created.<t,N,fork,M> The edge is defined here, where t is set to the start time of process M; if there are three system entities A, B, and C in the log, then for such events, two edges are established: A→B and B→C.

[0022] Furthermore, the data compression of the reasoned data includes:

[0023] Discard edges constructed from non-critical system call events;

[0024] Only model the data within the window where threat hunting is required;

[0025] Further compression was performed using a time-window-based source graph compression algorithm.

[0026] Furthermore, the source graph compression algorithm based on time windows includes:

[0027] First, all edges are traversed, and edges with the same edge type, whose corresponding events have the same process PID, unique identifier, and unique identifier of the operation object are placed in the same set, thus dividing the edges into several groups. Then, all edges in each group are sorted according to the occurrence time attribute. After sorting, the edges in each group are merged. If the time interval between two consecutive edges is less than the timestamp threshold, they are merged. Each time they are merged, only the edge with the earliest occurrence time and the edge with the latest occurrence time are retained, and the number of edges merged with the earliest occurrence edge is updated. If the time interval between two connected edges is greater than the given threshold, the merging process is restarted.

[0028] Furthermore, the construction of the source map based on the compressed data includes:

[0029] Traverse the node and edge datasets and construct the source graph according to the source graph pattern specification; reduce the source graph by removing isolated nodes; re-index all nodes in the source graph to ensure that the remaining nodes in the source graph are numbered consecutively.

[0030] Furthermore, the visualization of the source map of the constructed system audit logs includes:

[0031] The constructed source graph is visualized using the graph database HugeGraph.

[0032] Another aspect of the present invention proposes a source graph construction device based on system audit logs, comprising:

[0033] The data pattern design module is used to design data patterns based on the analyzed and processed system audit logs. The data patterns include the attribute information and format specifications required for the nodes and edges in the source graph.

[0034] The missing reasoning module is used to reason and improve the missing node or edge records or the attribute information corresponding to nodes and edges after the data pattern design.

[0035] The source data compression module is used to compress the data after the reasoning is completed;

[0036] The source graph construction module is used to construct source graphs based on compressed data and visualize the constructed source graphs of system audit logs.

[0037] Furthermore, the data pattern designed based on the analyzed and processed system audit logs includes:

[0038] Based on general data specifications, analyzed and processed system audit logs, and source map construction requirements, the essential attributes of nodes for source map-based source analysis are determined.

[0039] Analyze the event types in the system audit log data, classify them according to source node and destination node types, and summarize the interaction types between different types of nodes;

[0040] Based on general data specifications, analyzed and processed system audit logs, and source graph construction requirements, the essential attributes of edges for source graph source analysis are determined.

[0041] Construct the common attribute information of nodes and edges in the source graph.

[0042] Furthermore, the missing reasoning module is specifically used for:

[0043] For missing key attribute values ​​of nodes and edges, the path information of all entity objects in the event is first counted. After the count is completed, all path information of the entity is preprocessed and then stored in a set collection for deduplication. After deduplication, the set is converted into a list collection. If more than one path information remains, abnormal path analysis is performed, and the final path information of the entity is determined. If only one path information exists, the path information is stored as the path information of the object.

[0044] For missing node records, complete the node records as follows:

[0045] Retrieve information about node Y involved in the event from the known entity data. If node Y is not found, treat node Y as an unknown node. Based on the system call type, infer the type information of the unknown node Y involved and determine the value of the type C of node Y. If the type of node Y cannot be inferred based on the system call type, set the type X of Y to an unknown type.

[0046] If a parent process N exists in process node M, but no record of a fork relationship between N and M can be found in the log, then it is considered that a fork event between N and M is missing, and a new fork record needs to be created.<t,N,fork,M> The edge is defined here, where t is set to the start time of process M; if there are three system entities A, B, and C in the log, then for such events, two edges are established: A→B and B→C.

[0047] Furthermore, the traceability data compression module is specifically used for:

[0048] Discard edges constructed from non-critical system call events;

[0049] Only model the data within the window where threat hunting is required;

[0050] Further compression was performed using a time-window-based source graph compression algorithm.

[0051] Furthermore, the source graph compression algorithm based on time windows includes:

[0052] First, all edges are traversed, and edges with the same edge type, whose corresponding events have the same process PID, unique identifier, and unique identifier of the operation object are placed in the same set, thus dividing the edges into several groups. Then, all edges in each group are sorted according to the occurrence time attribute. After sorting, the edges in each group are merged. If the time interval between two consecutive edges is less than the timestamp threshold, they are merged. Each time they are merged, only the edge with the earliest occurrence time and the edge with the latest occurrence time are retained, and the number of edges merged with the earliest occurrence edge is updated. If the time interval between two connected edges is greater than the given threshold, the merging process is restarted.

[0053] Furthermore, the construction of the source map based on the compressed data includes:

[0054] Traverse the node and edge datasets and construct the source graph according to the source graph pattern specification; reduce the source graph by removing isolated nodes; re-index all nodes in the source graph to ensure that the remaining nodes in the source graph are numbered consecutively.

[0055] Furthermore, the visualization of the source map of the constructed system audit logs includes:

[0056] The constructed source graph is visualized using the graph database HugeGraph.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] This invention first designs a data model based on system audit logs; then, it infers and improves the missing node and edge records or the corresponding attribute information of nodes and edges after the data model design; it compresses the improved data; finally, it constructs a source graph based on the compressed data and visualizes the constructed source graph of the system audit logs. This invention improves the reproducibility of the source graph construction process for system audit logs by providing a standardized description of the process. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating a method for constructing a source graph based on system audit logs according to an embodiment of the present invention.

[0060] Figure 2 This is a comparison of the source data compression algorithm based on time windows before and after compression in an embodiment of the present invention;

[0061] Figure 3 The source map in this embodiment of the invention is visualized using HugeGraph;

[0062] Figure 4 This is a schematic diagram of a source map construction device based on system audit logs according to an embodiment of the present invention. Detailed Implementation

[0063] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments:

[0064] like Figure 1 As shown, a method for constructing a source graph based on system audit logs includes:

[0065] The data pattern is designed based on the system audit log. The data pattern includes the attribute information and format specifications required for the nodes and edges in the traceability graph. The nodes correspond to entities in the system audit log, and the edges correspond to events in the system audit log.

[0066] Reasoning and improvement are carried out on missing node and edge records or corresponding attribute information after the data model design.

[0067] Compress the data after the reasoning has been refined.

[0068] A source graph is constructed based on the compressed data, and the source graph of the constructed system audit logs is visualized.

[0069] Specifically, defining the data pattern of node and edge attribute information in the source graph is crucial for threat discovery using source graphs, and the quality of the data pattern will affect the final source analysis results. First, the essential attributes of system entities and events are analyzed, and then these attributes are expanded to meet threat hunting requirements. Next, this invention summarizes the key attributes and record missing issues existing in the source graph construction process and proposes an aggregation-based path missing reasoning algorithm. Then, this invention proposes three data compression schemes and details how the designed time window-based source data compression algorithm achieves efficient and lossless data compression. Finally, the processed source data is used to construct the source graph according to the designed iterative source graph construction algorithm. At this point, the source graph construction is complete, serving as the foundation for subsequent threat hunting. As one possible implementation, this invention uses the DARPA TC dataset as an example to detail the source graph construction process.

[0070] 1. Dataset Overview and Analysis

[0071] 1.1 Overview of DARPA TC

[0072] The DARPA Transparent Computing (TC) project, initiated in 2015 by the U.S. Defense Advanced Research Projects Agency (DARPA), is a five-year project aimed at enabling early detection of sophisticated attacks by labeling and tracking causal relationships across enterprise activities. The project comprises five technical areas, described in Table 1.

[0073] Table 1. Description of the five technical fields of DARPA TC

[0074]

[0075]

[0076] The DARPA TC project has partially released the data, which mainly includes: (1) terminal log data collected by the TA1 team, including FiveDirection, MARPLE, and CADETS, which covers Windows, Linux, and FreeBSD systems; and (2) detailed attack process reports provided by the TA5.1 team.

[0077] Over five years, the DARPA TC project organized five engagements. DARPA publicly released the data and attack reports collected during the third and fifth engagements. Table 2 shows the timeline and objectives of the five DARPA TC engagements.

[0078] Table 2. Dates, attack details, and targets of the five DARPA TC engagements.

[0079]

[0080] 1.2 Source tracing data analysis

[0081] The purpose of source data analysis is to determine the data characteristics of the collected log data, uncover their commonalities, and facilitate subsequent use in data schema design, missing data inference, data compression, etc. This invention uses data collected by the FiveDirection team in the DARPA TC TA1 organization as an example to analyze the data and find the following problems:

[0082] (1) Important attribute information of the collected entity objects is missing. For example, the file path information and the command line content information of the process in the data are important information for tracing the source, but the collected data cannot directly and effectively capture this information, which will bring great difficulties to the source tracing analysis.

[0083] (2) Most attribute collection results are empty or the same value, which cannot be modeled as the features of the entity, making it difficult to further discover the anomalies and not very meaningful for tracing the source;

[0084] (3) The main type of the collected log information is mostly threads. Due to the fine granularity of threads, the dependency explosion phenomenon is extremely serious. At the same time, it is difficult to find the relationship between the objects interacting between different threads of the same process. For example, for threads A and B of process X, if A is related to entity C and B is related to entity D, then to find the relationship between C and D, it is necessary to go through C→A→X→B→D to find the relationship between C and D.

[0085] (4) The data is massive, and direct processing will bring a huge computational and storage burden to the system, so the data needs to be further compressed.

[0086] Based on the above analysis, the present invention proposes the following processing steps for traceability data:

[0087] (1) Aggregate the important attribute fields of the traceability data to determine the enumeration range of the attributes. For example: event type attributes, various entity type attributes, attributes that need to be determined whether to retain, etc.

[0088] (2) Identify missing fields in the source data and establish a preliminary completion plan. For example, if path information is missing in entity objects such as files, relevant information can be found from event type data and aggregated for analysis and reasoning. The reasoned information can then be filled into the path attribute of the file object.

[0089] (3) Design entity type data patterns according to data characteristics and traceability diagram input format, fill data according to traceability data patterns, and store all entity object data;

[0090] (4) Design an event type data pattern based on the data characteristics and the source diagram input format, promote all thread type data in the event to the corresponding process type data, update the thread number and other fields in the event, and store all event data;

[0091] (5) Supplement and improve the parent-child relationship events between all processes and their parent processes. This step can effectively alleviate the problem of interrupted tracing clues caused by the loss of fork type data.

[0092] (6) Store all the unknown entities mentioned above, and infer all possible subject-object type combinations of the system call based on the system call type information. If it is unique, the type of the unknown entity can be inferred. If there are multiple possible combinations, further judgment is made based on the path information of the object. Entities that cannot be judged effectively are set as unknown entity types.

[0093] (7) Merge all entity type data according to type;

[0094] (8) Compress event type data by using a time window-based source data compression algorithm and retaining key system calls to compress the source data.

[0095] The above processing steps for traceability data effectively solve problems such as missing data, invalid data values, and massive data volume in traceability data, laying a solid foundation for building a robust and lossless traceability graph.

[0096] 2 Data Model Design

[0097] The data schema defines the attribute information and format specifications required for nodes and edges in the source graph. In the source graph, entity records in the source data need to be modeled as nodes of corresponding types, and event records in the source data need to be modeled as edges.

[0098] 2.1 Essential Attributes of Traceability Data

[0099] The data collected by the DARPA TC TA1 team meets the Common Data Schema (CDM) specification. Based on the CDM specification, the valid data collected by the DARPA TC TA1 team, the requirements for constructing the source graph, and the necessary data for attack source tracing, this study has determined the following essential entity attributes for source tracing analysis using the source graph, as shown in Table 3.

[0100] Table 3 Required Attributes for Traceability Entity Type Data

[0101]

[0102]

[0103] By analyzing the event types in the log data and classifying them according to the source node and destination node types, the interactions between different types of nodes can be summarized as shown in Table 4.

[0104] Table 4 System Entity Interaction Types

[0105]

[0106]

[0107] Based on the Common Data Model (CDM), the valid data collected by the DARPA TC TA1 team, the requirements for constructing the source graph, and the necessary data for attack source tracing, the following essential attributes of edges for source tracing analysis using the source graph were determined, as shown in Table 5.

[0108] Table 5. Attribute information of edges in the source graph.

[0109]

[0110]

[0111] 2.2 Source Tracing Analysis Extended Attributes

[0112] Tables 3 and 5 show the essential attribute fields specific to different types of nodes and the attribute fields of edges, respectively. To facilitate attack differentiation, association with expert knowledge, and flexible expansion, each node and edge must also include the attributes shown in Table 6.

[0113] Table 6. Common attribute information of nodes and edges in the source graph.

[0114]

[0115] These extended properties can depict different attack stories, while also enabling the association of nodes and edges in the source graph with expert knowledge, facilitating the observation of the context of the attack story, and allowing for the abstraction of attacks from different levels, which helps to grasp the current attack situation and assists in threat hunting.

[0116] 3. Inference based on missing key data

[0117] Data missing inference refers to a method used in data analysis to infer missing data by utilizing known data information when some data is missing. In this study, there are two types of missing data: (1) missing key attribute values ​​of nodes and edges; and (2) missing records of key nodes and edges.

[0118] 3.1 Inference based on missing key attributes

[0119] Missing key attribute values ​​for nodes and edges indicate that some attribute information in entity or event records is invalid or missing. When using source graphs for threat hunting, the completeness of attribute information is crucial for discovering known threats and conducting manual source tracing analysis. This invention takes the problem of missing entity path attributes in source tracing data as an example and proposes an aggregation-based path missing inference algorithm, as shown in Algorithm 1.

[0120]

[0121]

[0122] The algorithm first counts the path information of all entity objects in the event. After the count is complete, the path information of each entity needs to be normalized. Line 13 of the algorithm first preprocesses all the path information of the entity, then stores it in a set collection for deduplication. After deduplication, the set needs to be converted into a list collection. If more than one path information remains, abnormal path analysis is performed in line 15 of the algorithm to determine the final path information of the entity. If only one path information exists, that path information is stored as the path information of the object.

[0123] The algorithm described above can effectively aggregate and infer missing attribute values ​​from event data, providing methodological guidance for solving the problem of missing key attributes.

[0124] 3.2 Inference based on missing key records

[0125] The second type of problem, which involves missing records of key nodes and key edges, can be mainly divided into missing node records and missing edge records.

[0126] (1) Missing node records

[0127] For a node record missing, that is, for a log containing data related to a node Y of type X, but the relevant information for node Y cannot be found in the node data of type X, this study defines it as a node record missing.

[0128] This study employs the following steps to complete such node records:

[0129] ① Retrieve information about node Y involved in the event from the known entity database S. If node Y is not found, treat node Y as an unknown node and proceed to step ②.

[0130] ② Based on the system call (event) type, infer the type information of the unknown node Y involved, and determine the value of the type X of node Y. If the type of node Y cannot be inferred from the system call type, then set the type X of Y to unknown type.

[0131] (2) Missing edge records

[0132] Missing edge records refer to situations where the tracing graph constructed from the original log contains missing edges, making direct and effective tracing impossible. Missing edge records can be categorized as follows:

[0133] ① If a parent process N exists in process node M, but no record of a fork relationship between N and M can be found in the log, then it is considered that a fork event between N and M is missing, and a new fork event needs to be created.<t,N,fork,M> The edge, where t is set to the start time of process M;

[0134] ② If the log contains three system entities A, B, and C, for this type of event, two edges need to be established: A→B and B→C. For example, if there is a log entry where firefox.exe renames a.tmp to test.word at time t, then in addition to establishing...<t,firefox.exe,rename,a.tmp> Outside the border, a...<t,a.tmp,reame_to,test.word> The edge.

[0135] For scenario ②, there might be a missing information about the parent process N. The attribute information of this process can be supplemented using the key attribute reasoning method described in the previous section. For the process's start time t, let the earliest occurrence time of all its related events be t1, and the earliest start time of all its child processes be t2. This study assumes t = min{t1, t2}, meaning the earliest occurrence time of both is set as the start time of this unknown process.

[0136] Thus, this invention completes the work related to reasoning about missing key data, which will improve the source data information and lay a solid foundation for the construction of a high-quality source map.

[0137] 4. Source data compression

[0138] The sheer volume of data presents a significant challenge when constructing source graphs. Directly using this data for source graph construction incurs enormous system overhead. Furthermore, large-scale source graphs cannot be used for anomaly pattern recognition using methods such as graph neural networks and clustering. Presenting them directly to users also leads to problems such as an excessive number of paths between entities, making it difficult to locate critical paths. To address these issues, data compression becomes crucial. Source graph data compression refers to merging source node and edge data without affecting detection and source tracing effectiveness, thereby reducing the size of the source graph.

[0139] 4.1 Method Design

[0140] To reduce the size of traceability data, this invention employs the following three solutions:

[0141] (1) Discard the edges constructed by accompanying non-critical system call events;

[0142] (2) Only model the data within the window where threat hunting is required;

[0143] (3) Further compression is performed using a time window-based source graph compression algorithm.

[0144] Discarding accompanying non-critical edges refers to using expert knowledge to discard some edges constructed from non-critical system call events that will not affect the tracing results. For example, for system calls such as check_file_attributes, open, and close, if an attack occurs, system calls such as read and write will definitely exist to achieve the goal. Therefore, discarding the large number of non-critical edges such as check_file_attributes, open, and close will not affect the final tracing results. Therefore, this study chooses to discard the edges of these system calls.

[0145] Currently, many studies use User and Entity Behavior Analytics (UEBA) technology to establish a baseline of normal system behavior to identify anomalous time periods within a system. This study's threat hunting approach is based on the ability to identify time periods of anomalous behavior within the system. Therefore, this paper assumes that the current system can provide time periods of anomalous behavior, but the specific range of these time periods is not within the scope of this study. Thus, this research only models system entity data and event data within the window required for threat hunting. This approach effectively reduces the data size needed to construct the source map.

[0146] In addition to the data compression schemes mentioned above, this study argues that when a thread is promoted to its corresponding process node during modeling, there will be a large number of consecutive edges with the same source node, destination node, and attribute information. Furthermore, if a process engages in a large number of communications with a remote network connection within a short period, the edges corresponding to these events will also have the same source node, destination node, and edge attribute information. All of these situations can be merged. Therefore, this study proposes a time-window-based source-tracing data compression algorithm, the algorithm flow of which is shown in Algorithm 2.

[0147] The algorithm first iterates through all edges, grouping them into sets based on their type, the corresponding event's PID, entity, object1, and object2. Then, it sorts all edges within each group according to their occurrence time. After sorting, it merges edges within each group. If the time interval between two consecutive edges is less than a timestamp threshold, they are merged. During each merge, only the earliest and latest occurrence edges are retained, easily identifying the time interval of event interactions. The algorithm also updates the number of edges merged from the earliest occurrence edge. If the time interval between two connected edges exceeds a given threshold, the merging process restarts.

[0148] 4.2 Compression Ratio Analysis

[0149] The above algorithm is used to compress the source data, effectively reducing its size. For example, for the interaction between Firefox and 16.54.116.146:80 as shown in the figure, the result of directly constructing the source graph is as follows: Figure 2 As shown on the left, after compression, as Figure 2 As shown on the right, this compression algorithm reduces 412 edges to 4 edges, which not only facilitates the presentation of results to threat hunters but also preserves the interaction information between Firefox and 16.54.116.146:80. After compression of the source data using the time window-based source data compression algorithm, the size of the source data has been significantly reduced.

[0150]

[0151]

[0152] When conducting threat hunting on DARPA TC engagement#3 FiveDirection data, the UEBA approach was used. After establishing a normal baseline for the system, the data was analyzed. It was found that the behavior pattern of the system audit logs from 9:50 AM to 10:50 AM New York time on April 11, 2018, differed from the log data patterns of other time periods. Therefore, this study concludes that there was abnormal behavior in the system during this period, requiring further threat hunting. This study selected data from this period for data compression to facilitate subsequent threat hunting research. The compression rate test results for different system calls are shown in Table 7.

[0153] Table 7 shows the compression ratio results of the time window-based source data compression algorithm for different system calls.

[0154]

[0155]

[0156] The results show that the overall compression rate of the source data is 69.11%. After compression, the size of the source graph is reduced by more than half. Moreover, it can be found that the compression rate of critical system calls such as fork and execute is low, which means that critical system calls can be highlighted more effectively.

[0157] 5. Source Map Construction

[0158] At this point, the source data processing is complete. The source data processed in the previous sections has become more concise and complete. This section will introduce the construction of a source graph, detailing how to transform source data into a source graph.

[0159] Suppose that the currently processed traceability data contains a set of node data of entity types, V = {V1, V2, ..., V...} m}, where V i Let represent the set of nodes of class i, i∈N, 1≤i≤m. Suppose the currently processed source data contains edge data E={E1,E2,…,E...} representing events. n}, E j Let V represent the set of edges of type j, where j∈N, 1≤j≤n. According to the traceability data pattern specification, each node v in V must contain the attribute "node type" node_type, and each edge data e in E must contain the attributes "event occurrence time" time, "edge type" edge_type, "source node of information flow" src_node, and "destination node of information flow" dst_type.

[0160]

[0161]

[0162] This paper proposes Algorithm 3 for constructing the source graph. First, the algorithm traverses the node and edge datasets and constructs the source graph according to the source graph pattern specification. Lines 9-15 of the algorithm further reduce the source graph by removing isolated nodes. Line 16 of the algorithm re-indexes all nodes in the source graph to ensure that the remaining nodes in the source graph are numbered consecutively.

[0163] At this point, the source graph is complete. As one possible implementation, this invention uses the graph database HugeGraph to visualize the constructed source graph. HugeGraph is an open-source distributed graph database system characterized by high scalability, high performance, and low latency. It can store and manage large-scale graph data and supports efficient graph query and analysis operations.

[0164] This study uses the compressed data from the previous section to construct the source map, which is shown below. Figure 3 As shown, due to aesthetic considerations and limitations of visualization technology, many connecting edges between nodes are not displayed. Therefore, the displayed source graph is still a collection of several subgraphs of the constructed source graph.

[0165] In summary, this invention proposes a method for constructing a source graph based on system audit logs. Taking the publicly available dataset DARPA TC as an example, the entire process of constructing the source graph using the proposed scheme is introduced. First, the system audit log data is analyzed. After the analysis, based on the characteristics of the source data and the source graph, the source data pattern is designed. In the data preprocessing stage, key data missing inference is first performed, and the problems of missing key attributes and missing key records are introduced, along with corresponding solutions. Then, source data compression is introduced, and three data compression schemes are proposed. Finally, the source graph is constructed using a source graph construction algorithm. The source graph construction is thus completed.

[0166] Based on the above embodiments, such as Figure 4 As shown, the present invention also proposes a source graph construction device based on system audit logs, comprising:

[0167] The data pattern design module is used to design data patterns based on the analyzed and processed system audit logs. The data patterns include the attribute information and format specifications required for the nodes and edges in the source graph.

[0168] The missing reasoning module is used to reason and improve the missing node or edge records or the attribute information corresponding to nodes and edges after the data pattern design.

[0169] The source data compression module is used to compress the data after the reasoning is completed;

[0170] The source graph construction module is used to construct source graphs based on compressed data and visualize the constructed source graphs of system audit logs.

[0171] Furthermore, the data pattern designed based on the analyzed and processed system audit logs includes:

[0172] Based on general data specifications, analyzed and processed system audit logs, and source map construction requirements, the essential attributes of nodes for source map-based source analysis are determined.

[0173] Analyze the event types in the system audit log data, classify them according to source node and destination node types, and summarize the interaction types between different types of nodes;

[0174] Based on general data specifications, analyzed and processed system audit logs, and source graph construction requirements, the essential attributes of edges for source graph source analysis are determined.

[0175] Construct the common attribute information of nodes and edges in the source graph.

[0176] Furthermore, the missing reasoning module is specifically used for:

[0177] For missing key attribute values ​​of nodes and edges, the path information of all entity objects in the event is first counted. After the count is completed, all path information of the entity is preprocessed and then stored in a set collection for deduplication. After deduplication, the set is converted into a list collection. If more than one path information remains, abnormal path analysis is performed, and the final path information of the entity is determined. If only one path information exists, the path information is stored as the path information of the object.

[0178] For missing node records, complete the node records as follows:

[0179] Retrieve information about node Y involved in the event from the known entity data. If node Y is not found, treat node Y as an unknown node. Based on the system call type, infer the type information of the unknown node Y involved and determine the value of the type X of node Y. If the type of node Y cannot be inferred based on the system call type, set the type X of Y to an unknown type.

[0180] If a parent process N exists in process node M, but no record of a fork relationship between N and M can be found in the log, then it is considered that a fork event between N and M is missing, and a new fork record needs to be created.<t,N,fork,M> The edge is defined here, where t is set to the start time of process M; if there are three system entities A, B, and C in the log, then for such events, two edges are established: A→B and B→C.

[0181] Furthermore, the traceability data compression module is specifically used for:

[0182] Discard edges constructed from non-critical system call events;

[0183] Only model the data within the window where threat hunting is required;

[0184] Further compression was performed using a time-window-based source graph compression algorithm.

[0185] Furthermore, the source graph compression algorithm based on time windows includes:

[0186] First, all edges are traversed, and edges with the same edge type, whose corresponding events have the same process PID, unique identifier, and unique identifier of the operation object are placed in the same set, thus dividing the edges into several groups. Then, all edges in each group are sorted according to the occurrence time attribute. After sorting, the edges in each group are merged. If the time interval between two consecutive edges is less than the timestamp threshold, they are merged. Each time they are merged, only the edge with the earliest occurrence time and the edge with the latest occurrence time are retained, and the number of edges merged with the earliest occurrence edge is updated. If the time interval between two connected edges is greater than the given threshold, the merging process is restarted.

[0187] Furthermore, the construction of the source map based on the compressed data includes:

[0188] Traverse the node and edge datasets and construct the source graph according to the source graph pattern specification; reduce the source graph by removing isolated nodes; re-index all nodes in the source graph to ensure that the remaining nodes in the source graph are numbered consecutively.

[0189] Furthermore, the visualization of the source map of the constructed system audit logs includes:

[0190] The constructed source graph is visualized using the graph database HugeGraph.

[0191] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a source graph based on system audit logs, characterized in that, include: The data model is designed based on the system audit log. The data model includes the attribute information and format specifications required for the nodes and edges in the traceability graph. The nodes correspond to entities in the system audit log, and the edges correspond to events in the system audit log; Reasoning and improvement are carried out on missing node and edge records or corresponding attribute information after the data model design. Compress the data after the reasoning has been refined. A source map is constructed based on the compressed data, and the source map of the constructed system audit logs is visualized. The process of reasoning and improving missing node and edge records or corresponding attribute information after data pattern design includes: For missing key attribute values ​​of nodes and edges, the path information of all entity objects in the event is first counted. After the count is completed, all path information of the entity is preprocessed and then stored in a set collection for deduplication. After deduplication, the set is converted into a list collection. If more than one path information remains, abnormal path analysis is performed, and the final path information of the entity is determined. If only one path information exists, the path information is stored as the path information of the object. For missing node records, complete the node records as follows: Retrieve the nodes involved in the event from known entity data. Information, if no node found Then this node Treat them as unknown nodes; infer the involved unknown nodes based on the system call type. Type information to determine the node Type The value of the node cannot be inferred from the system call type. The type will be Type Set to unknown type; If edge records are missing, in the process node There is a parent process However, it cannot be found in the logs. and between The record of the relationship is considered to exist. and Between The event is missing, and a line needs to be established. The edge, here Set as process The start time; if there are three system entities in the log. , , For such events, two edges are established: and .

2. The method for constructing a source graph based on system audit logs according to claim 1, characterized in that, The data model designed based on system audit logs includes: Based on general data specifications, analyzed and processed system audit logs, and source map construction requirements, the essential attributes of nodes for source map-based source analysis are determined. Analyze the event types in the system audit log data, classify them according to source node and destination node types, and summarize the interaction types between different types of nodes; Based on general data specifications, analyzed and processed system audit logs, and source graph construction requirements, the essential attributes of edges for source graph source analysis are determined. Construct the common attribute information of nodes and edges in the source graph.

3. The method for constructing a source graph based on system audit logs according to claim 1, characterized in that, The data compression of the data after the reasoning is improved includes: Discard edges constructed from non-critical system call events; Only model the data within the window where threat hunting is required; Further compression was performed using a time-window-based source graph compression algorithm.

4. The method for constructing a source graph based on system audit logs according to claim 3, characterized in that, The source graph compression algorithm based on time windows includes: First, all edges are traversed, and edges with the same edge type, whose corresponding events have the same process PID, unique identifier, and unique identifier of the operation object are placed in the same set, thus dividing the edges into several groups. Then, all edges in each group are sorted according to the occurrence time attribute. After sorting, the edges in each group are merged. If the time interval between two consecutive edges is less than the timestamp threshold, they are merged. Each time they are merged, only the edge with the earliest occurrence time and the edge with the latest occurrence time are retained, and the number of edges merged with the earliest occurrence edge is updated. If the time interval between two connected edges is greater than the given threshold, the merging process is restarted.

5. The method for constructing a source graph based on system audit logs according to claim 1, characterized in that, The construction of the source map based on the compressed data includes: Traverse the node and edge datasets and construct the source graph according to the source graph pattern specification; reduce the source graph by removing isolated nodes; re-index all nodes in the source graph to ensure that the remaining nodes in the source graph are numbered consecutively.

6. The method for constructing a source graph based on system audit logs according to claim 1, characterized in that, The visualization of the source map of the constructed system audit logs includes: The constructed source graph is visualized using the graph database HugeGraph.

7. A source graph construction device based on system audit logs, characterized in that, include: The data pattern design module is used to design data patterns based on the analyzed and processed system audit logs. The data patterns include the attribute information and format specifications required for the nodes and edges in the source graph. The missing reasoning module is used to reason and improve the missing node or edge records or the attribute information corresponding to nodes and edges after the data pattern design. The source data compression module is used to compress the data after the reasoning is completed; The source graph construction module is used to construct source graphs based on compressed data and visualize the constructed source graphs of system audit logs; The missing reasoning module is specifically used for: For missing key attribute values ​​of nodes and edges, the path information of all entity objects in the event is first counted. After the count is completed, all path information of the entity is preprocessed and then stored in a set collection for deduplication. After deduplication, the set is converted into a list collection. If more than one path information remains, abnormal path analysis is performed, and the final path information of the entity is determined. If only one path information exists, the path information is stored as the path information of the object. For missing node records, complete the node records as follows: Retrieve the nodes involved in the event from known entity data. Information, if no node found Then this node Treat it as an unknown node; Based on the system call type, infer the unknown nodes involved. Type information to determine the node Type The value of the node cannot be inferred from the system call type. The type will be Type Set to unknown type; If edge records are missing, in the process node There is a parent process However, it cannot be found in the logs. and between The record of the relationship is considered to exist. and Between The event is missing, and a line needs to be established. The edge, here Set as process The start time; if there are three system entities in the log. , , For such events, two edges are established: and .

Images