Method and device for reducing provenance graph

Abstract

The present invention discloses a method and device for reducing a traceability graph, the method comprising: performing co-occurrence analysis on the historical operation data of the host, obtaining the application log of the system interacting with the outside world, parsing and converting the application log into a normalized log according to a preset field format, extracting entities from the normalized log using a predefined rule set, and for each normalized log, using a common field to establish a mapping relationship between the entity and the node in the traceability graph, so as to obtain the target node associated with the entity in the traceability graph, and then using the target node as a trigger point, using a random walk-based algorithm to capture the context of the target node, extracting a traceability subgraph starting from the target node for each target node, and combining all the traceability subgraphs to form a reduced traceability graph. The above method can achieve the reduction of the traceability graph, and solves the problem that the amount of traceability graph data in the prior art is huge, which affects the accuracy of attack traceability detection.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method and device for reducing a traceability graph. Background Art

[0002] Advanced persistent threats (APT) have recently become one of the most critical cyberspace threats facing enterprises and institutions. APT attacks are covert and long-running computer network intrusions that usually have a certain purpose. It has a low-frequency, low-speed attack mode and a remote control point attack strategy. APT attacks usually use zero-day vulnerabilities, and after gaining control of the target system, they will hide in the system for a long time, usually several months, in order to arouse suspicion. Due to the serious consequences of APT attacks, their detection is particularly important for network security.

[0003] There are many ways to detect APT, among which the detection method based on traceability graph is widely used. The traceability graph is a directed graph, in which the nodes represent system entities and the edges represent system calls, which contains the causal relationship between information flow and system time in the system. The traceability graph can be constructed through system logs (for example, audit logs, application logs), so it can capture almost all system behaviors. The above characteristics make it a suitable data source for APT detection. UNICORN is the most advanced APT detection method based on traceability graph. It uses the traceability graph representing the normal behavior of the system to train the model, and then uses the model to check whether the traceability graph representing the current system behavior contains APT attacks. In most cases, its detection accuracy exceeds 94%. However, the amount of data in the traceability graph is usually large. A host can generate more than 35GB of data in one day, while in actual applications, a high-throughput host may generate more than 1GB of data in just 5 minutes. In addition, since APT attacks usually last for a long time, the corresponding traceability data volume will also increase, making the traceability graph larger, resulting in a longer traceability detection time.

[0004] Related technologies can use some data reduction methods to alleviate the data size problem faced by APT detection methods based on provenance graphs, mainly including graph structure-based reduction methods and graph embedding-based reduction methods. Graph structure-based reduction methods attempt to remove redundant nodes and edges while retaining the connectivity (that is, causality) of the provenance graph. However, due to the low-frequency and low-speed attack mode of APT, the part of the provenance graph related to detection is usually discontinuous. If connectivity needs to be retained, too much information will be retained, so that it is impossible to obtain a sufficient reduction rate for timely APT detection. Graph embedding-based reduction methods use graph embedding technology to map the provenance graph to a low-dimensional space that retains graph information, which can usually achieve a very high reduction rate. However, because the information of the entire provenance graph is embedded in the low-dimensional space, and there is undoubtedly noise in the graph, it is difficult to achieve satisfactory results based on this detection method. Summary of the invention

[0005] The present invention provides a method and device for reducing a traceability graph, which captures the context of a node by establishing a mapping relationship between entities in an application log and nodes in a traceability graph, and uses the generated subgraph as input to an attack traceability detection method, thereby solving the problem in the prior art that the amount of traceability graph data is huge and affects the accuracy of attack traceability detection. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for reducing a provenance graph, the method comprising:

[0007] By co-occurring analysis of the host's historical operation data, the application logs of the system's interaction with the outside world are obtained;

[0008] Parsing and converting the application log into a normalized log according to a preset field format, and extracting entities from the normalized log using a predefined rule set;

[0009] For each normalized log, a mapping relationship between the entity and a node in a traceability graph is established using a common field to obtain a target node associated with the entity in the traceability graph, wherein the traceability graph is generated by capturing the information flow of the system kernel using a traceability tool;

[0010] Taking the target node as a trigger point, a random walk-based algorithm is used to capture the context of the target node, extracting a provenance subgraph starting from the target node for each target node, and combining all the provenance subgraphs to form a reduced provenance graph.

[0011] Optionally, the co-occurrence analysis of the host's historical operation data can be performed to obtain application logs of the system's interaction with the outside world, including:

[0012] By performing co-occurrence analysis on the historical operation data of the host, a list of co-occurring applications is obtained from the applications that the system interacts with the outside world;

[0013] A target application whose co-occurrence frequency ranks before a first preset value is selected from the co-occurrence application list, and logs of the target application are collected.

[0014] Optionally, collecting the logs of the target application includes:

[0015] By querying the process corresponding to the target application interacting with the outside world on the host, the application log of the target application is collected;

[0016] For a target application whose application log cannot be obtained, an audit tool is used to collect system calls of the target application to obtain the audit log of the target application.

[0017] Optionally, parsing and converting the application log into a normalized log according to a preset field format includes:

[0018] By parsing the application log, obtaining data information of a preset field from the application log;

[0019] The data information of the preset fields is normalized into a log table according to a preset field format to form a normalized log.

[0020] Optionally, the predefined rule set includes a default rule and a custom extended rule, and the extracting entities from the normalized log using the predefined rule set includes:

[0021] Using the default rule to determine if the preset field of the normalized log contains a known attack signature and / or a critical system command, then extracting a matching field from the normalized log as a precise entity;

[0022] Otherwise, the custom extension rule is used to extract the noun and the timestamp of the request from the preset field of the normalized log as the result entity.

[0023] Optionally, the using the default rule to determine if the preset field of the normalized log contains a known attack signature and / or a critical system command, then extracting a matching field from the normalized log as a precise entity includes:

[0024] Using the default rule to determine if the preset field of the normalized log contains a known attack signature, then checking whether the application field in the normalized log matches the known attack signature, and if so, extracting the application field as a precise entity;

[0025] The default rule is used to determine if the preset field of the normalized log contains a critical system command, and then the critical system command contained in the application field and the preset field in the normalized log is processed.

[0026] Optionally, the extracting a noun and a timestamp of a request thereof from a preset field of the normalized log using the custom extension rule as a result entity includes:

[0027] Using the custom extension rule to mark the preset field appearing in the normalized log;

[0028] Part-of-speech analysis is performed in combination with natural language processing to extract a noun and a timestamp of the request from the preset field as a result entity.

[0029] Optionally, for each normalized log, using a common field to establish a mapping relationship between the entity and a node in the traceability graph to obtain a target node associated with the entity in the traceability graph includes:

[0030] For each normalized log, select the name and the timestamp of the request as the common fields;

[0031] If the name of the entity is the same as the name of the node in the traceability graph, and the difference in the timestamps requested between the two is within a preset range, the entity is mapped to the corresponding node in the traceability graph, and the corresponding node in the traceability graph is the target node associated with the entity in the traceability graph.

[0032] Optionally, the target node is used as a trigger point, a random walk-based algorithm is used to capture the context of the target node, and a tracing subgraph starting from the target node is extracted for each target node, including:

[0033] Performing random walk with the target node as a trigger point, and recording the access frequency of the node during the random walk;

[0034] The passing nodes are sorted according to their access frequencies, the nodes sorted before a second preset value are retained as the context of the target node, and for each target node, a tracing subgraph starting from the target node is extracted.

[0035] In a second aspect, an embodiment of the present invention provides a device for reducing a provenance graph, the device comprising:

[0036] An acquisition unit, used to acquire application logs of the interaction between the system and the outside world by performing co-occurrence analysis on the historical operation data of the host;

[0037] An extraction unit, configured to parse and convert the application log into a normalized log according to a preset field format, and extract entities from the normalized log using a predefined rule set;

[0038] An establishing unit is used to establish, for each normalized log, a mapping relationship between the entity and a node in a traceability graph using a common field to obtain a target node associated with the entity in the traceability graph, wherein the traceability graph is generated by capturing the information flow of a system kernel using a traceability tool;

[0039] A capture unit is used to use the target node as a trigger point, adopt a random walk-based algorithm to capture the context of the target node, extract a tracing subgraph starting from the target node for each target node, and combine all tracing subgraphs to form a reduced tracing graph.

[0040] Optionally, the acquiring unit includes:

[0041] An analysis module, used to obtain a list of co-occurring applications from applications that interact between the system and the outside world by performing co-occurrence analysis on historical operation data of the host;

[0042] The collection module is used to select a target application whose co-occurrence frequency ranks before a first preset value from the co-occurrence application list, and collect logs of the target application.

[0043] Optionally, the collection module is specifically used to collect application logs of the target application by querying the process corresponding to the target application interacting with the outside world on the host;

[0044] The collection module is further specifically used to use an audit tool to collect system calls of a target application for which application logs cannot be obtained, so as to obtain the audit log of the target application.

[0045] Optionally, the extraction unit includes:

[0046] A parsing module, used for parsing the application log to obtain data information of a preset field from the application log;

[0047] The normalization module is used to normalize the data information of the preset fields into a log table according to a preset field format to form a standardized log.

[0048] Optionally, the predefined rule set includes a default rule and a custom extended rule, and the extraction unit is specifically used to use the default rule to determine if a preset field of the normalized log contains a known attack signature and / or a critical system command, and then extract a matching field from the normalized log as a precise entity;

[0049] The extraction unit is specifically further configured to use the custom extension rule to extract a noun and a timestamp of its request from a preset field of the normalized log as a result entity.

[0050] Optionally, the extraction unit is further configured to use the default rule to determine if the preset field of the normalized log contains a known attack signature, then check whether the application field in the normalized log matches the known attack signature, and if so, extract the application field as a precise entity;

[0051] The extraction unit is further specifically used to use the default rule to determine if the preset field of the normalized log contains a critical system command, and then process the critical system commands contained in the application field and the preset field in the normalized log.

[0052] Optionally, the extraction unit is further configured to use the custom extension rule to mark a preset field appearing in the normalized log;

[0053] The extraction unit is further specifically used to perform part-of-speech analysis in combination with natural language processing to extract a noun and its requested timestamp from the preset field as a result entity.

[0054] Optionally, the establishing unit includes:

[0055] A selection module is used to select the name and the requested timestamp as common fields for each normalized log;

[0056] A mapping module is used to map the entity to a corresponding node in the traceability graph if the name of the entity has the same name as a node in the traceability graph and the difference in the timestamps requested between the two is within a preset range. The corresponding node in the traceability graph is the target node associated with the entity in the traceability graph.

[0057] Optionally, the capture unit includes:

[0058] A recording module, used to perform random walk with the target node as a trigger point, and record the access frequency of the node during the random walk;

[0059] A sorting module is used to sort the passing nodes according to their access frequencies, retain the nodes sorted before a second preset value as the context of the target node, and extract a tracing subgraph starting from the target node for each target node.

[0060] In a third aspect, an embodiment of the present invention provides a storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method described in the first aspect.

[0061] In a fourth aspect, an embodiment of the present invention provides a device for reducing a provenance graph, including:

[0062] one or more processors;

[0063] a storage device for storing one or more programs,

[0064] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.

[0065] As can be seen from the above content, the method and device for reducing the traceability graph provided by the embodiment of the present invention obtains the application log of the system interacting with the outside world by co-occurrence analysis of the historical operation data of the host, and parses and converts the application log into a normalized log according to a preset field format, and extracts entities from the normalized log using a predefined rule set. For each normalized log, a mapping relationship between the entity and the node in the traceability graph is established using a common field to obtain the target node associated with the entity in the traceability graph. The traceability graph is generated by capturing the information flow of the system kernel using a traceability tool, and then the target node is used as a trigger point, and an algorithm based on random walk is used to capture the context of the target node. For each target node, a traceability subgraph starting from the target node is extracted, and all traceability subgraphs are combined to form a reduced traceability graph. Compared with the reduction method based on graph structure and the reduction method based on graph embedding in the prior art, the embodiment of the present invention captures the context of the node by establishing a mapping relationship between the entity in the application log and the node in the traceability graph, and uses the generated subgraph as the input of the attack tracing detection method, thereby effectively alleviating the problem of huge traceability graph data volume and improving the detection accuracy of attack tracing.

[0066] In addition, the technical effects that can be achieved by this embodiment include:

[0067] (1) The provenance graph reduction method adopts a random walk-based algorithm to effectively capture the context of entities in the provenance graph, and its computational complexity is greatly reduced.

[0068] (2) The provenance graph reduction method is extensively evaluated in simulated APT scenarios and real-world APT scenarios using different APT detection methods. The results show that the provenance graph reduction method in the embodiment of the present invention achieves a high compression rate.

[0069] (3) The provenance graph reduction method is helpful for APT detection and can be integrated with different APT detection methods. It also improves the accuracy of traditional anomaly detection methods and the most advanced APT detectors.

[0070] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0072] Figure 1 A flowchart of a method for reducing a traceability graph provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of the overall working process of the traceability graph reduction method provided by an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of data distribution before and after the traceability graph is reduced according to an embodiment of the present invention;

[0075] Figure 4a-4b A schematic diagram of the runtime overhead of APT-1 based on the experimental scenario provided for the embodiment of the present invention;

[0076] Figure 5 A block diagram of a device for reducing a provenance graph provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0077] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0079] The present invention provides a method and device for reducing a traceability graph, which captures the context of the node by establishing a mapping relationship between the entity in the application log and the node in the traceability graph, and uses the generated subgraph as the input of the attack traceability detection method, thereby solving the problem in the prior art that the amount of traceability graph data is huge and affects the accuracy of attack traceability detection. Advanced persistent attacks APT are difficult to detect due to their novelty, concealment and long-term characteristics, posing a great threat to Wang Lina's safety. The detection method based on the traceability graph can effectively detect APT, but due to the large amount of data, it is impossible to achieve good accuracy and effect. In order to reduce the amount of data, the existing traceability graph reduction methods are all general or forensic analysis-oriented reduction algorithms, which cannot achieve good reduction effects in APT detection scenarios. Here we propose an efficient application log-guided provenance graph reduction method, which uses application logs to identify entities related to attacks, and then uses a random walk algorithm to explore the context of nodes matching the entities in the provenance graph, and generates a reduced subgraph based on this. Experimental evaluation shows that this method has a good data reduction effect, with a maximum reduction rate of 150.46 times, and only 16% short-term overhead. This method can be integrated with different APT detection methods, and can improve the performance of the current leading APT detection methods by an average of 14.3%, and can improve the performance of traditional attack detection methods in APT detection scenarios by up to 68.42%.

[0080] Before introducing the specific implementation process of the present invention, in order to facilitate understanding of the implementation process, the method of the embodiment of the present invention mainly refers to the following factors:

[0081] 1. The attacker relies on processes, files, and sockets in the target system to launch attacks. In almost all scenarios, the attacker will inevitably establish a connection with the target system and interact with the system's processes and files.

[0082] 2. Not all applications are related to the attack. There are many nodes representing processes, files, and sockets in the traceability graph. Which nodes can be removed without affecting the detection effect? ​​Here, we refer to the APT kill chain and find that attackers only use public-facing applications, such as web servers and download tools. Therefore, the target can be narrowed down to public-facing applications.

[0083] 3. Application logs reflect the behavior of the process. Because the application is the entry point of the attack, there must be a lot of high-level semantic information related to the attack in the application log. In practice, it is found that the application log has many of the same entities as its underlying traceability graph, which means that the entities in the application log can be mapped to the nodes in the traceability graph. Therefore, with the help of the application log, it is possible to find out which nodes of the traceability graph are related to the detection.

[0084] 4. To avoid detection, the actual attack process will only last for a short time. Although APT attackers may hide for a long time after infiltrating the system, when they actually launch an attack, they usually choose to complete it in the shortest possible time to avoid being detected. Therefore, only a small context related to the attack entry needs to be captured, thus achieving a high reduction rate.

[0085] Generally speaking, APT detection can be performed by only focusing on the nodes related to the application program interacting with the interface and their limited context in the traceability graph. The embodiments of the present invention are described in detail below.

[0086] Figure 1 A flowchart of a method for reducing a traceability graph provided by an embodiment of the present invention. The method may include the following steps:

[0087] S100: Obtain application logs of the interaction between the system and the outside world by performing co-occurrence analysis on historical operation data of the host.

[0088] In today's enterprise environment, microservice architecture has been widely used, and each host is likely to run only one or a few applications that interact with the outside world. In other words, you only need to pay attention to the processes corresponding to the few applications that interact with the outside world on each host and collect their logs. However, collecting logs only for public-facing applications will lead to information loss, because there may be other applications in the system that implicitly depend on the processes of applications that interact with the outside world. Therefore, co-occurrence analysis is performed through the historical running data of the host to obtain the top K applications that co-occur with applications that interact with the outside world, and their logs are collected for subsequent analysis.

[0089] Specifically, we can perform co-occurrence analysis on the historical operation data of the host, obtain a list of co-occurring applications from the applications that the system interacts with the outside world, further select the target application whose co-occurrence frequency ranks before the first preset value from the list of co-occurring applications, and collect the logs of the target application. Here, co-occurrence analysis is considered to be the selection of applications that co-occur in the process of the system interacting with the outside world within the same time range, which can reasonably expand the range of possible attack paths and consider as many contexts as possible.

[0090] The logs here include application logs and audit logs. Specifically, in the process of collecting the logs of the target application, the application logs of the target application can be collected by querying the process corresponding to the target application that interacts with the outside world on the host. For the target application that cannot obtain the application log, the audit tool is used to collect the system calls of the target application to obtain the audit log of the target application.

[0091] Exemplarily, for target applications for which application logs cannot be obtained, such as commercial software that does not provide application logs and applications that delete logs for production environment performance, audit tools (eg, Linux audit framework) are used here to collect their system calls.

[0092] Specifically in this step, a list of applications that need attention and their logs are obtained, including application logs and audit logs. The specific log examples are shown below.

[0093] Application log example:

[0094]

[0095] Audit log example:

[0096]

[0097] In actual application scenarios, the above process is mainly implemented based on Python. First, the traceability graph and application logs are collected from alternative data sources. In practical applications, it is believed that predefined rules can also be selected according to different scenarios. In an embodiment of the present invention, CamFlow is used as a reference implementation. It should be noted that for applications that do not have carefully designed logs, it is conservatively assumed that system call logs related to system objects (such as processes, files, and sockets) can also be utilized. Xanthus is an automated tool that orchestrates virtual machines to reproduce scenarios and generate real logs. This function is used to establish three real attack data sets. However, Xanthus only supports Linux and macOS and cannot fully meet the requirements of cross-platform and versatility. Therefore, in the implementation process of the embodiment of the present invention, Xanthus is expanded and modified, and some practical options are added to make it compatible with Windows to meet the requirements of cross-platform and versatility.

[0098] In order to find out the processes that need to be tracked during the whole system tracing process, the embodiment of the present invention introduces a possible solution, namely, target process co-occurrence analysis. Its purpose is to abstract other processes that may be active at the same time as the attack entry process, so as to reasonably expand the scope of suspicious processes. Here, atop is selected as the data source to be analyzed. Specifically, the logs extracted by the atop parser are used as the corpus. The process names in the logs are then divided into corresponding units with the same timestamp. This allows the construction of a co-occurrence matrix of process names and the generation of a node dictionary and an edge dictionary. The node dictionary contains node names and node weights (frequency), and the edge dictionary contains starting points, end points, and edge weights (frequency). After filtering out system processes (systemd, auditd, etc.), the processes with the largest K edge weights among all weights among all edges related to the target process are taken as processes that need to be tracked.

[0099] S110: Parsing and converting the application log into a normalized log according to a preset field format, and extracting entities from the normalized log using a predefined rule set.

[0100] Among them, the preset field format is equivalent to the filling format set for the fields in the log. In the process of parsing and converting the application log into a normalized log according to the preset field format, the application log can be parsed to obtain the data information of the preset field from the application log, and then the data information of the preset field can be normalized to the log table according to the preset field format to form a normalized log.

[0101] Exemplarily, an example of converting application logs into a standardized format is shown in Table 1 below. The header shows the fields of the standardized log, {Time, PID, PNAME, IP, Protocol, Path, Msg}. Time represents the timestamp of the request, PID represents the process number when the application is running, PNAME represents the process name of the application, IP represents the IP address from which the request comes, Protocol represents the protocol used for this request, and Path represents the system file path or URL involved. All application logs (including corresponding audit logs) are parsed and converted into standardized logs. In addition, network traffic information is used to fill in the Msg field.

[0102] Table 1. Normalized log example

[0103] No Time PID PNAME IP Protocol Path Msg 1 t1 p1 apache2 IP1 HTTP a.com SELECT * FROM tablel 2 t2 p2 smbd IP2 SMB / usr / sbin / smbd / =′nohup telnetd-l / bin / sh-p 4444` 3 t3 p3 TermServices.exe IP3 RDP - \x02\xf0\x80\x7f\x65\x82\x01\x94\x04

[0104] Here, the normalization process of data information is further explained with the application log example and audit log example in the above text. In the second line of the application log example above, the log of apache2 shows that at time t1, a request from IP1 is sent to apache2 and attempts to access a.com. In lines 1-4 of the audit log example above, the audit log shows p1 of apache2's pid, and the requested network traffic information is SELECT*FROM tablel. So we can get normalized log 1, which is the first line in Table 1. In line 7 of the application log example above, the log of smbd shows that at time t2, a request from IP2 is sent to smbd. In lines 5-8 of the audit log example above, the relevant audit log shows that smbd's pid is p2, smbd's path is smbd and the requested network traffic information is / =`nohup telnetd-1 / bin / sh-p 4444`. So we can get normalized log 2, which is the second line in Table 1. Because TermServices.exe has no application log, only its audit log is used here. In lines 10-11 of the audit log example above, the audit log shows that the pid of TermServices.exe is p3, a request from IP3 is sent and the network traffic information of the request is \x02\xf0\x80\x7f\x65\x82\x01\x94\x04, and the normalized log 3 can be obtained, which is the third line in Table 1.

[0105] Furthermore, here we can develop 14 parsers specifically for application logs. If a new application is added, we only need to write a new parser for that application. Because the application log format usually does not change much, the modification cost is relatively low.

[0106] Specifically, after obtaining the normalized logs, entities are extracted from the normalized logs using a set of predetermined rules. These entities are extracted from logs of applications interacting with the outside world that may become targets of attackers. Therefore, these entities are likely to be related to the attackers.

[0107] In practical applications, the main goal of application log parsing is to generate standardized logs that may expose detection-related entities. The existence of multi-level logs requires multiple data parsers corresponding to different data sources. In the traceability graph reduction method provided by the present invention, 14 regular expression-based parsers are used without the need for more complex context-free grammars or even context-sensitive grammars.

[0108] During the parsing process, it is considered that some fields with the same meaning are often different in different log formats, so a generalization operation is implemented to further normalize the logs. For example, version numbers, timestamp formats, and paths are usually variable information even if they have the same meaning. The format of the normalized log has been mentioned before. The Path field and Message field of the normalized log usually contain system commands and files, which are usually the key to extracting detection-related entities.

[0109] Among them, the predefined rule set includes default rules and custom extended rules. In the process of extracting entities from the normalized log using the predefined rule set, the default rule can be used to determine if the preset field of the normalized log contains a known attack signature and / or a key system command, then the matching field is extracted from the normalized log as a precise entity, otherwise, the custom extended rule is used to extract the noun and the timestamp of its request from the preset field of the normalized log as the result entity. In the process of extracting precise entities, the default rule can be used to determine if the preset field of the normalized log contains a known attack signature, then check whether the application field in the normalized log matches the known attack signature, and if so, extract the application field as a precise entity; the default rule is used to determine if the preset field of the normalized log contains a key system command, then process the application field in the normalized log and the key system command contained in the preset field. In the process of extracting result entities, the custom extended rule can be used to mark the preset field that appears in the normalized log; and the part-of-speech analysis is performed in combination with natural language processing to extract the noun and the timestamp of its request from the preset field as the result entity.

[0110] An example of the process is to use the following predefined rules, applied sequentially to each normalized log:

[0111] 1. If the field Msg contains the signature of a known attack, the PNAME, Protocol, and Path (if not empty) fields will be checked to see if they match the known attack. If they match, the PID, PNAME, and Path (if not empty) will be extracted as entities. For example, the Msg in the normalized log 3 of Table 1 is part of the message from the MS12-020 vulnerability. Then it is found that PNAME is TermService and Protocol is rdp, which is consistent with the known attack, so the extracted entity is {p3, TermService}.

[0112] 2. If the Msg field contains some critical system commands, then PID, PNAME, Path, and the critical system commands in Msg will be processed. This step comes from the intuition that "attackers will use system built-in commands to manipulate system entities." Therefore, if the system is attacked, the fields Path and Msg in the normalized log may contain some system commands, such as cp, rm, and ls. For example, in the normalized log 2 of Table 1, / bin / sh in Msg is one of the critical system commands, and it seems that someone is trying to manipulate it, so the extracted entity is {p2, smbd, / usr / sbin / smbd, / bin / sh}.

[0113] 3. If the above two conditions are not met, then extract the entities appearing in the Msg field. Msg contains semantics about user behavior, sometimes in a structured text format. Therefore, natural language processing (NLP) can be used to tag Msg and perform part-of-speech analysis to extract nouns and their Time from Msg as entities. For example, table1 in the Msg of the normalized log 2 of Table 1 is a noun. So the resulting entity is {p1, table1}.

[0114] As can be seen above, the first two rules are for specific problems, and the last one is the default rule (or catch-all rule). Problem-specific rules are designed to extract a precise set of entities from normalized logs, and the default rule is used when all problem-specific rules are not applicable. In addition, the rule set is extensible, and new rules can be easily added here.

[0115] It can be understood that the extraction of detection-related entities is crucial to the traceability graph reduction method of the present invention and greatly affects the reduced traceability graph. At the same time, the definition of detection-related entities relies heavily on predefined rules. When a file is read or written using a system command, the file name is first assumed to be a detection-related entity. In addition to the file name disclosed by the traceability graph and the application log, the table entity disclosed by the application message is also considered to be a detection-related entity. Specifically, as mentioned above, the application message contains content like SELECT*FROM table1. It is obvious that the operation on table 1 is likely to be malicious. As for normal application messages, they are also considered to contain semantic information about the user's expected behavior. The application layer messages are regarded as sentences. In order to extract entities in the application layer log, Spacy is used because it contains a fast entity recognition model. In addition to collecting detection-related entities, their timestamps will be recorded, which will later be used to match the trigger points on the traceability graph and the names of the detection-related entities.

[0116] S120: For each normalized log, a mapping relationship between the entity and the node in the traceability graph is established using a common field to obtain a target node associated with the entity in the traceability graph.

[0117] The traceability graph is generated by using the traceability tool to capture the information flow of the system kernel. Here, the traceability graph is a directed acyclic graph with time as the direction, representing the causal relationship between the subject (process, thread, etc.) and the object (file, registry, network socket, etc.) in the system. No matter how long the time interval between two events is, the traceability graph can express the causal relationship between the two events, so security experts can use it to complete attack investigations.

[0118] For each normalized log, after obtaining its corresponding entity, the nodes directly related to these entities can be found in the traceability graph according to the name and time of the entity. These nodes are called trigger points. Specifically, for each normalized log, the name and requested timestamp can be selected as common fields; if the name of the entity is the same as the node in the traceability graph, and the difference in the requested timestamp between the two is within a preset range, the entity will be mapped to the corresponding node in the traceability graph, and the corresponding node in the traceability graph will be the target node associated with the entity in the traceability graph.

[0119] S130: Taking the target node as a trigger point, a random walk-based algorithm is used to capture the context of the target node, extracting a tracing subgraph starting from the target node for each target node, and combining all tracing subgraphs to form a reduced tracing graph.

[0120] Here, based on the trigger point, a random walk-based algorithm (as shown in Algorithm 1 below) can be used to capture the context of the trigger point. For each trigger point, Algorithm 1 outputs the subgraphs extracted from these points in the original provenance graph, and then combines the subgraphs extracted from all trigger points to form a reduced provenance graph.

[0121] Algorithm 1: Graph Reduction Algorithm

[0122]

[0123] The dependency explosion problem can be effectively avoided by using a random walk-based algorithm. It is noted here that many detection methods rely on the stability of the data, and random walks obviously cannot meet the requirements. Therefore, when designing the algorithm, the weight of the trigger point neighbors is increased by retaining nodes with higher frequencies during random walks to capture as many contexts of the entity as possible. This is also consistent with what was mentioned above, that the actual attack only lasts for a short time to avoid being discovered, and usually only the context of the entity related to the attack needs to be focused on.

[0124] In addition, some applications may not provide logs, or the information in the logs is insufficient, resulting in few matched trigger points. At this time, the generated subgraph may not support the subsequent APT detection process. To solve this problem, when the number of trigger points is found to be less than the predefined threshold, Algorithm 1 will be downgraded to breadth-first search (BFS) and possibly capture the context of the entity more.

[0125] Specifically, the target node can be used as a trigger point for random walking, and the access frequency of the nodes passed through can be recorded during the random walking process; the nodes passed through can be sorted according to their access frequency, and the nodes sorted before the second preset value can be retained as the context of the target node, and the tracing subgraph starting from the target node can be extracted for each target node. For example, during the random walking process, the trigger point is point v, and the surrounding nodes are repeatedly randomly accessed. There will be a probability of 1 / q to access the x1 node, a probability of 2 / q to access the x2 node, and a probability of 3 / q to access the x3 node. The node x3 with the highest probability of being visited is retained as the context of the target node, forming a tracing subgraph starting from the target node.

[0126] In practical applications, after extracting (detection-related) entities, their common fields are used to map them to the provenance graph of the entire system. The currently selected common fields are name and timestamp. That is, if a (detection-related) entity has the same name as a node in the provenance graph, and the timestamp difference between them is within a certain range, the entity is mapped to that node. This node will be selected as the starting point for the next step of graph reduction. The above process is repeated until all entities are mapped and all nodes they are mapped to are selected. The selected nodes are called "trigger points".

[0127] The specific graph reduction process is performed using NetworkX, a graph processing library written in Python. Inspired by the node2vec algorithm, the trigger point is used as the starting point to generate multiple node sequences, and the frequency of the node is retained in the frequency dictionary. After sorting in descending order, only the nodes with the highest frequency are retained. Considering that the context of the node is usually more important, when the number of trigger points is below a certain threshold, the traceability graph reduction algorithm in this application degenerates into a breadth-first search (BFS) to retain more context.

[0128] In summary, in order to solve the problem of huge amount of traceability graph data, the relevant technical solutions can use many traceability graph reduction methods, but there are still many shortcomings. First of all, since these methods are general or oriented to forensic analysis rather than APT detection, the efficiency of traceability graph reduction is relatively low, usually only 10x reduction rate, that is, reduced to 1 / 10 of the original data. The general or forensic analysis-oriented provenance graph reduction methods require that the reduced provenance graph can still retain the causal relationship (that is, retain the connectivity of the provenance graph), which will generate redundant information; secondly, these provenance graph reduction methods rely on the entire provenance graph, which requires memory and hard disk space of the same size as the image data. However, due to the huge scale of graph data in APT scenarios, the practicality of the methods is relatively low; thirdly, these provenance graph reduction methods are only designed for certain specific scenarios, and the universality of the scenarios is cross-cutting. For example, NodeMerge learns the file access mode of the process to merge redundant nodes, but since the file access mode in each scenario is different, this method may not be applicable to other scenarios. Finally, almost all methods are based on the reduction of the provenance graph itself, without making full use of other information that can be introduced to obtain a higher reduction rate.

[0129] However, an ideal data reduction method should have the following characteristics: high reduction rate, low overhead, and no negative impact on the APT detection effect. It is generally believed that a reduction algorithm tailored for APT detection (rather than a general or forensic analysis-oriented) can achieve a higher reduction rate because it can reduce more information irrelevant to detection. Outside the CIA, experience has shown that attackers can only use applications that interact with the outside world to launch attacks. Application logs contain high-level semantic information related to APT attacks and can be used for detection. In addition, in the same system, application logs and traceability graphs share many of the same entities. Therefore, the high-level semantics related to APT detection in application logs can be integrated into the traceability graph to achieve more effective data reduction. An embodiment of the present invention proposes an application log-guided traceability graph reduction method for efficient APT detection, which removes redundant information in the traceability graph, can achieve a high reduction rate, and can maintain the accuracy of the APT detection method. And use the following premise assumptions: First, assume the integrity of the data (mainly application logs and traceability data). That is, it is assumed that the attacker can destroy the application, but the attacker cannot modify the data that has been collected and stored, ensuring the integrity of the data beyond the text range. Secondly, it is assumed that the system has not been attacked before the provenance graph compression method is deployed. Finally, it is assumed that the hardware, kernel, running programs and implementation of the provenance graph compression method are correct. Specifically, the provenance graph reduction method first collects the logs of the applications that interact with the outside world and the provenance graph of the system, then extracts entities from the application logs and establishes a mapping relationship between them and the nodes in the provenance graph. Then, starting from the nodes in the provenance graph that have established a mapping relationship with the entities, a random walk-based algorithm is used to capture the context of the nodes, that is, the subgraph, which corresponds to the context of the entities in the application log. Finally, the generated subgraph is used as the input of the APT detection method. In this way, the parts of the provenance graph that are not related to APT can be deleted and the provenance graph can be reduced to a size suitable for APT detection.

[0130] In actual application scenarios, the overall working process of the traceability graph reduction method provided by the embodiment of the present invention is as follows: Figure 2 As shown, the first step is log filtering: co-occurrence analysis is performed on the host for applications that interact with the outside world, applications that interact with the outside world or a set of applications related to the outside world are found, and then application logs are obtained; then log parsing is performed: the logs are parsed into a standard format; further entity extraction is performed: entities are extracted from the normalized logs according to predefined rules; finally, entity mapping and graph reduction are performed: the extracted entities are mapped to nodes in the traceability graph, and their contexts are captured based on the biased walking algorithm proposed in the present invention, and finally a subgraph is filtered out from the original traceability graph, and the obtained subgraph can be used as the input of the APT detection method.

[0131] Furthermore, in order to verify the traceability effect achieved by the traceability graph reduction method in the embodiment of the present invention, a pre-set attack data set can also be used to evaluate the effect of traceability graph reduction. Specifically, in the process of data set selection, three types of attacks were implemented based on detailed reports of real-world APT activities and collected logs. In order to simulate APT scenarios, the typical APT kill chain can be strictly followed, including 7 main stages, namely collecting information, making plans, transmitting tools, infiltration, installing backdoors, controlling hosts, and subsequent operations. The specific generated data sets are as follows, and the detailed data set features are shown in Table 2:

[0132] 1. Supply chain dataset. In this dataset, two common supply chain attack scenarios are simulated.

[0133] SC-1 scenario. The attacker first identified a remote enterprise CI server that frequently used wget to download Debian packages from different repositories. The attacker discovered that the wget version on the server was 1.17, which was vulnerable to arbitrary remote file uploads (CVE-2016-4971) when the victim requested a malicious URL to the compromised server. A common remote access trojan was embedded in the Debian packages and compromised one of the repositories. As a result, any request to download a seemingly legitimate package was unknowingly redirected to the attacker's FTP server, which had the package embedded with the trojan. The CI server installed the malicious trojan software while downloading and installing the compromised package. The trojan then established a C&C channel with the attacker, creating a reverse TCP shell session on the CI server. Ultimately, the attacker modified the CI server configuration settings to gain control over the CI deployment output.

[0134] Scenario SC-2. The scenario setup is similar to SC-1. However, the attacker exploited a different vulnerability originating from Bash version 4.3. In short, the vulnerability allows attackers to execute arbitrary code in Bash scripts (CVE-2014-6271) by adding a trailing string after a function definition. Both of the above scenarios represent a supply chain breach.

[0135] 2. APT-1 dataset. Compared with the above two attacks, the attack scenario of APT-1 is more complex. The attacker first used a vulnerability scanner to find a SQL injection vulnerability in the corporate website, thereby gaining control of the web server. The attacker then used the web server to perform a slow intranet scan and found that the target OA server had an EternalBlue (MS17-010) vulnerability. The attacker then used the vulnerability to log in to the OA server. The attacker obtained root access to the OA server by uploading and executing MS15-015. Finally, key information was stolen and returned.

[0136] Table 2. Dataset characteristics

[0137]

[0138] Specifically, all the experiments above were run on a Linux host, using an 8-core Intel Xeon Cascade Lake 8255C CPU (2.50 GHz) and 32 GB RAM. In order to systematically evaluate the provenance graph reduction method of the embodiment of the present invention, the accuracy, precision, recall and F1 value before and after reduction are compared to verify the preservation of the provenance graph after reduction. In addition, the runtime overhead is also evaluated to determine whether the overhead of the provenance graph reduction method of the embodiment of the present invention is reasonable.

[0139] Furthermore, in order to verify whether the provenance graph reduction method of the embodiment of the present invention has a low negative impact on the performance of the APT detector. The most advanced provenance graph-based APT detection method UNICORN is selected here. UNICORN is a most advanced APT-based anomaly detection algorithm using provenance graphs. It uses a novel graph sketching technology to summarize the provenance graph into an incrementally updateable and fixed-size structure, through which long-term behaviors and their evolution can be modeled. Therefore, this method can effectively detect the detection results of APT when the provenance graph changes.

[0140] Regarding the data reduction rate, in order to evaluate the effectiveness of the provenance graph reduction method of the embodiment of the present invention in data reduction, the provenance graph reduction method of the embodiment of the present invention is applied to the above three data sets and compared with the other four data reduction methods LogGC, CPR, FD / SD, and GS / SS. The reduced data size and reduction rate are shown in Table 3. On average, the reduction rate of the provenance graph reduction method of the embodiment of the present invention is 78.14x, which is higher than that of other methods.

[0141] Table 3. Reduction results

[0142]

[0143] Since we took the size of the application log into account when calculating the reduction rate, the reduction rate of the APT-1 dataset seems to be lower than that of other methods. If only the size of the reduced provenance graph itself is considered, the reduction rate of APT-1 will reach 98.74x, which is much higher than the current results. The same is true for other datasets. In fact, in enterprises, application logs are usually collected for analysis, such as debugging, user information statistics, and so on. Therefore, the provenance graph reduction method of an embodiment of the present invention will have a higher reduction rate in the real world.

[0144] In addition, for the SC-2 and APT-1 datasets, the reduction rate of this method is relatively low. Considering that the reduction rate is inversely proportional to the number of entities matched between the application log and the provenance graph. In the SC-2 dataset, the Bash log contains almost all user operations, resulting in more matching entities and a lower reduction rate. For the APT-1 dataset, the scenario is much more complicated, and there will be more matching entities, so the reduction rate is lower. However, since the provenance graph reduction method of an embodiment of the present invention can be used together with other reduction methods, redundant entities can be removed by applying methods such as LogGC and NodeMerge, which can delete redundant entities in the provenance graph.

[0145] As for the impact on APT detection methods, the provenance graph reduction method of the embodiment of the present invention and other four reduction methods (LogGC, CPR, FD / SD and GS / SS) are applied to three data sets. Table 4 shows the changes in evaluation indicators (accuracy, precision, recall, F1-Score) before and after UNICORN reduces each data set. From Table 4, it can be seen that other methods have more or less negative impact on performance, but the provenance graph reduction method of the embodiment of the present invention has been improved for all standards.

[0146] Table 4. Evaluation results on UNICORN

[0147]

[0148] Since the attacker knows the target to be attacked in advance, and generally knows at least the general process of the target system supply chain, the attacker no longer needs to search for vulnerabilities in the system. In this case, the traces left by the attacker in the system will be greatly reduced, which brings some difficulties to the detection of supply chain attacks. In addition, the APT attacks that can be launched in the APT-1 dataset scenario are covert and complex. This is one of the reasons why UNICORN performs poorly on the original provenance graph. Because the provenance graph reduction method of the embodiment of the present invention retains the information related to the detection while deleting the information irrelevant to the attack, this has a positive impact on the performance of UNICORN.

[0149] Furthermore, in order to better understand the impact of APT detection performance, the well-known dimensionality reduction method t-SNE can be used to visualize the graph data before and after dimensionality reduction. The specific data distribution before and after the traceability graph is as follows: Figure 3 As shown, the data distribution before reduction is represented by a triangle shape, and the data distribution after reduction is represented by a square shape. Figure 3As can be seen in the figure, the data distribution after reduction is more distinguishable, which improves the accuracy of the detector. The original graphs with and without attacks are overlapping and inseparable. After reduction, the attack graph is in the shape of a straight line and a plane, while the normal graph is in the shape of a straight line and a plane overlapping with the attack graph. Although the overlap problem is not completely eliminated, a reasonable hyperplane can still be found to separate most of the attack data from the normal data. It is considered acceptable here that the existence of overlap is acceptable because some benign operations will definitely be performed in network attacks.

[0150] Regarding runtime overhead, although the main purpose of the provenance graph reduction method provided in the embodiment of the present invention is to retain subgraphs related to detection, it is better to maintain a low runtime overhead. In this embodiment, biased random walk is used as the first selection method to generate subgraphs. Specifically, the Alias ​​method can be used, which is a discrete sampling method with a time complexity of O(1). At the same time, it cannot be ignored that when there are fewer trigger points, in order to better preserve contextual information, the subgraph generation algorithm will degenerate into a BFS algorithm, and the fixed-step BFS traversal is obviously an algorithm with a time complexity of O(1). In summary, the overall time complexity of the provenance graph reduction method can be guaranteed to be O(N).

[0151] Here, the CPU and memory usage of the provenance graph reduction method in the embodiment of the present invention are measured to gain a deeper understanding of whether the provenance graph reduction method has a reasonable overhead. Figure 4a-4b The following figure shows the runtime overhead of the experimental scenario based on APT-1, which reduces the traceability map by 1.8GB in one round. In terms of CPU utilization, we mainly focus on the average CPU utilization of UNICORN running with the baseline configuration over a long period of time. Figure 4a It can be seen that the average CPU utilization is around 16%. Figure 4b As shown, the provenance graph reduction method in the embodiment of the present invention requires an average of 1320MB of memory during runtime, and the maximum memory space occupied is 5,773MB, which is acceptable in most cases. In general, the runtime overhead is very low.

[0152] It is understandable that in addition to UNICORN, other graph-based anomaly detection methods are also experimented with for the support of traditional detection methods.

[0153] 1. ODDBALL: ODDBALL is a graph-based anomaly detection algorithm that finds anomalies in static weighted graphs based on the egonet model. It is a fast, unsupervised method to detect abnormal nodes in weighted graphs and assign an "outlier value" to each node, which can be further used to evaluate the degree of abnormality of the node.

[0154] 2. Anomaly detection based on graph embedding: Traditional graph embedding is a method of finding a low-dimensional representation of a graph while maintaining certain properties. There have been some studies that use graph embedding for anomaly detection. For graph embedding methods, Node2Vec, LINE, and SDNE are implemented here. The goal of Node2Vec is to learn the mapping of nodes to the low-dimensional space of features by maximizing the probability of occurrence of subsequent nodes in a fixed-length random walk. LINE explicitly defines two functions for the first-order and second-order approximations, respectively, and minimizes the combination of the two. SDNE uses a deep autoencoder to maintain first-order and second-order network neighbors.

[0155] LOF and IF are further selected to generate outliers after graph embedding. LOF proposes to identify density-based local anomaly factors by assigning a degree to each object that it is likely to be an outlier, which is called the local outlier factor (LOF) of the object. IF Simply put, the isolation forest method builds a set of itrees for a given dataset, and then the anomalies are those instances with a shorter average path length on the itree.

[0156] In order to divide the data into attack data and normal data, the number of outliers is used as an indicator, and XGBoost is used as a binary classification algorithm. XGBoost is a state-of-the-art method for solving many binary classification problems, and is currently available in an open source package. It introduces a novel sparsity-aware algorithm for parallel tree learning, while the theoretically reasonable weighted quantile sketch process enables the processing of instance weights in approximate tree learning.

[0157] Table 5 fully illustrates that the provenance graph reduction method of the embodiment of the present invention is applicable to various graph-based anomaly detection methods. At the same time, previous work has explained that traditional methods may not be suitable for APT attack detection tasks because they have poor modeling capabilities for long-term system behaviors. However, the method of reducing graphs proposed here can promote the performance of traditional methods and also expand the versatility of provenance graph reduction. As shown in the above experiments, static graphs and streaming graphs are considered in the present invention, which shows that provenance graph reduction does not require a specific type of graph. In addition, the provenance graph reduction method of the embodiment of the present invention is designed to be applicable to almost all types of logs. Due to the presence of pre-built parsers, the provenance graph reduction method has almost no requirements on the format of the log. In addition, in an enterprise environment, the runtime overhead is reasonable. Therefore, the provenance graph reduction method of the embodiment of the present invention can be applied to various scenarios and enterprise environments. In addition, the provenance graph reduction method is orthogonal to other methods, so it can theoretically be used with them to obtain a higher reduction rate.

[0158] Table 5. Support for traditional detection methods

[0159]

[0160] The effect of the provenance graph reduction method in the embodiment of the present invention is systematically evaluated in simulated APT scenarios and APT scenarios from the real world. The evaluation results show that the provenance graph compression method provided in the embodiment of the present invention can achieve an average reduction rate of 78.14, a maximum reduction rate of 150.46, and only 16% CPU occupancy in a short period of time. In addition, the provenance graph reduction method can be integrated with different APT detection methods without affecting the performance of the detection method. In the evaluation scenario, the accuracy of UNICORN, the most advanced APT detection method, was improved by an average of 14.3%, and the accuracy of several traditional anomaly detection methods was improved by up to 68.42%. This shows that traditional anomaly detection methods can use the provenance graph reduced by the provenance graph reduction method in the embodiment of the present invention to detect APT.

[0161] The method for reducing a traceability graph provided by an embodiment of the present invention performs a co-occurrence analysis on the historical operation data of the host to obtain an application log of the system interacting with the outside world, parses and converts the application log into a normalized log according to a preset field format, and extracts entities from the normalized log using a predefined rule set. For each normalized log, a mapping relationship between the entity and the node in the traceability graph is established using a common field to obtain a target node associated with the entity in the traceability graph. The traceability graph is generated by capturing the information flow of the system kernel using a traceability tool, and then the target node is used as a trigger point, and an algorithm based on a random walk is used to capture the context of the target node. For each target node, a traceability subgraph starting from the target node is extracted, and all traceability subgraphs are combined to form a reduced traceability graph. Compared with the reduction method based on graph structure and the reduction method based on graph embedding in the prior art, the embodiment of the present invention captures the context of the node by establishing a mapping relationship between the entity in the application log and the node in the traceability graph, and uses the generated subgraph as the input of the attack traceability detection method, thereby effectively alleviating the problem of huge traceability graph data volume and improving the detection accuracy of attack traceability.

[0162] It is further explained that the provenance graph reduction algorithm in the embodiment of the present invention can also achieve more beneficial effects:

[0163] Regarding the possibility of conversion to a streaming algorithm: Although intrusion detection and forensic analysis are usually performed after the fact, since they are usually tasks with strict time limits, it is still necessary to design a real-time source graph reduction method. The source graph reduction method in the embodiment of the present invention can be directly extended to a streaming algorithm. Specifically, at any point in time, the source graph reduction method in the present invention obtains the source graph of the entire system and other multi-level system information logs, and can continuously match and generate trigger points on the graph based on the name and timestamp of the extracted entity, so that the source graph reduction method can generate subgraphs at runtime. At the same time, in the above-mentioned experiments, experiments were also conducted on UNICORN, which proved the potential of the source graph reduction method in the present invention to facilitate streaming APT detection.

[0164] Regarding the effect of filtering: The effect of process filtering may affect the performance of the traceability graph reduction method in the present invention. In a specific implementation, atop data can be used to perform co-occurrence analysis to filter out applications that should be considered. However, since atop is a coarse-grained system analysis tool, the filtering process may not be accurate.

[0165] Processing of performance bottlenecks: Modern APT attacks always generate a large amount of log data. Extracting detection-related entities from such a large amount of data is a difficult task, which places higher demands on computing performance. The implementation of the traceability graph reduction method in this invention uses SpaCy, which is good at large-scale information extraction tasks. However, there is still no guarantee that the throughput of this method will be generally applicable to all scenarios.

[0166] Based on the above embodiment, another embodiment of the present invention provides a device for reducing a provenance graph, such as Figure 5 As shown, the device comprises:

[0167] The acquisition unit 20 may be used to acquire application logs of the system interacting with the outside world by performing co-occurrence analysis on the historical operation data of the host;

[0168] The extraction unit 22 may be used to parse and convert the application log into a normalized log according to a preset field format, and extract entities from the normalized log using a predefined rule set;

[0169] The establishing unit 24 may be used to establish, for each normalized log, a mapping relationship between the entity and a node in a traceability graph using a common field to obtain a target node associated with the entity in the traceability graph, wherein the traceability graph is generated by capturing the information flow of a system kernel using a traceability tool;

[0170] The capture unit 26 can be used to use the target node as a trigger point, adopt a random walk-based algorithm to capture the context of the target node, extract a tracing subgraph starting from the target node for each target node, and combine all tracing subgraphs to form a reduced tracing graph.

[0171] In a specific application scenario, the acquisition unit 20 includes:

[0172] The analysis module can be used to obtain a list of co-occurring applications from applications that interact with the system and the outside world by performing co-occurrence analysis on the historical operation data of the host;

[0173] The collection module may be used to select a target application whose co-occurrence frequency ranks before a first preset value from the co-occurrence application list, and collect logs of the target application.

[0174] In a specific application scenario, the collection module can be specifically used to collect application logs of the target application by querying the process corresponding to the target application interacting with the outside world on the host;

[0175] The collection module may also be specifically used to collect system calls of a target application for which application logs cannot be obtained by using an audit tool to obtain the audit log of the target application.

[0176] In a specific application scenario, the extraction unit 22 includes:

[0177] A parsing module, which can be used to obtain data information of a preset field from the application log by parsing the application log;

[0178] The normalization module can be used to normalize the data information of the preset fields into a log table according to a preset field format to form a standardized log.

[0179] In a specific application scenario, the predefined rule set includes a default rule and a custom extended rule, and the extraction unit 22 can be specifically used to use the default rule to determine if the preset field of the normalized log contains a known attack signature and / or a key system command, and then extract the matching field from the normalized log as a precise entity;

[0180] The extraction unit 22 may also be specifically configured to extract a noun and a timestamp of a request from a preset field of the normalized log as a result entity using the custom extension rule.

[0181] In a specific application scenario, the extraction unit 22 may also be used to use the default rule to determine if the preset field of the normalized log contains a known attack signature, then check whether the application field in the normalized log matches the known attack signature, and if so, extract the application field as a precise entity;

[0182] The extraction unit 22 may be specifically configured to use the default rule to determine if the preset field of the normalized log contains a critical system command, and then process the critical system command contained in the application field and the preset field in the normalized log.

[0183] In a specific application scenario, the extraction unit 22 may also be used to use the custom extension rule to mark a preset field that appears in the normalized log;

[0184] The extraction unit 22 may be specifically configured to perform part-of-speech analysis in combination with natural language processing to extract a noun and a timestamp of the request from the preset field as a result entity.

[0185] In a specific application scenario, the establishing unit 24 includes:

[0186] The selection module can be used to select the name and request timestamp as common fields for each normalized log;

[0187] The mapping module can be used to map the entity to the corresponding node in the traceability graph if the name of the entity has the same name as the node in the traceability graph and the difference in the timestamps requested between the two is within a preset range. The node is the target node associated with the entity in the traceability graph.

[0188] In a specific application scenario, the capture unit 26 includes:

[0189] A recording module can be used to perform a random walk using the target node as a trigger point, and record the access frequency of the passed nodes during the random walk;

[0190] The sorting module can be used to sort the passing nodes according to their access frequencies, retain the nodes sorted before the second preset value as the context of the target node, and extract the tracing subgraph starting from the target node for each target node.

[0191] Based on the above method embodiment, another embodiment of the present invention provides a storage medium on which executable instructions are stored. When the instructions are executed by a processor, the processor implements the above method.

[0192] Based on the above embodiment, another embodiment of the present invention provides a reduction of a provenance graph, including:

[0193] one or more processors;

[0194] a storage device for storing one or more programs,

[0195] When the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0196] The above device embodiment corresponds to the method embodiment and has the same technical effect as the method embodiment. For specific description, please refer to the method embodiment. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment part, which will not be repeated here. It can be understood by those skilled in the art that the accompanying drawings are only schematic diagrams of one embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0197] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0198] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for reducing a provenance graph, characterized in that: The method comprises: By co-occurring analysis of the host's historical operation data, the application logs of the system's interaction with the outside world are obtained; Parsing and converting the application log into a normalized log according to a preset field format, and extracting entities from the normalized log using a predefined rule set; For each normalized log, a mapping relationship between the entity and a node in a traceability graph is established using a common field to obtain a target node associated with the entity in the traceability graph, wherein the traceability graph is generated by capturing the information flow of the system kernel using a traceability tool; Taking the target node as a trigger point, a random walk-based algorithm is used to capture the context of the target node, extracting a provenance subgraph starting from the target node for each target node, and combining all the provenance subgraphs to form a reduced provenance graph.

2. The method according to claim 1, characterized in that By performing co-occurrence analysis on the historical operation data of the host, we can obtain application logs of the system's interaction with the outside world, including: By performing co-occurrence analysis on the historical operation data of the host, a list of co-occurring applications is obtained from the applications that the system interacts with the outside world; A target application whose co-occurrence frequency ranks before a first preset value is selected from the co-occurrence application list, and logs of the target application are collected.

3. The method according to claim 2, characterized in that The collecting of the logs of the target application comprises: By querying the process corresponding to the target application interacting with the outside world on the host, the application log of the target application is collected; For a target application whose application log cannot be obtained, an audit tool is used to collect system calls of the target application to obtain the audit log of the target application.

4. The method according to claim 1, characterized in that The step of parsing and converting the application log into a normalized log according to a preset field format includes: By parsing the application log, obtaining data information of a preset field from the application log; The data information of the preset fields is normalized into a log table according to a preset field format to form a normalized log.

5. The method according to claim 1, characterized in that The predefined rule set includes a default rule and a custom extended rule, and extracting entities from the normalized log using the predefined rule set includes: Using the default rule to determine if the preset field of the normalized log contains a known attack signature and / or a critical system command, then extracting a matching field from the normalized log as a precise entity; Otherwise, the custom extension rule is used to extract the noun and the timestamp of the request from the preset field of the normalized log as the result entity.

6. The method according to claim 5, characterized in that The using the default rule to determine if the preset field of the normalized log contains a known attack signature and / or a key system command, then extracting a matching field from the normalized log as a precise entity, includes: Using the default rule to determine if the preset field of the normalized log contains a known attack signature, then checking whether the application field in the normalized log matches the known attack signature, and if so, extracting the application field as a precise entity; The default rule is used to determine if the preset field of the normalized log contains a critical system command, and then the critical system command contained in the application field and the preset field in the normalized log is processed.

7. The method according to claim 5, characterized in that The step of using the custom extension rule to extract a noun and a timestamp of a request from a preset field of the normalized log as a result entity includes: Using the custom extension rule to mark the preset field appearing in the normalized log; Part-of-speech analysis is performed in combination with natural language processing to extract nouns and their requested timestamps from the preset fields as result entities.

8. The method according to any one of claims 1 to 7, characterized in that For each normalized log, using a common field to establish a mapping relationship between the entity and a node in the traceability graph to obtain a target node associated with the entity in the traceability graph includes: For each normalized log, select the name and the timestamp of the request as the common fields; If the name of the entity is the same as the name of the node in the traceability graph, and the difference in the timestamps requested between the two is within a preset range, the entity is mapped to the corresponding node in the traceability graph, and the corresponding node in the traceability graph is the target node associated with the entity in the traceability graph.

9. The method according to any one of claims 1 to 7, characterized in that The target node is used as a trigger point, a random walk-based algorithm is used to capture the context of the target node, and a tracing subgraph starting from the target node is extracted for each target node, including: Performing random walk with the target node as a trigger point, and recording the access frequency of the node during the random walk; The passing nodes are sorted according to their access frequencies, the nodes sorted before a second preset value are retained as the context of the target node, and for each target node, a tracing subgraph starting from the target node is extracted.

10. A device for reducing a provenance graph, characterized in that: The device comprises: An acquisition unit, used to acquire application logs of the interaction between the system and the outside world by performing co-occurrence analysis on the historical operation data of the host; An extraction unit, configured to parse and convert the application log into a normalized log according to a preset field format, and extract entities from the normalized log using a predefined rule set; An establishing unit is used to establish, for each normalized log, a mapping relationship between the entity and a node in a traceability graph using a common field to obtain a target node associated with the entity in the traceability graph, wherein the traceability graph is generated by capturing the information flow of a system kernel using a traceability tool; A capture unit is used to use the target node as a trigger point, adopt a random walk-based algorithm to capture the context of the target node, extract a tracing subgraph starting from the target node for each target node, and combine all tracing subgraphs to form a reduced tracing graph.

Images