Real-time traceability graph compression method and device capable of reserving causal association

Abstract

The invention relates to the technical field of network security, and provides a real-time traceability graph compression method and device capable of reserving causal association. The method comprises the following steps: acquiring an event stream from a log information source, analyzing the event stream into an edge and a source node and a destination node corresponding to the edge, and acquiring initial state information of the source node and the destination node; updating the initial state information of the source node and the destination node based on the information flow of the edge to generate target state information; and judging whether the edge is compressed based on the target state information and the initial state information so as to realize compression of the traceability graph. The method is based on node dependence, under the condition that the causal relationship is reserved, the application scene and the compression effect of the compression algorithm are enhanced, the method is suitable for real-time traceability graph construction and storage scenes, graph storage pressure can be reduced, and subsequent threat detection and attack investigation are facilitated.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a real-time provenance graph compression method and device that retains causal associations. Background Art

[0002] Complex network attacks represented by APT attacks are difficult to detect and fully analyze with traditional detection methods due to their multi-step, long cycle, and strong concealment. The provenance graph is usually constructed from system audit logs, where system entities in the logs are represented as nodes and the interaction behaviors between nodes are represented as edges. Since the relationship between system entities and their behaviors is fully recorded, the provenance graph not only has great advantages in the detection of multi-step complex network attacks, but is also often used as a forensic tool to help security personnel investigate attacks and grasp the ins and outs of network attack events. Therefore, in recent years, provenance graphs have been widely studied and applied to threat detection and attack investigation. Provenance graph compression is an indispensable work in the field of provenance graph work, which is used to alleviate the storage and computing pressure caused by massive data. However, in the compression of provenance graphs, there is mutual exclusivity between the fidelity of information and the compression effect. Designing a compression algorithm with good compression effect and high information retention has great application value.

[0003] Existing traceability graph compression methods can be divided into entity-driven, event-driven, and attack investigation-driven compression. Entity-driven compression compresses the graph by removing or merging nodes. For example, some isolated entities that do not affect the analysis results can be safely deleted, and entities with the same functions and attributes can be merged to a certain extent. Event-driven compression compresses the graph by removing or merging edges. For example, event edges are summarized and compressed based on statistical information such as event type and quantity. Attack investigation-driven compression compresses the graph by removing nodes or edges that are not related to the attack event. For example, based on the assumed attack mode, benign event edges or nodes that are not related to the network attack are removed.

[0004] Attack investigation-driven compression methods often lose too much potential threat behavior information due to attack assumptions. Since the number of edges accounts for a much larger proportion than the number of nodes in the construction of the provenance graph, edge compression has greater compression potential. Story-driven compression algorithms often have better compression effects and have a more significant effect on alleviating dependency explosion and reducing overhead. However, the existing mainstream event-driven compression limits the effectiveness of the compression algorithm due to strict reachability preservation, and is not suitable for real-time provenance graph construction scenarios.

[0005] In summary, the existing compression algorithms have the problems of insufficient compression effect and loss of traceability graph information, which is not conducive to subsequent attack traceability analysis. Summary of the invention

[0006] In order to at least solve the problems of weak compression effect and loss of traceability graph information in existing traceability graph compression methods, the present invention provides a real-time traceability graph compression method and device that retains causal associations. Based on the definition of node dependencies and the determination of node dependency changes, the method further expands the application scenarios of traceability graph compression while retaining causal relationships, and is suitable for real-time traceability graph construction and compression.

[0007] In a first aspect, the present invention provides a real-time provenance graph compression method that retains causal associations, comprising:

[0008] Obtain an event stream from a log information source, parse the event stream into edges and source nodes and destination nodes corresponding to the edges, and obtain initial state information of the source nodes and the destination nodes;

[0009] Based on the information flow of the edge, the initial state information of the source node and the destination node is updated to generate target state information;

[0010] Based on the target state information and the initial state information, it is determined whether the edge is to be compressed, so as to achieve compression of the traceability graph.

[0011] Further, the initial state information includes initial node full dependency information and initial write node information;

[0012] Wherein, the initial node full dependency information utilizes a set to record node dependencies, and the node write node information is the node information written with unchanged node dependencies;

[0013] Correspondingly, the target state information includes target node full dependency information and target write node information.

[0014] Further, the initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, including:

[0015] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0016] The target node full dependency information of the source node is equal to the initial node full dependency information of the source node;

[0017] The target node full dependency information of the destination node is equal to the union of the initial node full dependency information of the source node, the initial node full dependency of the source node and the target node minus the set of the target node. The target node full dependency information of the destination node is calculated by formula (1):

[0018]

[0019] in, Indicates the initial node full dependency information of the source node, Src indicates the source node, Indicates the initial node full dependency information of the destination node, Dst represents the destination node, The target node full dependency information representing the destination node;

[0020] The target write node information of the destination node is empty when the target node full dependency information of the destination node is different from the initial node full dependency information of the destination node, and is equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (2):

[0021]

[0022] Further, or the initial state information includes initial node partial dependency information and initial write node information;

[0023] The initial node partial dependency information is queue information of length L, and the initial write node information is write node information with unchanged node dependency;

[0024] Correspondingly, the target state information includes target node partial dependency information and target write node information.

[0025] Furthermore, the initial node partial dependency information and the target node partial dependency information also include node version information;

[0026] Correspondingly, the initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, including:

[0027] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0028] The target node partial dependency information of the source node is equal to the initial node partial dependency information of the source node;

[0029] The target node part of the destination node depends on the information update as follows:

[0030] First, use the append function to add the initial node partial dependency information of the destination node, the initial node partial dependency information of the source node, and the source node to the end of the list in sequence;

[0031] Then, the pruning function is used to delete the duplicate information near the head of the list, and the tail of the list with a dependency length of L is retained as the target node partial dependency information of the destination node;

[0032] The target node partial dependency information of the destination node is updated by formula (3):

[0033]

[0034] in, The target node part of the destination node is dependent on the information, Indicates the initial node partial dependency information of the destination node, Indicates the initial node partial dependency information of the source node, Src indicates the source node, append indicates the adding function, and pruning indicates the reducing function;

[0035] The target write node information of the destination node is empty when the target node partial dependency information of the destination node is different from the initial node partial dependency information of the destination node, and is equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (4):

[0036]

[0037] Among them, dp new Indicates the newly added node partial dependency information of the destination node. If dp new If it is not empty, the target node information of the destination node is empty. new If it is empty, the target write node information of the destination node is equal to the initial write node information of the destination node.

[0038] Further, judging whether to compress the edge based on the target state information and the initial state information to achieve compression of the traceability graph includes:

[0039] It is determined whether the initial state information and the target state information of the destination node and the source node are the same; if they are the same, the edge is compressed.

[0040] Further, judging whether to compress the edge based on the target state information and the initial state information to achieve compression of the traceability graph includes:

[0041] Determine whether the initial state information and the target state information of the destination node and the source node are the same, and if they are the same, perform edge compression on the edge;

[0042] If they are different, update the destination node version.

[0043] Further, the initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, and further includes:

[0044] When the dp new is not empty, and the node X∈dp new When the node partial dependency information of node X contains the destination node N V , and depends on dp new , then node X depends on N V Updated version N v+1 , use formula (5) to determine the dependency version update of node X:

[0045]

[0046] Among them, NV represents the destination node, Represents the node partial dependency information of node X.

[0047] In a second aspect, the present invention provides a real-time provenance graph compression device that retains causal associations, comprising:

[0048] A node status recording module is used to obtain an event stream from a log information source, parse the event stream into edges and source nodes and destination nodes corresponding to the edges, and obtain initial status information of the source nodes and the destination nodes;

[0049] A node status update module, used to update the initial status information of the source node and the destination node based on the information flow of the edge to generate target status information;

[0050] An edge compression module is used to determine whether the edge is to be compressed based on the target state information and the initial state information, so as to achieve compression of the traceability graph.

[0051] According to a third aspect, an electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the method described above when executing the program.

[0052] Beneficial effects of the present invention:

[0053] Based on node dependency, the present invention enhances the application scenarios and compression effects of the compression algorithm while retaining the causal relationship. The present invention is suitable for real-time traceability graph construction and storage scenarios, which can reduce graph storage pressure and facilitate subsequent threat detection and attack investigation.

[0054] (1) In terms of compression effect, it is not uncommon for node A to repeatedly write to node B when the dependency remains unchanged, such as when a process continuously writes information to a file, or when a process and a network node cyclically read and write. In the traceability graph, the number of edges is often several times or even dozens of times that of nodes. In previous studies, when a node has an input edge, it is considered that the node dependency has changed, ignoring the situation where the node has information flow input but the dependency remains unchanged, which limits the application scenarios of graph compression. The present invention expands the edge compression range that keeps the causal relationship unchanged by recording the node dependencies, thereby enhancing the graph compression capability.

[0055] (2) From the perspective of causality, if the results of forward analysis and backward analysis remain unchanged, it means that the causal relationship between nodes is preserved and the compressed traceability graph can still be used in attack investigation scenarios.

[0056] (3) The present invention divides node dependencies into node full dependency records and node partial dependency records for two traceability graph construction scenarios. For graph construction scenarios that do not consider overhead, the node full dependency recording method can be used to record node dependencies in sets. For real-time traceability graph compression scenarios, the node partial dependency recording method can be used to record the latest node dependencies in lists, because scenarios where two or three nodes interact cyclically in system activities are more common, and higher compression effects can be achieved with lower overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of a real-time provenance graph compression method that retains causality provided by an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of a framework of a real-time provenance graph compression method for preserving causal relationships provided by an embodiment of the present invention;

[0059] Figure 3 A schematic diagram of node full dependency state changes provided by an embodiment of the present invention;

[0060] Figure 4 A schematic diagram of partial dependency transfer of version-free nodes provided by an embodiment of the present invention;

[0061] Figure 5 A schematic diagram of partial dependency transfer of versioned nodes provided by an embodiment of the present invention;

[0062] Figure 6 One of the schematic diagrams of node version update in the node partial dependency information of the non-destination node provided by the embodiment of the present invention;

[0063] Figure 7 A second schematic diagram of node version update in the node partial dependency information of a non-destination node provided by an embodiment of the present invention;

[0064] Figure 8 A schematic diagram of a compression method implementation flow provided in an embodiment of the present invention;

[0065] Fig. 9 A schematic diagram of state changes of a cyclic interactive compression scene provided by an embodiment of the present invention;

[0066] Fig.10 A schematic diagram of information flow compression between multiple nodes provided by an embodiment of the present invention;

[0067] Fig.11 A schematic diagram of compression of interactive information flow between two nodes provided by an embodiment of the present invention;

[0068] Fig.12 A schematic diagram of multi-node interactive information flow compression provided by an embodiment of the present invention;

[0069] Fig.13 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] The key concepts that need to be stated in this invention are as follows:

[0072] Event: An event is a log record of the behavior between system entities with event stamps extracted from sources such as system logs or application logs. Each event contains time, subject (such as process entity), object (such as file entity, socket entity, etc.) and interaction behavior between entities (such as read, write, etc.).

[0073] Information flow: Information flow is the flow of information between entities. Information flow includes control flow and data flow. If process A forks to generate child process B, there is control flow between A and B, and the direction of information transmission is from A to B. If process C writes node information to file D, there is data flow between C and D, and the direction of information transmission is from C to D.

[0074] Node: A node is a representation of an entity parsed from an event in a traceability graph. For each edge, the parsed nodes can be divided into source nodes and destination nodes. The information flow of the source node points to the destination node. The node is represented by the letter N, the source node is represented by Src, and the destination node is represented by Dst.

[0075] Edge: Edge is the representation of information flow between nodes in the traceability graph. The edge in the traceability graph is a directed edge, and the direction of the edge is the direction of the information flow. For each event, the parsed edge has attributes such as time and interaction behavior. The edge is represented by the letter E.

[0076] Provenance graph: The provenance graph is extracted from the log source and is a directed graph representation of the information flow interaction between entities. The provenance graph G is composed of nodes and edges of event analysis, and can usually be expressed as G = (N, E).

[0077] Node dependency: The dependency of a node refers to the source of the node's information flow. If there are one or more time-increasing edges between node A and node B, so that the information flow of A is transmitted to B, then B depends on A, and node A is the information source of node B. The dependencies of a node are all the information sources of the node.

[0078] Causality: There is a causal relationship between two nodes that have a dependency relationship.

[0079] Forward analysis: Given a detection node A, forward analysis finds all nodes in the traceability graph that depend on the detection node A based on causal relationships.

[0080] Backward analysis: Given a detection node A, backward analysis traces all nodes that affect detection point A (i.e., the dependencies of node A) based on causal relationships.

[0081] Attack investigation: In the field of traceability, attack investigation usually refers to the reconstruction of network attack scenarios based on threat points. Forward / backward analysis is often used in attack investigation to discover the source of a threat and the impact of the threat.

[0082] The compression basis of the present invention is that when the dependency of node A remains unchanged, repeated writing of node B is redundant, and only the first edge between node A and node B can be retained, so the causal relationship between nodes required for subsequent attack investigation can be correctly retained.

[0083] The present invention is mainly used for the construction and compression of the traceability graph. The log information source comes from the real-time audit log of the host or the offline data set. The algorithm processes the log event stream in the form of time sequence. The event can be parsed into directed edges between Src, Dst and nodes. Figure 1 As shown, a real-time provenance graph compression method for retaining causal association provided by an embodiment of the present invention includes:

[0084] S1: Obtain the event stream from the log information source, parse the event stream into edges and the source nodes and destination nodes corresponding to the edges, and obtain the initial state information of the source nodes and destination nodes.

[0085] S2: Update the initial state information of the source node and the destination node based on the information flow of the edge to generate the target state information;

[0086] S3: Determine whether to compress the edge based on the target state information and the initial state information to achieve compression of the traceability graph.

[0087] Specifically, Figure 2 As shown in the figure, the real-time traceability graph compression method that retains causal associations uses a framework: First, obtain the event stream from the log source (log information source) and parse each event into nodes and edges of the traceability graph. Then, obtain the source node status information and destination node status information corresponding to the edge, and update the node status information according to the information flow. If the source node has been written to the destination node without changing the dependency, it means that this edge can be compressed. Finally, the traceability graph is constructed using the nodes generated by the event and the edges retained by the algorithm.

[0088] The method provided by the embodiment of the present invention is based on node dependency, and enhances the application scenario and compression effect of the compression algorithm while retaining the causal relationship. The present invention is suitable for real-time traceability graph construction and storage scenarios, which can reduce graph storage pressure and assist in subsequent threat detection and attack investigation.

[0089] As a real-time traceability graph compression method that retains causal relationships in an embodiment of the present invention, further, the initial state information includes initial node full dependency information and initial write node information; wherein the initial node full dependency information uses a set to record node dependencies, and the node write node information is write node information whose node dependencies remain unchanged; the target state information includes target node full dependency information and target write node information.

[0090] Accordingly, the initial state information of the source node and the destination node is updated based on the edge information flow to generate the target state information, including:

[0091] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0092] The target node full dependency information of the source node is equal to the initial node full dependency information of the source node;

[0093] The target node full dependency information of the destination node is equal to the union of the initial node full dependency information of the source node, the initial node full dependency of the source node and the target node minus the set of the target node. The target node full dependency information of the destination node is calculated by formula (1):

[0094]

[0095] in, Indicates the initial node full dependency information of the source node, Src indicates the source node, Indicates the initial node full dependency information of the destination node, Dst represents the destination node, The target node full dependency information representing the destination node;

[0096] The target write node information of the destination node is empty when the target node full dependency information of the destination node is different from the initial node full dependency information of the destination node. If they are the same, they are equal to the initial write node information of the destination node. The update of the target write node information of the destination node is determined by formula (2):

[0097]

[0098] It can be understood that the node full dependency recording method and the corresponding state update method provided in this embodiment are suitable for graph construction scenarios that require maximum edge compression but do not consider overhead. The edges in the traceability graph are compressed using changes in dependency information, thereby enhancing the compression effect.

[0099] As a real-time traceability graph compression method that retains causal relationships in an embodiment of the present invention, further, or the initial state information includes initial node partial dependency information and initial write node information; wherein, the initial node partial dependency information is queue information of length L, and the initial write node information is write node information with unchanged node dependency; the target state information includes target node partial dependency information and target write node information.

[0100] Accordingly, the initial node distribution dependency information and the target node partial dependency information also include node version information;

[0101] Based on the edge information flow, the initial state information of the source node and the destination node is updated to generate the target state information, including:

[0102] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0103] The target node partial dependency information of the source node is equal to the initial node partial dependency information of the source node;

[0104] The target node part of the destination node depends on the information update as follows:

[0105] First, use the append function to add the initial node partial dependency information of the destination node, the initial node partial dependency information of the source node, and the source node to the end of the list in sequence;

[0106] Then use the pruning function to delete the duplicate information near the head of the list, and keep the tail of the list with a dependency length of L as the target node partial dependency information of the destination node;

[0107] The target node partial dependency information of the destination node is updated by formula (3):

[0108]

[0109] in, The target node part of the destination node is dependent on the information, Indicates the initial node partial dependency information of the destination node, Indicates the initial node partial dependency information of the source node, Src indicates the source node, append indicates the adding function, and pruning indicates the reducing function;

[0110] The target write node information of the destination node is empty when the target node partial dependency information of the destination node is different from the initial node partial dependency information of the destination node. If they are the same, they are equal to the initial write node information of the destination node. The update of the target write node information of the destination node is determined by formula (4):

[0111]

[0112] Among them, dp new Indicates the newly added node partial dependency information of the destination node, which is used to determine whether the node partial dependency information of the destination node has changed. If dp new If it is not empty, the target node information of the destination node is empty. new If it is empty, the target write node information of the destination node is equal to the initial write node information of the destination node.

[0113] It can be understood that the node partial dependency information recording method provided in this embodiment is suitable for compression in the construction of the traceability graph and compression of the real-time traceability graph, because the scenario of two or three nodes interacting cyclically in system activities is more common. It can be used to record the occurrence of real-time event flows in the traceability graph, and achieve a higher compression effect with a lower overhead.

[0114] As a real-time provenance graph compression method that retains causal association in an embodiment of the present invention, further, judging whether to compress edges based on target state information and initial state information to achieve compression of the provenance graph includes:

[0115] Determine whether the initial state information and target state information of the destination node are the same as those of the source node. If they are the same, compression will be performed simultaneously.

[0116] Furthermore, if the node dependency information is partial node dependency information, it also includes:

[0117] If the initial status information and the target status information are different, the destination node version is updated.

[0118] As a real-time traceability graph compression method for retaining causal association in an embodiment of the present invention, further, based on the information flow of the edge, the initial state information of the source node and the destination node is updated to generate the target state information, and further includes:

[0119] When dp new is not empty, and the node X∈dp new When the node partial dependency information of node X contains the destination node N V , and depends on dp new , then node X depends on N V Updated version N v+1 , use formula (5) to determine the dependency version update of node X:

[0120]

[0121] Among them, NV represents the destination node, Represents the node partial dependency information of node X.

[0122] The following is a further explanation from four aspects: node status recording, node status update, edge compression judgment and applicable scenarios of compression algorithm.

[0123] 1. Node status record

[0124] The node status information is necessary node information in the algorithm implementation process. For node N, its status information mainly includes two parts. The first part is the node dependency information DP of node N, which records the dependent nodes of N. The second part is the write node information WN of node N, which records the write node information of N whose dependency remains unchanged, that is, when the dependency of node N remains unchanged, the node N as the source node Src, and all the destination nodes Dst into which the information flows. By recording the node dependency information DP and the write node information WN of the node, it can be determined whether the node dependency information has changed and whether the edge can be compressed to support the functional implementation of the compression algorithm.

[0125] The dependency information of nodes will continue to accumulate as the information flow is transmitted, causing the memory and CPU overhead of the algorithm to continue to increase as the volume of the traceability graph grows. Recording all node dependency information DP of the node can compress the edges to the greatest extent while ensuring that the causal relationship remains unchanged, but it is only applicable to scenarios with shorter events and appropriate graph sizes. Therefore, an embodiment of the present invention provides a recording form that retains partial node dependency information, ensuring low storage and computing overhead, which can be used to construct compression scenarios for real-time traceability graphs.

[0126] For the above two traceability graph construction scenarios, the node dependency information record can be divided into node full dependency information ALL_DP record and node partial dependency information PART_DP record. Table 1 shows the recording content and function of the node status information, where the node full dependency information ALL_DP or node partial dependency information PART_DP can be selected to write the node information WN as the node status information. For the graph construction scenario that requires maximum compression of edges and does not consider the overhead, the node full dependency information ALL_DP recording method can be used to record node dependencies in a collection. For the real-time traceability graph compression scenario, the node partial dependency information PART_DP recording method can be used to record the latest node dependencies in a list. The node partial dependency information PART_DP can choose to configure the length L of the node dependency information recorded, with a minimum of 1. If the dependency length is L, the algorithm can compress the redundant cyclic interactions between L+1 nodes. Generally speaking, L can be configured to be 1 or 2, because the scenario of two or three nodes cyclically interacting in system activities is more common, and a higher compression effect can be achieved with a lower overhead.

[0127] Table 1 Node status information table

[0128] Node status information content effect WN Node Collection Determine whether Dst is in WN to implement compression judgment. ALL_DP (optional) Node Collection Used for the transfer of dependent states and determination of dependent changes PART_DP (optional) Node queue (length L) Used for the transfer of dependent states and determination of dependent changes

[0129] When event parsing finds a new node, the write node information WN and node dependency information DP of the new node state are set to empty, and then the state information of the updated node is transmitted through the edge information, and finally it is determined whether the edge is compressible.

[0130] 2. Node status update

[0131] After obtaining the state information of the node, the next step is to update the state of the node according to the information flow transmission of the edge to achieve edge compression. The operation of state update is also mainly aimed at the node dependency information DP of the node and the write node information WN of the node. The node dependency information DP related operations of the node mainly involve the transmission and update of dependency, that is, for each edge, how the node dependency information DP of the source node Src is transmitted to the destination node Dst and how to update the node dependency information DP of the destination node. For the write node information WN of the node, when the source node Src writes the destination node Dst, the write node information WN of the source node Src should be added with the destination node Dst, and the write node information WN of the destination node Dst needs to be cleared when its dependency changes. Therefore, the state update of the write node information WN mainly involves the change judgment problem of the node dependency information DP. Since the dependency representation forms of the node full dependency information ALL_DP and the node partial dependency information PART_DP are different, the state update for the two cases is described in detail below.

[0132] 2.1 Status information update in case of full node dependency information ALL_DP

[0133] In the node full dependency information ALL_DP, the node dependency information DP is represented by a set. Each element in the set is a unique identifier of a node. For each edge parsed from the event, use formula (1) to pass the node dependency information between nodes: For an edge with source node Src and destination node Dst, let the node initial dependencies be and After the edge information is transmitted, the destination node Dst depends on The result is the union operation of the DP of the source node Src with Src itself, then the union operation with the DP of Dst, and finally the difference operation with Dst itself. Then we use formula (2) as the judgment condition for whether the semantics has changed: if the dependency set of the destination node has not changed before and after the event, it is considered that the semantics has not changed, otherwise it has changed, and the target write node information of the destination node is updated according to whether it has changed.

[0134]

[0135]

[0136] For example, Figure 3 As shown in the figure, the state change of node dependency information DP in the case of full node dependency information ALL_DP is shown; in the figure, P1 and P2 are process nodes, F is a file node, when T=1, P1 writes file F; when T=2, P2 reads file F. The initial dependencies of P1, F, and P2 are the empty set {}, the set containing the dependency node R {R}, and the empty set {}. After time T1, the DP of F is calculated as {R, P1} according to formula (1). If DP changes, the WN state of F is set to empty. After time T2, the dependency of P2 is calculated as {R, P1, F}, DP changes, and the WN state is set to empty.

[0137] 2.2 Status information update in case of node partial dependency information PART_DP

[0138] In the node partial dependency information PART_DP, the node dependencies are represented by a list, the maximum length of the list is L, and a maximum of L dependent nodes can be stored. Since PART_DP cannot record all dependencies of a node, there may be a situation where dependencies are changed but not detected due to missing dependencies. For example, Figure 4 As shown, the P dependency at time T=1 and time T=3 has changed, but due to the length limitation of the dependency list, the dependency appears to be the same.

[0139] To solve the above problem, the present invention adds node version information as a solution. Whenever a node dependency change is detected, the node version is increased and the node version information N is added. v As the node identifier in the dependency record, the dependency changes of the node will reflect the version difference and can always be passed to Dst. For example, Figure 5 As shown in the figure, NV represents node plus version, N represents node, and V represents version. Since the dependency of F changes when T=2, the F-0 version is increased to F-1. Therefore, although the dependency record length is only 1, the node dependency change can be transmitted through the version, and the node dependency change of P can be accurately detected to accurately update WN.

[0140] Therefore, in the node partial dependency information in PART DP, the node identifier with version information is used as the element of the dependency list. For the edge with source node Src and destination node Dst, let the node initial dependencies be and We use formula (3) to transfer node dependency information between nodes: First, use the append function to add the DP of Dst, the DP of Src, and Src to the end of the list in sequence (the end represents the latest node dependency). Then use the pruning function to delete the duplicate nodes near the head of the list and keep the end of the list with a dependency length of L as the destination node dependency after the state is updated. We use formula (4) to perform a difference operation to obtain the newly added node dependency information dp after the Dst state is updated new , using dp new As a criterion for determining whether a node dependency has changed: If dp new If it is not empty, it means that the node dependency has changed, otherwise, the node dependency has not changed.

[0141]

[0142]

[0143] In PART_DP, when node N v Dependencies change and versions increase to N v+1 , there exists a set dp that causes the dependency change new , if it causes N v Dependent nodes X∈dp new Depends on node N v , and also depends on dp new , then node X should also depend on N v+1 ,To prevent this dependency from being lost, we use formula (5) to make a judgment and update the dependency version of X. The retention of this dependency can eliminate the redundant edges of the cyclic scenario.

[0144]

[0145] For example, Figure 6 As shown in the figure, an example scenario of dependency version update is shown (NV represents node plus version): process P interacts with socket S through send and recv information. When T=2, since S-1, which causes the dependency change of P, also depends on S-1 (it is assumed that the node always depends on itself) and P-0, the dependency P-0 of S-1 is updated to P-1. Then, the subsequent process P writing S will not cause the dependency change of S.

[0146] For example, Figure 7 As shown in the figure, an example scenario of dependency version update is shown: when an event occurs, the dependency information of node B changes. At this time, dp new ={C, D, A}, then for each newly added dependency information dp new The nodes in the dp new In this example, node C depends on dp new As well as node B-0, it meets the conditions in formula (5), so B-0 is updated to B-1. Node A and node C do not meet formula (5), so there is no need to update their node partial dependency information.

[0147] 3. Edge compression determination

[0148] By recording and updating the node's status information, the edge compression judgment can be performed to implement the compression algorithm. The edge compression judgment process is as follows: For an edge with a source node of Src and a destination node of Dst, if the WN of Src contains Dst, it means that the Src node has been written to Dst while the dependency remains unchanged, so this edge can be compressed without affecting the causal relationship. Because Src has not changed and has been written to Dst, the information flow of this edge has been parsed, so there is no need to update the status after the edge is compressed. If the compression judgment fails, the information flow of the edge needs to be processed to update the node status information of Src and Dst.

[0149] The overall process of the algorithm is as follows Figure 8 As shown: for each edge Ei, the source node is Si, the destination node is Di, and the initialization information of each node includes WN and DP. WN is the node set written when the node dependency remains unchanged, and DP records the node dependency information (refer to the above content for the specific implementation of the dependency in the step).

[0150] In order to facilitate the understanding of the algorithm process, the example Fig. 9The node state change process in a cyclic compression scenario is demonstrated: In the figure, process P (node ​​P) and network socket S (node ​​S) exchange information, and the initial node states DP and WN are both empty. When T=1, P sends information to S, and P's WN adds S node. Due to dependency transmission, S's DP adds P-0, and the dependency transmission changes, so S version changes from 0 to 1, and the write node set WN with unchanged dependency is set to empty. Since P's WN does not contain S before writing, this edge can be uncompressed. When T=2, P receives S's information, then P's dependency changes to S-1, the version changes from 0 to 1, and WN is set to empty. S node WN adds node P, and since node S that causes P to change depends on P-0 before the dependency change and S-1 that causes the dependency change, the dependency version is updated, and the dependency is changed from P-0 to P-1. When T=3, P sends information to S, WN adds node S, and since S's dependency has not changed, there are no other changes. When T=4, since S's WN includes P, this edge can be compressed when P reads information from S. If the subsequent dependencies between the two nodes do not change, the subsequent mutual read and write edges can be compressed.

[0151] 4. Compression algorithm applicable scenarios

[0152] The final compression method is suitable for several compression scenarios.

[0153] 4.1 Compression of read and write scenarios between multiple nodes

[0154] In normal system activity, there will be scenarios where a process continuously reads information from a file and writes to another file. For example, Fig.10 As shown in the figure, the directed edges in the figure represent the direction of information flow, and the T on the edge represents the event of the event. Process A writes node information to file B at T=1, 2, 4, and 6, and process C reads information from file B at T=3, 5, and 7. Process A receives new information input at T=5, which causes semantic changes. Then, the three edges of process A at T=1, 2, and 4 before the semantic change at T=5 only need to retain the edge corresponding to T=1. Similarly, the two edges of file B at T=3 and 5 before the semantic change at T=6 only need to retain the edge corresponding to T=1. It should be noted that A is equivalent to the node in the above method.

[0155] 4.2 Compression of interaction scenarios between multiple nodes

[0156] The read and write interaction between processes and network nodes is also a common scenario. This compression method is applicable to the mutual reading and writing between multiple nodes. If the number of node dependencies is L, it can solve the cyclic read and write redundant edges between L+1 nodes. Fig.11As shown in the figure, network socket S and process A exchange information, assuming that the node dependency retention length L is 1. When T=3, network socket S and process A have the same semantic information, and write to each other's node when the dependency remains unchanged. Therefore, after T=3, since the node dependency does not change, the mutual read and write operations of the two nodes can be compressed. For the interaction between multiple nodes, it is also a common scenario for the parent-child process to interact with each other through a pipeline. For example, Fig.12 As shown, process A and process B exchange information through pipeline P. At T=6, the node dependency remains unchanged, and the subsequent event edges of A transmitting information from P to B and B transmitting information from P to A can be compressed. It should be noted that process A, process B and pipeline P are equivalent to the nodes in the above method.

[0157] In summary, the embodiments of the present invention aim at the construction and compression scenarios of the traceability graph, based on the definition of node dependency, compressing the event edges while keeping the source node dependency unchanged. The present invention is applicable to the compression of various traceability graph structures and can be used for real-time graph construction. While retaining the traceability graph attack investigation capability, it improves the effect of traceability graph compression.

[0158] From the compression effect, it is not uncommon to repeatedly write to node B when the dependency of node A remains unchanged, such as a process continuously writing information to a file, or a process and network nodes cyclically reading and writing. In the traceability graph, the number of edges is often several times or even dozens of times that of nodes. In previous studies, if a node has an input edge, it is considered that the node dependency has changed, ignoring the situation where the node has information flow input but the dependency remains unchanged, which limits the application scenarios of graph compression. By recording the dependencies of nodes, the present invention expands the range of edge compression that keeps the causal relationship unchanged, thereby enhancing the ability of graph compression.

[0159] From the perspective of causality, if the results of forward analysis and backward analysis remain unchanged, it means that the causal relationship between nodes is preserved, and the compressed traceability graph can still be used in attack investigation scenarios. The following is a demonstration from the perspective of forward analysis and backward analysis:

[0160] For each node A, if an anomaly is found at time T, the forward analysis will traverse the outgoing edges from node A forward with a time greater than T. If node A is found to be abnormal at time T, there must be new information flow input to node A that causes the node abnormality, that is, the node dependency has changed. According to the method provided by the present invention, all outgoing edges after the node dependency changes will not be compressed, and all nodes corresponding to the outgoing edges after the anomaly occurs (at time T) will be retained, ensuring that the results of the forward analysis can be fully retained;

[0161] Similarly, the backward analysis traverses all incoming edges from node A whose post-item traversal time is less than T. Since the deleted incoming edges are redundant edges that have been written to node A and whose source node dependencies remain unchanged, the nodes corresponding to all input edges before the exception occurs (time T) will not be confirmed, ensuring the integrity of the forward impact analysis.

[0162] The embodiment of the present invention further provides a real-time provenance graph compression device that retains causal association, including:

[0163] The node status recording module is used to obtain the event stream from the log information source, parse the event stream into edges and the source nodes and destination nodes corresponding to the edges, and obtain the initial status information of the source nodes and the destination nodes;

[0164] The node state update module is used to update the initial state information of the source node and the destination node based on the information flow of the edge to generate the target state information;

[0165] The edge compression module is used to determine whether the edge should be compressed based on the target state information and the initial state information to achieve compression of the traceability graph.

[0166] like Fig.13 As shown, an embodiment of the present invention further provides an electronic device, which may include: a processor (processor) 1301, a communication interface (Communications Interface) 1302, a memory (memory) 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The processor 1301 can call the logic instructions in the memory 1303 to execute the method provided in the above embodiment, for example, including:

[0167] Obtain the event stream from the log information source, parse the event stream into edges and the source nodes and destination nodes corresponding to the edges, and obtain the initial state information of the source nodes and the destination nodes; update the initial state information of the source nodes and the destination nodes based on the information flow of the edges to generate the target state information; determine whether to compress the edges based on the target state information and the initial state information to achieve compression of the traceability graph.

[0168] In addition, when the logic instructions in the above-mentioned memory 1303 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time provenance graph compression method that preserves causal relationships, characterized in that: include: Obtain an event stream from a log information source, parse the event stream into edges and source nodes and destination nodes corresponding to the edges, and obtain initial state information of the source nodes and the destination nodes; Based on the information flow of the edge, the initial state information of the source node and the destination node is updated to generate target state information; Based on the target state information and the initial state information, it is determined whether the edge is to be compressed, so as to achieve compression of the traceability graph.

2. A real-time traceability graph compression method that retains causal association according to claim 1, characterized in that: The initial state information includes initial node full dependency information and initial write node information; Wherein, the initial node full dependency information utilizes a set to record node dependencies, and the node write node information is the node information written with unchanged node dependencies; Correspondingly, the target state information includes target node full dependency information and target write node information.

3. A real-time traceability graph compression method that retains causal association according to claim 2, characterized in that: The initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, including: The target write node information of the source node is equal to the initial write node information of the source node plus the target node; The target node full dependency information of the source node is equal to the initial node full dependency information of the source node; The target node full dependency information of the destination node is equal to the union of the initial node full dependency information of the source node, the initial node full dependency of the source node and the target node minus the set of the target node. The target node full dependency information of the destination node is calculated by formula (1): in, Indicates the initial node full dependency information of the source node, Src indicates the source node, Indicates the initial node full dependency information of the destination node, Dst represents the destination node, The target node full dependency information representing the destination node; The target write node information of the destination node is empty when the target node full dependency information of the destination node is different from the initial node full dependency information of the destination node, and is equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (2):

4. The method for compressing a real-time provenance graph with causal association preservation according to claim 1, characterized in that: Or the initial state information includes initial node partial dependency information and initial write node information; The initial node partial dependency information is queue information of length L, and the initial write node information is write node information with unchanged node dependency; Correspondingly, the target state information includes target node partial dependency information and target write node information.

5. A real-time traceability graph compression method that retains causal association according to claim 4, characterized in that: The initial node partial dependency information and the target node partial dependency information also include node version information; Correspondingly, the initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, including: The target write node information of the source node is equal to the initial write node information of the source node plus the target node; The target node partial dependency information of the source node is equal to the initial node partial dependency information of the source node; The target node part of the destination node depends on the information update as follows: First, use the append function to add the initial node partial dependency information of the destination node, the initial node partial dependency information of the source node, and the source node to the end of the list in sequence; Then, the pruning function is used to delete the duplicate information near the head of the list, and the tail of the list with a dependency length of L is retained as the target node partial dependency information of the destination node; The target node partial dependency information of the destination node is updated by formula (3): in, The target node part of the destination node is dependent on the information, Indicates the initial node partial dependency information of the destination node, Indicates the initial node partial dependency information of the source node, Src indicates the source node, append indicates the adding function, and pruning indicates the reducing function; The target write node information of the destination node is empty when the target node partial dependency information of the destination node is different from the initial node partial dependency information of the destination node, and is equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (4): Among them, dp new Indicates the newly added node partial dependency information of the destination node. If dp new If it is not empty, the target node information of the destination node is empty. new If it is empty, the target write node information of the destination node is equal to the initial write node information of the destination node.

6. A real-time traceability graph compression method that retains causal association according to claim 2 or 3, characterized in that: Determining whether to compress the edge based on the target state information and the initial state information to achieve compression of the traceability graph includes: It is determined whether the initial state information and the target state information of the destination node are the same as those of the source node; if they are the same, the edge is compressed.

7. A real-time traceability graph compression method that retains causal association according to claim 5, characterized in that: Determining whether to compress the edge based on the target state information and the initial state information to achieve compression of the traceability graph includes: Determine whether the initial state information and the target state information of the destination node are the same as those of the source node, and if they are the same, perform edge compression on the edge; If they are different, update the destination node version.

8. A real-time traceability graph compression method that retains causal association according to claim 7, characterized in that: The method further comprises: When the dp new is not empty, and the node X∈dp new When the node partial dependency information of node X contains the destination node N V , and depends on dp new , then node X depends on N V Updated version N v+1 , use formula (5) to determine the dependency version update of node X: Among them, N V represents the destination node, Represents the node partial dependency information of node X.

9. A real-time provenance graph compression device that retains causal associations, characterized in that: include: A node status recording module is used to obtain an event stream from a log information source, parse the event stream into edges and source nodes and destination nodes corresponding to the edges, and obtain initial status information of the source nodes and the destination nodes; A node status update module, used to update the initial status information of the source node and the destination node based on the information flow of the edge to generate target status information; An edge compression module is used to determine whether the edge is to be compressed based on the target state information and the initial state information, so as to achieve compression of the traceability graph.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

Images