A real-time provenance graph compression method and device preserving causality

Abstract

The present application relates to the technical field of network security, and provides a real-time traceability graph compression method and device for preserving causal association. The method comprises: obtaining an event stream from a log information source, parsing the event stream into edges and source nodes and destination nodes corresponding to the edges, and obtaining initial state information of the source nodes and the destination nodes; updating the initial state information of the source nodes and the destination nodes based on the information flow of the edges to generate target state information; and judging whether the edges are compressed based on the target state information and the initial state information to achieve compression of the traceability graph. The present application is based on node dependency, enhances the application scenarios and compression effect of the compression algorithm in the case of preserving causal relationship, is suitable for real-time traceability graph construction and storage scenarios, can reduce graph storage pressure, and is conducive to subsequent threat detection and attack investigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a real-time provenance graph compression method and device for preserving causal association. BACKGROUND

[0002] Complex network attacks represented by APT attacks are difficult to be detected and analyzed completely by traditional detection methods due to their characteristics such as multi-step, long cycle and strong concealment. A provenance graph is usually constructed from system audit logs, and the system entities in the logs are represented as nodes, and the interaction behaviors between the nodes are represented as edges. Since the provenance graph completely records the association between system entities and their behaviors, it not only has great advantages in multi-step complex network attack detection, but also often serves as a kind of evidence tool to help security personnel investigate attacks and grasp the origin and development of network attack events. Therefore, in recent years, provenance graphs have been widely studied and applied to threat detection and attack investigation. Provenance graph compression is an indispensable work in the field of provenance graph work, which is used to alleviate the storage and computing pressure caused by massive data. However, in provenance graph compression, there is a mutual exclusivity between information fidelity and compression effect, and it has great application value to design a compression algorithm with good compression effect and high information retention.

[0003] Existing provenance graph compression methods can be divided into entity-driven, event-driven and attack investigation-driven compression. Entity-driven compression compresses the graph by removing or merging nodes, for example, some isolated entities that do not affect the analysis results can be safely deleted, and entities with the same function and attributes can be merged to a certain extent. Event-driven compression compresses the graph by removing or merging edges. For example, according to the statistical information of the retained event type and quantity, the event edges are summarized and compressed. Attack investigation-driven compression compresses the graph by removing nodes or edges irrelevant to attack events. For example, according to the assumed attack mode, benign event edges or nodes irrelevant to network attacks are removed.

[0004] Attack investigation-driven compression methods often lose too much potential threat behavior information due to attack assumptions. Since the number of edges accounts for a much larger proportion than the number of nodes in the construction of a provenance graph, edge compression has greater compression potential, so event-driven compression algorithms often have better compression effect, and have more significant effect on relieving dependence explosion and reducing overhead. However, the existing mainstream event-driven compression strictly preserves reachability, which limits the effect of the compression algorithm, and is not suitable for real-time provenance graph construction scenarios.

[0005] In summary, the existing compression algorithms have the problems of insufficient compression effect and loss of provenance graph information, which are not conducive to subsequent attack provenance analysis. SUMMARY

[0006] In order to at least solve the problems of the existing traceability graph compression method that the compression effect is not outstanding and the traceability graph information is lost, the present application provides a real-time traceability graph compression method and device for preserving causal association, which is based on the definition of node dependence and the judgment of node dependence change, further expands the application scene of traceability graph compression while preserving the causal relationship, and is suitable for real-time traceability graph construction and compression.

[0007] In the first aspect, the present application provides a real-time traceability graph compression method for preserving causal association, which comprises:

[0008] Obtaining an event stream from a log information source, parsing the event stream into edges and corresponding source nodes and destination nodes of the edges, and obtaining initial state information of the source nodes and the destination nodes;

[0009] Updating the initial state information of the source nodes and the destination nodes based on the information flow of the edges to generate target state information;

[0010] Judging whether the edges are compressed based on the target state information and the initial state information to realize compression of the traceability graph.

[0011] Further, the initial state information comprises initial node full dependence information and initial write node information.

[0012] Wherein, the initial node full dependence information records node dependence by using a set, and the node write node information is write node information that is invariant for node dependence.

[0013] Correspondingly, the target state information comprises target node full dependence information and target write node information.

[0014] Further, updating the initial state information of the source nodes and the destination nodes based on the information flow of the edges to generate target state information comprises:

[0015] The target write node information of the source node is equal to the initial write node information of the source node plus the target node.

[0016] The target node full dependence information of the source node is equal to the initial node full dependence information of the source node.

[0017] The target node full dependence information of the destination node is equal to the initial node full dependence information of the source node, the union set of the initial node full dependence information of the source node and the target node minus the set of the target node, and the target node full dependence information of the destination node is calculated by formula (1):

[0018]

[0019] Wherein, Src represents the initial node's full dependency information for the source node. The initial node dependencies of the destination node are represented by Dst, where Dst represents the destination node. This represents the target node's full dependency information;

[0020] The target write node information of the destination node is empty when the target node full dependency information of the destination node is not the same as the initial node full dependency information of the destination node; otherwise, it is equal to the initial write node information of the destination node. The update of the target write node information of the destination node is determined by formula (2):

[0021]

[0022] Furthermore, the initial state information may include initial node partial dependency information and initial write node information;

[0023] Wherein, the initial node partial dependency information is queue information of length L, and the initial write node information is write node information with unchanged node dependencies;

[0024] Correspondingly, the target state information includes target node partial dependency information and target write node information.

[0025] Furthermore, the initial node partial dependency information and the target node partial dependency information also include node version information;

[0026] Correspondingly, the initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, including:

[0027] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0028] The target node partial dependency information of the source node is equal to the initial node partial dependency information of the source node;

[0029] The target node's dependency information is updated as follows:

[0030] First, the append function is used to add the initial node partial dependency information of the destination node, the initial node partial dependency information of the source node, and the source node to the end of the list in sequence;

[0031] Then, the pruning function is used to delete duplicate information near the beginning of the list, and the tail of the list with a dependency length of L is retained as the target node partial dependency information of the target node.

[0032] The target node's partial dependency information is updated using formula (3):

[0033]

[0034] in, This indicates the target node's partial dependency information. This indicates the initial node's dependency information for the destination node. This represents the initial node dependency information of the source node, Src represents the source node, append represents the append function, and pruning represents the pruning function;

[0035] The target write node information of the destination node is empty when the target node partial dependency information of the destination node is different from the initial node partial dependency information of the destination node; if they are the same, it is equal to the initial write node information of the destination node. The update of the target write node information of the destination node is determined by formula (4):

[0036]

[0037] Among them, dp new This represents the dependency information of the newly added node in the destination node, if dp new If dp is not empty, then the target node's write node information is empty. new If empty, the target write node information of the destination node is equal to the initial write node information of the destination node.

[0038] Further, determining whether the edges should be compressed based on the target state information and the initial state information to achieve the compression of the source graph includes:

[0039] Determine whether the initial state information and target state information of the destination node are the same as those of the source node. If they are the same, compress the edges.

[0040] Further, determining whether the edges should be compressed based on the target state information and the initial state information to achieve the compression of the source graph includes:

[0041] Determine whether the initial state information and target state information of the destination node are the same as those of the source node. If they are the same, compress the edges.

[0042] If they are different, then update the version of the destination node.

[0043] Furthermore, updating the initial state information of the source node and the destination node based on the information flow of the edges to generate target state information also includes:

[0044] When the dp new Not empty, and node X∈dp new If the node dependency information of node X includes the destination node N, then... V And depends on dp new Then node X depends on N V Updated version N v+1 The dependency version update of node X is determined by formula (5):

[0045]

[0046] Where NV represents the destination node, This represents the node-specific dependency information for node X.

[0047] Secondly, the present invention provides a real-time source graph compression device that preserves causal relationships, comprising:

[0048] The node state recording module is used to obtain the event stream from the log information source, parse the event stream into edges and the source nodes and destination nodes corresponding to the edges, and obtain the initial state information of the source nodes and the destination nodes.

[0049] The node state update module is used to update the initial state information of the source node and the destination node based on the information flow of the edge, and generate target state information;

[0050] An edge compression module is used to determine whether the edge should be compressed based on the target state information and the initial state information, so as to realize the compression of the source graph.

[0051] Thirdly, an electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the program to implement the method described above.

[0052] The beneficial effects of this invention are:

[0053] This invention, based on node dependency, enhances the application scenarios and compression effect of the compression algorithm while preserving causal relationships. It is suitable for real-time source graph construction and storage scenarios, reducing graph storage pressure and facilitating subsequent threat detection and attack investigation.

[0054] (1) From the perspective of compression effect, it is not uncommon for node A to repeatedly write to node B when its dependencies remain unchanged, such as when a process continuously writes information to a file, or when a process and a network node repeatedly read and write. Moreover, in a source graph, the number of edges is often several times or even tens of times greater than the number of nodes. In previous studies, the presence of an input edge to a node was considered a change in its dependencies, ignoring the situation where a node has an input flow but its dependencies remain unchanged, thus limiting the application scenarios of graph compression. This invention expands the range of edge compression that preserves causal relationships by recording the dependencies of nodes, thereby enhancing the graph compression capability.

[0055] (2) From the perspective of causality, if the results of forward analysis and backward analysis remain unchanged, it means that the causal relationship between nodes is preserved, and the compressed source graph can still be used in attack investigation scenarios.

[0056] (3) This invention addresses two source graph construction scenarios by dividing node dependencies into full node dependency recording and partial node dependency recording. For graph construction scenarios where overhead is not considered, the full node dependency recording method can be used, with a set recording node dependencies. For real-time source graph compression scenarios, the partial node dependency recording method can be used, with a list recording the latest node dependencies, because scenarios where two or three nodes interact cyclically are more common in system activities, allowing for higher compression effects with lower overhead. Attached Figure Description

[0057] Figure 1 A flowchart illustrating a real-time source graph compression method that preserves causal relationships, provided in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the framework for a real-time source graph compression method that preserves causal relationships, provided in an embodiment of the present invention;

[0059] Figure 3 This is a schematic diagram of node full dependency state changes provided in an embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of partial dependency propagation for versionless nodes provided in an embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the dependency propagation of version nodes provided in an embodiment of the present invention;

[0062] Figure 6 This is one of the schematic diagrams illustrating node version updates in the node dependency information of a non-destination node provided in an embodiment of the present invention;

[0063] Figure 7 The second schematic diagram of node version update in the node dependency information of non-destination node provided in the embodiments of the present invention;

[0064] Figure 8 This is a schematic diagram illustrating the implementation process of the compression method provided in this embodiment of the invention;

[0065] Figure 9 This is a schematic diagram of the state changes in a cyclic interactive compression scenario provided in an embodiment of the present invention;

[0066] Figure 10 This is a schematic diagram of information flow compression between multiple nodes provided in an embodiment of the present invention;

[0067] Figure 11 This is a schematic diagram of the compression of interactive information flow between two nodes provided in an embodiment of the present invention;

[0068] Figure 12 This is a schematic diagram of multi-node interactive information flow compression provided in an embodiment of the present invention;

[0069] Figure 13 This is a schematic diagram of an electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0071] The key concepts that need to be declared in this invention are as follows:

[0072] Events: Events are log records of the behavior between system entities, extracted from sources such as system logs or application logs, and bearing event stamps. Each event includes a time, a subject (such as a process entity), an object (such as a file entity, a socket entity, etc.), and the interaction behavior between the entities (such as reading, writing, etc.).

[0073] Information flow: Information flow is the movement of information between entities. Information flow includes control flow and data flow. If process A forks to create child process B, then there is a control flow between A and B, and the information is transferred from A to B. If process C writes node information to file D, then there is a data flow between C and D, and the information is transferred from C to D.

[0074] Node: A node is the representation of an entity parsed from an event in the source graph. For each edge, the parsed nodes can be divided into source nodes and destination nodes. The information flow of the source node points to the destination node. Nodes are represented by the letter N, source nodes by Src, and destination nodes by Dst.

[0075] Edges: Edges are the representation of information flow between nodes in a source graph. Edges in a source graph are directed, and their direction indicates the direction of information flow. For each event, the parsed edge possesses attributes such as time and interaction behavior. Edges are represented by the letter E.

[0076] Source graph: A source graph is a directed graph representation of the information flow and interaction between entities, extracted from the log source. A source graph G consists of nodes and edges from event resolution, and can usually be represented as G = (N, E).

[0077] Node dependency: Node dependency refers to the source of information flow to a node. If there is one or more time-increasing edges between node A and node B that allow information flow from A to B, then B depends on A, and node A is the source of information for node B. A node's dependency is all of its information sources.

[0078] Causal relationship: There is a causal relationship between two nodes that have a dependency relationship.

[0079] Forward analysis: Given a detection node A, forward analysis finds all nodes in the source graph that depend on detection node A based on causal relationships.

[0080] Backward analysis: Given a detection node A, backward analysis traces all nodes that affect detection node A based on causal relationships (i.e., the dependencies of node A).

[0081] Attack investigation: In the field of attribution mapping, attack investigation typically refers to reconstructing network attack scenarios based on threat points. Forward / backward analysis is frequently used in attack investigations to discover the source of a threat and its impact.

[0082] The compression principle of this invention is that when the dependency of node A remains unchanged, repeated writing to node B is redundant. Only the first edge between node A and node B can be retained, which can correctly preserve the causal relationship between nodes required for subsequent attack investigations.

[0083] This invention is primarily used for the construction and compression of source graphs. Log information sources include real-time audit logs from the host machine or offline datasets. The algorithm processes log event streams arranged chronologically, and events can be parsed into Src, Dst, and directed edges between nodes. For example... Figure 1 As shown, an embodiment of the present invention provides a real-time source graph compression method that preserves causal relationships, comprising:

[0084] S1: Obtain the event stream from the log information source, parse the event stream into edges and the corresponding source and destination nodes, and obtain the initial state information of the source and destination nodes.

[0085] S2: Update the initial state information of the source node and the destination node based on the edge information flow to generate the target state information;

[0086] S3: Determine whether to compress edges based on target state information and initial state information to achieve source graph compression.

[0087] Specifically, such as Figure 2 As shown, the real-time source graph compression method that preserves causal relationships uses the following framework: First, it obtains the event stream from the log source (log information source) and parses each event into nodes and edges of the source graph. Then, it obtains the source node state information and destination node state information corresponding to each edge, and updates the node state information according to the information stream. If a source node has already been written to the destination node without changing its dependencies, then this edge can be compressed. Finally, it constructs the source graph using the nodes generated by the events and the edges preserved by the algorithm.

[0088] The method provided in this invention is based on node dependency, which enhances the application scenarios and compression effect of the compression algorithm while preserving causal relationships. This invention is applicable to real-time source graph construction and storage scenarios, reducing graph storage pressure and facilitating subsequent threat detection and attack investigation.

[0089] As a real-time source graph compression method that preserves causal relationships in this embodiment of the invention, the initial state information further includes initial node full dependency information and initial write node information; wherein, the initial node full dependency information uses a set to record node dependencies, and the node write node information is write node information with unchanged node dependencies; the target state information includes target node full dependency information and target write node information.

[0090] Correspondingly, the edge-based information flow updates the initial state information of the source and destination nodes to generate target state information, including:

[0091] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0092] The target node full dependency information of the source node is equal to the initial node full dependency information of the source node;

[0093] The target node's total dependency information is equal to the initial node total dependency information of the source node, the union of the initial node total dependencies of the source node and the target node, minus the target node's set. The target node's total dependency information is calculated using formula (1):

[0094]

[0095] in, Src represents the initial node's full dependency information for the source node. The initial node dependencies of the destination node are represented by Dst, where Dst represents the destination node. This represents the target node's full dependency information;

[0096] The target write node information of the destination node is empty if the target node full dependency information of the destination node is not simultaneously empty with the initial node full dependency information of the destination node, and equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (2):

[0097]

[0098] It is understood that the node full dependency recording method and corresponding state update method provided in this embodiment are suitable for graph construction scenarios that require maximum edge compression but do not consider overhead. By utilizing changes in dependency information to compress edges in the source graph, the compression effect is enhanced.

[0099] As a real-time source graph compression method that preserves causal relationships in this embodiment of the invention, the initial state information may further include initial node partial dependency information and initial write node information; wherein, the initial node partial dependency information is queue information of length L, and the initial write node information is write node information with unchanged node dependencies; the target state information includes target node partial dependency information and target write node information.

[0100] Correspondingly, the initial node distribution dependency information and the target node partial dependency information also include node version information;

[0101] The edge-based information flow updates the initial state information of the source and destination nodes to generate target state information, including:

[0102] The target write node information of the source node is equal to the initial write node information of the source node plus the target node;

[0103] The target node partial dependency information of the source node is equal to the initial node partial dependency information of the source node;

[0104] The target node's dependency information is updated as follows:

[0105] First, use the append function to add the initial node dependency information of the destination node, the initial node dependency information of the source node, and the source node to the end of the list in sequence;

[0106] Then, the pruning function is used to delete duplicate information near the beginning of the list, retaining the target node part of the tail of the list with a dependency length of L as the target node;

[0107] The target node's partial dependency information is updated using formula (3):

[0108]

[0109] in, This indicates the target node's partial dependency information. This indicates the initial node's dependency information for the destination node. This represents the initial node dependency information of the source node, Src represents the source node, append represents the append function, and pruning represents the pruning function;

[0110] The target write node information of the destination node is empty if the target node part dependency information and the initial node part dependency information of the destination node are not both empty, and if they are the same, it is equal to the initial write node information of the destination node. The update of the target write node information of the destination node is determined by formula (4):

[0111]

[0112] Among them, dp new This represents the partial dependency information of the newly added node in the destination node, used to determine whether the partial dependency information of the destination node has changed. If dp new If dp is not empty, then the target node's write node information is empty. new If empty, the target write node information of the destination node is equal to the initial write node information of the destination node.

[0113] It is understood that the node-dependent information recording method provided in this embodiment is suitable for compression in the construction of source graphs and for real-time compression of source graphs, because scenarios where two or three nodes interact cyclically are more common in system activities. It can be used to record the real-time event flow of source graphs, achieving a high compression effect with low overhead.

[0114] As a real-time source graph compression method that preserves causal relationships in this embodiment of the invention, further, it determines whether edges should be compressed based on target state information and initial state information to achieve source graph compression, including:

[0115] Determine whether the initial state information and target state information of the destination node are the same as those of the source node. If they are the same, compress the edges.

[0116] Furthermore, if the node dependency information is partial node dependency information, it also includes:

[0117] If the initial state information and the target state information are different, then the target node version is updated.

[0118] As a real-time source graph compression method that preserves causal relationships in this embodiment of the invention, further, updating the initial state information of the source node and the destination node based on the edge information flow to generate target state information, also includes:

[0119] When dp new Not empty, and node X∈dp new If the node dependency information of node X includes the destination node N, then... V And depends on dp new Then node X depends on N V Updated version N v+1 The dependency version update of node X is determined by formula (5):

[0120]

[0121] Where NV represents the destination node, This represents the node-specific dependency information for node X.

[0122] The following sections will further explain this from four aspects: node state recording, node state updating, edge compression determination, and applicable scenarios for compression algorithms.

[0123] 1. Node state record

[0124] Node state information is essential for the algorithm implementation. For node N, its state information mainly consists of two parts. The first part is node dependency information DP, which records the nodes that N depends on. The second part is node write node information WN, which records the write node information when N's dependencies remain unchanged. That is, when node N, as the source node Src, has unchanged dependencies, all destination nodes Dst to which information flows. By recording the node dependency information DP and write node information WN, it is possible to determine whether the node dependency information has changed and whether the edges can be compressed, thus supporting the implementation of the compression algorithm.

[0125] The dependency information of nodes accumulates continuously as information flows, causing the memory and CPU overhead of the algorithm to increase with the size of the source graph. Dynamic programming (DP), which records all node dependency information, can compress edges to the greatest extent while maintaining causal relationships, but it is only suitable for scenarios with short events and appropriate graph size. Therefore, this embodiment of the invention provides a recording format that retains some node dependency information, ensuring low storage and computational overhead, and can be used in real-time source graph construction and compression scenarios.

[0126] For the two source graph construction scenarios mentioned above, node dependency information recording can be divided into ALL_DP (all-dependency information) and PART_DP (partial-dependency information) recording. Table 1 shows the recording content and function of node state information. Either ALL_DP or PART_DP can be combined with node information WN as node state information. For graph construction scenarios requiring maximum edge compression and disregarding overhead, the ALL_DP method can be used, recording node dependencies in a set. For real-time source graph compression scenarios, the PART_DP method can be used, recording the latest node dependencies in a list. The length L of the PART_DP node dependency information can be configured, with a minimum of 1. If the dependency length is L, the algorithm can compress redundant cyclic interactions between L+1 nodes. Generally, L can be configured to 1 or 2, as scenarios with two or three nodes interacting cyclically are more common in system activities, achieving higher compression effects with lower overhead.

[0127] Table 1 Node Status Information Table

[0128] Node state information Content Action WN Node set Judge whether Dst is in WN, realize compression decision. ALL_DP (optional) Node set Used for transmission of dependency state, decision of dependency change PART_DP (optional) Node queue (length L) Used for transmission of dependency state, decision of dependency change

[0129] When the event parsing discovers a new node, the node information WN (written node information) and the node dependency information DP (node ​​dependency information) of the new node are both set to empty. Then, the state information of the updated node is passed through the edge information, and finally, it is determined whether the edge is compressible.

[0130] 2. Node state update

[0131] After obtaining the node's state information, the next step is to update the node's state based on the edge information flow to achieve edge compression. State update operations mainly target the node's dependency information (DP) and the node's write node information (WN). Operations related to the node's dependency information (DP) primarily involve dependency propagation and updating; that is, for each edge, how is the dependency information (DP) of the source node (Src) passed to the destination node (Dst), and how is the dependency information (DP) of the destination node updated? For the node's write node information (WN), when the source node (Src) writes to the destination node (Dst), the write node information (WN) of the source node (Src) should be added to the write node information of the destination node (Dst), and the write node information (WN) of the destination node (Dst) needs to be cleared when its dependencies change. Therefore, the state update of the write node information (WN) mainly involves determining changes in the node dependency information (DP). Since the dependency representations of the node's full dependency information (ALL_DP) and the node's partial dependency information (PART_DP) are different, the state update for both cases will be explained in detail below.

[0132] 2.1 Node Dependency Information ALL_DP Case State Information Update

[0133] In the node dependency information ALL_DP, the node dependency information DP is represented by a set. Each element in the set is a unique identifier for a node. For each edge parsed from the event, formula (1) is used to transmit the node dependency information between nodes: For an edge with source node Src and destination node Dst, let the initial dependencies of the nodes be respectively and After the edge information is passed, the destination node Dst depends on... The result is obtained by performing a union operation between the DP of the source node Src and Src itself, then a union operation between the DP of Dst and the source node Src, and finally a difference operation between the source node Src and Dst itself. Then we use formula (2) as the judgment condition for whether the semantics have changed: if the dependency set of the destination node has not changed before and after the event, then the semantics are considered to have not changed; otherwise, they have changed. The target write node information of the destination node is updated according to whether it has changed.

[0134]

[0135]

[0136] For example, such as Figure 3 The diagram illustrates the state changes of node dependency information DP under the condition of ALL_DP. In the diagram, P1 and P2 are process nodes, and F is a file node. At T=1, P1 writes to file F; at T=2, P2 reads from file F. The initial dependencies of P1, F, and P2 are an empty set {}, a set containing dependent node R {R}, and an empty set {}, respectively. After time T1, the DP of F is calculated as {R, P1} according to formula (1). DP changes, and the WN state of F is set to empty. After time T2, the dependency of P2 is calculated as {R, P1, F}. DP changes, and the WN state is set to empty.

[0137] 2.2 Node Dependency Information: State Information Update in PART_DP Case

[0138] In the node dependency information PART_DP, node dependencies are represented by a list with a maximum length of L, storing at most L dependent nodes. Because PART_DP cannot record all node dependencies, dependency changes may go undetected due to missing dependencies. For example, ... Figure 4 As shown, the P dependency changes at times T=1 and T=3, but due to length limitations, the dependency list appears to have the same dependency.

[0139] To address the aforementioned problems, this invention adds node version information as a solution. Whenever a change in node dependency is detected, the node's version is incremented, and the node is appended with version information N. v As a node identifier in the dependency record, changes to the node's dependencies will reflect version differences and will always be passed to Dst. For example, ... Figure 5 As shown, NV represents a node plus version, N represents a node, and V represents a version. Since the dependency F changes when T=2, and version F-0 is added to F-1, even though the dependency record length is only 1, the change in node dependency can be transmitted through the version, accurately detecting the change in node dependency of P, so as to accurately update WN.

[0140] Therefore, in PART DP, the node dependency information is stored in the PART portion, and the node identifier with version information is used as an element of the dependency list. For an edge with source node Src and destination node Dst, let the initial dependencies of the nodes be as follows: and We use formula (3) to pass node dependency information between nodes: First, we use the append function to add the DP of Dst, the DP of Src, and Src to the end of the list in sequence (the end represents the latest node dependency). Then, we use the pruning function to delete duplicate nodes near the beginning of the list and keep the end of the list with a dependency length of L as the destination node dependency after the state update. We use formula (4) to perform a difference operation to obtain the new node dependency information dp after the Dst state update. new Using DP new As a condition for determining whether node dependencies have changed: if dp new If the value is not empty, it means that the node dependency has changed; otherwise, the node dependency has not changed.

[0141]

[0142]

[0143] In PART_DP, when node N v Dependencies change and versions increase to N v+1 There exists a set dp that causes changes in dependencies. new If it causes N v The node X∈dp depends on the change new Depends on node N v It also depends on dp new Then node X should also depend on N. v+1 To prevent the loss of this dependency, we use formula (5) to make a judgment and update the dependency version of X. The preservation of this dependency can eliminate redundant edges in the cyclic scenario.

[0144]

[0145] For example, such as Figure 6 As shown, an example scenario of dependency version update is demonstrated (NV represents node plus version): Process P and socket S exchange information via send and recv. When T=2, since S-1, which causes the dependency change of P, also depends on S-1 (assuming that the node always depends on itself) and P-0, updating the dependency P-0 of S-1 to P-1 will not cause the dependency change of S when process P writes to S.

[0146] For example, such as Figure 7 The example shown illustrates a scenario where a dependency on a version update occurs: after an event happens, the dependency information of node B changes, and at this point, dp... new ={C, D, A}, then for each newly added dependency information dp new The nodes in the dp are judged according to formula (5). new The nodes in the process perform dependency version updates. In this example, node C depends on dp. new Node B-0 meets the conditions in formula (5), so B-0 is updated to B-1. Nodes A and C do not meet the conditions in formula (5), so their node dependency information does not need to be updated.

[0147] 3. Edge compression determination

[0148] By recording and updating the state information of nodes, edge compression can be determined, thus implementing a compression algorithm. The edge compression determination process is as follows: For an edge with source node Src and destination node Dst, if Src's WN contains Dst, it means that the Src node has already been written to Dst without changing its dependencies. Therefore, this edge can be compressed without affecting the causal relationship. Since Src has not changed and has already been written to Dst, the information flow of this edge has been parsed, so no further state update is needed after edge compression. If the compression determination fails, the information flow of the edge needs to be processed, and the node state information of Src and Dst needs to be updated.

[0149] The overall process of the algorithm is as follows: Figure 8 As shown: For each edge Ei, the source node is Si, and the destination node is Di. The initialization information of each node includes WN and DP. WN is the set of nodes written while keeping node dependencies unchanged, and DP records node dependency information (refer to the above content for specific implementation details related to dependencies in the steps).

[0150] To facilitate understanding the algorithm process, an example is provided. Figure 9This demonstrates a node state change process in a cyclic compression scenario: In the diagram, process P (node ​​P) interacts with network socket S (node ​​S). Initially, both node DP and WN are empty. When T=1, P sends information to S, and node S is added to P's WN. Due to dependency transitivity, P-0 is added to S's DP, and the dependency changes, so S's version changes from 0 to 1, and the set of write nodes with unchanged dependencies, WN, is set to empty. Since P's WN did not contain S before writing, this edge does not need to be compressed. When T=2, P receives information from S, so P's dependency becomes S-1, its version changes from 0 to 1, and WN is set to empty. Node P is added to S's WN. Since node S, which caused the change in P, depends on P-0 before the dependency change and S-1, which caused the dependency change, the dependency version is updated, changing the dependency from P-0 to P-1. When T=3, P sends information to S, and node S is added to WN. Since S's dependency has not changed, there are no other changes. When T=4, since S's WN contains P, this edge can be compressed when P reads information from S. If the dependencies between the two subsequent nodes remain unchanged, then all subsequent mutual read-write edges can be compressed.

[0151] 4. Applicable Scenarios for Compression Algorithms

[0152] The final compression method is applicable to the following compression scenarios.

[0153] 4.1 Compression in multi-node read / write scenarios

[0154] During normal system activity, scenarios may occur where processes continuously read information from one file and write it to another. For example, ... Figure 10 As shown in the diagram, the directed edges represent the direction of information flow, and T on the edge represents the event that occurs. Process A writes node information to file B at T=1, 2, 4, and 6; process C reads information from file B at T=3, 5, and 7; and process A receives new information input at T=5, causing a semantic change. Therefore, for process A before the semantic change at T=5, only the edge corresponding to T=1 needs to be retained. Similarly, for file B before the semantic change at T=6, only the edge corresponding to T=1 needs to be retained. It should be noted that the nodes in process A are equivalent to those in the above method.

[0155] 4.2 Compression of Interaction Scenarios Between Multiple Nodes

[0156] Read-write interactions between processes and network nodes are also a common scenario. This compression method is suitable for mutual read-write interactions between multiple nodes. If the number of node dependencies to be retained is L, it can resolve redundant edges in cyclic read-write operations between L+1 nodes. Figure 11As shown, network socket S exchanges information with process A. Assume the node dependency retention length L is 1. Then, at T=3, network socket S and process A have the same semantic information, and both processes write to each other's nodes without changing dependencies. Therefore, after T=3, since the node dependencies remain unchanged, the mutual read and write operations between the two nodes can be compressed. For multi-node interactions, parent and child processes using pipes for read and write operations is also a common scenario. For example,... Figure 12 As shown, process A and process B exchange information through pipe P. At T=6, the node dependency remains unchanged, and subsequent event edges involving A transmitting information to B via P and B transmitting information to A via P can be compressed. It should be noted that process A, process B, and pipe P are equivalent to nodes in the above method.

[0157] In summary, this invention addresses the construction and compression of source graphs by compressing event edges based on the definition of node dependencies, while maintaining the source node dependencies. This invention is applicable to the compression of various source graph structures and can be used for real-time graph construction. It improves the compression effect of source graphs while preserving their attack investigation capabilities.

[0158] Verifying the compression effect reveals that it's not uncommon for node A to repeatedly write to node B when its dependencies remain unchanged, such as a process continuously writing information to a file or a process repeatedly reading and writing to network nodes. Furthermore, in a source graph, the number of edges is often several times, or even tens of times, greater than the number of nodes. Previous studies assumed a change in node dependencies simply because a node had an input edge, ignoring cases where a node had an input flow but its dependencies remained unchanged, thus limiting the application scenarios of graph compression. This invention, by recording node dependencies, expands the range of edge compression that preserves causal relationships, thereby enhancing the capabilities of graph compression.

[0159] From a causal perspective, if the results of forward and backward analysis remain unchanged, it indicates that the causal relationships between nodes are preserved, and the compressed source graph can still be used in attack investigation scenarios. This will be demonstrated below from the perspectives of forward and backward analysis:

[0160] For each node A, if an anomaly is detected at time T, the forward analysis traverses the outgoing edges of node A with a time greater than T. If an anomaly is detected at time T, then node A must have received new information flow input causing the anomaly, meaning the node's dependencies have changed. As the method provided by this invention shows, all outgoing edges after the node's dependencies have changed will not be compressed; therefore, all nodes corresponding to outgoing edges after the anomaly occurs (at time T) will be preserved, ensuring that the results of the forward analysis are fully retained.

[0161] Similarly, the backward analysis traverses all incoming edges from node A with a time less than T. Since the deleted incoming edges are all redundant edges that have already been written to node A and whose source node dependencies remain unchanged, the nodes corresponding to all input edges before the anomaly (time T) will also be invalid, thus ensuring the integrity of the forward impact analysis.

[0162] This invention also provides a real-time source graph compression device that preserves causal relationships, comprising:

[0163] The node state recording module is used to obtain the event stream from the log information source, parse the event stream into edges and the corresponding source and destination nodes, and obtain the initial state information of the source and destination nodes.

[0164] The node state update module is used to update the initial state information of the source node and the destination node based on the information flow of the edge, and generate the target state information.

[0165] The edge compression module is used to determine whether to compress edges based on the target state information and the initial state information, so as to realize the compression of the source graph.

[0166] like Figure 13 As shown, this embodiment of the invention also provides an electronic device, which may include: a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The processor 1301 can call logical instructions in the memory 1303 to execute the methods provided in the above embodiments, such as including:

[0167] The event stream is obtained from the log information source, parsed into edges and their corresponding source and destination nodes, and the initial state information of the source and destination nodes is obtained. The initial state information of the source and destination nodes is updated based on the edge information stream to generate target state information. Based on the target state information and the initial state information, it is determined whether the edges should be compressed to achieve the compression of the source graph.

[0168] Furthermore, when the logical instructions in the aforementioned memory 1303 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A real-time source graph compression method that preserves causal relationships, characterized in that, include: The event stream is obtained from the log information source, and the event stream is parsed into edges and the corresponding source nodes and destination nodes. The initial state information of the source nodes and the destination nodes is obtained. The initial state information includes initial node full dependency information and initial write node information. The initial node full dependency information uses a set to record node dependencies, and the node write node information is write node information with unchanged node dependencies. The initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information; the target state information includes target node full dependency information and target write node information; specifically including: The target write node information of the source node is equal to the initial write node information of the source node plus the target node; The target node full dependency information of the source node is equal to the initial node full dependency information of the source node; The target node full dependency information of the destination node is equal to the initial node full dependency information of the source node, the union of the initial node full dependencies of the source node and the target node, minus the set of the target node. The target node full dependency information of the destination node is calculated using formula (1): in, This represents the initial node's full dependency information for the source node. Indicates the source node, This indicates the initial node's full dependency information for the destination node. Indicates the destination node. This represents the target node's full dependency information; The target write node information of the destination node is empty if the target node full dependency information of the destination node is not the same as the initial node full dependency information of the destination node, and is equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (2): ； Based on the target state information and the initial state information, it is determined whether the edges should be compressed in order to achieve the compression of the source graph.

2. The real-time source graph compression method that preserves causal relationships according to claim 1, characterized in that, Alternatively, the initial state information may include initial node dependency information and initial write node information; Wherein, the initial node partial dependency information is queue information of length L, and the initial write node information is write node information with unchanged node dependencies; Correspondingly, the target state information includes target node partial dependency information and target write node information.

3. The real-time source graph compression method that preserves causal relationships according to claim 2, characterized in that, The initial node partial dependency information and the target node partial dependency information also include node version information; Correspondingly, the initial state information of the source node and the destination node is updated based on the information flow of the edge to generate target state information, including: The target write node information of the source node is equal to the initial write node information of the source node plus the target node; The target node partial dependency information of the source node is equal to the initial node partial dependency information of the source node; The target node's dependency information is updated as follows: First, the append function is used to add the initial node partial dependency information of the destination node, the initial node partial dependency information of the source node, and the source node to the end of the list in sequence; Then, the pruning function is used to delete duplicate information near the beginning of the list, and the tail of the list with a dependency length of L is retained as the target node partial dependency information of the target node. The target node dependency information of the target node is updated using formula (3): in, This indicates the target node's partial dependency information. This indicates the initial node's dependency information for the destination node. This represents the initial node dependency information of the source node. Indicates the source node, This indicates the addition of a function. Represents the reduction function; The target write node information of the destination node is empty when the target node partial dependency information of the destination node is different from the initial node partial dependency information of the destination node; if they are the same, it is equal to the initial write node information of the destination node. The update of the target write node information of the destination node is determined by formula (4): in, This indicates the dependency information of the newly added node in the destination node. If not empty, the target node information to be written to is empty. If empty, the target write node information of the destination node is equal to the initial write node information of the destination node.

4. The real-time source graph compression method that preserves causal relationships according to claim 1, characterized in that, Determining whether to compress the edges based on the target state information and the initial state information to achieve the compression of the source graph includes: Determine whether the initial state information and target state information of the destination node are the same as those of the source node. If they are the same, compress the edges.

5. The real-time source graph compression method that preserves causal relationships according to claim 3, characterized in that, Determining whether to compress the edges based on the target state information and the initial state information to achieve the compression of the source graph includes: Determine whether the initial state information and target state information of the destination node are the same as those of the source node. If they are the same, compress the edges. If they are different, then update the target node version.

6. The real-time source graph compression method that preserves causal relationships according to claim 5, characterized in that, The method further includes: When the Not empty, and node When, if node The node dependency information includes the destination node. And dependent on Then the node Depends on Updated version Nodes are determined using formula (5). Dependency version updates: in, Indicates the destination node. Represents a node The node part depends on information.

7. A real-time source graph compression device that preserves causal relationships, characterized in that, include: The node state recording module is used to obtain an event stream from the log information source, parse the event stream into edges and the corresponding source and destination nodes, and obtain the initial state information of the source and destination nodes; wherein, the initial state information includes initial node full dependency information and initial write node information; wherein the initial node full dependency information uses a set to record node dependencies, and the node write node information is write node information with unchanged node dependencies; A node state update module is used to update the initial state information of the source node and the destination node based on the information flow of the edge, generating target state information; the target state information includes target node full dependency information and target writing node information; the node state update module is specifically used for: The target write node information of the source node is equal to the initial write node information of the source node plus the target node; The target node full dependency information of the source node is equal to the initial node full dependency information of the source node; The target node full dependency information of the destination node is equal to the initial node full dependency information of the source node, the union of the initial node full dependencies of the source node and the target node, minus the set of the target node. The target node full dependency information of the destination node is calculated using formula (1): in, This represents the initial node's full dependency information for the source node. Indicates the source node, This indicates the initial node's full dependency information for the destination node. Indicates the destination node. This represents the target node's full dependency information; The target write node information of the destination node is empty if the target node full dependency information of the destination node is not the same as the initial node full dependency information of the destination node, and is equal to the initial write node information of the destination node if they are the same. The update of the target write node information of the destination node is determined by formula (2): ； An edge compression module is used to determine whether the edge should be compressed based on the target state information and the initial state information, so as to realize the compression of the source graph.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-6.

Images