A graph index migration method and device based on DCG

Abstract

The present invention discloses a graph index migration method and device based on DCG. DCG is a method for constructing indexes in a dynamic graph pattern matching algorithm. Index migration can migrate valid indexes in the original DCG to the DCG corresponding to the query graph and discard invalid indexes. The graph index migration method based on DCG is mainly divided into the following four parts: generation of a query tree, determination of a matching order, construction of an index, adjustment of a query tree and migration of an index. The method proposed in the present invention is mainly used in network security APT attack pattern recognition scenarios, and can be used when the initial data is small. Therefore, the method can ensure its effectiveness for a long time, ensure millisecond-level return results, and save a lot of storage space.

Description

Technical Field

[0001] The present invention relates to the field of streaming graph data processing, and in particular to a DCG-based graph index migration method and device. Background Art

[0002] Graph pattern matching technology solves the problem of finding all subgraphs within a data graph that satisfy the constraints of a given pattern graph. Early graph pattern matching algorithms could only be used on static data graphs, meaning that the structure and content of the data graphs did not change over time.

[0003] However, with the rapid development of information technology, mobile applications that touch every aspect of daily life are proliferating, generating massive amounts of data. As time passes, the graphs constructed using this data constantly change. In some current application areas, graph pattern matching algorithms must deliver rapid matching results despite the ever-changing structure and content of the graph, ensuring real-time and effective results. How to handle these constantly changing graphs and deliver accurate matching results with minimal latency is a current research hotspot.

[0004] Dynamic graph pattern matching is a key technology currently used to analyze dynamic graph changes. It solves the following problems: given a pattern graph, in a dynamically changing data graph, for each data graph insertion, it can identify in real time the newly added subgraph that matches the pattern graph; and for each data graph deletion, it can identify in real time the deleted subgraph that matches the pattern graph. Compared to traditional methods, dynamic graph pattern matching is more suitable for scenarios with high real-time requirements. For example, in recommendation systems, after a user browses some content, the application often needs to quickly provide recommendations based on the user's current and previously browsed content. In the field of network security monitoring, faced with a constant influx of access requests, algorithms must be able to quickly and accurately identify whether a new request is likely to be a malicious attack, thereby quickly determining whether access should be blocked. Furthermore, in the industrial sector, some large enterprises have also launched their own distributed parallel graph processing systems to solve large-scale dynamic graph matching problems.

[0005] In network security APT attack pattern recognition, some traditional classic attack patterns will first be abstractly modeled into the form of pattern graphs, and then analyzed based on the real-time network packet data received online, so as to perform real-time dynamic graph pattern matching.

[0006] DCG (Data-Centric Graph) is a method for building indexes in algorithms that solve dynamic graph pattern matching problems. Its specific definition is as follows:

[0007] A Data-Centric Graph (DCG) is a multigraph in which multiple edges can exist between two nodes. Suppose there is a data graph G = (V, E, L), a pattern graph Q = (V, E, L), and a data center graph D = (V, E, L). D also needs to meet the following conditions:

[0008] (1) Require v∈VG

[0009] (2) Requires L(e)∈ID(V) and Type(e)∈{Null,Implicit,Explicit}

[0010] (3) Require And v1=v3,v2=v4

[0011] In the second condition above, lD(VQ) represents the ID of all nodes in the pattern graph, and Type(e) represents the edge type of the data center graph. Each edge is one of the three types mentioned above.

[0012] Path: Given a graph G = (V, E, L), if starting from a node v in the graph, a finite number of traversals can lead to another node v`, then the path 〈v, ..., v`〉 between these two nodes is denoted as v→v`. Furthermore, if the length of this path is greater than 2, the path 〈v, ..., vp, v`〉 can also be denoted as v→vp.v`.

[0013] Definition of edge types in DCG: Given a data graph G, a pattern graph Q, and a data center graph D, after converting the pattern graph into a tree, denote the root node of the pattern tree as us, and denote the node in the data graph that matches us as vs. For an edge e = (v, v`, ID(u`)) in D, check the following two conditions:

[0014] (1) This path can match a path us→Parent(u).u` in the pattern graph.

[0015] (2) Require And subtree can completely match subtree`

[0016] If both of the above conditions are met, the edge is called an explicit edge. If only the first condition is met, the edge is called an implicit edge. If neither condition is met, there is no need to create this edge in D and it is called a null edge.

[0017] Currently, many dynamic graph pattern matching algorithms save intermediate results by building indexes. By building indexes, when the data graph structure or content is updated, the algorithm does not need to execute the static graph pattern matching algorithm on the entire graph, which greatly speeds up the dynamic graph pattern matching. However, in actual use scenarios, dynamic graph pattern matching algorithms need to formulate a series of matching strategies based on the data graph before building the index. When the initial data is small, the formulated strategies may not guarantee their performance after a certain number of updates to the data graph. Therefore, there is a need for a dynamic graph pattern matching technology that can solve the above problems. It can effectively solve the problems that may arise in the actual use of dynamic graph pattern matching algorithms and ensure the long-term effectiveness of the algorithm. Summary of the Invention

[0018] This paper aims to address the shortcomings of existing dynamic graph pattern matching techniques by proposing a DCG-based graph index migration technique for streaming big data processing. This method allows the algorithm to construct indexes in the DCG even when the initial data graph is insufficient, and migrates the indexes to the new DCG after subsequent data graph updates, ensuring the long-term effectiveness of the algorithm.

[0019] With this processing technology, the indexes in the original DCG can be migrated to the new DCG with the adjusted query tree. The migration operation can retain valid indexes and delete invalid indexes, greatly reducing the excessive space usage and reduced algorithm matching speed caused by invalid indexes.

[0020] The object of the present invention is achieved through the following technical solutions: In a first aspect, the present invention provides a DCG-based graph index migration method, the method comprising the following steps:

[0021] (1) Generation of query tree

[0022] (1.1) Construct a pattern graph and select a point in the pattern graph as the root node of the pattern tree;

[0023] (1.2) Use the root node to perform a breadth traversal of the pattern graph. During the traversal, a pattern tree is constructed based on the access order and the index is maintained. The constructed pattern tree is regarded as a directed graph. For edges whose actual directions are opposite to the parent-child relationship in the tree, their corresponding flags are set to true, indicating that the direction is opposite to the actual direction. For edges that exist in the pattern graph but not in the pattern tree, they are recorded as non-tree edges corresponding to the current pattern tree.

[0024] (2) Determination of matching order

[0025] After the pattern tree is generated according to step (1), the order of matching pattern tree nodes in the matching process is determined; the subsequent matching pattern nodes are selected by the index maintained during the matching process of step (1);

[0026] (3) Building an index

[0027] When a node in the data graph can be matched with the root node of the pattern graph, and there is a corresponding explicit edge between the node and the virtual node in the DCG, it means that there is a subgraph in the data graph that matches the pattern graph, and the result matching starts from this node.

[0028] (4) Adjust the query tree and migrate the index

[0029] Set an index number threshold. When the number of indexes affected by a graph operation reaches the index number threshold, the pattern tree is adjusted. During the adjustment process, first execute step (1) to reselect the root node of the pattern tree. The entire pattern tree needs to be rebuilt from the newly selected root node. Then execute step (2) to re-determine the current matching order and migrate the indexes in the original DCG to the new DCG. When migrating, first migrate the stable index that can actually produce matching results. After migrating the stable index, migrate the remaining indexes. At this time, it is necessary to judge the validity of these indexes before migrating or discarding them. When the migration is completed, use the new index to match the results.

[0030] Furthermore, in step (1), the node with the least number of matches and the lowest degree in the pattern graph is selected as the root node.

[0031] Furthermore, in step (4), if the reselected root node is the same as the current root node, it means that there is no need to adjust the pattern tree and migrate the index.

[0032] Furthermore, in step (4), given a pattern tree Q and a DCG, find the edges between the data nodes and virtual nodes that match the root node of the pattern tree in the DCG. If an edge e is an explicit edge, then starting from the child node of e, find a subgraph sub composed entirely of explicit edges in the DCG. If this subgraph can match Q, then all DCG edges in sub are called stable indexes.

[0033] In the second aspect, the present invention also provides a DCG-based graph index migration device, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the DCG-based graph index migration method.

[0034] In a third aspect, the present invention further provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the DCG-based graph index migration method is implemented.

[0035] The beneficial effects of the present invention are:

[0036] 1) The method of converting the directed query graph into a single pattern tree can avoid the problem of poor matching performance of DCG-based graph pattern matching methods on directed graphs.

[0037] 2) After a certain amount of updates to the data graph, the query tree is adjusted and index migration is performed. This can avoid storing too many invalid indexes in the system and allow the system to use more effective query trees for index construction and result matching. This reduces the number of invalid indexes stored in the system, optimizes the distribution of indexes, and improves the algorithm's efficiency in time and space. This makes the algorithm more usable in engineering practice and can guarantee its performance for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 An instance that generates a directed query graph into a single query tree.

[0039] Figure 2 To migrate various possible situations in unstable indexes.

[0040] Figure 3 This is a structural diagram of a DCG-based graph index migration device provided by the present invention. DETAILED DESCRIPTION

[0041] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0042] This invention is primarily applicable to the field of streaming big data processing for network security APT attack pattern recognition. It adjusts the query graph used and migrates valid indexes based on the current graph matching situation. For example, the validity of the query tree generated by the query graph can be determined based on the current index construction and the matching results, and the query tree can be adjusted and the index migrated. The invention provides a method for generating a query tree, using the query tree to construct an index in a distributed graph (DCG), and then determining whether the index remains valid in the new DCG based on the current index construction situation. This allows the system to adaptively adjust the query tree structure, delete invalid indexes, ensure algorithm performance, and reduce system overhead. Since the invention is targeted at network security APT attack pattern recognition scenarios, the practical significance of the corresponding abstract algorithm model in the current application scenario of the invention is explained here. A node represents a host node in a network environment, and the node ID is the unique host identifier in that network environment. A directed edge between nodes represents the presence of an attack between them, and the direction of the edge represents the direction of the attack. The data stream accessed by the algorithm model is network data packets detected by network sensors, from which individual element events can be parsed, representing an attack path initiated by host A to host B, abstracted as a directed edge in the algorithm description. The original point-edge is still used for the convenience of description, and no further details are given. The present invention provides a graph index migration method based on DCG, which includes the following steps:

[0043] (1) Generation of pattern diagram

[0044] When performing graph pattern matching for online businesses, we first identify the attack pattern we want to identify and then model it as a graph. The nodes in the graph represent host nodes in the network. Directed edges between nodes indicate the presence of an attack between them, and the direction of the edge represents the direction of the attack.

[0045] (2) Generation of query tree:

[0046] First, a point is selected as the root node of the pattern tree, and then the root node is used to perform a breadth traversal of the pattern graph, and the pattern tree is constructed according to the access order during the traversal. In this method, the pattern graph generates a pattern tree in two steps. The first step is to select a point as the root node of the pattern tree, and the second step is to use the root node to perform a breadth traversal of the pattern graph, and the pattern tree is constructed according to the access order during the traversal. Usually, the root node needs to select a point with the least number of matches and the lowest degree in the data graph, which can ensure that the search tree has as few branches as possible. In the constructed pattern tree, a reverse mark is set for each edge. The constructed pattern tree is regarded as a directed graph. For the edges whose actual direction is opposite to the parent-child relationship in the tree, the corresponding mark will be set to true, indicating that it is opposite to the actual direction. For edges that exist in the pattern graph but do not exist in the pattern tree, they are recorded as non-tree edges corresponding to the current pattern tree. In this way, a directed pattern graph can be converted into a single pattern tree. Figure 1 This figure shows how a directed query graph is converted into a query tree based on the data graph. The left side shows the data graph, the center shows the initial directed query graph, and the right side shows the generated query tree. Edges in the generated query tree that are opposite in direction to the actual edges are indicated by dashed lines.

[0047] (3) Determination of matching order:

[0048] After generating the pattern tree according to the previous method, the next step is to determine the order in which the pattern tree nodes are matched during the matching process. Different matching orders can significantly impact algorithm performance, as the matching process is essentially a deep search of the data graph. A good matching order can significantly reduce the number of inefficient recursive function calls, thereby speeding up graph operations. This method uses the index maintained during the previous matching process to select subsequent matching pattern nodes. Specifically, the set of edges in the pattern tree is denoted as edgeSet. All edges in edgeSet are traversed, each time finding an edge whose parent node exists in the current matching order and whose pattern edge has the fewest explicit indexes. Having the fewest explicit indexes indicates that the subtree rooted at that node has the fewest matches in the data graph, thus minimizing the generation of useless intermediate results. Therefore, that node is selected as the next matching point, the edge is removed from edgeSet, and the child nodes of the current edge are added to the corresponding matching order set. This process is repeated until edgeSet is empty.

[0049] (4) Build index:

[0050] After determining the matching order, we need to begin indexing the DCG. This indexing method is the same as that used in common DCG-based dynamic graph matching algorithms. Specifically, when a node in the data graph matches the root node of the pattern graph, and there is a corresponding explicit edge between that node and the virtual node in the DCG, it indicates that a subgraph in the data graph matches the pattern graph. At this point, we can begin matching results from that node.

[0051] (5) Adjust the query tree and migrate the index:

[0052] After processing a certain number of graph updates, the originally determined pattern tree may need to change. In the present invention, a threshold is set. When the number of indexes affected by a graph operation reaches the threshold, the pattern tree is adjusted. The process of adjusting the query tree and determining the matching order is the same as the above steps. During the adjustment process, the second step is first executed to reselect the root node of the pattern tree. If the selected root node is the same as the current root node, it means that the pattern tree does not need to be adjusted. If they are different, the entire pattern tree needs to be rebuilt from the current newly selected root node. After completing the adjustment of the pattern tree, the third step is executed to determine the current matching order. After completing the above two steps, if the pattern tree has changed, the index in the original DCG needs to be migrated to the new DCG. When migrating, it is necessary to migrate the stable index first. The so-called stable index is the index that can actually produce matching results. This type of index will not be invalid no matter how the pattern tree is adjusted. Specifically, given a pattern tree Q and a DCG, find the edges between the data nodes and virtual nodes that match the root node of the pattern tree in the DCG. If an edge e is an explicit edge, then starting from the child node of e, find a subgraph sub composed entirely of explicit edges in the DCG that can match Q. All DCG edges in sub are called stable indexes. Because they can already produce matching results, they are definitely not useless indexes. After migrating the stable indexes, the remaining indexes need to be migrated. At this time, it is necessary to judge the validity of these indexes before migrating or discarding them. When the migration is completed, the new index can be used to match the results. Figure 2 The following shows the different indexes in the original DCG and the new DCG. The specific migration process is described in detail below.

[0053] a) After determining the new pattern tree, you're ready to begin migrating indices. At this point, you can directly migrate all indices that meet the stable edge definition without further judgment. This is because the resulting matching results remain unchanged regardless of how the pattern tree changes, so stable edges will not become invalid indices as the pattern tree changes.

[0054] b) Then the formal migration process begins. At this time, you need to process the graph operations while migrating the original index. The current index is still used during the migration process. There are four possible situations during the migration process, namely Figure 2 The initial query tree is as follows: Figure 2 As shown on the far left, assuming the pattern tree before adjustment is as follows Figure 2 As shown, R is the root node of the pattern tree, and R contains N1 to N kThe subtree connected to each of these k child nodes is represented by the triangle below, and the subtree can be empty. For the first case, assume that the parent node of the pattern edge corresponding to the index to be migrated is A, node A and the root node NR of the new pattern tree are both in the subtree of R's child node N, and NR is A's ancestor node. This is because the path from node A to the new root node NR will be shortened, and A's subtree has not changed. According to the definition of the index type, it can be known that the existing index type will not change. At this time, the index result can be directly migrated to the DCG corresponding to the new pattern tree; for the second case, assume that the parent node of the pattern edge corresponding to the index to be migrated is A, node A and the root node NR of the new pattern tree are both in the subtree of R's child node N, and A is NR's ancestor node. It uses a node in the subtree of the root node of the original pattern tree as the root node of the new pattern tree. This is slightly more complicated because the direction of the index is reversed. In this case, if the index is explicit, it should be discarded. Determining whether the sub-pattern tree rooted at A matches the data graph is computationally expensive, so the index should be rebuilt during the matching process. For implicit indexes, the current index needs to be determined to match the root node. If so, the implicit index can be retained and migrated to the DCG corresponding to the new pattern tree. For the third case, assume that the parent node of the pattern edge corresponding to the index to be migrated is A. Node A and the root node NR of the new pattern tree are both in the sub-tree rooted at R's child node N, and A and NR share a common ancestor node C. The original path from node A to the root node is roughly A->C->N->R. In the adjusted pattern tree, the path from node A to the root node becomes A->C->NR. In this case, while the sub-tree of node A remains unchanged, a portion of the path from node A to the root node has been replaced. Therefore, the only requirement is to determine whether the index remains valid after replacing this portion of the path. Specifically, the requirement is to determine whether this replaced path can find a matching path p in the data graph, and whether p can be added to the index. If the conditions are met, the index can be migrated to the DCG corresponding to the new pattern tree; otherwise, the index is deleted. In the fourth case, assume that the parent node of the pattern edge corresponding to the index to be migrated is A, and node A and the root node NR of the new pattern tree are in subtrees rooted at different child nodes of R. A is in the subtree of node N, and NR is in the subtree of node N'. In the new pattern tree, the path from node A to root node NR has an additional path, R->N'->NR, while the subtree of node N remains unchanged. This case is similar to the third case, so the handling method is the same and will not be repeated here.

[0055] c) After all indexes are migrated, the new indexes are used to overwrite the original indexes, and the new indexes are used for matching starting from the next graph operation.

[0056] The pseudo code for migrating a stable index is as follows:

[0057]

[0058] The pseudo code for migrating an unstable index is as follows:

[0059]

[0060]

[0061] Corresponding to the aforementioned embodiment of a DCG-based graph index migration method, the present invention also provides an embodiment of a DCG-based graph index migration device.

[0062] See also Figure 3 A DCG-based graph index migration device provided by an embodiment of the present invention includes a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a DCG-based graph index migration method in the above embodiment.

[0063] The embodiment of the graph index migration device based on DCG provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown in the figure, a hardware structure diagram of a device with data processing capability where a graph index migration device based on DCG is provided by the present invention is located. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0064] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0065] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0066] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a DCG-based graph index migration method in the above embodiment is implemented.

[0067] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0068] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A DCG-based graph index migration method, characterized in that: The method comprises the following steps: (1) Generation of query tree (1.1) Construct a pattern graph and select a point in the pattern graph as the root node of the pattern tree; (1.2) Use the root node to perform breadth traversal of the pattern graph, build a pattern tree according to the access order during the traversal process, and maintain the index; The constructed pattern tree is regarded as a directed graph. For the edges whose actual directions are opposite to the parent-child relationships in the tree, their corresponding flags are set to true, indicating that they are opposite to the actual directions. For the edges that exist in the pattern graph but not in the pattern tree, they are recorded as non-tree edges corresponding to the current pattern tree. (2) Determination of matching order After the pattern tree is generated according to step (1), the order of matching pattern tree nodes in the matching process is determined; the subsequent matching pattern nodes are selected by the index maintained during the matching process of step (1); (3) Building an index When a node in the data graph can be matched with the root node of the pattern graph, and there is a corresponding explicit edge between the node and the virtual node in the DCG, it means that there is a subgraph in the data graph that matches the pattern graph, and the result matching starts from this node; (4) Adjust the query tree and migrate the index Set an index number threshold. When the number of indexes affected by a graph operation reaches the index number threshold, the pattern tree is adjusted. During the adjustment process, first execute step (1) to reselect the pattern tree root node. The entire pattern tree needs to be rebuilt from the newly selected root node. Then execute step (2) to redetermine the current matching order and migrate the index in the original DCG to the new DCG; During migration, you first need to migrate the stable index that can actually generate matching results. After migrating the stable index, you need to migrate the remaining indexes. At this time, you need to determine the validity of these indexes before migrating or discarding them. After the migration is complete, use the new index to match the results.

2. A DCG-based graph index migration method according to claim 1, characterized in that: In step (1), the node with the least number of matches and the lowest degree in the pattern graph is selected as the root node.

3. The DCG-based graph index migration method according to claim 1, characterized in that: In step (4), if the reselected root node is the same as the current root node, it means that there is no need to adjust the pattern tree and migrate the index.

4. The DCG-based graph index migration method according to claim 1, characterized in that: In step (4), given a pattern tree Q and a DCG, find the edges between the data nodes and virtual nodes that match the root node of the pattern tree in the DCG. If an edge e is an explicit edge, then starting from the child node of e, find a subgraph sub composed entirely of explicit edges in the DCG. If this subgraph can match Q, then all DCG edges in sub are called stable indexes.

5. A graph index migration device based on DCG, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a DCG-based graph index migration method as described in any one of claims 1 to 4.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a DCG-based graph index migration method as described in any one of claims 1 to 4 is implemented.

Images