Distributed sampling method, device and equipment for triangle counting in dynamic graph streams

Abstract

Embodiments of the present application disclose a distributed sampling method and device for dynamic graph stream triangle counting and equipment. The method adopts Master-Worker-Aggregator architecture, the master node guides the edge sampling behavior of the worker node by using the edge hash value, plans the sample graph of the worker node, simultaneously, all the worker nodes are regarded as a cluster as a whole, when a triangle is sampled by any worker node, it is equivalent to that the cluster samples the triangle, and the corresponding counting information is sent to the aggregation node for processing, so that the dependence relationship between the sample edge sets of different machines is effectively utilized, the influence of the covariance caused by the shared edge triangle is significantly reduced, and the accuracy of the estimation result is improved, and compared with the existing simple parallel single machine flow algorithm, smaller estimation error can also be obtained by using shorter running time.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of triangle counting technology, and in particular to a distributed sampling method, apparatus, and device for triangle counting in dynamic graph streams. Background Art

[0002] In the real world, graphs are widely used in various fields because they provide an effective way to represent complex relationships and structures. Nowadays, large amounts of data are often modeled and analyzed in the form of graphs. This data includes social media networks, collaboration and co-authorship networks, transportation networks, and various types of biological networks. For example, most social networks have large user bases, typically ranging from millions to billions. In the era of big data, studying and analyzing these massive networks presents significant memory challenges. A common strategy to address this is to analyze these networks using sampling techniques to obtain approximate results. Triangle counting is a core problem in network science. It is a computationally complex but important graph statistic. Real-world networks often contain a large number of triangles, while randomly generated networks rarely exhibit this characteristic. Triangle counting involves many important metrics in network analysis, including the clustering coefficient of individual vertices, the clustering coefficient of the entire network, transitivity ratio, and triangle connectivity. Triangle counting can provide a deep understanding of network structure and relationships. For example, in social networks, triangle counting is used to discover communities, detect fake accounts, and determine the roles of users in the network. Triangle counting is also used in many graph data mining and database applications, such as detecting web spam, revealing hidden topic relationships in the web, mining dense subgraphs, database query optimization, link recommendation, anomaly detection, etc.

[0003] There are many accurate stand-alone triangle counting algorithms. However, due to their high computation time and memory requirements, accurate stand-alone triangle counting algorithms are not suitable for processing large graphs. In order to process large graphs, another widely used method is to extend the triangle counting algorithm to a distributed environment to count large graphs. However, unlike streaming algorithms, these distributed counting algorithms need to obtain information about all edges in the graph at once. Therefore, they are not suitable for streaming. Figure 3Because edges in streaming graphs are continuously received over time, it's impossible to obtain all edge data at once. However, many real-world graphs are better represented by streaming graphs, as these graphs constantly change over time. The streaming graph sampling algorithm estimates the number of triangles in the graph stream by traversing the input edge stream once. During the traversal, the triangle count estimate is incrementally updated as each edge arrives, requiring only a portion of the graph's edges to be stored. Therefore, the streaming graph sampling algorithm is well-suited for maintaining and updating approximate triangle counts in dynamic graph streams.

[0004] In dynamic graph data streams, the graph's topology is constantly changing, including edge addition and deletion operations. However, existing streaming algorithms for approximate triangle counting primarily focus on processing graph data streams involving only edge insertion operations. Most streaming graph sampling algorithms can only handle edge insertion operations, not edge deletion operations. Only a few streaming graph sampling algorithms are capable of handling edge deletions, but these algorithms suffer from low estimation accuracy. Furthermore, these algorithms that do handle edge deletions are designed to run on a single computer. To leverage multiple machines to obtain more accurate triangle count estimates, a simple and effective approach is to directly parallelize existing single-machine streaming sampling algorithms. Specifically, the graph data can be broadcast to all machines, and then the single-machine streaming algorithm can be run independently on each machine. Since each machine can provide an unbiased estimate of the triangle count, a more accurate triangle count estimate can be obtained by averaging the estimates provided by these machines. This approach reduces the variance of the estimate, thereby improving the accuracy of the estimate.

[0005] However, the estimation error of existing single-machine streaming algorithms is significantly affected by the covariance of triangle pairs sharing edges. Although simple parallelization of single-machine streaming algorithms can improve estimation accuracy to a certain extent, it cannot effectively reduce the covariance caused by triangle pairs sharing edges. Currently, no other streaming algorithm can effectively utilize multiple machines to work together to achieve faster or more accurate estimation of the number of triangles in dynamic graph data streams. Summary of the Invention

[0006] Embodiments of the present invention provide a distributed sampling method, apparatus, and device for dynamic graph flow triangle counting to solve the problem that the estimation error of the existing dynamic flow graph sampling algorithm is significantly affected by the covariance caused by shared edge triangle pairs, resulting in inaccurate estimation results.

[0007] In a first aspect, an embodiment of the present invention provides a distributed sampling method for triangle counting in dynamic graph streams, the method comprising:

[0008] For each edge operation of the dynamic graph stream to be processed, a master node randomly generates a hash value of the edge operation according to a preset hash function mapping range, and sends the edge operation and the hash value to multiple worker nodes; the edge operation types include adding edges and deleting edges;

[0009] Determine, by the working node, a sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the sampling triangle determined according to the edge operation to an aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the working node;

[0010] The aggregation node determines a global triangle count estimate of the entire graph and a local triangle count estimate associated with each vertex in the graph based on the received counting information.

[0011] Optionally, determining a sampling triangle formed by the input edge of the edge operation in the current local sample graph includes:

[0012] Common neighbor vertices of the two vertices of the input edge are determined in the local sample graph, and each of the common neighbor vertices and the two vertices of the input edge form a sampling triangle.

[0013] Optionally, the sending of the counting information of the sampled triangles determined according to the edge operation to the aggregation node includes:

[0014] For each of the sampled triangles, the estimated change in the number of local triangles associated with the common neighbor vertices is determined based on the edge operation and the sampling probability of the sampled triangle and sent to the aggregation node, and the sum of the estimated change in the number of local triangles of all the common neighbor vertices is counted as the estimated change in the number of local triangles associated with the two vertices of the input edge and the estimated change in the number of global triangles and sent to the aggregation node.

[0015] Optionally, if the edge operation type is adding an edge, the estimated change in the number of local triangles associated with the common neighbor vertices is 1 / q[uvw]; if the edge operation type is deleting an edge, the estimated change in the number of local triangles associated with the common neighbor vertices is -1 / q[uvw]; where q[uvw] = |W| / m 2 ; q[uvw] represents the sampling probability of the sampling triangle {u,v,w} formed by vertices u, v and w, |W| represents the number of working nodes, and m represents the number of mappable hash values.

[0016] Optionally, the method further includes:

[0017] If the number of available worker nodes is greater than the number of mappable hash values, the available worker nodes are divided into multiple groups so that the number of target worker nodes in each group is less than or equal to the number of mappable hash values, and a target master node and a secondary aggregation node are configured for each group of target worker nodes.

[0018] For each edge operation of the dynamic graph stream to be processed, each target master node randomly generates a target hash value of the edge operation according to the preset hash function mapping range, and sends the edge operation and each target hash value to each target working node of the corresponding group;

[0019] Determine, by the target working node, a local sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the local sampling triangle determined according to the edge operation to the corresponding secondary aggregation node; if the received target hash value corresponds to the identifier of the target working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the target working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the target working node;

[0020] Each secondary aggregation node determines a group global triangle number estimation value and a group local triangle number estimation value according to the received counting information sent by each target working node of the corresponding group, and sends them to the main aggregation node;

[0021] The main aggregation node determines the global triangle number estimation value of the current entire graph based on the received global triangle number estimation values ​​of each group, and determines the local triangle number estimation value related to each vertex in the graph based on the received local triangle number estimation values ​​of each group.

[0022] Optionally, dividing the available working nodes into multiple groups includes:

[0023] Define k1 as the integer part of |W'| / m, and k2 as the remainder part of |W'| / m, where |W'| represents the number of available work nodes and m represents the number of mappable hash values. If k2=0, then the available work nodes are divided into k1 groups, each of which includes m target work nodes. If k2≠0, then the available work nodes are divided into k1+1 groups, with the first k1 groups each including m target work nodes, and the last group including k2 target work nodes.

[0024] Correspondingly, if k2=0, determining the estimated value of the global number of triangles of the entire current graph based on the received estimated values ​​of the global number of triangles of each group includes:

[0025]

[0026] The determining of the local triangle number estimation value related to each vertex in the graph according to the received local triangle number estimation value of each group comprises:

[0027]

[0028] Correspondingly, if k2≠0, the determining of the global triangle number estimation value of the current entire graph according to the received global triangle number estimation value of each group comprises:

[0029]

[0030] The determining of the local triangle number estimation value related to each vertex in the graph according to the received local triangle number estimation value of each group comprises:

[0031]

[0032] Wherein, represents the global triangle number estimation value at time t, represents the jth group of global triangle number estimation value at time t, represents the local triangle number estimation value related to vertex u at time t, represents the jth group of local triangle number estimation value at time t, represents the k1th group of global triangle number estimation value calculated at time t, represents the k1+1th group of global triangle number estimation value at time t, represents the k1th group of local triangle number estimation value related to vertex u calculated at time t, represents the k1+1th group of local triangle number estimation value at time t, w1 represents the first weight, and w2 represents the second weight.

[0033] In a second aspect, the embodiment of the present application further provides a distributed sampling device for dynamic graph stream triangle counting, which comprises:

[0034] An edge operation distribution module is configured to, for each edge operation of a to-be-processed dynamic graph stream, generate a hash value of the edge operation randomly according to a preset hash function mapping range through a master node, and send the edge operation and the hash value to a plurality of worker nodes; the type of the edge operation comprises adding an edge and deleting an edge;

[0035] an edge operation processing module, configured to determine, through the working node, a sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the sampling triangle determined according to the edge operation to an aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the working node;

[0036] The triangle count estimation module is used to determine the global triangle count estimation value of the current entire graph and the local triangle count estimation value related to each vertex in the graph through the aggregation node according to the received counting information.

[0037] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0038] one or more processors;

[0039] a memory for storing one or more programs;

[0040] When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed sampling method for triangle counting of dynamic graph streams provided by any embodiment of the present invention.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the distributed sampling method for triangle counting of dynamic graph streams provided by any embodiment of the present invention.

[0042] In a fifth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program. When the program is executed by a processor, it implements the distributed sampling method for triangle counting of dynamic graph streams provided by any embodiment of the present invention.

[0043] An embodiment of the present invention provides a distributed sampling method for triangle counting in dynamic graph streams. For each edge operation of the dynamic graph stream to be processed, a hash value of the edge operation is first randomly generated by the master node according to a preset hash function mapping range, and the edge operation and the generated hash value are sent to multiple working nodes. Each working node then determines the sampling triangle formed by the corresponding input edge in its current local sample graph based on the received edge operation, and sends the counting information of the sampling triangle determined based on the edge operation to the aggregation node. After that, each working node determines whether the received hash value corresponds to its own identifier. If so, and the received edge operation type is to add an edge, the corresponding input edge is inserted into its local sample graph. At the same time, if the received edge operation type is to delete an edge, the hash value is not considered and the corresponding input edge is directly removed from its local sample graph. After receiving the counting information sent by the working node, the aggregation node can update the global triangle number estimate of the current entire graph and the local triangle number estimate related to each vertex in the graph based on the counting information. The distributed sampling method for triangle counting in dynamic graph streams provided by an embodiment of the present invention utilizes edge hash values ​​to guide the edge sampling behavior of working nodes, plans the sample graph of working nodes, and treats all working nodes as a cluster as a whole. When a triangle is sampled by any working node, it is equivalent to the cluster sampling the triangle, and the corresponding counting information is sent to the aggregation node for processing, thereby effectively utilizing the dependency between sample edge sets of different machines, significantly reducing the impact of the covariance caused by shared edge triangles, and thus improving the accuracy of the estimation results. At the same time, compared with the existing simple parallelized single-machine stream algorithm, a shorter running time can also be used to obtain a smaller estimation error. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flowchart of a distributed sampling method for triangle counting in dynamic graph streams provided in the first embodiment of the present invention;

[0045] Figure 2 This is an example of a global triangle and a local triangle in the figure provided in the first embodiment of the present invention;

[0046] Figure 3 This is an example of the change in the number of triangles in a dynamic graph stream provided by the first embodiment of the present invention;

[0047] Figure 4 This is a diagram of the EHADS algorithm framework provided in Example 1 of the present invention;

[0048] Figure 5 Eight different types of triangle pairs in the dynamic graph stream provided in the first embodiment of the present invention;

[0049] Figure 6 This is a diagram of the EHADS-g algorithm framework provided in Example 1 of the present invention;

[0050] Figure 7 Global absolute percentage error experimental results of various algorithms provided for the first embodiment of the present application on different data sets;

[0051] Figure 8 Root mean square error experimental results of various algorithms provided for the first embodiment of the present application on different data sets;

[0052] Figure 9 Running time experimental results of various algorithms provided for the first embodiment of the present application on different data sets;

[0053] Figure 10 Scalability experimental results of the EHADS algorithm provided for the first embodiment of the present application on different data sets;

[0054] Figure 11 Structural schematic diagram of the distributed sampling device for dynamic graph stream triangle counting provided for the second embodiment of the present application;

[0055] Figure 12 Structural schematic diagram of the computer device provided for the third embodiment of the present application. DETAILED DESCRIPTION

[0056] The present application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0057] Before discussing the example embodiments in more detail, it is noted that some example embodiments are described as processes or methods depicted as flowcharts. Although the processes are described in a particular sequential order, many of the steps can be performed in parallel, concurrently or in any order. In addition, the order of the steps can be re-arranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figure, which can also be performed after the processes are terminated. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0058] Embodiment One

[0059] Figure 1This is a flowchart of the distributed sampling method for triangle counting of dynamic graph streams provided in the first embodiment of the present invention. This embodiment is applicable to the situation where triangle counting of dynamic graph streams (i.e., a series of edge operations) is performed in real time in a distributed manner through multiple machines. The following realistic conditions are specifically considered: no prior knowledge, i.e., the algorithm does not have any prior knowledge about the input graph stream before processing the dynamic graph stream, such as information on the number of vertices and edges; no shared environment, i.e., the stored data on each machine are independent of each other, and the data stored on one machine cannot be accessed or shared by other machines; single traversal, i.e., each machine follows the principle of single traversal and processes the edge operations in the dynamic graph stream one by one in chronological order. Each machine can only access the edge operations currently being processed and the edges in the locally stored sample graph, but cannot access the edge operations that have been processed. This method can be executed by the distributed sampling device for triangle counting of dynamic graph streams provided in the embodiment of the present invention. The device can be implemented in hardware and / or software and can generally be integrated into a computer device. As Figure 1 As shown, the specific steps include:

[0060] S11. For each edge operation of the dynamic graph stream to be processed, the master node randomly generates a hash value of the edge operation according to a preset hash function mapping range, and sends the edge operation and the hash value to multiple working nodes; the types of the edge operations include adding edges and deleting edges.

[0061] S12. Determine, through the working node, the sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send the counting information of the sampling triangle determined according to the edge operation to the aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is to add an edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is to delete an edge, remove the input edge from the local sample graph of the working node.

[0062] S13. Determine, by the aggregation node according to the received counting information, a global triangle number estimate of the entire graph and a local triangle number estimate associated with each vertex in the graph.

[0063] This embodiment designs a distributed sampling algorithm (EHADS) based on the Master-Worker-Aggregator architecture for estimating the global and local triangle counts in dynamic graph streams. The EHADS algorithm architecture can include a master node, multiple worker nodes, and an aggregator node. The master node can be responsible for distributing edge operations, the worker nodes can be responsible for performing triangle sampling tasks with the highest time complexity and calculating the number of sampled triangles, and the aggregator node can calculate an unbiased estimate of the number of triangles based on the number of triangle samples provided by the worker nodes and the probability that a triangle is sampled by the worker node cluster.

[0064] Based on the problem to be solved, the distributed estimation problem of the number of global and local triangles in a dynamic graph stream can be defined as follows: The prerequisite is that there is a dynamic graph stream to be processed (e 1 ,e 2 ,…,e t ), and |W| worker nodes, W represents the set of worker nodes participating in the EHADS algorithm; the computational task is to calculate the actual global number of triangles |T in the dynamic graph flow to be processed for each time t∈{t1,t2,…} t The estimated value of | And every vertex u∈V t The actual number of local triangles |T t The estimated value of [u]| The optimization goal is to minimize the estimator and The deviation and variance of the algorithm are used to ensure that the estimated value given by the algorithm is as close to the real value as possible. The dynamic graph flow to be processed can be a dynamic undirected graph flow, and the dynamic graph flow to be processed Π can be defined as a time-based edge operation sequence (e 1 ,e 2 ,…,e t ), e t Denotes the edge operation arriving at time t. Let G t =(V t ,E t ) represents the real graph at time t, V t Representation graph G t The vertex set in E t Representation graph G t The edge set in graph G t It is generated gradually by operating on the first t edges in the dynamic graph stream Π to be processed, and the initial graph is an empty graph. For each edge {u,v}∈E t , that is, the edge between vertex u and vertex v, u≠v∈V t , then you can use e t={u,v,δ} represents the edge operation at time t, where the types of edge operations include adding edges and deleting edges, and δ∈{+,-} can be used to represent adding new edges (+) or deleting existing edges (-). t For three vertices u, v and w, if there is an edge between any two vertices, that is, e uv ,e uw ,e vw ∈E t , then the triangle formed by vertices u, v and w can be represented by {u, v, w}. In addition, T t Representation graph G t The set of global triangles, that is, graph G t All triangles in T t [u] represents graph G t For each vertex u∈V t The set of relevant local triangles, that is, all triangles containing vertex u. For example, Figure 2 As shown, vertex a has two local triangles, {a, b, d} and {a, c, d}, and there are a total of 6 global triangles in the entire graph.

[0065] Furthermore, you can use A t Represents the set of triangles added to the dynamic graph stream π to be processed, specifically A can be defined t is the set of triangles that have been added to the graph at or before time t, which can be expressed as:

[0066]

[0067] Among them, a time identifier s is introduced to distinguish triangles with the same vertex composition but added to the graph at different time points. t Represents the set of triangles deleted in the dynamic graph stream π to be processed, specifically D t is the set of triangles deleted from the graph at or before time t, which can be expressed as:

[0068]

[0069] Among them, the time identifier s is also introduced to distinguish triangles with the same vertex composition but deleted at different time points. Similarly, when it comes to each vertex u∈V t When Represents the set of triangles added containing vertex u, and uses Represents the set of deleted triangles containing vertex u. It should be noted that triangles composed of the same vertices can be added or deleted multiple times. For example, Figure 3As shown, the black solid line represents the edge that existed before the current moment, the green solid line represents the edge added at the current moment, and the red dotted line represents the edge deleted at the current moment. At time t, the edge {b,c} is added to the graph, which triggers the addition of two triangles {a,b,c} and {b,c,e} in the graph. At time t+2, the edge {b,c} in the graph is deleted, resulting in the removal of two triangles {a,b,c} and {b,c,e} in the graph. At time t+3, the edge {b,c} is re-added to the graph, which again triggers the addition of two triangles {a,b,c} and {b,c,e} in the graph. According to the above definition, the set of triangles added at time t is A t -A t-1 Similarly, the set of triangles deleted at time t is D t -D t-1 , in the exemplary Figure 3 In the set A t+3 -A t-1 That is, the set D includes the following elements: ({a,b,c},t), ({b,c,e},t), ({a,c,d},t+1), ({a,b,c},t+3), ({b,c,e},t+3). t+3 -D t-1 That is, it includes the following elements: ({a,b,c},t+2), ({b,c,e},t+2).

[0070] For the EHADS algorithm, Figure 4 As shown, the black edges in the dynamic graph stream Π to be processed represent the add edge operation, and the red edges represent the delete edge operation. These edge operations are input in chronological order. In the processing of the graph stream, the edge operation can flow from the data source node to the master node based on the time axis. The master node then uses a specific edge hash distribution strategy to determine which worker node will process and sample each edge operation. The worker node can calculate the number of triangles based on its own local sample graph and update the local sample graph based on the distribution of the master node. The aggregation node is responsible for aggregating the triangle counts from all worker nodes and performing corresponding processing to obtain a real-time estimate of the number of global and local triangles in the entire graph stream. Each worker node can store a local sample graph, which can be accessed through S i Represents the sample edge set currently saved by working node i, which can be further obtained by Represents the set of sample edges maintained by working node i at time t, and can be obtained through G i =(V i ,E i ) represents the sample edge set S currently stored by working node i i For each vertex u∈V in the graph i , can be obtained by N i[u] represents G i The neighbor vertex set of vertex u in the vertices can be further obtained by Represents the sample edge set at time t Composition diagram The input of the EHADS algorithm can include the dynamic graph stream Π to be processed and the preset hash function mapping range, and the output can include the global triangle number estimate An estimate of the number of local triangles associated with each vertex u

[0071] Specifically, for the master node, for each edge operation from the data source node, a random edge hash function h(·) can be used to uniformly and randomly map it according to the preset hash function mapping range to obtain the hash value of the edge operation. The correspondence between the hash value and the working node identifier can be established in advance. After obtaining the hash value of the edge operation, the corresponding working node can be determined based on the hash value. The number of mappable hash values ​​in the preset hash function mapping range is m. Some or all of the mappable hash values ​​can be mapped one-to-one to the identifiers of the working nodes participating in the EHADS algorithm. The probability that the edge operation is sampled by any of the working nodes is p=1 / m. When the number of available working nodes is less than or equal to the number of mappable hash values, all available working nodes can participate in the EHADS algorithm. When the number of available working nodes is greater than the number of mappable hash values, m working nodes can be selected to participate in the EHADS algorithm. Preferably, the preset hash function mapping range is the integer set {1, 2, ..., m}, and correspondingly, the identifications of the working nodes participating in the EHADS algorithm are {1, 2, ..., |W|}, that is, the hash value obtained by mapping can be directly used as the identification of the corresponding working node, then P(h(e t )=i)=1 / m, where i∈{1,2,…,|W|} and m≥|W|. After generating the hash value of the edge operation being processed, the master node can compare the edge operation and the corresponding hash value (e t ,h(e t )) is sent to all worker nodes participating in the EHADS algorithm. In particular, the master node can also generate corresponding hash values ​​for only edge operations of the add type and send them to the worker nodes, and for edge operations of the delete type, only send the edge operation to the worker nodes. For each delete edge operation, the local sample graphs of all worker nodes can be updated.

[0072] For example, the pseudo code of the EHADS algorithm for the master node is as follows:

[0073]

[0074]

[0075] For each working node participating in the EHADS algorithm, the edge sample set S stored locally can be firstly i Initialized to empty. Then, whenever an edge operation (e t ={u,v,δ},h(e t )), the Update function can be called to update the estimate maintained by the aggregation node. The update process can specifically determine the count information of the sampling triangle formed by the edge {u,v} and the two edges in the current local sample graph and send it to the aggregation node. Subsequently, if the received edge operation e t It is an add edge operation (δ=+) and the hash value corresponds to the identifier of the current working node (e.g. the hash value is the identifier of the current working node h(e t )=i), then the working node can call the Insert function to sample the edge {u, v}, that is, insert the input edge into the local sample graph of the working node itself. Correspondingly, if the received edge operation e t It is an edge addition operation and the hash value does not correspond to the identifier of this working node (such as h(e t )≠i), then this working node does not sample the edge {u,v}. If the received edge operation e t If it is a delete edge operation (δ=-), the work node can call the Delete function to remove the edge {u, v} from the local sample graph of the work node itself. That is, regardless of whether the hash value of the delete edge operation corresponds to the identifier of the work node, the Delete function is executed to remove the edge {u, v}.

[0076] Optionally, determining the sampling triangle formed by the input edge of the edge operation in the current local sample graph includes: determining the common neighbor vertices of the two vertices of the input edge in the local sample graph, each of the common neighbor vertices and the two vertices of the input edge forming a sampling triangle.

[0077] Specifically, for the current input edge {u, v}, each working node i can determine its current local sample graph G i The common neighbor vertices N of vertex u and vertex v i [u]∩N i [v] is used to determine the sampling triangle formed. Each common neighbor vertex w means the existence of the triangle {u, v, w}, that is, the working node i successfully sampled the triangle {u, v, w}.

[0078] Further optionally, the counting information of the sampled triangles determined according to the edge operation is sent to the aggregation node, including: for each of the sampled triangles, the change in the estimated value of the local triangle number related to the common neighbor vertex is determined according to the edge operation and the sampling probability of the sampled triangle, and sent to the aggregation node, and the sum of the estimated value changes of the local triangle number of all the common neighbor vertices is counted as the estimated value change of the local triangle number related to the two vertices of the input edge and the global estimated value change of the triangle number is sent to the aggregation node.

[0079] Specifically, for each common neighbor vertex w of the input edge {u, v} discovered by the working node i, the working node i needs to send the corresponding counting information to the aggregation node, so as to use this counting information to update the global triangle number estimate and the local triangle number estimate related to vertices u, v and w. Specifically, each time a common neighbor vertex w is discovered, the change in the local triangle number estimate related to the common neighbor vertex w is determined based on the type of the current edge operation and the sampling probability of the corresponding sampled triangle and sent directly to the aggregation node, while the change in the local triangle number estimate is counted locally. After traversing all discovered common neighbor vertices, the sum of the counted changes in the local triangle number estimate related to all common neighbor vertices is sent to the aggregation node as the change in the local triangle number estimate related to vertex u and vertex v and the change in the global triangle number estimate, so as to save network communication. The delta sent to the aggregation node can carry the object identifier. The delta of the local triangle count estimate associated with a common neighbor vertex w can be recorded as (w, x), the delta of the local triangle count estimate associated with vertex u can be recorded as (u, sum), the delta of the local triangle count estimate associated with vertex v can be recorded as (v, sum), and the delta of the global triangle count estimate can be recorded as (#, sum). In addition, the sum value can be initialized to 0 at the beginning of each call to the Update function.

[0080] Further optionally, if the type of the edge operation is to add an edge, the estimated change in the number of local triangles associated with the common neighbor vertices is 1 / q[uvw]; if the type of the edge operation is to delete an edge, the estimated change in the number of local triangles associated with the common neighbor vertices is -1 / q[uvw]; where q[uvw] = |W| / m 2 ; q[uvw] represents the sampling probability of the sampling triangle {u,v,w} formed by vertices u, v and w, |W| represents the number of working nodes, and m represents the number of mappable hash values.

[0081] Specifically, if the current edge operation e tIt is an edge addition operation. When the worker node i finds the common neighbor vertex w, it means that the worker node i has sampled the added triangle ({u,v,w},t)∈A t At this time, the working node i needs to send the relevant counting information to the aggregation node to increase the corresponding estimated value. The increase can be set to 1 / q[uvw]. t It is an edge deletion operation. When the working node i finds the common neighbor vertex w, it means that the working node i has sampled the deleted triangle ({u,v,w},t)∈D t At this time, the worker node i needs to send the relevant counting information to the aggregation node to reduce the corresponding estimated value. The reduction can be set to 1 / q[uvw]. Then the change in the estimated value of the number of local triangles related to each common neighbor vertex can be set to 1 / q[uvw] and -1 / q[uvw], corresponding to the above x, where the value of q[uvw] is |W| / m 2 This update mechanism can ensure that the mathematical expectation of the random variable corresponding to each triangle in the dynamic graph stream Π to be processed is exactly 1 or -1. For each added or deleted sampling triangle {u, v, w}, q[uvw] is the probability that the EHADS algorithm samples the triangle {u, v, w}. When the sample edge set S i When the first two edges {u,w} and {v,w} exist in S (i.e., the first two edges that appear in the dynamic graph stream π to be processed), the working node i can successfully sample the triangle {u,v,w}. i The probability is 1 / m 2 , that is, P(h({u,w})=i∧h({v,w})=i)=1 / m 2 Since each edge operation is broadcasted by the master node to all worker nodes, the probability of a triangle being sampled is not affected by the third arriving edge. As long as the edges {u, w} and {v, w} are mapped to the same worker node by the edge hash function h(·), the EHADS algorithm can successfully sample the triangle {u, v, w}. Therefore, the sampling probability of sampling the triangle {u, v, w} is:

[0082]

[0083] Moreover, each edge deletion operation in the graph stream will update the local sample graph of each working node. Therefore, triangles with the same vertex composition but added or deleted at different time points have the same probability of being sampled.

[0084] Exemplarily, the pseudo code of the EHADS algorithm for the working node (each working node i∈W) is as follows:

[0085]

[0086] For the aggregation node, the counting information sent by all working nodes participating in the EHADS algorithm can be collected and summarized, and corresponding processing can be performed, such as calculating and updating the corresponding global triangle number estimate or vertex-related local triangle number estimate based on the received object identifier. Specifically, the calculation can be achieved by adding the current locally stored estimate (initially 0) to the corresponding received change, so as to obtain an unbiased estimate of the global triangle number of the entire graph and the local triangle number related to each vertex in the graph in real time.

[0087] Exemplarily, the pseudo code of the EHADS algorithm for the aggregation node part is as follows:

[0088]

[0089]

[0090] In the EHADS algorithm, when each worker node samples the triangle {u, v, w}, it can immediately send the corresponding counting information to the aggregation node to update the global triangle number estimate and the local triangle number estimate related to each vertex. Furthermore, processing can be performed based on user query operations. When the user does not query the number of triangles in the graph stream, a lazy aggregation strategy can be introduced to minimize the network communication between the worker node and the aggregation node. Specifically, before the user issues a triangle number query request, the worker node can temporarily partially aggregate the counting information obtained based on multiple edge operations locally to avoid sending the counting information of each edge operation to the aggregation node. When the user issues a triangle number query request, the worker node will then send the pre-aggregated local counting results to the aggregation node for final global summary and calculation. This can reduce unnecessary network communication before the user query, improving algorithm efficiency and resource utilization.

[0091] In the EHADS algorithm, the master node uses a random edge hash function to map each edge operation to each worker node with equal probability (there is also a probability that it will not be mapped to any worker node). This ensures a balanced distribution of sample edges, that is, the mathematical expectation of the number of sample edges stored locally by all worker nodes is equal. For any time t, when m≥|W|, the EHADS algorithm can provide G t An unbiased estimate of the number of global and local triangles in , defining and They represent that the EHADS algorithm processes e t Given later and Then we can get:

[0092]

[0093] The types of triangle pairs in the current graph stream are defined to provide an accurate classification of triangle pairs, thereby providing a theoretical basis for variance research. For each added or deleted triangle ({u, v, w}, s) ∈ A t ∪D t , introduce indicator function 1 ({u,v,w},s) Represents the add edge operation associated with the first edge reached by the triangle, and the indicator function 2 ({u,v,w},s) represents the add edge operation associated with the second edge of the triangle. Then for any two different triangles σ≠ω∈A t ∪D t , the type of its ordered pair is defined as follows:

[0094]

[0095] For example, Figure 5 As shown, the cases where triangles in the dynamic graph stream to be processed share edges can be divided into eight types: Figure 5 The green edge in the middle represents an added triangle σ∈A t The third arriving edge in the graph, the red edge represents a deleted triangle σ∈D t The third arriving edge in . Specifically, Figure 5 The triangle pairs of types 1-4 share only one edge, and the corresponding edge operation for this shared edge is (t3,{u,v,+}), while the triangle pairs of types 5-8 share two edges, and the corresponding edge operations are (t1,{u,w,+}) and (t2,{v,w,+}). Figure 5 Only one case of each type of triangle pair is shown in Figure 3; in reality, there may be multiple cases of each type.

[0096] For any time t≥1, define A t ∪D t The number of triangle pairs of type i in , and also define A t [u]∪D t The number of triangle pairs of type i in [u]. and The variance of satisfies:

[0097]

[0098] It can be seen that the EHADS algorithm effectively reduces the covariance caused by triangle pairs that share edges. In particular, when m = |W|, the EHADS algorithm can completely eliminate the covariance caused by triangle pairs of types 1-4 mentioned above.

[0099] The complexity of the EHADS algorithm is analyzed. Assuming that each working node participating in the EHADS algorithm uses an adjacency list to store the local sample graph in the local memory, the time complexity of the master node is O(t·|W|), the space complexity is O(1), and the time complexity of the working node is O(t 2 / m), the space complexity is O(t / m), and the time complexity of aggregating nodes is O(t 2 |W| / m), the space complexity is O(|V t |).

[0100] On the basis of the above technical solution, optionally, the method further includes: if the number of available working nodes is greater than the number of mappable hash values, the available working nodes are divided into multiple groups so that the number of target working nodes in each group is less than or equal to the number of mappable hash values, and a target master node and a sub-aggregation node are configured for each group of target working nodes; for each edge operation of the dynamic graph flow to be processed, the target hash value of the edge operation is randomly generated by each target master node according to the preset hash function mapping range, and the edge operation and each target hash value are sent to each target working node of the corresponding group; the local sampling triangle formed by the input edge of the edge operation in the current local sample graph is determined by the target working node, and the counting information of the local sampling triangle is determined according to the edge operation and sent. to the corresponding secondary aggregation node; if the received target hash value corresponds to the identifier of the target working node and the type of the edge operation is to add an edge, the input edge is inserted into the local sample graph of the target working node; if the type of the edge operation is to delete an edge, the input edge is removed from the local sample graph of the target working node; each secondary aggregation node determines the grouped global triangle number estimate and the grouped local triangle number estimate according to the received counting information sent by each target working node of the corresponding group, and sends them to the main aggregation node; the main aggregation node determines the global triangle number estimate of the current entire graph according to the received global triangle number estimate of each group, and determines the local triangle number estimate related to each vertex in the graph according to the received local triangle number estimate of each group.

[0101] Specifically, when the number of available working nodes is greater than the number of mappable hash values, that is, there are a large number of machines available to calculate the estimated value of the number of triangles in the dynamic graph stream Π to be processed, the above-mentioned EHADS algorithm can be improved to obtain the EHADS-g algorithm. Figure 6As shown, the EHADS-g algorithm can adopt a new Master-Worker-Aggregator architecture with multiple master nodes and hierarchical aggregators (multiple secondary aggregation nodes and one master aggregation node), the master node set is M, and the secondary aggregation node set is C, so as to give full play to the distributed processing capabilities of the available work node cluster and provide more accurate estimates. Specifically, the available work nodes can be divided into multiple groups so that the number of target work nodes in each group is less than or equal to the number of mappable hash values, and a target master node and a secondary aggregation node are configured for each group of target work nodes. For each edge operation of the dynamic graph stream Π to be processed, each group of target master nodes, target work nodes and secondary aggregation nodes can respectively use the above-mentioned EHADS algorithm to obtain the group global triangle number estimate and the group local triangle number estimate of the group and send them to the master aggregation node, so that the master aggregation node finally determines the global triangle number estimate of the current entire graph and the local triangle number estimate related to each vertex in the graph based on the received group global triangle number estimate and group local triangle number estimate of each group. The input of the EHADS-g algorithm can include the dynamic graph stream Π to be processed and the preset hash function mapping range, and the output can include the global triangle number estimate An estimate of the number of local triangles associated with each vertex u Among them, the j-th target master node uses its own edge hash function h j (·) Generate the target hash value for the jth group of target worker nodes. The edge hash functions of each target master node are independent, so the triangle count estimates provided by each group of target worker nodes are also independent. The master aggregator can ultimately aggregate the estimates based on the preset weights of each group. Specifically, it can assign greater weights to estimates with smaller variances, i.e., it can assign greater weights to groups with a larger number of target worker nodes.

[0102] Further optionally, the dividing the available working nodes into multiple groups includes: defining k1 as the integer part of |W'| / m and k2 as the remainder part of |W'| / m, where |W'| represents the number of available working nodes and m represents the number of mappable hash values; if k2=0, the available working nodes are divided into k1 groups, each group including m target working nodes; if k2≠0, the available working nodes are divided into k1+1 groups, the first k1 groups each including m target working nodes, and the last group including k2 target working nodes; accordingly, if k2=0, determining the global triangle number estimate of the current entire graph based on the received global triangle number estimate of each group includes:

[0103]

[0104] The step of determining the estimated value of the number of local triangles associated with each vertex in the graph based on the received estimated values ​​of the number of local triangles in each group includes:

[0105]

[0106] Correspondingly, if k2≠0, determining the estimated value of the global number of triangles of the entire current graph based on the received estimated values ​​of the global number of triangles of each group includes:

[0107]

[0108] The step of determining the estimated value of the number of local triangles associated with each vertex in the graph based on the received estimated values ​​of the number of local triangles in each group includes:

[0109]

[0110] in, represents the estimated value of the global number of triangles at time t, represents the estimated number of global triangles in the jth group at time t, represents the estimated number of local triangles associated with vertex u at time t, represents the estimated number of local triangles in the jth group at time t, represents the estimated value of the global number of triangles between groups calculated by the first k1 groups at time t, represents the estimated number of global triangles in the k1+1th group at time t, represents the estimated number of local triangles between groups related to vertex u calculated by the first k1 groups at time t, represents the estimated number of local triangles in the k1+1th group at time t, w1 represents the first weight, and w2 represents the second weight.

[0111] Specifically, when |W'|>m, we can define k2=|W'|%m, that is, |W'|=k1m+k2, where k1≥1 and 0≤k2 <m。EHADS-g算法将|W’|个可用工作节点分成了k1或k1+1组,前k1组每组包括m个目标工作节点,若有第k1+1组,则最后一组包括k2个目标工作节点。从而可以尽可能的减小共享边的三角形对引起的协方差,提高最终估计值的准确性。同时根据数学期望的线性性质,可以推导出 and It's G tUnbiased estimates of the number of global triangles and the number of local triangles in . Among them, the first weight w1 and the second weight w2 are the weights of two given independent and unbiased estimates. The estimate with smaller variance should be given a larger weight. Obviously, The variance is less than The variance of The variance is less than The variance of .

[0112] For example, the pseudo code of the EHADS-g algorithm is as follows:

[0113] Target master node (each target master node j∈M):

[0114]

[0115] Target worker nodes (each target worker node i∈W'):

[0116]

[0117]

[0118] Secondary aggregation node (each secondary aggregation node j∈C):

[0119]

[0120] Master aggregation node:

[0121]

[0122] Based on the above technical solution, the performance of the EHADS algorithm was tested using a real-world graph dataset to evaluate its accuracy and speed compared to existing streaming algorithms. The algorithm also verified that its runtime scales linearly with the number of edge operations in the input graph stream. In the experiments, self-loops and the direction of each edge were excluded. The dataset used in the experiments is available from the Stanford Network Analysis Platform (SNAP). The original graph dataset was processed and converted into a streaming graph. An edge add operation was created for each edge in the order in which the edges were input. Since the original dataset does not include edge deletion information, 20% of the edges in the graph were randomly selected and edge delete operations were introduced at random positions after the corresponding edge add operations. The experiments were performed on a cluster of 32 machines, each equipped with a 2.60 GHz Intel(R) Xeon(R) E5-2650 v2 CPU and 128 GB of memory. The EHADS algorithm and a series of baseline algorithms can be implemented using the Scala programming language and the Spark framework. These baseline algorithms include existing single-machine streaming algorithms such as Triest, ThinkD, WRS, and RFES. In the WRS algorithm, the relative size α of the waiting room can be set to 10%. Furthermore, because the auxiliary space required by the RFES algorithm typically significantly exceeds the pre-defined sample size (by several or even dozens of times), to ensure experimental fairness, the EHADS algorithm is only compared with the RFES algorithm in the Q2 experiment. All baseline algorithms are further simply parallelized, running them independently on multiple machines. For each baseline algorithm, the final estimate is averaged across all worker nodes. Clearly, the variance of the estimates provided by the simply parallelized algorithms is smaller than that of the estimates provided by the single-machine streaming algorithms. In the experiments, the EHADS algorithm is compared with these simply parallelized streaming algorithms.

[0123] By τ and Indicates the actual number of global triangles and local triangles at the end of the input graph stream, and is passed through and Represents the estimated values ​​of global and local triangles obtained by the algorithm. To evaluate the accuracy of the estimated values ​​provided by each algorithm, the global absolute percentage error (GAPE) and the root mean square error (RMSE) can be used. In the experiment, each evaluation metric for all algorithms was calculated multiple times (e.g., 20 times) and the average value was recorded to ensure the reliability of the experimental results. Among them, the global absolute percentage error is used to measure the difference between the estimated value of the global triangle number and the actual value. The lower the metric, the better. It is expressed as:

[0124]

[0125] The root mean square error is used to measure the closeness between the estimated value of the number of local triangles and the true value. The lower the indicator, the better. It is expressed as:

[0126]

[0127] The number of working nodes of each algorithm is fixed at 20, and then the sample size of the working nodes is changed, and the sample size of each working node does not exceed 25% of the edge set size of the dataset. Then the performance of the EHADS algorithm and each baseline algorithm in terms of accuracy is compared. Figure 7 and Figure 8 As shown in the figure, as the sample size of the worker nodes increases, the errors of all distributed streaming algorithms in estimating the global and local triangle counts show a downward trend. In estimating the global triangle count, the EHADS algorithm consistently maintains the lowest estimation error across the six datasets. In estimating the local triangle count, the EHADS algorithm also exhibits the smallest error. It is worth noting that in the Email-Enron and Gowalla datasets, the improvement in the local triangle count estimation accuracy of the EHADS algorithm is not significant compared to the WRSDel-Parallel and ThinkDacc-Parallel algorithms. In these two datasets, when the sample size of the worker nodes is low, the RMSE indicator of the EHADS algorithm is slightly higher than that of the WRSDel-Parallel and ThinkDacc-Parallel algorithms.

[0128] The performance of the EHADS algorithm and various baseline algorithms in terms of running time is further compared, and the number of working nodes of all algorithms is set to 10. Figure 9 The figure shows the running time (in seconds) of each algorithm when the GAPE index is less than 1% (that is, when the estimation accuracy reaches 99%). It is worth noting that the sample size used by the algorithm is not restricted in the experiment, that is, the memory space usage of all algorithms is different. Figure 9 It can be clearly seen that under the same estimation accuracy, the EHADS algorithm also shows the shortest running time. This shows that the EHADS algorithm can more effectively utilize computing resources while maintaining high accuracy, making it more advantageous in terms of running time.

[0129] Further verification of the scalability of the EHADS algorithm, that is, to study the running time of the EHADS algorithm under different edge operation quantities. The parallel degree of the EHADS algorithm is set to 10, and the sample capacity of each worker node is set to 10% of the edge set size of the data set. For the three data sets of Gowalla, DBLP and NotreDame, the EHADS algorithm needs to process the edge operation quantity t=r·|E|, and the value range of r is {0.1, 0.2, …, 0.9, 1.0}, r represents the proportion of the edge operation quantity to the edge set size of the data set. As shown in Figure 10 It can be seen that the running time of the EHADS algorithm shows a linear correlation trend with the edge operation quantity on the Gowalla, DBLP and NotreDame data sets.

[0130] The technical scheme provided by the embodiment of the application is that, for each edge operation of the to-be-processed dynamic graph stream, the master node first generates a hash value of the edge operation by randomly generating the hash value of the edge operation according to a preset hash function mapping range, and sends the edge operation and the generated hash value to a plurality of worker nodes. Then each worker node determines a sampling triangle formed by the corresponding input edge in its current local sample graph according to the received edge operation, and sends the counting information of the sampling triangle to the aggregation node according to the edge operation. After that, each worker node judges whether the received hash value corresponds to the identity of the worker node itself, and if yes and the type of the received edge operation is an added edge, the corresponding input edge is inserted into the local sample graph of the worker node, and if the type of the received edge operation is a deleted edge, the corresponding input edge is removed from the local sample graph of the worker node without considering the hash value. After receiving the counting information sent by the worker nodes, the aggregation node can update the global triangle number estimation value of the current entire graph and the local triangle number estimation value related to each vertex in the graph according to the counting information. By using the edge hash value to guide the edge sampling behavior of the worker nodes, the sample graph of the worker nodes is planned, and all the worker nodes are used as a cluster as a whole. When a triangle is sampled by any worker node, it is equivalent to that the cluster samples the triangle, and the corresponding counting information is sent to the aggregation node for processing. Therefore, the dependence relationship between different machine sample edge sets is effectively utilized, the influence of the covariance caused by the shared edge triangle is significantly reduced, and the accuracy of the estimation result is improved. Compared with the existing simple parallel single-machine stream algorithm, a smaller estimation error can also be obtained using a shorter running time.

[0131] Embodiment two

[0132] Figure 11This is a schematic diagram of the structure of a distributed sampling device for triangle counting in dynamic graph streams provided in the second embodiment of the present invention. The device can be implemented in hardware and / or software and can generally be integrated into a computer device to execute the distributed sampling method for triangle counting in dynamic graph streams provided in any embodiment of the present invention. Figure 11 As shown, the device includes:

[0133] The edge operation distribution module 100 is configured to randomly generate a hash value of each edge operation of the dynamic graph stream to be processed according to a preset hash function mapping range through a master node, and send the edge operation and the hash value to multiple worker nodes; the edge operation types include adding an edge and deleting an edge;

[0134] The edge operation processing module 200 is configured to determine, through the working node, the sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send the counting information of the sampling triangle determined according to the edge operation to the aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is to add an edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is to delete an edge, remove the input edge from the local sample graph of the working node;

[0135] The triangle count estimation module 300 is configured to determine, through the aggregation node, a global triangle count estimation value of the entire graph and a local triangle count estimation value associated with each vertex in the graph based on the received count information.

[0136] The technical solution provided by the embodiment of the present invention is that for each edge operation of the dynamic graph stream to be processed, the master node first randomly generates a hash value of the edge operation according to the preset hash function mapping range, and sends the edge operation and the generated hash value to multiple working nodes. Then, each working node determines the sampling triangle formed by the corresponding input edge in its current local sample graph based on the received edge operation, and sends the counting information of the sampling triangle determined based on the edge operation to the aggregation node. After that, each working node determines whether the received hash value corresponds to its own identifier. If so and the received edge operation type is to add an edge, the corresponding input edge is inserted into its local sample graph. At the same time, if the received edge operation type is to delete an edge, the hash value is not considered and the corresponding input edge is directly removed from its local sample graph. After receiving the counting information sent by the working node, the aggregation node can update the global triangle number estimate of the current entire graph and the local triangle number estimate related to each vertex in the graph based on the counting information. By utilizing edge hash values ​​to guide the edge sampling behavior of working nodes, planning the sample graph of working nodes, and treating all working nodes as a cluster as a whole, when a triangle is sampled by any working node, it is equivalent to the cluster sampling the triangle, and the corresponding counting information is sent to the aggregation node for processing, thereby effectively utilizing the dependency between sample edge sets of different machines, significantly reducing the impact of the covariance caused by shared edge triangles, and thus improving the accuracy of the estimation results. At the same time, compared with the existing simple parallelized single-machine flow algorithm, a shorter running time can also be used to obtain a smaller estimation error.

[0137] On the basis of the above technical solution, optionally, the edge operation processing module 200 includes:

[0138] The sampling triangle determining unit is configured to determine common neighbor vertices of the two vertices of the input edge in the local sample graph, wherein each of the common neighbor vertices and the two vertices of the input edge form a sampling triangle.

[0139] On the basis of the above technical solution, optionally, the edge operation processing module 200 further includes:

[0140] A counting information determination unit is used to determine, for each of the sampled triangles, the change in the estimated value of the local triangle number associated with the common neighbor vertex based on the edge operation and the sampling probability of the sampled triangle, and send it to the aggregation node; and to count the sum of the estimated value changes of the local triangle number of all the common neighbor vertices as the change in the estimated value of the local triangle number associated with the two vertices of the input edge and the change in the estimated value of the global triangle number, and send it to the aggregation node.

[0141] On the basis of the above technical solution, optionally, if the type of the edge operation is to add an edge, the change in the estimated value of the number of local triangles associated with the common neighbor vertices is 1 / q[uvw]; if the type of the edge operation is to delete an edge, the change in the estimated value of the number of local triangles associated with the common neighbor vertices is -1 / q[uvw]; wherein, q[uvw] = |W| / m 2 ; q[uvw] represents the sampling probability of the sampling triangle {u,v,w} formed by vertices u, v and w, |W| represents the number of working nodes, and m represents the number of mappable hash values.

[0142] Based on the above technical solution, optionally, the distributed sampling device for triangle counting of dynamic graph streams further includes:

[0143] A working node partitioning module is configured to, if the number of available working nodes is greater than the number of mappable hash values, partition the available working nodes into multiple groups such that the number of target working nodes in each group is less than or equal to the number of mappable hash values, and configure a target master node and a secondary aggregation node for each group of target working nodes;

[0144] The target working master node working module is used to randomly generate a target hash value of each edge operation of the dynamic graph flow to be processed according to the preset hash function mapping range through each target master node, and send the edge operation and each target hash value to each target working node of the corresponding group;

[0145] a target working node working module, configured to determine, through the target working node, a local sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the local sampling triangle determined according to the edge operation to a corresponding secondary aggregation node; if the received target hash value corresponds to the identifier of the target working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the target working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the target working node;

[0146] The secondary aggregation node working module is used to determine the estimated value of the number of global triangles in the group and the estimated value of the number of local triangles in the group according to the received counting information sent by each target working node of the corresponding group through each secondary aggregation node, and send them to the main aggregation node;

[0147] The main aggregation node working module is used to determine the global triangle number estimation value of the current entire graph based on the received global triangle number estimation values ​​of each group through the main aggregation node, and determine the local triangle number estimation value related to each vertex in the graph based on the received local triangle number estimation values ​​of each group.

[0148] Based on the above technical solution, optionally, the working node division module is specifically used to:

[0149] Define k1 as the integer part of |W'| / m, and k2 as the remainder part of |W'| / m, where |W'| represents the number of available work nodes and m represents the number of mappable hash values. If k2=0, then the available work nodes are divided into k1 groups, each of which includes m target work nodes. If k2≠0, then the available work nodes are divided into k1+1 groups, with the first k1 groups each including m target work nodes, and the last group including k2 target work nodes.

[0150] Correspondingly, if k2=0, the main aggregation node working module is specifically used to:

[0151]

[0152]

[0153] Correspondingly, if k2≠0, the main aggregation node working module is specifically used to:

[0154]

[0155] in, represents the estimated value of the global number of triangles at time t, represents the estimated number of global triangles in the jth group at time t, represents the estimated number of local triangles associated with vertex u at time t, represents the estimated number of local triangles in the jth group at time t, represents the estimated value of the global number of triangles between groups calculated by the first k1 groups at time t, represents the estimated number of global triangles in the k1+1th group at time t, represents the estimated number of local triangles between groups related to vertex u calculated by the first k1 groups at time t, represents the estimated number of local triangles in the k1+1th group at time t, w1 represents the first weight, and w2 represents the second weight.

[0156] The distributed sampling device for triangle counting of dynamic graph streams provided by an embodiment of the present invention can execute the distributed sampling method for triangle counting of dynamic graph streams provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0157] It is worth noting that in the above-mentioned embodiment of the distributed sampling device for triangle counting of dynamic graph streams, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0158] Example 3

[0159] Figure 12 The structural diagram of the computer device provided in the third embodiment of the present invention shows a block diagram of an exemplary computer device suitable for implementing the embodiment of the present invention. Figure 12 The computer device shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention. Figure 12 As shown, the computer device includes a processor 121, a memory 122, an input device 123 and an output device 124; the number of processors 121 in the computer device can be one or more. Figure 12 Taking a processor 121 as an example, the processor 121, memory 122, input device 123 and output device 124 in the computer device can be connected through a bus or other means. Figure 12 The bus connection is taken as an example.

[0160] The memory 122, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the distributed sampling method for triangle counting in dynamic graph streams in the embodiments of the present invention (for example, the edge operation distribution module 100, edge operation processing module 200, and triangle count estimation module 300 in the distributed sampling apparatus for triangle counting in dynamic graph streams). The processor 121 executes the software programs, instructions, and modules stored in the memory 122 to execute various functional applications and data processing of the computer device, thereby implementing the above-mentioned distributed sampling method for triangle counting in dynamic graph streams.

[0161] The memory 122 may primarily include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the computer device, etc. Furthermore, the memory 122 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some instances, the memory 122 may further include memory remotely located relative to the processor 121, and these remote memories may be connected to the computer device via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The input device 123 may be used to obtain each edge operation of the dynamic graph stream to be processed, and to generate key signal input related to user settings and function control of the computer device. The output device 124 may include a display screen, which may be used to display the result of the triangle number estimation value to the user.

[0163] Example 4

[0164] A fourth embodiment of the present invention further provides a storage medium containing computer-executable instructions. When executed by a computer processor, the computer-executable instructions are used to perform a distributed sampling method for triangle counting in dynamic graph streams. The method includes:

[0165] For each edge operation of the dynamic graph stream to be processed, a master node randomly generates a hash value of the edge operation according to a preset hash function mapping range, and sends the edge operation and the hash value to multiple worker nodes; the edge operation types include adding edges and deleting edges;

[0166] Determine, by the working node, a sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the sampling triangle determined according to the edge operation to an aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the working node;

[0167] The aggregation node determines a global triangle count estimate of the entire graph and a local triangle count estimate associated with each vertex in the graph based on the received counting information.

[0168] The storage medium can be any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROMs, floppy disks, or tape devices; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disks or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system that is connected to the computer system via a network (such as the Internet). The second computer system may provide program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (e.g., in different computer systems connected via a network). The storage medium may store program instructions (e.g., embodied as a computer program) that may be executed by one or more processors.

[0169] Of course, the storage medium containing computer-executable instructions provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the distributed sampling method for triangle counting of dynamic graph streams provided by any embodiment of the present invention.

[0170] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0171] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0172] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0173] Example 5

[0174] Embodiment 5 of the present invention also provides a computer program product, which includes a computer program (also referred to as code, instructions). The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it is used to execute the distributed sampling method for triangle counting of dynamic graph streams provided in any of the above embodiments, and has the corresponding beneficial effects of the execution method.

[0175] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A distributed sampling method for triangle counting in dynamic graph streams, characterized by: include: For each edge operation of the dynamic graph stream to be processed, a master node randomly generates a hash value of the edge operation according to a preset hash function mapping range, and sends the edge operation and the hash value to multiple worker nodes; the edge operation types include adding edges and deleting edges; Determine, by the working node, a sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the sampling triangle determined according to the edge operation to an aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the working node; Determining, by the aggregation node, a global triangle count estimate for the entire graph and a local triangle count estimate associated with each vertex in the graph based on the received counting information; The method further comprises: If the number of available worker nodes is greater than the number of mappable hash values, the available worker nodes are divided into multiple groups such that the number of target worker nodes in each group is less than or equal to the number of mappable hash values, and each group of target worker nodes is assigned a target master node and a secondary aggregation node; wherein the available worker nodes represent machines that can be used to calculate an estimate of the number of triangles in the dynamic graph stream to be processed; For each edge operation of the dynamic graph stream to be processed, each target master node randomly generates a target hash value of the edge operation according to the preset hash function mapping range, and sends the edge operation and each target hash value to each target working node of the corresponding group; Determine, by the target working node, a local sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the local sampling triangle determined according to the edge operation to the corresponding secondary aggregation node; if the received target hash value corresponds to the identifier of the target working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the target working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the target working node; Each secondary aggregation node determines a group global triangle number estimation value and a group local triangle number estimation value according to the received counting information sent by each target working node of the corresponding group, and sends them to the main aggregation node; The main aggregation node determines the global triangle number estimation value of the current entire graph based on the received global triangle number estimation values ​​of each group, and determines the local triangle number estimation value related to each vertex in the graph based on the received local triangle number estimation values ​​of each group.

2. The distributed sampling method for triangle counting in dynamic graph streams according to claim 1 is characterized in that: The determining of a sampling triangle formed by the input edge of the edge operation in the current local sample graph includes: Common neighbor vertices of the two vertices of the input edge are determined in the local sample graph, and each of the common neighbor vertices and the two vertices of the input edge form a sampling triangle.

3. The distributed sampling method for triangle counting in dynamic graph streams according to claim 2, characterized in that: The step of determining the count information of the sampled triangles according to the edge operation and sending it to the aggregation node comprises: For each of the sampled triangles, the estimated change in the number of local triangles associated with the common neighbor vertices is determined based on the edge operation and the sampling probability of the sampled triangle and sent to the aggregation node, and the sum of the estimated change in the number of local triangles of all the common neighbor vertices is counted as the estimated change in the number of local triangles associated with the two vertices of the input edge and the estimated change in the number of global triangles and sent to the aggregation node.

4. The distributed sampling method for triangle counting in dynamic graph streams according to claim 3, characterized in that: If the edge operation type is to add an edge, the estimated change in the number of local triangles associated with the common neighbor vertices is 1 / q[uvw]; if the edge operation type is to delete an edge, the estimated change in the number of local triangles associated with the common neighbor vertices is -1 / q[uvw]; where q[uvw] = |W| / m 2 ; q[uvw] represents the sampling probability of the sampling triangle {u,v,w} formed by vertices u, v and w, |W| represents the number of working nodes, and m represents the number of mappable hash values.

5. The distributed sampling method for triangle counting in dynamic graph streams according to claim 1, characterized in that: The available working nodes are divided into multiple groups, including: Define k1 as the integer part of |W'| / m, and k2 as the remainder part of |W'| / m, where |W'| represents the number of available work nodes and m represents the number of mappable hash values. If k2=0, then the available work nodes are divided into k1 groups, each of which includes m target work nodes. If k2≠0, then the available work nodes are divided into k1+1 groups, with the first k1 groups each including m target work nodes, and the last group including k2 target work nodes. Correspondingly, if k2=0, determining the estimated value of the global number of triangles of the entire current graph based on the received estimated values ​​of the global number of triangles of each group includes: The step of determining the estimated value of the number of local triangles associated with each vertex in the graph based on the received estimated values ​​of the number of local triangles in each group includes: Correspondingly, if k2≠0, determining the estimated value of the global number of triangles of the entire current graph based on the received estimated values ​​of the global number of triangles of each group includes: The step of determining the estimated value of the number of local triangles associated with each vertex in the graph based on the received estimated values ​​of the number of local triangles in each group includes: in, represents the estimated value of the global number of triangles at time t, represents the estimated number of global triangles in the jth group at time t, represents the estimated number of local triangles associated with vertex u at time t, represents the estimated number of local triangles in the jth group at time t, represents the estimated value of the global number of triangles between groups calculated by the first k1 groups at time t, represents the estimated number of global triangles in the k1+1th group at time t, represents the estimated number of local triangles between groups related to vertex u calculated by the first k1 groups at time t, represents the estimated number of local triangles in the k1+1th group at time t, w1 represents the first weight, and w2 represents the second weight.

6. A distributed sampling device for triangle counting in dynamic graph streams, characterized in that: include: An edge operation distribution module is configured to randomly generate a hash value of each edge operation of a dynamic graph stream to be processed according to a preset hash function mapping range through a master node, and send the edge operation and the hash value to multiple worker nodes; the edge operation types include adding an edge and deleting an edge; an edge operation processing module, configured to determine, through the working node, a sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the sampling triangle determined according to the edge operation to an aggregation node; if the hash value corresponds to the identifier of the working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the working node; a triangle count estimation module, configured to determine, through the aggregation node, a global triangle count estimation value of the entire graph and a local triangle count estimation value associated with each vertex in the graph based on the received count information; The device further comprises: a worker node partitioning module configured to, if the number of available worker nodes is greater than the number of mappable hash values, partition the available worker nodes into multiple groups such that the number of target worker nodes in each group is less than or equal to the number of mappable hash values, and configure a target master node and a secondary aggregation node for each group of target worker nodes; wherein the available worker nodes represent machines that can be used to calculate an estimate of the number of triangles in a dynamic graph stream to be processed; A target work master node work module is configured to randomly generate a target hash value of each edge operation of the dynamic graph stream to be processed according to the preset hash function mapping range through each target master node, and send the edge operation and each target hash value to each target work node of the corresponding group; a target working node working module, configured to determine, through the target working node, a local sampling triangle formed by the input edge of the edge operation in the current local sample graph, and send counting information of the local sampling triangle determined according to the edge operation to a corresponding secondary aggregation node; if the received target hash value corresponds to the identifier of the target working node and the type of the edge operation is add edge, insert the input edge into the local sample graph of the target working node; if the type of the edge operation is delete edge, remove the input edge from the local sample graph of the target working node; The secondary aggregation node working module is used to determine the estimated value of the number of global triangles in the group and the estimated value of the number of local triangles in the group according to the received counting information sent by each target working node of the corresponding group through each secondary aggregation node, and send them to the main aggregation node; The main aggregation node working module is used to determine the global triangle number estimation value of the current entire graph based on the received global triangle number estimation values ​​of each group through the main aggregation node, and determine the local triangle number estimation value related to each vertex in the graph based on the received local triangle number estimation values ​​of each group.

7. A computer device, characterized in that: include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the distributed sampling method for triangle counting of dynamic graph streams as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the distributed sampling method for triangle counting of dynamic graph streams as described in any one of claims 1 to 5 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the distributed sampling method for triangle counting in dynamic graph streams as claimed in any one of claims 1 to 5.

Images