A method and apparatus for using data in a graph computing hybrid computing mode

Abstract

This invention relates to the field of graph computing, and particularly to a method and apparatus for using data in a hybrid graph computing mode. It mainly includes: partitioning vertices according to vertex and edge weights, assigning vertices in each partition to corresponding computing nodes; storing the partitioned vertices in each computing node, using a row-column sparse compression matrix to store the adjacency relationships between vertices and edges, and reconstructing the vertices and edges stored in each computing node according to the adjacency relationships so that the reconstructed vertices and edges can adapt to all computing modes; obtaining the current computing mode based on the number of currently active vertices and edges, and using the reconstructed vertices and edges for data processing under the current computing mode. This invention can flexibly and efficiently convert the storage representation of graph data in a hybrid graph computing mode system, meeting the computational pre-processing requirements of different graph computing modes.

Description

Technical Field

[0001] This invention relates to the field of graph computing, and in particular to a method and apparatus for using data under a graph computing hybrid computing mode. Background Technology

[0002] Distributed graph computing is a method for processing large-scale graph data. It achieves high performance, high availability, and high scalability by distributing graph data, such as vertices and edges, across multiple computing nodes for parallel processing. The core concepts of distributed graph computing include graphs, graph databases, graph algorithms, and distributed computing frameworks. The core task of distributed graph computing is to perform graph algorithm analysis on a specified graph in a graph database based on a distributed graph computing framework. The core algorithm principles include graph data distribution, parallelization of graph algorithms, and data exchange and synchronization, with graph data distribution being a crucial step in the algorithm execution process.

[0003] Graph computation has two typical processing modes based on the direction of information flow: push mode and pull mode. In push mode, each vertex participating in the computation passes messages along outgoing edges; while in pull mode, all vertices retrieve messages from adjacent vertices along incoming edges. Each mode has its advantages and disadvantages: (1) Push mode can achieve selective scheduling, thus skipping edges that do not need to participate in the computation when there are few edges originating from active vertices. The disadvantage is that locks or atomic operations are needed to ensure the correctness of data modification in a concurrent environment, introducing additional overhead; (2) The advantage of pull mode is that there is no contention for data modification, but all edges must be traversed, even though most edges do not participate in the computation in many cases.

[0004] In graph computing systems, hybrid computing modes can be employed, such as a combination of fixed push mode and adaptive push-pull mode, or a combination of fixed pull mode and adaptive push-pull mode. When using the adaptive push-pull mode, mode switching can be performed after algorithm verification. In practice, the specific computing mode to be used can be comprehensively considered based on user-specified parameters, server memory configuration, graph data size, and algorithm applicability. Clearly, a hybrid graph computing system that simultaneously supports both fixed and adaptive push-pull modes has a wider application scope and higher execution efficiency. However, when using a hybrid graph computing mode, the graph computing system must support two graph data partitioning and representation methods, and current data storage representations and graph algorithms cannot meet these prerequisites.

[0005] Therefore, how to overcome the shortcomings of existing technologies and solve the problem that the existing data usage methods of graph computing cannot meet the needs of hybrid graph computing modes is a problem to be solved in this technical field. Summary of the Invention

[0006] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention solves the problem that the data usage method of existing graph computing cannot meet the needs of hybrid graph computing mode.

[0007] The embodiments of the present invention adopt the following technical solutions:

[0008] In a first aspect, the present invention provides a method for using data under a graph computing hybrid computing mode, specifically: partitioning vertices according to the weights of vertices and edges, and assigning the vertices in each partition to corresponding computing nodes; each computing node stores the partitioned vertices, uses a row-column sparse compression matrix to store the adjacency relationships of vertices and edges, and performs segmentation and reconstruction of the vertices and edges stored in each computing node according to the adjacency relationships so that the reconstructed vertices and edges can adapt to all computing modes; obtaining the current computing mode according to the number of currently active vertices and edges, and using the reconstructed vertices and edges to perform data processing under the current computing mode.

[0009] Preferably, partitioning vertices according to the weights of vertices and edges specifically includes: obtaining the weights of each vertex and edge; calculating the sum of the weighted total weights of all vertices on each computing node and the weighted total weights of all edges associated with it; and dividing the vertices into a corresponding number of partitions according to the original IDs of the vertices based on the number of computing nodes, wherein the sum of the weighted weights of all vertices and all edges associated with them in each partition is the same.

[0010] The preferred method of using a row-column sparse compression matrix to store the adjacency relationship between vertices and edges specifically includes: each computing node obtains the adjacent edges of the partitioned vertices from the data source, and uses an existence sequence to record whether each vertex has an adjacent edge; wherein, the adjacent edge includes: the starting point of the edge, the ending point of the edge, and the edge weight; for all vertices with adjacent edges, the starting point or ending point of the adjacent edge is stored sequentially in the adjacent edge sequence, the position start and end values ​​of the adjacent edge of each vertex in the adjacent edge sequence are stored in the offset sequence, the weight of the adjacent edge is stored in the edge weight sequence, and the existence sequence, adjacent edge sequence, offset sequence, and edge weight sequence are combined to form a row-column sparse compression matrix.

[0011] Preferably, each computing node stores the partitioned vertices, specifically including: obtaining the original IDs of all vertices in the graph, each computing node mapping the original IDs of the partitioned vertices to computing IDs in parallel, and storing the computing ID of each partitioned vertex; wherein, the computing ID is a continuously incrementing and non-repeating integer starting from 0.

[0012] Preferably, the step of segmenting and reconstructing the vertices and edges stored in each computing node according to the adjacency relationship specifically includes: segmenting and reconstructing the vertices and edges based on the computing ID of the vertex and the adjacency relationship.

[0013] Preferably, the step of segmenting and reconstructing vertices and edges based on edge segmentation specifically includes: traversing the row-column sparse compression matrix and counting the number of outgoing or incoming edges that need to be reconstructed for all vertices on each computing node; calculating the total offset of the adjacent edges of all vertices on each computing node in the row-column sparse compression matrix and arranging the total offset in ascending order according to the computing ID of the vertex; traversing the adjacent edge sequence and flipping and recombining the starting and ending points of the adjacent edges of the partitioned vertices on each computing node according to the total offset, so as to complete the segmentation and reconstruction of vertices and edges from the partitioned vertices to all vertices, wherein the number of outgoing edges after reconstruction of each vertex is consistent with the sum of the number of edges with that vertex as the starting point before reconstruction, and the number of incoming edges after reconstruction of each vertex is consistent with the sum of the number of edges with that vertex as the ending point before reconstruction.

[0014] Preferably, the step of reconstructing vertices and edges based on edge segmentation further includes one or more of the following data processing methods: deduplication of adjacent vertices, sorting of adjacent vertex IDs, and load balancing.

[0015] Preferably, when the calculation mode is the adaptive push-pull calculation mode, obtaining the current calculation mode based on the number of currently active vertices and edges specifically includes: for each calculation node, obtaining the total number of incoming edges and outgoing edges of all vertices, and the total weighted number of incoming edges and outgoing edges of active vertices; when the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is not less than the weighted number of outgoing edges of all vertices, switching to the pull mode in the adaptive push-pull calculation mode; when the total number of incoming edges of active vertices is not greater than the weighted number of outgoing edges of all vertices, switching to the pull mode in the adaptive push-pull calculation mode; when the total number of incoming edges of active vertices is not greater than the weighted number of outgoing edges of all vertices, switching to the pull mode in the adaptive push-pull calculation mode. If the weighted number of incoming edges to a vertex is less than the weighted number of outgoing edges to all vertices, switch to push mode in adaptive push-pull calculation mode; if the total number of incoming edges to active vertices is greater than the weighted number of incoming edges to all vertices, and the total number of outgoing edges to active vertices is less than the weighted number of outgoing edges to all vertices, determine whether the ratio of the total number of incoming edges to the total number of active vertices is greater than or equal to the ratio of the total number of edges to the total number of vertices. If the determination is true, switch to pull mode in adaptive push-pull calculation mode; otherwise, switch to push mode in adaptive push-pull calculation mode.

[0016] In a second aspect, the present invention provides a data usage apparatus under a graph computing hybrid computing mode, specifically comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the data usage method under the graph computing hybrid computing mode in the first aspect.

[0017] Thirdly, the present invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors to perform the method provided in the first aspect.

[0018] Fourthly, a chip is provided, comprising: a processor and an interface for calling and running a computer program stored in memory, performing the method as provided in the first aspect.

[0019] Fifthly, a computer program product containing instructions is provided that, when executed on a computer or processor, causes the computer or processor to perform the method provided in the first aspect.

[0020] Compared with existing technologies, the advantages of this invention are as follows: During the graph data partitioning stage, weighted partitioning is used to improve the data balance of each computing node. Vertex and edge data are segmented and reconstructed into data structures suitable for all computing modes. The current computing mode is adaptively selected based on the number of active vertices and edges. This allows for the coordinated switching of different computing modes within the graph computing system, enabling the storage and use of graph data in different modes with only one data load. This invention can flexibly and efficiently convert the storage representation of graph data in a hybrid graph computing mode system, meeting the computational pre-requisites of different graph computing modes. In a preferred embodiment, an optimized row-column sparse compression matrix is ​​used to further reduce the data storage pressure on computing nodes. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0022] Figure 1 A flowchart illustrating a data usage method under a graph computing hybrid computing mode, as provided in an embodiment of the present invention;

[0023] Figure 2 A flowchart illustrating another data usage method under a graph computing hybrid computing mode provided in this embodiment of the invention;

[0024] Figure 3 A flowchart illustrating another data usage method under a graph computing hybrid computing mode provided in this embodiment of the invention;

[0025] Figure 4 This is a schematic diagram of a graph structure in a real-world scenario provided in the method of this invention.

[0026] Figure 5 This is a schematic diagram of the data storage structure in a push mode of a certain actual scenario in the method provided by the embodiments of the present invention;

[0027] Figure 6 This is a schematic diagram of the data storage structure in a real-world scenario pull mode of the method provided in this embodiment of the invention;

[0028] Figure 7 This is a schematic diagram of a data utilization device under a graph computing hybrid computing mode provided in an embodiment of the present invention;

[0029] The accompanying figure is labeled as follows:

[0030] 11: Processor; 12: Memory. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] This invention is an architecture of a specific functional system. Therefore, the specific embodiments mainly describe the functional logic relationship of each structural module, and do not limit the specific software and hardware implementation methods.

[0033] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Example 1:

[0035] Hybrid graph computing systems require selecting the appropriate computing mode based on algorithms and parameters during the graph data partitioning phase, thus determining the data storage representation. The method provided in this embodiment enables the graph computing system to coordinate switching between different computing modes and complete the graph data storage and representation conversion between different modes, provided that data is loaded only once.

[0036] like Figure 1 As shown, the specific steps of the data usage method under the graph computing hybrid computing mode provided in this embodiment of the invention are as follows:

[0037] Step 101: Partition the vertices according to the weights of the vertices and edges, and assign the vertices in each partition to the corresponding computing nodes.

[0038] Graph data distribution, also known as graph data partitioning, distributes graph data across multiple computer nodes to achieve parallel processing and load balancing. Since graph data mainly contains vertex and edge data, graph data partitioning can generally be divided into vertex-cut and edge-cut.

[0039] Vertex partitioning: After vertex partitioning, each edge is saved only once, and vertices are divided into primary vertices and virtual vertices according to the divided regions. This method increases storage space and requires handling synchronization issues, but it reduces system network communication.

[0040] Edge partitioning: After edge partitioning, vertices are saved once, while the cut edges are stored separately on two machines. In edge-based operations, for two vertices in different partitions, network communication is required to exchange data. In contrast to vertex partitioning, edge partitioning increases the amount of data transmitted over the network, but requires less storage space.

[0041] In the method provided in this embodiment, to meet the needs of subsequent data transformation, an edge partitioning method is used to divide the vertices of the entire graph into corresponding computing nodes according to a partitioning strategy. Specifically, when using the adaptive push-pull mode, vertices assigned to a computing node are called principal vertices of that computing node, and vertices not assigned to that computing node are called virtual vertices.

[0042] In real-world scenarios, graph data is extremely complex and massive. To reduce the memory storage pressure on graph data, the necessary graph data is first filtered out. Then, the vertices in the graph data are mapped by ID, transforming the original IDs of the filtered vertices, which are not continuously distributed, into continuous computed IDs. In specific implementation, the master node in the compute node obtains the original IDs of all vertices in the graph. Under multi-threading, each loading thread executes in parallel, and each compute node maps the original IDs of the partitioned vertices to computed IDs in parallel, and saves the computed ID of each partitioned vertex. The computed ID is a continuously incrementing, non-repeating integer starting from 0.

[0043] Step 102: Each computing node stores the partitioned vertices, uses a row-column sparse compression matrix to store the adjacency relationship between vertices and edges, and performs segmentation and reconstruction on the vertices and edges stored in each computing node according to the adjacency relationship so that the reconstructed vertices and edges can adapt to all computing modes.

[0044] In typical scenarios, when each computing node performs data loading, it loads the specified directional edges of the vertices it has divided based on the direction parameters, and uses a row-column sparse compression matrix to store the neighbor information of the points.

[0045] In this embodiment, due to the use of a hybrid computing mode, the loaded data also needs to be segmented and reconstructed. In some scenarios, when the computing mode only uses a fixed push mode or a fixed pull mode, no segmentation and reconstruction are performed, and the saved graph data is the data structure of the corresponding computing mode. In other scenarios, when the computing mode uses an adaptive push-pull mode, vertices and edges are segmented and reconstructed based on the vertex computing ID and adjacency relationship to adapt to the data usage needs of all computing modes. The specific conversion method will be detailed later.

[0046] The graph data that has been partitioned and reconstructed can be acquired and processed through a unified data acquisition interface, data transformation and processing interface to support the reading and loading of data sources from graph databases, files and real-time data under three computing modes.

[0047] Step 103: Obtain the current computation mode based on the number of active vertices and edges, and use the reconstructed vertices and edges to perform data processing under the current computation mode.

[0048] In real-world scenarios, the computation mode can be obtained based on preset computation mode parameters, or it can be switched in real time during data processing based on the number of active vertices and edges. Once the computation mode is determined, data processing operations can be performed on the graph data at each computation node, including optional preprocessing operations such as adjacent vertex deduplication, adjacent vertex ID sorting, and load balancing. After preprocessing, the graph data is then processed according to the specified algorithm parameters, which include, but are not limited to, the algorithm name, graph name, number of iterations, and computation mode.

[0049] After steps 101-103 provided in this embodiment, in the case of multiple computing modes mixed together, data storage and reorganization in a specific way can be achieved so that the data usage needs of different computing modes can be met with only one load.

[0050] In step 101, in order to preserve the locality of data during the mapping stage, vertex data under each label can be obtained based on the vertex label ID, and the range of calculated IDs for continuously mapped vertices can be divided for each label.

[0051] The mapping rules are as follows, where N is the total number of vertices:

[0052] f(x)=X→Y,X∈{long,string},Y∈{int}&[0,N).

[0053] The mapping pseudocode is as follows:

[0054]

[0055] As can be seen, the original IDs before mapping use long or string data types, each requiring a large amount of storage space, and the string type is not easy to calculate or sort; the calculated IDs after mapping use int data types, occupying less storage space, and are easy to calculate and sort, making them more suitable for calculation and transmission in graph computing systems.

[0056] In real-world scenarios, graph data exhibits local clustering characteristics; adjacent and similar vertices are often connected. Therefore, it's necessary to group related vertices into a single partition to preserve data locality. Thus, a contiguous block partitioning strategy is proposed and adopted, with the following partitioning rules: P i =∪V i V i ∈V& Continuous.

[0057] In existing technologies, a contiguous block partitioning strategy is typically used to evenly distribute the vertices of the entire graph among corresponding computing nodes, assigning each node a unique partition number, starting from 0 and incrementing. Based on the partition number order, each computing node stores the vertex IDs belonging to its partition, ensuring the partitioned vertices are contiguous. The corresponding vertex belonging to itself is called a principal vertex; otherwise, it is a virtual vertex. This vertex partitioning method preserves data locality. However, graph data generally exhibits power-law properties, meaning a few vertices may hold a majority of edges, potentially leading to super vertices. During computation, the number of edges determines the computational load, while the number of vertices determines the complexity of random memory accesses in a single computation. Therefore, the contiguous block partitioning method can cause graph data imbalance, impacting the computational efficiency of distributed systems.

[0058] The method provided in this embodiment employs a hybrid load balancing strategy based on a continuous block partitioning strategy, using the weighted harmonic sum of vertices and edges as the boundary to divide vertices. Specifically, the weights of each vertex and edge are obtained, and the weighted sum of the total weight of all vertices and the total weight of all edges associated with each computing node is calculated. Based on the number of computing nodes, vertices are divided into a corresponding number of partitions according to their original vertex IDs, where the sum of the weighted weights of all vertices and all edges associated with each partition is the same. This approach ensures a balanced data load across the partitioned graph for each computing node. The calculation method for the sum of the weighted weights of all vertices and all edges associated with each partition is as follows:

[0059] λ·|V i |+|E i |, i∈P;

[0060] Among them, |V i| represents the total weight of the vertices partitioned on node i, where λ is the weighted value of the total vertex weights, λ=ω·(p-1), and the ω parameter is used to adjust the weighted value λ of the vertex. In general scenarios, ω defaults to a value of 8. |E i | represents the total weight of all edges associated with each vertex on node i.

[0061] After the entire graph is partitioned, each computing node loads the graph data. Based on the partitioned vertex range, it retrieves adjacent edges for each vertex from the data source. Adjacent edges include the edge's starting point, ending point, and weight. The storage method of graph data is crucial to the performance and efficiency of graph databases. Common methods include adjacency lists, adjacency matrices, adjacency multi-lists, and compressed row-column matrices. Since most real-world graph data is sparse, and compressed row-column matrices offer advantages such as high compression ratios, high computational efficiency, and support for matrix operations, most graph computing systems use compressed row matrices (CSR) or compressed column matrices (CSC) as the graph storage method. Compressed row matrices and compressed column matrices correspond to outgoing and incoming graph data storage, respectively.

[0062] The row-column sparse compression matrix consists of three one-dimensional arrays: an offset array records the range of adjacent edges of a vertex within the adjacent edge array, an adjacent edge array records the start or end point of adjacent edges, and a weight array records edge weights. By default, an edge partitioning method is used, and the row-column sparse compression matrix stores the adjacent edge information of the computed node itself, without generating virtual vertices. However, real graph data is sparsity-based. Using the original row-column sparse compression matrix data structure for sparse data would result in redundant data storage in the offset array. Therefore, in this embodiment, as... Figure 2 As shown, a sparse compression matrix with double compression is used for graph data storage.

[0063] Step 201: Each computing node obtains the adjacent edges of the partitioned vertices from the data source and uses an existence array to record whether each vertex has adjacent edges.

[0064] Compared to existing row-column sparse compression matrices, the BitSet data structure provided in this embodiment adds a two-dimensional existence array. This existence array contains a long[] and an int[], where long[] is used to record whether a vertex has adjacent edges, and int[] is used to quickly obtain the start and end values ​​of the adjacent edge offset of a vertex.

[0065] Step 202: For all vertices with adjacent edges, store the start or end point of the adjacent edge in the adjacent edge array in sequence, store the start and end position values ​​of the adjacent edge of each vertex in the adjacent edge array in the offset array, store the weight of the adjacent edge in the edge weight array, and form a row and column sparse compression matrix by combining the existing array, the adjacent edge array, the offset array, and the edge weight array.

[0066] In practice, sparse compression matrices can be implemented using either BitSet or Array data structures.

[0067] When the number of isolated vertices without adjacent edges exceeds a certain limit, the memory footprint of the row-column sparse compression matrix using the BitSet data structure is smaller. Therefore, if |V'|>|V| / δ, then the row-column sparse compression matrix of the BitSet data structure containing existing data is used; otherwise, the row-column sparse compression matrix of the Array data structure without existing arrays is used.

[0068] In a certain scenario, there is a vertex V and an isolated point X. The following inequality exists:

[0069]

[0070] Here, V' refers to the number of isolated vertices, and V is the total number of vertices. The left side of the inequality is the row and column sparse compression matrix of the Array data structure, and the right side is the row and column sparse compression matrix of the BitSet data structure. The inequality represents a comparison of their memory usage. It can be inferred that when the number of isolated vertices X is greater than 3 / 64 of the total number of vertices V, the graph data needs to be compressed a second time using BitSet to achieve less memory usage.

[0071] After selecting a suitable data structure, the corresponding data can be written into the offset array, adjacent edge array, and weight array according to the corresponding data structure. Then, the above arrays and the existence array are combined to form the row and column sparse compression matrix in this embodiment.

[0072] After steps 201-202 provided in this embodiment, the compressed storage of graph data can be completed using the row and column sparse compression matrix provided in this embodiment.

[0073] After data is loaded and compressed for storage, if the adaptive push-pull mode is used, the data compression and storage requirements caused by the change in graph partitioning strategy need to be addressed. Therefore, this embodiment provides a partitioning and reconstruction method based on edge partitioning, which quickly implements vertex partitioning strategy conversion and data recompression on known edge partitioning to meet the pre-data requirements of the adaptive push-pull mode.

[0074] like Figure 3 As shown, the following method can be used to segment and reconstruct vertices and edges based on edge segmentation.

[0075] Step 301: Traverse the row and column sparse compression matrix and count the number of outgoing or incoming edges that need to be reconstructed for all vertices on each computing node.

[0076] During reconstruction, the starting and ending points of edges need to be swapped. However, the starting and ending points may not be on the same computation node. Therefore, it is necessary to traverse the entire row-column sparse compression matrix and count the number of outgoing or incoming edges of a vertex on all computation nodes for counting during reconstruction. Traverse the adjacent edge sequence in the row-column sparse compression matrix and count the number of outgoing edges (i.e., out-degree) or incoming edges (i.e., in-degree) of all vertices on each partition computation node. The number of outgoing or incoming edges of all vertices is represented as all_degree[n], where n is the number of vertices, and each array element represents the number of outgoing or incoming edges of a vertex.

[0077] Step 302: Calculate the total offset of the adjacent edges of all vertices on each computing node in the row and column sparse compression matrix, and sort the total offsets in ascending order of the computing ID of the vertex.

[0078] Similarly, it is also necessary to calculate the total offset of a vertex across all computation nodes. By calculating the total offset of the number of adjacent edges of each vertex in the entire graph across each partition computation node, the total number of adjacent edges that need to be reconstructed for each vertex can be obtained. To facilitate the traversal of vertices, they are sorted in ascending order according to their computation IDs. The total offset of all vertices after sorting is represented as all_offset[n+1], where n is the number of vertices, and each array element represents the total offset of a vertex.

[0079] Step 303: Traverse the adjacent edge sequence and flip and reassemble the start and end points of the adjacent edges of the partitioned vertex on each computing node according to the total offset to complete the partitioning and reconstruction of vertices and edges from the partitioned vertex to all vertices. The number of outgoing edges after reconstruction of each vertex is consistent with the sum of the number of incoming edges starting from that vertex before reconstruction, and the number of incoming edges after reconstruction of each vertex is consistent with the sum of the number of outgoing edges ending at that vertex before reconstruction.

[0080] The adjacent edge sequence in the row-column sparse compression matrix is ​​traversed. Based on the offset of each vertex, the starting or ending point of each edge can be determined. Then, the starting and ending points of the adjacent edges of the vertices within the partition are flipped and recombined. Flipping and recombining refers to using a vertex in the adjacent edge sequence as the core, writing the starting points of edges according to the offset sequence into a new adjacent edge sequence, and compressing the data according to the row-column sparse compression matrix storage method provided in this embodiment to generate a new row-column sparse compression matrix, ultimately completing data compression and recombining. Since the starting and ending points of adjacent edges are recombined during reconstruction without changing the connection relationship between vertices and adjacent edges, the number of outgoing edges after reconstruction of each vertex is the same as the sum of the number of edges with that vertex as the starting point before reconstruction, and the number of incoming edges after reconstruction of each vertex is the same as the sum of the number of edges with that vertex as the ending point before reconstruction; the total number of edges remains unchanged. The rules for flipping and recombining are determined according to the actual scenario requirements, such as mutual conversion between incoming and outgoing edges, undirected self-recombination, etc.

[0081] After steps 301-303 provided in this embodiment, the graph data can be segmented and reconstructed. The original row and column sparse compression matrix and the reconstructed row and column sparse compression matrix record the information of the starting point and ending point of the adjacent edge, respectively. Therefore, the required data can be directly obtained in both push mode and pull mode, so that the data meets the usage needs of all computing modes.

[0082] In a certain scenario, the current computing node contains vertex A, whose adjacent edges are A->B and A->C. Before reconstruction, this computing node only contains outgoing edge data of vertex A, that is, only data related to vertex A as an outgoing vertex, i.e., adjacent edge sequence A: [B,C], offset sequence [0,2]. This data can only be used in the fixed push mode. During the partitioning and reconstruction, it is necessary to count all vertices in each partition and flip the outgoing data to the incoming data for application in the pull mode of adaptive processing. When flipping A->B and A->C, it is necessary to count the number of outgoing and incoming edges of each vertex with the adjacent vertices as the core. For vertex A, its adjacent vertices are only B and C. Therefore, the number of incoming edges corresponding to each vertex is: A:0, B:1, C:1, and the corresponding offset sequence is [0,0,1,2]. During the flipping and reconstruction, the writing starts from the vertices of the entire graph in the current partition. If vertex A has no incoming edges, it is skipped. The writing result is a new adjacent edge sequence: A:[], B:[A], C[A]. This result indicates that vertex A has no incoming edges, vertex B's incoming edge has another vertex A, and vertex C's incoming edge has another vertex A. Therefore, when using the adaptive pull mode, all incoming edge adjacent vertices can be obtained directly from the reconstructed adjacent edge sequence. When the reconstructed adjacent edge sequence A:[], B:[A], C[A] is stored on the compute node, the edge information related to all vertices of the entire graph and vertices in the current partition will be stored after the splitting and reconstruction. A belongs to the current partition and is a principal vertex, while B and C do not belong to the current partition and are virtual vertices. The total number of edges after splitting and reconstruction will not change, but the form of edge representation will change, allowing the simultaneous acquisition of edge information from both principal and virtual vertices as starting points.

[0083] After the graph data is partitioned and reconstructed, the graph data storage representation satisfies the selected computation mode. At this point, in order to adapt to the data usage needs of a specific algorithm, one or more data processing steps can be performed on vertices and edges according to the characteristics of each algorithm and the current computation mode, including adjacent vertex deduplication, adjacent vertex ID sorting, and load balancing.

[0084] After the graph data loading and processing is completed, the system scheduling algorithm enters the actual graph computation process. In this embodiment, when using the adaptive push-pull computation mode, the current computation mode can be obtained based on the number of currently active vertices and edges to achieve adaptive computation mode switching.

[0085] For each computation node, obtain the total number of incoming and outgoing edges for all vertices, as well as the total weighted number of incoming and outgoing edges for all active vertices. Based on the current number of active vertices and edges, determine the current computation mode, where at least three computation modes exist.

[0086] In the following calculations: This refers to the total number of adjacent connected edges and the total number of adjacent outgoing edges of all active vertices in the i-th compute node. This refers to the total number of adjacent incoming edges and the total number of adjacent outgoing edges of all active vertices in the i-th computing node. This represents the total number of incoming edges to all vertices in the i-th computed node. Let E represent the total number of outgoing edges from all vertices in the i-th computation node; let V represent the weight of each edge, V represent the weight of each vertex, and p represent the number of partitions. ω is a harmonic threshold determined by the number of edges, vertices, and partitions. The more partitions there are, the more dispersed the edges become, and the fewer edges each vertex in the computation node holds. Therefore, the value of ω can be reduced when there are more partitions.

[0087] Scenario 1: When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is not less than the weighted number of outgoing edges of all vertices, switch to the pull mode in the adaptive push-pull calculation mode. That is, only when the following condition (1) is met, and condition (2) is not met, the pull mode is used:

[0088]

[0089]

[0090] in,

[0091] Scenario 2: When the total number of incoming edges of active vertices is not greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, switch to push mode in the adaptive push-pull calculation mode. That is, only when condition (2) is met, and condition (1) is not met, push mode is used.

[0092] Scenario 3: When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, determine whether the ratio of the total number of incoming edges of active vertices to the total number of active vertices is greater than or equal to the ratio of the total number of edges to the total number of vertices. If the determination is true, switch to the pull mode in the adaptive push-pull calculation mode; otherwise, switch to the push mode in the adaptive push-pull calculation mode.

[0093] If both conditions (1) and (2) above are true, then determine... If the expression is true, then select pull mode; otherwise, select push mode.

[0094] In hybrid graph computation systems, a Bulk Synchronous Parallel (BSP) framework is typically employed, where all algorithms are executed in multiple rounds of superstepping. Therefore, before each superstep, a mode switching computation is performed, using the formula above to determine the appropriate computation mode for the current round. The computation is iterated until the algorithm reaches convergence, at which point the algorithm terminates, and the results are written to a database and a result file is generated.

[0095] The data usage method under the hybrid graph computing mode provided in this embodiment, through the graph data storage and transformation method provided in the figure of this embodiment, transforms the graph data according to the selected computing mode, solves the data requirements of the hybrid graph computing mode system under different computing modes, integrates and coordinates the switching between multiple modes, and completes the data transformation work with only one data loading. It optimizes the data layer processing flow, so that the graph computing system under the hybrid graph computing mode can switch between different computing modes efficiently and flexibly. Compared with the single-mode graph computing system, the hybrid graph computing mode system has a wider application range and higher execution efficiency.

[0096] Example 2:

[0097] The data usage method based on the graph computing hybrid computing mode provided in Embodiment 1 can be implemented in certain specific embodiments through the specific implementation methods described in this embodiment. It is understood that the specific implementation methods provided in this embodiment are only used to illustrate the specific implementation process of the method in Embodiment 1 in certain specific scenarios and are not intended to limit the scope of protection.

[0098] In the scenario of this embodiment, given graph G<m,n> m is the number of vertices and n is the number of edges; the distributed graph computing cluster contains p computing nodes.

[0099] The master node retrieves the original vertex IDs of the entire graph and maps each original vertex ID to a continuously incrementing, non-repeating calculated ID starting from 0, storing it in an ID mapping data container. The mapping data container includes a Map.<Long,int> `int[]` represents the mapping relationship between the original `long` id and the calculated `int` id. Map<Long,int> The key is the original id of the long data type, and the value corresponding to the key is the calculated id of the int data type; the index of int[] is the calculated id of the int data type, and the element value is the original long data type id corresponding to the calculated id represented by the index.

[0100] All computational modes use a continuous block-based equal partitioning method to divide vertices. In graph G, the distributed graph computing cluster has p computing nodes, and the vertices are partitioned into {0, m / p, ..., m} according to the partitioning order of the computing nodes. When using an adaptive push-pull model, there are concepts of principal vertices and virtual vertices. For example, for computing node 0, the principal vertex range is [0, m / p), and other vertices on the current computing node are virtual vertices.

[0101] When partitioning the edges associated with each vertex, the fixed push-pull mode and the adaptive push-pull mode use different rules for edge partitioning. The fixed push-pull mode stores all adjacent edges of its own primary vertex; the adaptive push-pull mode includes push mode and pull mode. The push mode stores edges whose endpoint is the primary vertex, and the pull mode stores edges whose starting point is the primary vertex.

[0102] The edge division rules for fixed push-pull mode are as follows:

[0103] E i ={(src,dst,value)∈E|src∈V i};

[0104] The edge partitioning rules for adaptive push-pull mode are as follows:

[0105]

[0106] Among them, E i Let i be the set of edges of the i-th partition under the fixed push-pull mode. Let be the set of edges for the i-th partition in the push mode of the adaptive push-pull mode. V represents the sum of the edge sets of the i-th partition in the pull mode of the adaptive push-pull mode; i Let be the set of vertices in the i-th partition; src is the starting point of the edge, dst is the ending point of the edge, and value is the weight of the edge.

[0107] After the partitioning is completed, each computing node reads the data source data in parallel, submits the direction parameters according to the algorithm using the graph data, loads the edge data according to the partitioning range of its own point, and saves the edge information into the sparse row and column compression matrix using the edge partitioning rule of fixed push-pull mode.

[0108] In a specific scenario, the structure of graph G is as follows: Figure 4 As shown in the diagram, the numbers represent the computation IDs of the corresponding vertices. According to the vertex partitioning rules, vertices 0 and 1 are assigned to computation node 0. (Graph G)<m,n> The computation node is represented using a row-column sparse compression matrix as follows:

[0109] When going out: nbr = [1,2,3,5]; idx = [0,2,4].

[0110] In the in direction: nbr = [2,4,0,2]; idx = [0,2,4].

[0111] Wherein, nbr is an array of adjacent edges, which stores the endpoint of the outgoing edge or the starting point of the incoming edge for each vertex; idx is an array of offsets, which stores the offset of each vertex's adjacent edge in nbr.

[0112] As can be seen from the data in the row-column sparse compression matrix: in the outward direction, the endpoints of the edges adjacent to vertex 0 are vertices 1 and 2, and the endpoints of the edges adjacent to vertex 1 are vertices 3 and 5; in the inward direction, the starting points of the edges adjacent to vertex 0 are vertices 2 and 4, and the starting points of the edges adjacent to vertex 1 are vertices 0 and 2. Data storage can be completed in the same manner for each computation node.

[0113] After the compute node has loaded the partition data, it determines the compute mode parameters. If the adaptive push-pull mode is used, it performs data storage representation conversion.

[0114] For compute node 0, we first count the neighbor edges. We iterate through the offset array and count the number of neighbor edges of all vertices in the entire graph that are located in the partition compute node. The outgoing and incoming direction counts are given by: degree out =[0,1,1,1,0,1], degree in =[1,0,2,0,1,0]. Then, based on the statistical results, calculate the offset of all vertices in the current partition, with the outgoing and incoming directions being: offset. out =[0,0,1,2,3,3,4], offset in =[0,1,1,3,3,4,4], and finally, based on the offset result, traverse the adjacent edge array nbr to reorganize the data.

[0115] In push mode, a row-sparse compression matrix is ​​used, and the reconstruction result is as follows: Figure 5 As shown, where: nbr=[1,0,1,0],idx=[0,1,1,3,3,4,4].

[0116] Using column-sparse compression matrices in pull mode, the recombined result is as follows: Figure 6 As shown, where: nbr=[0,0,1,1],idx=[0,1,1,3,3,4,4].

[0117] The converted graph data has the same data storage format as before the conversion. However, the adaptive push-pull mode data storage represents the adjacency data of all graph vertices in the current computing node, and its adjacency points belong to the current partition.

[0118] After the graph data transformation is completed, data processing operations can be performed on the transformed graph data. Optional adjacent vertex deduplication, adjacent vertex ID sorting, and load balancing can be performed for specific algorithms. After completion, the algorithm enters the calculation process and finally completes the calculation.

[0119] In practice, a fixed push mode, a fixed pull mode, or an adaptive push-pull mode can be selected. The calculation mode can be selected adaptively or by binding empirical parameters to choose the optimal mode. When using the fixed push or fixed pull mode, the above data conversion process is unnecessary; when using the adaptive push-pull mode, the graph data needs to be converted.

[0120] In a specific scenario, a weakly connected component algorithm (WCP) is required for graph computation. The computation mode parameter is an adaptive push-pull mode, and the WCP algorithm performs iterative computation according to the BSP computation model. During the WCP computation process, there is a vertex convergence characteristic. In the first round of iteration, all vertices participate in the computation, and activeE = allledges. The pull mode is selected according to the aforementioned computation mode switching rules. After the first round of iteration, in each subsequent round, some vertices may converge and no longer participate in the computation. After each round of iteration, an appropriate push mode needs to be selected based on the number of active vertices and active edges, following the method in Example 1.

[0121] In another specific scenario, the PageRank algorithm is used for iterative calculations, with the calculation mode parameter being an adaptive push-pull mode. The PageRank algorithm is one in which all vertices of the graph participate in each iteration, and the number of active vertices does not decrease during the calculation process. Therefore, according to the mode switching rules, the algorithm selects the pull mode in each iteration until the algorithm converges or reaches the maximum number of iterations.

[0122] Example 3:

[0123] Based on the data usage method under the graph computing hybrid computing mode provided in Embodiments 1 and 2 above, the present invention also provides a data usage apparatus under the graph computing hybrid computing mode that can be used to implement the above methods, such as... Figure 7 The diagram shown is a schematic representation of the device architecture according to an embodiment of the present invention. The data utilization device in this embodiment, operating under a graph computing hybrid computing mode, includes one or more processors 11 and a memory 12. Figure 7 Take a processor 11 as an example.

[0124] Processor 11 and memory 12 can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0125] The memory 12 serves as a non-volatile computer-readable storage medium for data usage methods in a graph computing hybrid computing mode. It can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the data usage methods in the graph computing hybrid computing mode in Embodiments 1 and 2. The processor 11 executes various functional applications and data processing of the data usage device in the graph computing hybrid computing mode by running the non-volatile software programs, instructions, and modules stored in the memory 12, thereby implementing the data usage methods in the graph computing hybrid computing mode in Embodiments 1 and 2.

[0126] Memory 12 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 12 may optionally include memory remotely located relative to processor 11, which can be connected to processor 11 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0127] The program instructions / modules are stored in memory 12. When executed by one or more processors 11, they perform the data usage method in the graph computing hybrid computing mode described in Embodiments 1 and 2 above, for example, performing the above-described... Figures 1 to 3 The steps shown.

[0128] This invention also provides a non-volatile computer storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 7 One of the processors 11 can enable the one or more processors to execute the data usage method in the graph computing hybrid computing mode of Embodiments 1 and 2, for example, to perform the above-described... Figures 1 to 3 The steps shown.

[0129] It is worth noting that the information interaction and execution process between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiment of the present invention. For details, please refer to the description in the method embodiment of the present invention, and will not be repeated here.

[0130] Those skilled in the art will understand that all or part of the steps in the various methods of the embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for using data under a graph computing hybrid computing mode, characterized in that, include: The vertices are partitioned according to their weights and edges, and the vertices in each partition are assigned to corresponding computing nodes. This includes: obtaining the weights of each vertex and edge, calculating the sum of the weighted total weights of all vertices and all their associated edges on each computing node; and dividing the vertices into a corresponding number of partitions according to their original IDs based on the number of computing nodes, wherein the sum of the weighted weights of all vertices and all their associated edges in each partition is the same. Each computing node stores the partitioned vertices, including: obtaining the original IDs of all vertices in the graph; each computing node maps the original IDs of the partitioned vertices to computed IDs in parallel, and stores the computed ID of each partitioned vertex; wherein, the computed ID is a continuously incrementing and unique integer starting from 0; using a row-column sparse compression matrix to store the adjacency relationships of vertices and edges, and performing segmentation and reconstruction on the vertices and edges stored in each computing node according to the adjacency relationships, so that the reconstructed vertices and edges can adapt to all computing modes; including: traversing the row-column sparse compression matrix according to the computed IDs and adjacency relationships of the vertices, and counting the number of outgoing or incoming edges that need to be reconstructed for all vertices on each computing node; Calculate the total offset of the adjacent edges of all vertices on each computing node in the row and column sparse compression matrix, and sort the total offsets in ascending order of the computing ID of the vertices; Traverse the adjacent edge sequence, and flip and reorganize the start and end points of the adjacent edges of the partitioned vertex on each computing node according to the total offset, so as to complete the partitioning and reconstruction of vertices and edges from the partitioned vertex to all vertices. The number of outgoing edges after reconstruction of each vertex is consistent with the sum of the number of edges with the vertex as the starting point before reconstruction, and the number of incoming edges after reconstruction of each vertex is consistent with the sum of the number of edges with the vertex as the ending point before reconstruction. The current computation mode is obtained based on the number of active vertices and edges, and the reconstructed vertices and edges are used to perform data processing under the current computation mode.

2. The data usage method under the graph computing hybrid computing mode according to claim 1, characterized in that, The method of using a row-column sparse compression matrix to store the adjacency relationships of vertices and edges specifically includes: Each computing node obtains the adjacent edges of the partitioned vertices from the data source and uses an existence sequence to record whether each vertex has adjacent edges; where adjacent edges include: the start point of the edge, the end point of the edge, and the edge weight; For all vertices with adjacent edges, the start or end point of the adjacent edges is stored in the adjacent edge sequence in sequence. The start and end positions of the adjacent edges of each vertex in the adjacent edge sequence are stored in the offset sequence. The weights of the adjacent edges are stored in the edge weight sequence. The existing sequence, the adjacent edge sequence, the offset sequence, and the edge weight sequence are combined into a row-column sparse compression matrix.

3. The data usage method under the graph computing hybrid computing mode according to claim 1, characterized in that, Also includes: Perform one or more of the following data processing operations on vertices and edges: deduplication of adjacent vertices, sorting of adjacent vertices by ID, and load balancing.

4. The data usage method under the graph computing hybrid computing mode according to claim 1, characterized in that, When the calculation mode is adaptive push-pull calculation mode, obtaining the current calculation mode based on the number of currently active vertices and edges specifically includes: For each computing node, obtain the total number of incoming and outgoing edges for all vertices, as well as the total weighted number of incoming and outgoing edges for active vertices; When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is not less than the weighted number of outgoing edges of all vertices, switch to the pull mode in the adaptive push-pull calculation mode. When the total number of incoming edges of active vertices is not greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, switch to push mode in adaptive push-pull calculation mode. When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, determine whether the ratio of the total number of incoming edges of active vertices to the total number of active vertices is greater than or equal to the ratio of the total number of edges to the total number of vertices. If the determination is true, switch to the pull mode in the adaptive push-pull calculation mode; otherwise, switch to the push mode in the adaptive push-pull calculation mode.

5. A data utilization device in a graph computing hybrid computing mode, characterized in that: The method includes a memory and at least one processor, which are connected via a data bus. The memory stores instructions that can be executed by the at least one processor. After being executed by the processor, the instructions are used to perform a data usage method in a graph computing hybrid computing mode as described in any one of claims 1-4.

6. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, which are executed by one or more processors to perform the data usage method under the graph computing hybrid computing mode as described in any one of claims 1-4.

Images