Data use method and device in mixed calculation mode of graph calculation

Abstract

The invention relates to the field of graph calculation, in particular to a data use method and device in a graph calculation hybrid calculation mode. The method mainly comprises the following steps: partitioning vertexes according to weights of the vertexes and edges, and dividing the vertexes in each partition to corresponding computational nodes; each computing node stores the divided vertexes, a row and column sparse compression matrix is used for storing the adjacency relation of the vertexes and the edges, and the vertexes and the edges stored in each computing node are segmented and reconstructed according to the adjacency relation, so that the reconstructed vertexes and the reconstructed edges can adapt to all computing modes; and obtaining a current calculation mode according to the number of the currently active vertexes and the number of the currently active edges, and performing data processing in the current calculation mode by using the reconstructed vertexes and the reconstructed edges. According to the method, the storage representation mode of the graph data can be flexibly and efficiently converted in a mixed graph calculation mode system, and the calculation preposition requirements of different graph calculation modes are met.

Description

Technical Field

[0001] The present invention relates to the field of graph computing, and in particular to a method and device for using data in a graph computing hybrid computing mode. Background Art

[0002] Distributed graph computing is a method for processing large-scale graph data. It achieves high performance, high availability, and high scalability by distributing graph data such as vertices and edges on multiple computing nodes for parallel processing. The core concepts of distributed graph computing include graphs, graph databases, graph algorithms, and distributed computing frameworks. Performing graph algorithm analysis on a specified graph in a graph database based on a distributed graph computing framework is the core task of distributed graph computing. The core algorithm principles include graph data distribution, graph algorithm parallelization, and data exchange and synchronization. Graph data distribution is an important step in the algorithm execution process.

[0003] There are two typical processing modes for graph computing based on the direction of information flow: push mode and pull mode. When using the push mode, each vertex participating in the calculation transmits messages along the outgoing edge; the pull mode is the opposite, allowing all vertices to obtain messages from adjacent vertices along the incoming edge. Both modes have their own advantages and disadvantages: (1) The push mode can achieve selective scheduling, thereby skipping edges that do not need to participate in the calculation when there are fewer edges starting from active vertices. The disadvantage is that locks or atomic operations are required to ensure the correctness of data modifications in a concurrent environment, introducing additional overhead; (2) The advantage of the pull mode is that there is no competition in data modification, but all edges must be traversed, even though most edges do not participate in the calculation in many cases.

[0004] In the graph computing system, a hybrid computing mode can be used, for example: a hybrid computing mode of fixed push mode + adaptive push-pull mode, or a fixed pull mode + adaptive push-pull mode. When using the adaptive push-pull mode, you can also switch within the mode after the submission algorithm verification is completed. In the specific implementation, you can comprehensively consider the specific computing mode to be used based on the user-specified parameters or server memory configuration, graph data size, and algorithm applicability. Obviously, the hybrid graph computing mode system that supports both fixed push-pull mode and adaptive push-pull mode has a wider application level and support range, and higher execution efficiency. However, when using the hybrid graph computing mode, the graph computing system must support two graph data partitioning and representation methods. The current data storage representation and graph algorithm cannot meet the aforementioned prerequisite requirements.

[0005] In view of this, how to overcome the defects of the existing technology and solve the problem that the data usage method of the existing graph computing cannot meet the needs of the hybrid graph computing mode is a problem to be solved in this technical field. Summary of the invention

[0006] In response to the above defects or improvement needs of the prior art, the present invention solves the problem that the data usage method of the existing graph computing cannot meet the needs of the hybrid graph computing mode.

[0007] The embodiment of the present invention adopts the following technical solution:

[0008] In the first aspect, the present invention provides a method for using data in a hybrid computing mode of graph computing, specifically: partitioning vertices according to the weights of vertices and edges, and assigning the vertices in each partition to corresponding computing nodes; each computing node saves the divided vertices, and uses a row-column sparse compression matrix to save the adjacency relationship between vertices and edges, and splits and reconstructs the vertices and edges saved in each computing node according to the adjacency relationship, so that the reconstructed vertices and edges can adapt to all computing modes; the current computing mode is obtained according to the number of currently active vertices and edges, and the reconstructed vertices and edges are used to perform data processing in the current computing mode.

[0009] Preferably, partitioning the vertices according to the weights of the vertices and edges specifically includes: obtaining the weight of each vertex and edge, calculating the sum of the weighted total weight of all vertices on each computing node and the weighted total weight of all edges associated with it; according to the number of computing nodes, dividing the vertices into a corresponding number of partitions according to the original ID of the vertex, wherein the sum of the weighted weights of all vertices and all edges associated with it in each partition is the same.

[0010] Preferably, the use of a row-column sparse compressed matrix to save the adjacency relationship between vertices and edges specifically includes: each computing node obtains the adjacent edges of the divided vertices from a data source, and uses an existence sequence to record whether there are adjacent edges for each vertex; wherein the adjacent edges include: the starting point of the edge, the end point of the edge, and the edge weight; for all vertices with adjacent edges, the starting point or end point of the adjacent edge is saved in the adjacent edge sequence in sequence, the position start and end values ​​of the adjacent edge of each vertex in the adjacent edge sequence are saved in the offset sequence, the weight of the adjacent edge is saved in the edge weight sequence, and the existence sequence, the adjacent edge sequence, the offset sequence, and the edge weight sequence form a row-column sparse compressed matrix.

[0011] Preferably, each computing node saves the divided vertices, specifically including: obtaining the original IDs of all vertices in the graph, each computing node maps the original IDs of the divided vertices to calculation IDs in parallel, and saves the calculation ID of each divided vertex; wherein the calculation ID is an integer that increases continuously from 0 and is non-repeating.

[0012] Preferably, the splitting and reconstructing the vertices and edges stored in each computing node according to the adjacency relationship specifically includes: splitting and reconstructing the vertices and edges based on edge splitting according to the computation IDs and adjacency relationships of the vertices.

[0013] Preferably, the splitting and reconstruction of vertices and edges based on edge splitting specifically includes: traversing the row-column sparse compression matrix, counting the number of outgoing edges or incoming edges that need to be reconstructed for all vertices on each computing node; calculating the total offset of the adjacent edges of all vertices on each computing node in the row-column sparse compression matrix, and arranging the total offset in ascending order according to the calculation ID of the vertex; traversing the adjacent edge sequence, and flipping and reorganizing the starting point and end point of the adjacent edge of the partition vertex on each computing node according to the total offset, so as to complete the splitting and reconstruction of vertices and edges from the partition vertex to all vertices, wherein the number of outgoing edges of each vertex after reconstruction is consistent with the sum of the number of edges with the vertex as the starting point of the incoming edge before reconstruction, and the number of incoming edges of each vertex after reconstruction is consistent with the sum of the number of edges with the vertex as the end point of the outgoing edge before reconstruction.

[0014] Preferably, the splitting and reconstruction of vertices and edges based on edge splitting also includes: performing one or more data processing of adjacent vertex deduplication, adjacent vertex ID sorting and load balancing on vertices and edges.

[0015] Preferably, when the calculation mode is the adaptive push-pull calculation mode, the current calculation mode is obtained according to the current number of active vertices and edges, specifically including: for each calculation node, obtaining the total number of incoming edges and outgoing edges of all vertices, and the total number of weighted incoming edges and weighted outgoing edges of active vertices; when the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is not less than the weighted number of outgoing edges of all vertices, switching to the pull mode in the adaptive push-pull calculation mode; when the total number of incoming edges of active vertices is not greater than the weighted number of outgoing edges of all vertices, switching to the pull mode in the adaptive push-pull calculation mode. When the total number of incoming edges of the active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of the active vertices is less than the weighted number of outgoing edges of all vertices, switch to the push mode in the adaptive push-pull calculation mode; when the total number of incoming edges of the active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of the active vertices is less than the weighted number of outgoing edges of all vertices, determine whether the ratio of the total number of incoming edges of the active vertices to the total number of active vertices is greater than or equal to the ratio of the total number of edges to the total number of vertices, and when the judgment is established, switch to the pull mode in the adaptive push-pull calculation mode, otherwise switch to the push mode in the adaptive push-pull calculation mode.

[0016] In a second aspect, the present invention provides a data usage device in a graph computing hybrid computing mode, specifically: comprising at least one processor and a memory, at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the data usage method in the graph computing hybrid computing mode in the first aspect.

[0017] In a third aspect, the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors to complete the method provided in the first aspect.

[0018] In a fourth aspect, a chip is provided, comprising: a processor and an interface, for calling and running a computer program stored in the memory from a memory, and executing the method provided in the first aspect.

[0019] According to a fifth aspect, a computer program product comprising instructions is provided. When the instructions are executed on a computer or a processor, the computer or the processor executes the method provided in the first aspect.

[0020] Compared with the prior art, the beneficial effects of the present invention are: using weighted partitioning in the graph data division stage to improve the data balance of each computing node, splitting and reconstructing vertex and edge data into data structures suitable for all computing modes, adaptively selecting the current computing mode according to the number of active vertices and edges, and being able to coordinate the graph computing system to switch different computing modes on the premise of loading data only once, and complete the storage and use of graph data in different modes. The present invention can flexibly and efficiently convert the storage representation of graph data in a hybrid graph computing mode system to meet the computational pre-requirements of different graph computing modes. In the preferred scheme of this embodiment, an optimized row and column sparse compression matrix is ​​used to further reduce the data storage pressure of the computing nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flow chart of a method for using data in a graph computing hybrid computing mode provided by an embodiment of the present invention;

[0023] Figure 2 A flow chart of another method for using data in a graph computing hybrid computing mode provided by an embodiment of the present invention;

[0024] Figure 3 A flow chart of another method for using data in a graph computing hybrid computing mode provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a graph structure of a certain actual scenario in the method provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of a data storage structure in a certain actual scenario push mode in the method provided by an embodiment of the present invention;

[0027] Figure 6 A schematic diagram of a data storage structure in a certain actual scenario pull mode in the method provided by an embodiment of the present invention;

[0028] Figure 7 A schematic diagram of the structure of a data use device in a graph computing hybrid computing mode provided by an embodiment of the present invention;

[0029] The reference numerals are as follows:

[0030] 11: processor; 12: memory. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0033] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0034] Embodiment 1:

[0035] The hybrid graph computing mode system needs to select the corresponding computing mode according to the algorithm and parameters in the graph data partitioning stage to determine the data storage representation method. The method provided in this embodiment can coordinate the graph computing system to switch different computing modes and complete the conversion of graph data storage representation in different modes under the premise of loading data only once.

[0036] like Figure 1 As shown, the specific steps of the data use method in the graph computing hybrid computing mode provided by the embodiment of the present invention are as follows:

[0037] Step 101: partition the vertices according to the weights of the vertices and edges, and assign the vertices in each partition to corresponding computing nodes.

[0038] Graph data distribution is also called graph data partitioning, which distributes graph data on multiple computer nodes to achieve parallel data processing and load balancing. Since graph data mainly contains vertex and edge related data, graph data partitioning can usually be divided into vertex-cut and edge-cut.

[0039] Point segmentation: After point segmentation, each edge is saved only once, and the vertices are divided into main points and virtual points according to the divided areas. This method increases the storage space and requires processing synchronization issues, but it reduces system network communication.

[0040] Edge splitting: After edge splitting, the vertices are saved once, and the cut edges are saved on two machines. In edge-based operations, network communication is required to exchange data between two vertices in different partitions. In contrast to point splitting, edge splitting increases the amount of data transmitted over the network, but requires less storage space.

[0041] In the method provided in this embodiment, in order to meet the needs of subsequent data conversion, the edge segmentation method is used to divide the vertices of the entire graph into corresponding computing nodes according to the partition strategy. Among them, when using the adaptive push-pull mode, the vertex divided into a certain computing node is called the main point of the corresponding computing node, and the vertex not divided into the computing node is called a virtual point.

[0042] In real scenarios, graph data is very complex and large. In order to reduce the memory storage pressure of graph data, we first filter out the graph data to be used, and then perform ID mapping on the vertices in the graph data, mapping the original IDs of the discontinuous distribution of the filtered vertices to continuous calculation IDs. In the specific implementation, the master node in the computing node obtains the original IDs of all vertices in the graph, and each loading thread executes in parallel under multi-threading. Each computing node maps the original IDs of the divided vertices to calculation IDs in parallel, and saves the calculation ID of each divided vertex; where the calculation ID is an integer that is continuously incremented and non-repeating starting from 0.

[0043] Step 102: Each computing node saves the divided vertices, uses a row-column sparse compression matrix to save the adjacency relationship between vertices and edges, and splits and reconstructs the vertices and edges saved in each computing node according to the adjacency relationship, so that the reconstructed vertices and edges can adapt to all computing modes.

[0044] In general scenarios, when each computing node performs data loading, the computing node loads the specified direction edge of the divided vertex according to the direction parameter, and uses a row-column sparse compression matrix to save the adjacent point information of the point.

[0045] In this embodiment, due to the use of a hybrid computing mode, the loaded data also needs to be split and reconstructed. In some scenarios, when the computing mode only uses a fixed push mode or a fixed pull mode, no splitting and reconstruction is performed, and the saved graph data is the data structure of the corresponding computing mode; in other scenarios, when the computing mode uses an adaptive push-pull mode, the vertices and edges are split and reconstructed based on edge splitting according to the computing ID and adjacency relationship of the vertices to meet the data usage needs in all computing modes. The specific conversion method is described in detail later.

[0046] The split and reconstructed graph data can be acquired and processed through uniformly defined data acquisition interfaces, data conversion and processing interfaces to support the reading and loading of data sources of graph databases, files and real-time data in three computing modes.

[0047] Step 103: Obtain the current computing mode according to the number of currently active vertices and edges, and use the reconstructed vertices and edges to perform data processing under the current computing mode.

[0048] In actual scenarios, the computing mode can be obtained based on the preset computing mode parameters, or it can be switched in real time during data processing based on the number of active vertices and edges. After determining the computing mode, data processing operations can be performed on the graph data on each computing node, including optional preprocessing operations such as adjacent vertex deduplication, adjacent vertex ID sorting, and load balancing. After preprocessing, the graph data is processed according to the specified algorithm parameters, including but not limited to the algorithm name, graph name, number of iterations, computing mode, etc.

[0049] After steps 101 to 103 provided in this embodiment, when multiple computing modes are mixed, data usage requirements of different computing modes can be met by only loading once through specific data storage and reorganization.

[0050] In step 101, in order to preserve the locality of data in the mapping stage, the vertex data under each label can be obtained according to the label id of the vertex, and the calculation id range of the continuous mapping vertices can be divided for each label.

[0051] The mapping rules are as follows, where N is the total number of vertices:

[0052] f(x)=X→Y,X∈{long,string},Y∈{int}&[0,N).

[0053] The mapping pseudo code is as follows:

[0054]

[0055] It can be seen that the original id before mapping uses the long or string data type, each original id requires a large storage space, and the string type is not easy to calculate or sort; the calculated id after mapping uses the int data type, which occupies less storage space and is easy to calculate and sort, and is more suitable for calculation and transmission in the graph computing system.

[0056] In real scenarios, graph data contains local clustering characteristics. There are often connections between adjacent and similar vertices. It is necessary to divide related vertices into one partition as much as possible to retain data locality. Therefore, a continuous block partitioning strategy is given and adopted. The partitioning rules are as follows: i =∪V i ,V i ∈V&continuous.

[0057] In the prior art, the vertices of the entire graph are usually evenly distributed to the corresponding computing nodes based on the continuous block partitioning strategy, and each computing node is given a unique partition number, which starts to increase from 0. According to the order of the partition numbers of the computing nodes, each computing node saves the vertex ID sequence number that belongs to itself, and the divided vertices are continuous. The corresponding vertex belonging to itself is called the main point, otherwise it is a virtual point. Through this vertex partitioning method, data locality is retained. However, general graph data has a power law, and a few vertices may hold the majority of edges, and there may be super vertices. In the calculation process, the number of edges determines the amount of calculation, and the number of vertices determines the complexity of random memory access in a single calculation. Therefore, the continuous block equal division method will cause the problem of unbalanced graph data and affect the computing efficiency of the distributed system.

[0058] In the method provided in this embodiment, on the basis of the continuous block partitioning strategy, a hybrid balancing strategy is adopted to divide vertices using the weighted harmonic values ​​of vertices and edges as boundaries. Specifically, the weight of each vertex and edge is obtained, and the sum of the weighted total weight of all vertices on each computing node and the total weight of all edges associated with it is calculated; according to the number of computing nodes, the vertices are divided into a corresponding number of partitions according to the original ID of the vertex, wherein the sum of the weighted weights of all vertices and all edges associated with it in each partition is the same. In this way, the partition graph data load balance of each computing node can be guaranteed as much as possible, and the sum of the weighted weights of all vertices and all edges associated with it in each partition is calculated as follows:

[0059] λ·|V i |+|E i |, i∈P;

[0060] Among them, |V i| represents the total weight of the vertices divided on the computing node i, λ is the weighted value of the total weight of the vertex, λ=ω·(p-1), the ω parameter is used to adjust the weighted value λ of the vertex, and the default value of ω in general scenarios is 8. |E i | means calculating the total weight of all edges associated with each vertex on node i.

[0061] After the full graph is divided, each computing node performs graph data loading. According to the divided vertex range, the adjacent edges of each vertex are obtained from the data source. The adjacent edges include: the starting point of the edge, the end point of the edge, and the edge weight. The storage method of graph data is crucial to the performance and efficiency of the graph database. Common methods include adjacency lists, adjacency matrices, adjacency multi-tables, and row-column sparse compressed matrices. Since most real graph data are sparse graphs, and row-column sparse compressed matrices have the advantages of high compression ratio of graph data, efficient operation, and support for matrix operations, most graph computing systems use Compressed Sparse Row Matrix (abbreviated as CSR) or Compressed Sparse Column Matrix (abbreviated as CSC) as the graph storage method. The row sparse compressed matrix and the column sparse compressed matrix correspond to the outbound and inbound graph data storage, respectively.

[0062] The row-column sparse compression matrix is ​​composed of three one-dimensional arrays: the offset array records the range of the vertex adjacent edges in the adjacent edge array, the adjacent edge array records the starting point or end point of the vertex adjacent edges, and the weight array records the edge weight data. The edge segmentation method is used by default, and the row-column sparse compression matrix saves the adjacent edge information of the vertex of the calculation node itself, and does not generate virtual points. However, the real graph data is sparse, and the sparse data uses the original row-column sparse compression matrix data structure. The offset array will generate redundant data storage. Therefore, in this embodiment, if Figure 2 As shown, a row-column sparse compressed matrix with secondary compression is used for graph data storage.

[0063] Step 201: Each computing node obtains the adjacent edges of the divided vertices from the data source, and uses an existence array to record whether there is an adjacent edge for each vertex.

[0064] Compared with the existing row-column sparse compressed matrix, the BitSet row-column sparse compressed matrix provided in this embodiment adds a two-dimensional existence array to the BitSet data structure. The existence array contains a long[] and an int[]. The long[] is used to record whether the vertex has adjacent edges, and the int[] is used to quickly obtain the start and end values ​​of the adjacent edge offset of the vertex.

[0065] Step 202: For all vertices with adjacent edges, the starting point or end point of the adjacent edge is saved in the adjacent edge array in sequence, the starting and ending values ​​of the position of the adjacent edge of each vertex in the adjacent edge array is saved in the offset array, and the weight of the adjacent edge is saved in the edge weight array. The existing array, adjacent edge array, offset array and edge weight array form a row-column sparse compressed matrix.

[0066] In actual implementation, the row-column sparse compression matrix can be implemented using two data structures: BitSet and Array.

[0067] When the number of isolated points without adjacent edges exceeds a limit, the memory usage of the row-column sparse compressed matrix using the BitSet data structure is smaller. Therefore, if |V'|>|V| / δ, the row-column sparse compressed matrix of the BitSet data structure containing the existing data is used; otherwise, the row-column sparse compressed matrix of the Array data structure without the existing array is used.

[0068] In a certain scene, there is a vertex V, an isolated point X, and the following inequality exists:

[0069]

[0070] Among them, V' refers to the number of isolated points, and V is the total number of vertices. The left side of the inequality is the row-column sparse compression matrix of the Array data structure, and the right side of the formula is the row-column sparse compression matrix of the BitSet data structure. The inequality represents the memory usage comparison between the two. It can be inferred that when the number of isolated points X is greater than 3 / 64 of the total number of vertices V, the graph data needs to be compressed twice using BitSet to achieve less data memory usage.

[0071] After selecting a suitable data structure, the corresponding data can be written into the offset array, adjacent edge array and weight array according to the corresponding data structure, and then the above arrays and the existence array are combined to form a row-column sparse compressed matrix in this embodiment.

[0072] After step 201 to step 202 provided in this embodiment, the compressed storage of the graph data can be completed by using the row-column sparse compression matrix provided in this embodiment.

[0073] After the data is loaded and compressed for storage, if the adaptive push-pull mode is used, the data compression and storage requirements caused by the change of the graph segmentation strategy need to be resolved. Therefore, this embodiment provides a segmentation and reconstruction method based on edge segmentation to quickly implement point segmentation strategy conversion and data recompression on known edge segmentation to meet the pre-data requirements of the adaptive push-pull mode.

[0074] like Figure 3 As shown, the following method can be used to split and reconstruct vertices and edges based on edge splitting.

[0075] Step 301: traverse the row-column sparse compression matrix and count the number of outgoing edges or incoming edges that need to be reconstructed for all vertices on each computing node.

[0076] When reconstructing, the start and end points of the edge need to be swapped, but the start and end points may not be on the same computing node, so it is necessary to traverse the entire row-column sparse compression matrix and count the number of outgoing edges or incoming edges of a vertex on all computing nodes so as to count them during reconstruction. Traverse the adjacent edge sequence in the row-column sparse compression matrix and count the number of outgoing edges (i.e., out-degree) or incoming edges (i.e., in-degree) of the vertex in each partition computing node of the entire graph. The number of outgoing edges or incoming edges of all vertices is expressed as all_degree[n], where n is the number of vertices and each array element represents the number of outgoing edges or incoming edges of a vertex.

[0077] Step 302: Calculate the total offset of the adjacent edges of all vertices on each computing node in the row-column sparse compression matrix, and arrange the total offset in increasing order according to the computing ID of the vertex.

[0078] Similarly, it is also necessary to count the total offset of a vertex on all calculation nodes. By calculating the total offset of the number of adjacent edges of the whole graph vertex in each partition calculation node, the total number of adjacent edges that need to be reconstructed for each vertex can be obtained. In order to facilitate traversal of vertices, the vertices are arranged in ascending order according to the calculation ID size. The total offset of all sorted vertices is expressed as all_offset[n+1], where n is the number of vertices and each array element represents the total offset of a vertex.

[0079] Step 303: Traverse the adjacent edge sequence, and flip and reorganize the starting point and end point of the adjacent edge of the partition vertex on each computing node according to the total offset to complete the partitioning and reconstruction of the vertices and edges from the partition vertex to all vertices, wherein the number of outgoing edges of each vertex after reconstruction is consistent with the sum of the number of edges with the vertex as the starting point of the incoming edge before reconstruction, and the number of incoming edges of each vertex after reconstruction is consistent with the sum of the number of edges with the vertex as the end point of the outgoing edge before reconstruction.

[0080] Traverse the adjacent edge sequence in the row-column sparse compression matrix, and determine which edge each point in the adjacent edge sequence is the starting point or end point according to the offset of each vertex, and then flip and reorganize the starting point and end point of the adjacent edge of the vertex in the partition. Flipping and reorganizing means taking the vertex in the adjacent edge sequence as the core, writing the starting point of the edge into the new adjacent edge sequence according to the offset sequence, and compressing the data according to the row-column sparse compression matrix storage method provided in this embodiment, generating a new row-column sparse compression matrix, and finally completing data compression and reorganization. Since the starting point and end point of the adjacent edge are reorganized during reconstruction, the connection relationship between the vertex and the adjacent edge is not changed. Therefore, the number of outgoing edges after each vertex is reconstructed is consistent with the sum of the number of edges with the vertex as the starting point of the incoming edge before reconstruction, and the number of incoming edges after each vertex is reconstructed is consistent with the sum of the number of edges with the vertex as the end point of the outgoing edge before reconstruction, and the total number of edges remains the same. Among them, the rules of flipping and reorganization are determined according to the needs of the actual scenario, such as: mutual conversion of input and output, undirected self, etc.

[0081] After steps 301 to 303 provided in this embodiment, the segmentation and reconstruction of the graph data can be completed. The original row-column sparse compressed matrix and the reconstructed row-column sparse compressed matrix respectively record the information of the starting and ending points of the adjacent edges. Therefore, the required data can be directly obtained in both the push mode and the pull mode, so that the data meets the usage requirements of all computing modes.

[0082] In a certain scenario, the current computing node contains vertex A, and its adjacent edges are A->B and A->C. Before reconstruction, the computing node only contains the outbound edge data of vertex A, that is, it only contains the relevant data of vertex A as the outbound vertex, that is: adjacent edge sequence A: [B,C], offset sequence [0,2]. This data can only be used in fixed push mode. When splitting and reconstructing, it is necessary to count all vertices in each partition and flip the outbound data to the inbound data to apply to the pull mode in adaptation. When flipping A->B and A->C, it is necessary to count the number of outbound edges and the number of inbound edges of each vertex with the vertices of the adjacent edges as the core. For vertex A, its adjacent vertices are only B and C. Therefore, the number of inbound edges corresponding to each vertex is: A:0, B:1, C:1, and the corresponding offset sequence is [0,0,1,2]. When performing flip reorganization, write from the full-graph vertices in the current partition as the starting point. Vertex A is skipped if it has no incoming edges. The writing result is a new adjacent edge sequence: A:[],B:[A],C[A]. This result means: vertex A has no incoming edges, the incoming edge of vertex B is another vertex A, and the incoming edge of vertex C is another vertex A. Therefore, when using the adaptive pull mode, all incoming edge adjacent points can be directly obtained through the reconstructed adjacent edge sequence. When the reconstructed adjacent edge sequence A:[],B:[A],C[A] is saved on the computing node, the edge information related to all vertices in the full graph and the vertices in the current partition will be saved after splitting and reconstruction. A belongs to the current partition and is the main point, and B and C do not belong to the current partition and are virtual points. The total amount of edges will not change after splitting and reconstruction, but the expression of the edges will change, and the edge information of the main points and virtual points as the starting point can be obtained at the same time.

[0083] After the graph data is split and reconstructed, the graph data storage representation meets the selected computing mode. At this time, in order to adapt to the data usage needs of a specific algorithm, one or more data processing including adjacent vertex deduplication, adjacent vertex calculation ID sorting, and load balancing can be performed on vertices and edges according to the characteristics of each algorithm and the current computing mode.

[0084] After the graph data loading process is completed, the system scheduling algorithm enters the actual graph computing process. In this embodiment, when the adaptive push-pull computing mode is used, the current computing mode can be obtained according to the number of currently active vertices and edges to achieve adaptive computing mode switching.

[0085] For each computing node, obtain the total number of incoming edges and outgoing edges of all vertices, as well as the total number of weighted incoming edges and weighted outgoing edges of active vertices. Obtain the current computing mode based on the current number of active vertices and edges, where there are at least 3 computing modes.

[0086] In the following calculation: Refers to the total number of adjacent incoming edges and the total number of adjacent outgoing edges of all active vertices in the i-th computing node. Refers to the total number of adjacent incoming edges and the total number of adjacent outgoing edges of all active vertices in the i-th computing node; represents the total number of incoming edges of all vertices in the i-th computation node, represents the total number of outgoing edges of all vertices in the i-th computing node; E represents the weight of the edge, V represents the weight of the vertex, and p represents the number of partitions. ω is a reconciliation threshold determined by the number of edges, vertices, and partitions; when there are more partitions, the division of edges is more dispersed, and the vertex of each computing node holds fewer edges, so when there are more partitions, the value of ω can be reduced.

[0087] Scenario 1: When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is not less than the weighted number of outgoing edges of all vertices, switch to the pull mode in the adaptive push-pull computing mode. That is, the pull mode is used only when the following condition (1) is met and condition (2) is not met:

[0088]

[0089]

[0090] in,

[0091] Scenario 2: When the total number of incoming edges of active vertices is not greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, switch to the push mode in the adaptive push-pull computing mode. That is, when only condition (2) above is met and condition (1) is not met, use the push mode.

[0092] Scenario three: When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, determine whether the ratio of the total number of incoming edges of active vertices to the total number of active vertices is greater than or equal to the ratio of the total number of edges to the total number of vertices. If the judgment is true, switch to the pull mode in the adaptive push-pull computing mode; otherwise, switch to the push mode in the adaptive push-pull computing mode.

[0093] If both conditions (1) and (2) above are met, then If this formula is true, then the pull mode is selected, otherwise the push mode is selected.

[0094] In the hybrid graph computing mode system, the Bulk Synchronous Parallel computing framework (BSP) is usually used, and the computing process of all algorithms is divided into multiple rounds of superstep execution. Therefore, in the computing process, the mode switching calculation is performed before each round of superstep execution. The calculation mode used in the current round is selected by judging the above formula, and the calculation is iterated until the algorithm reaches the convergence condition, the algorithm ends, and the algorithm results are written into the database and the result file is generated.

[0095] The data usage method under the graph computing hybrid computing mode provided in this embodiment converts the graph data according to the selected computing mode through the graph data storage and conversion method provided in the graph of this embodiment, thereby solving the data requirements of the hybrid graph computing mode system under different computing modes, integrating and coordinating the switching between multiple modes, and completing the data conversion work by loading the data only once, optimizing the data layer processing flow, so that the graph computing system under the hybrid graph computing mode can switch between different computing modes efficiently and flexibly. Compared with the single-mode graph computing system, the hybrid graph computing mode system has a wider application range and higher execution efficiency.

[0096] Embodiment 2:

[0097] Based on the data usage method in the graph computing hybrid computing mode provided in Example 1, in some specific implementations, it can be implemented through the specific implementation of this embodiment. It can be understood that the specific implementation provided in this embodiment is only used to illustrate the specific implementation process of the method in Example 1 in certain specific scenarios, and is not intended to limit the scope of protection.

[0098] In the scenario of this embodiment, given graph G<m,n> , m is the number of vertices, n is the number of edges; the distributed graph computing cluster contains p computing nodes.

[0099] The master node obtains the original id of all vertices in the graph, maps each vertex original id to a calculated id that is continuously incremented and non-repeated starting from 0, and saves it in the id mapping data container. The mapping data container includes Map<Long,int> and int[], indicating the mapping relationship between the original long id and the calculated int id.<Long,int> The key is the original id of the long data type, and the value corresponding to the key is the calculated id of the int data type; the subscript of int[] is the calculated id of the int data type, and the element value is the original id of the long data type corresponding to the calculated id represented by the subscript.

[0100] All computing modes use the continuous block equalization method to divide vertices. In the graph G, the distributed graph computing cluster has p computing nodes, which are divided into {0, m / p..., m} according to the order of computing node partitioning. Among them, when using the adaptive push-pull model, there are the concepts of main points and virtual points. For example, for computing node 0, the main point range is [0, m / p), and other points are virtual points in the current computing node.

[0101] When dividing the edges associated with each vertex, the fixed push-pull mode and the adaptive push-pull mode have different rules for dividing the edges. The fixed push-pull mode stores all the adjacent edges of its own principal point; the adaptive push-pull mode includes the push mode and the pull mode. The push mode saves the edges whose end point is the principal point, and the pull mode saves the edges whose starting point is the principal point.

[0102] The fixed push-pull mode edge division rules are as follows:

[0103] E i ={(src,dst,value)∈E|src∈V i};

[0104] The rules for edge division in the adaptive push-pull mode are as follows:

[0105]

[0106] Among them, E i is the edge set of the i-th partition in fixed push-pull mode, is the edge set of the i-th partition in the push mode in the adaptive push-pull mode, is the edge set sum of the i-th partition in the pull mode in the adaptive push-pull mode; V i is the vertex set of the i-th partition; src is the starting point of the edge, dst is the end point of the edge, and value is the weight of the edge.

[0107] After the division is completed, each computing node reads the data source data in parallel, submits the direction parameters according to the algorithm using the graph data, divides the range according to the points to which it belongs, loads the edge data, and uses the edge division rules of the fixed push-pull mode to save the edge information into the sparse row and column compression matrix.

[0108] In a specific scenario, the structure of graph G is as follows Figure 4 As shown in the figure, the numbers in the figure represent the calculation ID of the corresponding vertex. According to the point partitioning rule, vertices 0 and 1 are divided into calculation nodes 0.<m,n> The row-column sparse compressed matrix used in this computation node is represented as follows:

[0109] In the outbound direction: nbr = [1, 2, 3, 5]; idx = [0, 2, 4].

[0110] In the inbound direction: nbr = [2, 4, 0, 2]; idx = [0, 2, 4].

[0111] Among them, nbr is the adjacent edge array, which stores the end point of the edge in the outbound direction or the starting point of the edge in the inbound direction of each vertex; idx is the offset array, which stores the offset of the adjacent edge of each vertex in nbr.

[0112] From the data in the row-column sparse compressed matrix, it can be seen that in the outbound direction, the endpoints of the adjacent edge of vertex 0 are vertex 1 and vertex 2, and the endpoints of the adjacent edge of vertex 1 are vertex 3 and vertex 5; in the inbound direction, the starting points of the adjacent edge of vertex 0 are vertex 2 and vertex 4, and the starting points of the adjacent edge of vertex 1 are vertex 0 and vertex 2. For each computing node, data storage can be completed in the same way as above.

[0113] After the computing node has loaded the partition data, it determines the computing mode parameters and, if the adaptive push-pull mode is used, performs data storage representation conversion.

[0114] For computing node 0, first count the neighbor edges. Traverse the offset array and count the number of neighbor edges of the vertex in the partition computing node. The statistical results of the outgoing and incoming directions are: degree out =[0,1,1,1,0,1], degree in =[1,0,2,0,1,0]. Then, based on the statistical results, the offset of the vertex of the entire graph in the current partition is calculated. The outgoing and incoming directions are: out =[0,0,1,2,3,3,4], offset in =[0,1,1,3,3,4,4], and finally traverse the adjacent edge array nbr according to the offset result to reorganize the data.

[0115] In push mode, row sparse compression matrix is ​​used, and the reorganization result is as follows Figure 5 As shown, where: nbr = [1,0,1,0], idx = [0,1,1,3,3,4,4].

[0116] In pull mode, column sparse compression matrix is ​​used, and the reorganization result is as follows Figure 6 As shown, where: nbr = [0,0,1,1], idx = [0,1,1,3,3,4,4].

[0117] The data storage format of the converted graph data is the same as that before the conversion. However, the data storage in the adaptive push-pull mode represents the adjacency data of the vertices of the entire graph at the current computing node, and its adjacency points belong to the current partition.

[0118] After completing the graph data conversion, you can also perform data processing operations on the converted graph data, and perform optional adjacent vertex deduplication, adjacent vertex ID sorting and load balancing for specific algorithms. After completion, the algorithm enters the calculation process and finally completes the calculation.

[0119] In actual implementation, you can choose a fixed push mode, a fixed pull mode, or an adaptive push-pull mode. In actual implementation, you can select the computing mode by adaptive selection, or you can select the optimal computing mode by empirical parameter binding. When using a fixed push mode or a fixed pull mode, the above data conversion process is not required; when using an adaptive push-pull mode, the graph data needs to be converted.

[0120] In a specific scenario, it is necessary to use a weakly connected component algorithm for graph calculations. The calculation mode parameter is an adaptive push-pull mode, and the weakly connected component algorithm performs iterative calculations according to the BSP calculation model. There is a characteristic of vertex convergence in the weakly connected component calculation process. In the first round of iterative calculations, all vertices participate in the calculation, activeE=alledges, and the pull mode is selected according to the above calculation mode switching rules; after the first round of iterative calculations, each round of iterative calculations may have vertex results that converge and no longer participate in the calculation. After each round of iterative calculations, it is necessary to select an appropriate push mode according to the number of active vertices and active edges according to the method in Example 1.

[0121] In another specific scenario, the PageRank webpage ranking algorithm is used for iterative calculation, and the calculation mode parameter is the adaptive push-pull mode. The PageRank webpage ranking algorithm is an algorithm in which all graph vertices participate in the iteration in each round. The number of active vertices will not decrease during the algorithm calculation process. Therefore, during the iterative calculation process, the algorithm selects the pull mode in each round of iterative calculation according to the mode switching rules until the algorithm results converge or the maximum number of iterations is reached.

[0122] Embodiment 3:

[0123] Based on the data use method in the graph computing hybrid computing mode provided in the above-mentioned embodiments 1 and 2, the present invention also provides a data use device in the graph computing hybrid computing mode that can be used to implement the above-mentioned method, such as Figure 7 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The data usage device in the graph computing hybrid computing mode of this embodiment includes one or more processors 11 and a memory 12. Figure 7 A processor 11 is taken as an example.

[0124] The processor 11 and the memory 12 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0125] The memory 12 is a non-volatile computer-readable storage medium as a method for using data in a graph computing hybrid computing mode, and can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the method for using data in a graph computing hybrid computing mode in Embodiments 1 to 2. The processor 11 executes various functional applications and data processing of the data using device in a graph computing hybrid computing mode by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, implements the method for using data in a graph computing hybrid computing mode in Embodiments 1 to 2.

[0126] The memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the processor 11 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0127] The program instructions / modules are stored in the memory 12. When executed by one or more processors 11, the data usage method in the graph computing hybrid computing mode in the above-mentioned embodiments 1 to 2 is executed, for example, the above-described Figures 1 to 3 The steps shown.

[0128] An embodiment of the present invention further provides a non-volatile computer storage medium, wherein the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by one or more processors, such as Figure 7 A processor 11 in the embodiment 1 can enable the one or more processors to execute the data use method in the graph computing hybrid computing mode in the embodiment 1 to the embodiment 2, for example, to execute the above-described Figures 1 to 3 The steps shown.

[0129] It is worth noting that the information interaction, execution process, etc. between the modules and units within the above-mentioned devices and systems are based on the same concept as the processing method embodiment of the present invention. The specific contents can be found in the description of the method embodiment of the present invention and will not be repeated here.

[0130] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data usage method in a graph computing hybrid computing mode, characterized in that: include: Partition the vertices according to the weights of the vertices and edges, and assign the vertices in each partition to the corresponding computing nodes; Each computing node saves the divided vertices, uses row-column sparse compression matrix to save the adjacency relationship between vertices and edges, and splits and reconstructs the vertices and edges saved in each computing node according to the adjacency relationship, so that the reconstructed vertices and edges can adapt to all computing modes; Get the current calculation mode based on the number of currently active vertices and edges, and use the reconstructed vertices and edges to process data in the current calculation mode.

2. The data use method in the graph computing hybrid computing mode according to claim 1 is characterized in that: Partitioning the vertices according to the weights of the vertices and the edges specifically includes: Get the weight of each vertex and edge, and calculate the sum of the weighted total weight of all vertices on each computational node and the weighted total weight of all its associated edges; According to the number of computing nodes, the vertices are divided into a corresponding number of partitions according to the original ids of the vertices, where the sum of the weighted weights of all vertices and all their associated edges in each partition is the same.

3. The data use method in the graph computing hybrid computing mode according to claim 1 is characterized in that: The method of using a row-column sparse compressed matrix to store the adjacency relationship between vertices and edges specifically includes: Each computing node obtains the adjacent edges of the divided vertices from the data source, and uses the existence sequence to record whether there are adjacent edges for each vertex; wherein the adjacent edges include: the starting point of the edge, the end point of the edge and the edge weight; For all vertices with adjacent edges, the starting point or end point of the adjacent edge is saved in the adjacent edge sequence in turn, the starting and ending values ​​of the adjacent edge of each vertex in the adjacent edge sequence are saved in the offset sequence, and the weight of the adjacent edge is saved in the edge weight sequence. The existence sequence, adjacent edge sequence, offset sequence and edge weight sequence are combined to form a row-column sparse compressed matrix.

4. The data use method in the graph computing hybrid computing mode according to claim 1, characterized in that: Each computing node stores the divided vertices, specifically including: Get the original IDs of all vertices in the graph. Each computing node maps the original IDs of the divided vertices to the calculation IDs in parallel, and saves the calculation IDs of each divided vertex. The calculation ID is an integer that increases continuously from 0 and does not repeat.

5. The data use method in the graph computing hybrid computing mode according to claim 4 is characterized in that: The segmentation and reconstruction of the vertices and edges stored in each computing node according to the adjacency relationship specifically includes: According to the calculated ID and adjacency relationship of the vertex, the vertices and edges are split and reconstructed based on edge splitting.

6. The data use method in the graph computing hybrid computing mode according to claim 5, characterized in that: The segmentation and reconstruction of vertices and edges based on edge segmentation specifically includes: Traverse the row and column sparse compression matrix and count the number of outgoing edges or incoming edges that need to be reconstructed for all vertices on each computing node; Calculate the total offset of the adjacent edges of all vertices on each computing node in the row-column sparse compression matrix, and arrange the total offset in increasing order of the computing id of the vertex; Traverse the adjacent edge sequence, and flip and reorganize the starting and ending points of the adjacent edges of the partitioned vertex on each computing node according to the total offset to complete the partitioning and reconstruction of the vertices and edges from the partitioned vertex to all vertices, where the number of outgoing edges of each vertex after reconstruction is consistent with the sum of the number of edges with the vertex as the starting point of the incoming edge before reconstruction, and the number of incoming edges of each vertex after reconstruction is consistent with the sum of the number of edges with the vertex as the end point of the outgoing edge before reconstruction.

7. The data use method in the graph computing hybrid computing mode according to claim 5, characterized in that: The segmentation and reconstruction of vertices and edges based on edge segmentation also includes: Perform one or more of the following data processing on vertices and edges: adjacent vertex deduplication, adjacent vertex ID sorting, and load balancing.

8. The data use method in the graph computing hybrid computing mode according to claim 1, characterized in that: When the calculation mode is the adaptive push-pull calculation mode, the current calculation mode is obtained according to the number of currently active vertices and edges, specifically including: For each compute node, obtain the total number of incoming edges and outgoing edges of all vertices, as well as the total number of weighted incoming edges and weighted outgoing edges of active vertices; When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is not less than the weighted number of outgoing edges of all vertices, switch to the pull mode in the adaptive push-pull calculation mode; When the total number of incoming edges of active vertices is not greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, switch to the push mode in the adaptive push-pull calculation mode; When the total number of incoming edges of active vertices is greater than the weighted number of incoming edges of all vertices, and the total number of outgoing edges of active vertices is less than the weighted number of outgoing edges of all vertices, determine whether the ratio of the total number of incoming edges of active vertices to the total number of active vertices is greater than or equal to the ratio of the total number of edges to the total number of vertices. If the judgment is established, switch to the pull mode in the adaptive push-pull calculation mode; otherwise, switch to the push mode in the adaptive push-pull calculation mode.

9. A data usage device in a graph computing hybrid computing mode, characterized in that: It includes at least one processor and a memory, and the at least one processor and the memory are connected via a data bus. The memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the data usage method in the graph computing hybrid computing mode described in any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the data usage method in the graph computing hybrid computing mode as described in any one of claims 1-8.

Images