Distributed graph calculation acceleration method based on self-increasing ID and sequential transmission

Abstract

The invention provides a method for accelerating information synchronization between nodes in a graph computing system. An original string type Vertex ID is converted into an int type self-increasing ID, and Values of vertexes are subjected to serialization and deserialization in sequence, so that the data transmission quantity and the computing power overhead are reduced; in a graph calculation framework, a master node is responsible for negotiating scheduling, a worker node is responsible for executing calculation, a hash algorithm is used for distributing Vertex to the worker node, the master node negotiates and distributes a self-increasing Vertex ID segment, after calculation is completed, only Vertex Value data is transmitted, the Vertex Value data is obtained through a fixed-size deserialization method, the performance is remarkably improved, the network resource overhead is reduced, the data transmission amount can be effectively reduced, and the data transmission efficiency is improved. And the overall efficiency of the graph calculation system is improved.

Description

Technical Field

[0001] This technology belongs to the field of computer science, and more specifically, is a technology for accelerating information synchronization between computing nodes in a distributed graph computing system. Background Art

[0002] In the era of big data, there are relationships between data. Since graphs are organizational structures that express complex relationships between things, they are needed in many application scenarios in real life, such as Taobao user friend relationship graphs, road maps, circuit diagrams, virus transmission networks, national power grids, literature networks, social networks, and knowledge graphs.

[0003] In order to obtain useful information from the correlation between these data, a large number of graph algorithms have emerged. They obtain important information hidden in the graph data through iterative processing of large graph data. As the core technology of the next generation of artificial intelligence, graph computing has been widely used in many fields such as medicine, education, military, and finance, and has attracted much attention from governments, global research and development institutions, and giant companies. It has become a new strategic commanding height in the global scientific and technological competition.

[0004] With the continuous development of Internet business, the scale of data is getting larger and larger, and it is often necessary to process graphs with tens of billions of vertices, hundreds of billions or even trillions of edges. In order to meet the growing demand for large-scale graph data processing, many excellent graph computing frameworks have emerged in the computer industry, such as Google's Pregel and CMU's GraphLab. Although these graph computing systems have significantly improved performance compared to MapReduce, Spark, etc., their computing efficiency is still very low, and sometimes even worse than well-optimized single-threaded programs.

[0005] Traditional graph computing frameworks need to transfer a large number of vertex IDs between distributed computing nodes, greatly increasing the amount of data that needs to be sent between distributed computing nodes, resulting in huge network and computing resource overhead for the entire system, and performance cannot keep up with the needs of expanding data scale. Summary of the invention

[0006] In view of this, the present application provides a method that can greatly reduce the network and computing power overhead of information synchronization between computing nodes and improve the performance of the graph computing system.

[0007] The graph computing framework of this application is a distributed framework. The service nodes are divided into two types, master and worker. The master is responsible for statistics, coordination and scheduling of the computing process. The amount of computing itself is very small. The worker is responsible for the actual storage and computing tasks. There is only one master service in the framework, and there are multiple worker services. The services use TPC long connections for two-way communication. The running framework is as follows:Figure 1 shown.

[0008] A graph is composed of vertices and edges. This system uses the point-cutting method to distribute the graph and divide a graph into multiple worker nodes to achieve distributed computing. Therefore, there are two types of Vertex: Master Vertex, which is the main vertex for graph computing. Each vertex has only one Master Vertex. The other is MirrorVertex, which is the mirror image of the main vertex on other worker nodes and is used to provide the necessary information required for computing. Due to computing needs, each worker must store the neighbor vertex information of all incoming edges corresponding to all Master Vertex.

[0009] In the computing framework of this application, the imported original business Vertex ID is not directly used in the computing process, but the original string-type Vertex ID is converted into an int-type auto-incrementing ID. In the subsequent graph computing tasks, only the int-type Vertex ID is used, and the int-type ID is only converted into the original string ID for output in the final output stage. To achieve this, the graph data import solution is as follows.

[0010] The first step of graph computing is to import graph data from the outside. The source can be files, databases and other media. Multiple workers import graph data in batches at the same time, first importing points and then edges.

[0011] The Master service provides an HTTP API for importing graph data. The corresponding API is called externally to specify the import method, data source, data volume, etc. The Master service splits the entire large import task into multiple small subtasks based on the received task parameters, and notifies the worker service that the task is ready and starts importing. The worker service requests the import subtask from the master service and obtains the parameters required for import, such as the file name to be read or the database address. The server assigns the task in the ready state to the worker and sets the task state to doing. After receiving the task parameters, the Worker starts to read the Vertex data, hashes the read Vertex ID, and takes the remainder of the number of workers, and uses the result as the worker node ID finally assigned to this Vertex, that is, workerID = hash (VertexID) % workerCount. The data of this Vertex is sent to the corresponding worker service to complete the Vertex allocation. When all the Vertex data in the task is allocated, the master service is notified, and the master service marks this subtask as done. When all subtasks are in the done state, the import task of the entire graph is completed.

[0012] While reading the Vertex information, the Worker service uses the above hash algorithm to assign the Vertex to the corresponding worker. On each worker, a sending buffer is established for all workers, and the assigned Vertex information is stored in the buffer area corresponding to the worker. After the buffer is full, the Vertex data is sent to the corresponding worker service through the network in batches. In order to speed up the sending speed, a double buffering mechanism can be used here, that is, two buffers are established for each worker, one for writing Vertex data, which automatically enters the sending state after it is full, and the other for sending, which automatically enters the waiting to be written state after sending.

[0013] After receiving the Vertex information from other Workers, the Worker service stores the Vertex information into the Vertex array. After the collection is completed, each worker counts the number of Vertexes assigned to itself and reports it to the master service. Through master negotiation, the starting ID of the self-incrementing ID of each worker node is obtained, as long as there are no duplicate Vertex IDs between workers. After negotiation with the master, each worker node obtains a Vertex ID segment, which is StartID (starting ID) to Start ID + number of Vertexes. After the allocation is completed, the master synchronizes the ID segment to each worker. After the worker obtains the self-incrementing ID segment, it can assign self-incrementing IDs to all Vertexes. For example, if there are 1 billion Vertexes in total and 5 workers, the Vertex ID segment allocation is as follows:

[0014] Worker1:[0,199,999,999]

[0015] Worker2:[200,000,000,399,999,999]

[0016] Worker3:[400,000,000,599,999,999]

[0017] Worker4:[600,000,000,799,999,999]

[0018] Worker5:[800,000,000,999,999,999]

[0019] The subscript of the Vertex array on each worker + the StartID of this worker is the auto-increment ID of the corresponding Vertex. Each worker establishes a map correspondence for the Vertex on this service to realize the mutual conversion of the auto-increment ID to the original ID. The structure of the auto-increment ID to the original ID is an array, the array subscript corresponds to the Vertex array subscript, and the content is the original ID of string type, that is: [] string IntIDToStringID. The structure of the original ID to the auto-increment ID is a Hash Map, that is: HashMap[string]int StringIDToIntID, the key is the original ID, and the value is the auto-increment ID.

[0020] So how does this solution reduce network and computing overhead in graph computing and improve the performance of the graph computing framework? The computing model of this solution is that in each round of calculation, the vertex Vertex uses its own Value and the Value of its neighboring Vertex for calculation, and all the information required for the calculation is stored in various data structures of the worker node (including Vertex array, Value array, etc.). In addition to the Vertex information assigned to the local machine (the Vertex assigned to this worker, that is, the Vertex directly managed by this worker is called the Master Vertex), each worker node also needs to store all the Vertex adjacent to the Master Vertex. These Vertex are called Mirror Vertex. The worker does not directly manage these Vertex, but the calculation process requires the information of these Mirror Vertex. These Mirror Vertex are backups of the Master Vertex in other workers. The storage of Vertex in the worker is as follows: Figure 3 shown.

[0021] After each round of calculation is completed, the Master Vertex in the worker will generate a new Value, which will cause the Value of the Master Vertex to be inconsistent with the Value of the Mirror Vertex on other corresponding workers, so the Value of the Master Vertex needs to be synchronized to the Value of the Mirror Vertex in other worker nodes.

[0022] In the existing graph computing framework, synchronizing the Value requires at least sending the key-value pair of Vertex ID and Vertex Value so that the peer worker can obtain the Value of the corresponding Vertex. However, this framework does not need to send the key-value pair of Vertex ID and Vertex Value like the existing architecture. Instead, it serializes the corresponding VertexValue into the data stream one by one starting from the Start ID in order. Since the size of the Vertex Value is fixed, the Vertex ID segment of each worker has been determined during the data import phase. Therefore, after sending it to the corresponding worker node, it only needs to query the Start ID through the worker ID, and directly deserialize the fixed-size data to obtain the data of the corresponding Vertex ID, eliminating the need to send the Vertex ID data. Deserialization also only needs to parse the Value of the corresponding Vertex, greatly improving the serialization and deserialization performance. The specific synchronization scheme is as follows:

[0023] Before the calculation begins, a Value array is initialized. The size is the number of all vertices. It is used to store the Value values ​​of all vertices. The order corresponds to the Vertex array. That is, the Value corresponding to the subscript of the Value array is the Value of the corresponding position in the Vertex array. In this way, if you want to get the Value of the Vertex with the self-incrementing ID vid1, you can directly use Value[vid1]. After the calculation is completed, the new value of the Master Vertex will be in the Value array. At this time, the new Value needs to be synchronized to the Mirror Value in other worker nodes before the subsequent iterative calculation can continue.

[0024] Before synchronization, a send buffer is initialized for each worker. The for loop traverses the Value array of the Master Vertex. The values ​​in the Value are serialized and directly stored in the send buffer. After the send buffer is full, it can be sent to the corresponding worker without writing Vertex ID and other information. After receiving the Value data stream, the receiving worker directly deserializes the value in the data stream according to the Value Type to obtain the value of the Value. By reading the worker ID corresponding to this connection, the StartID of the Vertex of the other worker can be obtained. Therefore, the Vertex ID corresponding to the deserialized Value is the sequence number of the deserialized Value (that is, the number of the Value of this connection) i+StartID. After calculating the Vertex ID, the new Value can be updated to the Value of the corresponding Mirror Vertex. After the Value of the Mirror Vertex is synchronized, the next round of iterative calculation can be started.

[0025] To realize the above graph computing functions, this framework contains the following main modules:

[0026] 1. Communication module: used for data exchange between master and worker, worker and worker, command sending and receiving, etc. In order to achieve fast, stable and efficient data communication, this framework uses TCP long connection as the main communication means. In addition, the master service needs to receive external calls, so the external API uses HTTP, which is easier to use and has better compatibility.

[0027] 2. Serialization and deserialization module: used to support the conversion between the Value data required for calculation and the network byte stream. In order to ensure the efficient conversion of large amounts of data, this system implements serialization and deserialization algorithms separately for all data types that need to be transmitted.

[0028] 3. Input and output module: used to support multiple graph data input methods, such as file system, MYSQL, HDFS, etc.

[0029] 4. Computing process management module: used to support the execution process management and scheduling of graph computing tasks

[0030] 5. Algorithm plug-in module: used to support user-defined algorithms, and implement custom algorithm writing based on the plug-in method without changing the framework.

[0031] 6. Graph data import module: used to implement the execution process management and scheduling of graph data import tasks.

[0032] 7. Graph data storage module: used to store the vertex and edge data required for graph calculation on the worker, as well as the conversion function between the original ID and the auto-increment ID.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] The present invention proposes a distributed graph computing acceleration method based on auto-incrementing ID and sequential transmission. The original string ID is converted into an integer auto-incrementing ID, and the vertex values ​​are serialized and deserialized sequentially, thereby avoiding the transmission of vertex IDs and complex data structures, greatly reducing the computing power overhead of serialization and deserialization and the amount of data transmitted over the network, improving the overall performance of the graph computing system, and enabling the graph computing system to support larger-scale graph computing business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 :System operation architecture diagram

[0036] Figure 2 : Vertex data import service interaction sequence diagram

[0037] Figure 3 :Vertex allocation and storage diagram on workers DETAILED DESCRIPTION

[0038] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0039] Take bank accounts as an example. The bank has a total of one billion accounts and the transfer relationship between accounts is abstracted into a graph. The vertices are accounts, the vertex IDs are the account numbers, and the edges are the account transfer relationships. One billion vertices are stored in an average of 100 files in an NFS distributed file system. The graph computing cluster consists of one master node and five worker nodes. The worker IDs are 0, 1, 2, 3, and 4 respectively. The execution algorithm is Page Rank.

[0040] The user sends an HTTP request to the master service, providing the NFS file address and path of the graph data. The master service divides the import task into 100 subtasks based on the 100 vertex files provided, and notifies the worker to import the graph data. The worker requests the master service for the import method, file address and other task information. The five workers simultaneously read the corresponding files to obtain the vertex data (account data). Then, the account is assigned to the corresponding worker according to the account number, corresponding to WorkerID = hash (account number) % 5, and the account information is written into the corresponding worker's send buffer, which is sent after it is full.

[0041] After receiving the assigned account information, the receiving worker initializes a Vertex array to store the received account information. After all import subtasks on the master are completed, the master notifies all workers to report the number of Vertex. The master allocates auto-increment ID segments based on the number of Vertex reported by the worker and sends the allocation results to all workers. After receiving the allocation results, the worker establishes a mapping relationship between the original ID and the auto-increment ID of the Vertex on this node. Assuming that the allocation is completely even, each worker should have 200 million points, and the auto-increment ID segment of each worker is as follows:

[0042] Worker0:[0,199,999,999]

[0043] Worker1:[200,000,000,399,999,999]

[0044] Worker2:[400,000,000,599,999,999]

[0045] Worker3:[600,000,000,799,999,999]

[0046] Worker4:[800,000,000,999,999,999]

[0047] After the vertex import is completed, the edge (transfer relationship) import begins. The process is similar to the vertex import. The master splits the subtasks and the worker performs the actual import task (since the edge import is not the focus of this invention, the description is relatively brief). After the edge is imported, each worker can obtain the neighbor Vertex corresponding to the current node Vertex (that is, other accounts that transfer money to me).

[0048] After the import task is completed, the algorithm execution phase begins. Here, the Page Rank algorithm is executed. Before the algorithm is executed, a Value array is created. The array size is the number of all points in this graph, that is, 1 billion. The array subscript is the auto-increment ID of Vertex. Value is of float32 type, and the required memory is 4GB. All Value array values ​​are initialized to 1.0 / 1000000000.

[0049] According to the definition of the Page Rank algorithm, the new value of each vertex is equal to the sum of the values ​​of all neighbors / the value of the neighbor divided by the number of its neighbors, that is, newValue = Σ(nv / nd), nv is the Value of the neighbor, and nd is the number of corresponding neighbors.

[0050] After calculating the new value, it is necessary to synchronize the new value to the Value array on other workers. Take worker0 as an example. This worker stores 200 million Vertex and its Value, so initialize the send buffer, serialize the Value of the 200 million Vertex vertices from front to back and write them into the send buffer, and send these data streams to the other 4 workers.

[0051] After receiving the Value data stream sent by woker0, the receiver deserializes it in float32 from front to back. Since the master has informed all workers of the worker's StartID, it knows that worker0's StartID = 0, and the order of ValueID = Value for each Value + StartID = i. After calculating the MirrorVertexID corresponding to the Value, this value can be updated to the Value array, that is, Value[ValueID] = newValue. After the Value value is synchronized, the next round of iterative calculation can be started until the iteration ends.

[0052] From the above example, we can see that, assuming that the bank account number is a string with an average length of 20 bytes, the total amount of data that needs to be transmitted is:

[0053] Existing system: total number of vertices * average length of vertex data packet * (number of workers - 1), the structure of the vertex serialized data packet is: ID Length (4B) | ID (20B) | ValueType (1B) | Value (4B). Therefore, the total data volume is 1000000000 * (4 + 20 + 1 + 4) * 4 = 140GB.

[0054] The method proposed in this application: 1000000000*4*4=16GB,

[0055] The amount of data that needs to be transmitted by this system is less than one-eighth of the original system, which has a huge performance improvement in serialization and data transmission.

Claims

1. A distributed graph computing acceleration method based on self-incrementing ID and sequential transmission, characterized in that: The following steps are involved: a) Importing graph data into a distributed graph computing framework, which includes a Master node and multiple Worker nodes; b) Assign a unique integer auto-increment ID to each imported vertex, and store the mapping relationship between the original string vertex ID and the auto-increment ID on each Worker node; c) After each round of iterative calculation is completed, each Worker node serializes the Value of all MasterVertex on it into the data stream in the order of self-incrementing ID, and sends the data stream to other Worker nodes; d) The receiving Worker node queries the starting Vertex ID of the corresponding auto-increment ID segment based on the ID of the sending Worker node; e) The receiving Worker node deserializes the corresponding Mirror Vertex value directly from the data stream based on the starting Vertex ID and the fixed size of the value type, and updates the Mirror Vertex value.

2. The method according to claim 1, characterized in that The method of allocating the auto-increment ID in step b) includes: a) The Vertex read by the Worker is distributed to each Worker node as the MasterVertex through the hash algorithm; b) Each Worker node counts the number of vertices on it and reports it to the Master node; The Master node negotiates and allocates non-overlapping auto-increment ID segments to each Worker node based on the number of vertices of each Worker node. c) Each Worker node assigns an auto-incrementing ID to the Master Vertex on it. The value of the auto-incrementing ID is the subscript of the vertex in the vertex array of the Worker node to which it belongs plus the subscript of the Worker. The starting ID of the node.

3. The method according to claim 1, characterized in that The method of serializing the data stream in step c) includes: a) Initialize a Value array with the same size as the vertex array of this Worker node, and store the Value of each MasterVertex in the corresponding position in the array; b) Serialize the values ​​in the Value array into the data stream in the order of the auto-incrementing ID.

4. The method according to claim 1, characterized in that: The method for deserializing the data stream in step e) includes: a) According to the ID of the sender Worker node, query the start of the corresponding auto-increment ID segment Vertex ID; b) Read values ​​from the data stream in sequence according to the fixed size of the Value type, and assign the read values ​​to the Value corresponding to the Mirror Vertex whose auto-increment ID is the start ID plus the read sequence number.

Images