A graph computing implementation method based on shared memory, terminal device and medium

Abstract

The present invention relates to a graph computing implementation method based on shared memory, a terminal device and a medium, wherein the method comprises: dividing the data of a graph structure into multiple slices; sorting all vertices in each slice, and treating the data corresponding to each vertex as a data block; duplicating each vertex multiple times to obtain a main vertex and multiple sub-vertices; obtaining a main queue and multiple sub-queues according to the arrangement order of each vertex in the slice, wherein the main queue is composed of the main vertices corresponding to all vertices contained in the slice, and the sub-queue is composed of the sub-vertices corresponding to all vertices contained in the slice; storing both the main queue and the sub-queue in the shared memory; and sequentially reading the vertices at the head of the main queue or the sub-queue for data processing during multi-threaded concurrent processing. The present invention can efficiently execute the flow and processing of data and supports concurrent reading.

Description

Technical Field

[0001] The present invention relates to the field of graph computing, and in particular to a graph computing implementation method based on shared memory, a terminal device and a medium. Background Art

[0002] Graph computing plays an important role in relationship building, user group analysis and discovery, attribute propagation, etc. In the era of big data, when the scale of a graph reaches a certain level, it is difficult for a single machine to solve large-scale graph computing. Therefore, it is of great significance to develop and debug graph algorithms for large-scale data.

[0003] Graph computing based on distributed computing is to divide graph data into multiple parts and store them in different computing nodes, and complete computing tasks through network communication and collaboration. Due to factors such as network transmission and coordination between nodes, although this method can support large-scale graph computing, the latency is relatively high. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a graph computing implementation method, terminal device and medium based on shared memory.

[0005] The specific plan is as follows:

[0006] A method for implementing graph computing based on shared memory includes the following steps:

[0007] S1: Divide the graph structure data into multiple shards;

[0008] S2: Sort all vertices in each shard and treat the data corresponding to each vertex as a data block;

[0009] S3: Duplicate each vertex multiple times to obtain a main vertex and multiple secondary vertices; obtain a main queue and multiple secondary queues according to the arrangement order of each vertex in the shard, the main queue is composed of the main vertex arrangement corresponding to all vertices contained in the shard, and the secondary queue is composed of one secondary vertex arrangement corresponding to all vertices contained in the shard; store both the main queue and the secondary queue in the shared memory;

[0010] S4: In the multi-threaded concurrent processing process, the vertices at the head of the main queue or the sub-queue are read sequentially for data processing. The data processing process adopts the GAS calculation model. When it reaches the Gather stage, the main vertex and the sub-vertex are calculated separately; when it reaches the Apply stage, the sub-vertex sends the calculation results of the Gather stage to the main vertex, which is summarized and further calculated by the main vertex. In the Scatter stage, the data of the main vertex is updated based on the further calculation results of the main vertex and synchronized to the sub-vertex; after the data in the vertex is updated, the edge data is updated.

[0011] Furthermore, the sorting in step S2 is performed using the PSW algorithm.

[0012] Furthermore, the shared memory is set to be able to be read by multiple processes, and only one process can write to the shared memory.

[0013] Furthermore, in the reading and writing of shared memory, the writing process writes from the end of the queue, and the reading process reads from the head of the queue. If the end of the queue is read, it waits for the writing process to complete before reading the data block at the corresponding position.

[0014] A terminal device for implementing graph computing based on shared memory includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method in the embodiment of the present invention are implemented.

[0015] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described above in an embodiment of the present invention are implemented.

[0016] The present invention adopts the above technical solution, can efficiently execute the flow and processing of data, supports concurrent reading, and avoids the context switching action when a large number of threads are switched in the traditional way, which causes a large amount of resource waste. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0018] To further illustrate various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in this field should be able to understand other possible implementations and advantages of the present invention.

[0019] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.

[0020] Embodiment 1:

[0021] The embodiment of the present invention provides a method for implementing graph computing based on shared memory, such as Figure 1 As shown, the method comprises the following steps:

[0022] S1: Divide the graph structure data into multiple shards.

[0023] Graph-structured data can be any type of data, such as social networks, knowledge graphs, etc.

[0024] Sharding can be implemented using existing methods. In this embodiment, the sharding process is as follows:

[0025] 1. Randomly select some nodes as the initial medoids (center points), which can be any nodes in the graph.

[0026] 2. For all non-medoids nodes, assign them to the cluster with the nearest medoid.

[0027] 3. For each cluster, calculate the sum of the distances from all nodes in it to the medoid of the cluster, and select the node with the smallest sum of distances as the new medoid.

[0028] 4. Repeat steps 2 and 3 until a certain condition is met, such as reaching the maximum number of iterations or the cluster no longer changes.

[0029] S2: Sort all vertices in each shard, and treat the data corresponding to each vertex as a data block.

[0030] In this embodiment, all vertices in each shard are sorted by the PSW algorithm, and the sorting can be performed based on the ID value of the vertex.

[0031] The final fragments are all data blocks composed of vertices.

[0032] S3: Copy each data block multiple times to obtain a primary data block and multiple secondary data blocks; according to the arrangement order of each vertex in the shard, obtain a primary queue and multiple secondary queues, the primary queue is composed of the primary data blocks corresponding to all the data blocks contained in the shard, and the secondary queue is composed of one secondary data block corresponding to all the data blocks contained in the shard; store both the primary queue and the secondary queue in the shared memory.

[0033] S4: During the multi-threaded concurrent processing, the data blocks at the head of the main queue or the sub-queue are sequentially read for data processing.

[0034] In this embodiment, data processing adopts the GAS (Gather Apply Scatter) computing model of distributed graph computing, and the computing process is divided into the following three steps:

[0035] Step 1: Gather phase: The main vertex and the secondary vertex collect the data of the adjacent edges and themselves for calculation. In this phase, the data on the vertices and edges are read-only.

[0036] Step 2: Apply phase: The secondary vertex sends the calculation results of the Gather phase to the primary vertex, which then summarizes and further calculates them.

[0037] Step 3: Scatter phase: Based on the further calculation results of the main vertex, the data of the main vertex is updated and synchronized to the secondary vertex; when the data in the vertex is updated, the edge data is updated.

[0038] In this embodiment, the shared memory is set to be able to be read by multiple processes, and only one process can write to the shared memory. In this way, the context switching of the kernel can be greatly reduced, and the throughput and performance of the system are greatly improved.

[0039] In the reading and writing of shared memory, the writing process writes from the end of the queue, and the reading process reads from the head of the queue. If the end of the queue is read, it means that the writing operation is in progress, and it is necessary to wait for the writing process to complete before reading the data block at the corresponding position.

[0040] Graph computing based on shared memory uses multiple threads to process data concurrently. Since these threads can directly access the same shared memory, the communication and synchronization operations between them are more efficient. Specifically, when processing data in parallel, each thread is assigned a certain number of data blocks for calculation, and these data blocks are usually stored in the shared memory in a continuous manner. There is no need for data copying or network transmission between threads. They only need to coordinate each other's computing operations through variables or locks in the shared memory, thus avoiding the overhead of context switching.

[0041] The embodiment of the present invention combines shared memory with graph computing, and performs data interaction and calculation through shared memory between different processes on the same computer. It has the advantages of low latency and high bandwidth, and is suitable for scenarios with small data volumes and requiring frequent communication, high concurrency, and low latency.

[0042] Embodiment 2:

[0043] The present invention also provides a terminal device for implementing graph computing based on shared memory, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned method embodiment of embodiment 1 of the present invention when executing the computer program.

[0044] Further, as an executable solution, the shared memory-based graph computing implementation terminal device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The shared memory-based graph computing implementation terminal device may include, but is not limited to, a processor and a memory. Those skilled in the art may understand that the composition structure of the shared memory-based graph computing implementation terminal device is merely an example of a shared memory-based graph computing implementation terminal device, and does not constitute a limitation on the shared memory-based graph computing implementation terminal device. It may include more or less components than the above, or a combination of certain components, or different components. For example, the shared memory-based graph computing implementation terminal device may also include input and output devices, network access devices, buses, etc., and the embodiments of the present invention do not limit this.

[0045] Further, as an executable solution, the processor may be a central processing unit (CPU) or other general-purpose processors. The general-purpose processor may be a microprocessor or any conventional processor, and the processor is the control center of the shared memory-based graph computing implementation terminal device, and uses various interfaces and lines to connect various parts of the entire shared memory-based graph computing implementation terminal device.

[0046] The memory can be used to store the computer program and / or module, and the processor implements the various functions of the terminal device based on the shared memory graph calculation by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the mobile phone, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0047] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method in the embodiment of the present invention are implemented.

[0048] If the module / unit for implementing terminal device integration based on shared memory graph computing is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc.

[0049] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. A method for implementing graph computing based on shared memory, characterized in that: The following steps are involved: S1: Divide the graph structure data into multiple shards, wherein the dividing into multiple shards includes: 1), randomly select multiple nodes as the initial center points; 2) For each non-central point among all nodes, assign it to the cluster where the central point closest to it is located; 3) For each cluster, calculate the sum of the distances from all nodes in the cluster to the center of the cluster, and select the node with the smallest sum of distances as the new center of the cluster; 4), repeating steps 2) and 3) until a predetermined stop condition is met, wherein the stop condition includes: reaching a maximum number of iterations or the cluster no longer changes; S2: Sort all vertices in each shard and treat the data corresponding to each vertex as a data block; S3: Duplicate each vertex multiple times to obtain a main vertex and multiple secondary vertices; obtain a main queue and multiple secondary queues according to the arrangement order of each vertex in the shard, the main queue is composed of the main vertex arrangement corresponding to all vertices contained in the shard, and the secondary queue is composed of one secondary vertex arrangement corresponding to all vertices contained in the shard; store both the main queue and the secondary queue in the shared memory; S4: In the multi-threaded concurrent processing process, the head vertex in the main queue or the sub-queue is read sequentially for data processing. The data processing process adopts the GAS calculation model. When it reaches the Gather stage, the main vertex and the sub-vertex are calculated separately; when it reaches the Apply stage, the sub-vertex sends the calculation results of the Gather stage to the main vertex, which is summarized and further calculated by the main vertex. In the Scatter stage, the data of the main vertex is updated based on the further calculation results of the main vertex and synchronized to the sub-vertex; after the data in the vertex is updated, the edge data is updated; The shared memory is set to be able to be read by multiple processes, but only one process can write to the shared memory.

2. The method for implementing graph computing based on shared memory according to claim 1, characterized in that: The sorting in step S2 is performed using the PSW algorithm.

3. The method for implementing graph computing based on shared memory according to claim 1, characterized in that: In the reading and writing of shared memory, the writing process writes from the end of the queue, and the reading process reads from the head of the queue. If the end of the queue is read, it waits for the writing process to complete before reading the data block at the corresponding position.

4. A terminal device for implementing graph computing based on shared memory, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 3 when executing the computer program.

5. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images