A single-machine-based large-scale graph data processing system

Abstract

The application provides a single-machine-based large-scale graph data processing system, comprising a data loading module, a data computing module, a data releasing module, a storage management module and a disk; the data loading module is used for obtaining a subgraph with an active state from the disk and transmitting the subgraph to the data computing module; the data computing module is used for updating the subgraph and transmitting a message generated by the update to the storage management module; the data computing module is also used for transmitting the subgraph to the data releasing module; the data releasing module is used for writing the subgraph into the disk; when the subgraph is written into the disk, the storage management module is used for setting the state of the subgraph as convergent. By applying a subgraph center-based computing model to a single-machine system and establishing a unique pipeline processing architecture, the architecture can overlap data I / O and CPU operation, thereby reducing the I / O cost of a traditional vertex center computing model, improving CPU utilization, and promoting sequential access to the disk.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a large-scale graph data processing system based on a single machine. Background Technology

[0002] In recent years, graph data has become a highly valued topic in data science and engineering due to its ease of abstracting real-world entities and relationships. It has been widely applied in various fields such as social network analysis, recommender systems, financial fraud detection, and drug discovery. Furthermore, graph data offers high flexibility; many problems that were originally modeled using matrices, relations, or other data structures can be transformed into graph data processing, further highlighting its importance. With the increasing prevalence of social media and mobile internet applications, the scale of abstract graph data generated or collected by computer systems is growing rapidly. This magnitude of growth poses extremely acute challenges to the large-scale data storage, analysis, and mining capabilities of modern computer systems.

[0003] Traditional large-scale graph computing systems use parallelization methods based on data partitioning, that is, integrating the resources of multiple computers to complete graph computation tasks. Although these computing systems play an important role in the field of large graph processing, due to the high maintenance and construction costs, only a few companies with large-scale computer clusters can perform large-scale graph computing. In addition, distributed computing systems are usually based on the assumption that using more computing nodes will reduce computation time, but in reality, this assumption does not always hold true. Adding computing nodes may lead to greater communication costs, thus failing to significantly improve system performance.

[0004] To address the practical needs of large-scale graph analysis and resource-constrained use cases, a series of large-scale graph processing systems based on single machines have been proposed. These systems utilize external storage as a memory extension to process large graphs and employ a vertex-centric computational model (which confines information transfer during computation to nodes) to improve data locality and simplify the user experience. Although the vertex-centric computational model is simple and easy to understand, it also suffers from high communication or I / O overhead. Summary of the Invention

[0005] In view of the above problems, this application is made to provide a large-scale graph data processing system based on a single machine that overcomes or at least partially solves the problems, comprising:

[0006] A large-scale graph data processing system based on a single machine includes a data loading module, a data calculation module, a data release module, a storage management module, and a disk; the disk stores large-scale graph data composed of several subgraphs; the storage management module stores state information corresponding to each subgraph; initially, the subgraph is active.

[0007] The data loading module is used to obtain the subgraph that is active from the disk and transmit the subgraph to the data calculation module;

[0008] The data calculation module is used to update the subgraph and transmit the update message to the storage management module;

[0009] When the updated subgraph changes, the data calculation module is also used to transmit the subgraph to the data release module;

[0010] When the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph to the disk;

[0011] When the subgraph is written to the disk, the storage management module sets the state of the subgraph to convergence.

[0012] Preferably, when the updated subgraph remains unchanged, the data calculation module is further configured to write the subgraph to the disk.

[0013] Preferably, when the subgraph is the last one in the current round of updates, the data release module is also used to transfer the subgraph to the data loading module.

[0014] Preferably, when the current round of updates ends, the storage management module is also used to set the state of the subgraph that received the message to active.

[0015] Preferably, when the current round of updates ends and there is no message cache in the storage management module, the data calculation module is further used to aggregate all the subgraphs to obtain updated large-scale graph data.

[0016] Preferably, the storage management module includes a message storage unit and a status management unit; the status management unit stores the status information.

[0017] The data calculation module is used to transmit the updated message to the message storage unit;

[0018] When the current round of updates ends, the state management unit sets the state of the subgraph that received the message to active.

[0019] Preferably, the data calculation module includes an aggregation calculation unit;

[0020] When the current round of updates ends and there is no message cache in the message storage unit, the aggregation calculation unit is used to aggregate all the subgraphs to obtain updated large-scale graph data.

[0021] Preferably, when the subgraph is acquired by the data loading module, the storage management module is further configured to set the state of the subgraph to await computation.

[0022] Preferably, when the subgraph is transmitted to the data calculation module, the storage management module is further configured to set the state of the subgraph to be being calculated.

[0023] Preferably, when the subgraph is transmitted to the data release module, the storage management module is further configured to set the state of the subgraph to be released.

[0024] This application has the following advantages:

[0025] In the embodiments of this application, in response to the high communication or I / O overhead of existing large-scale graph processing systems based on single machines, this application provides a solution that applies a subgraph-centric computational model to a single-machine system and establishes a pipelined processing architecture. Specifically, it provides: "A large-scale graph data processing system based on a single machine, comprising a data loading module, a data computation module, a data release module, a storage management module, and a disk; the disk stores large-scale graph data composed of several subgraphs; the storage management module stores state information corresponding to each subgraph; initially, the subgraphs are active." The data loading module is used to retrieve the active subgraph from the disk and transmit the subgraph to the data calculation module; the data calculation module is used to update the subgraph and transmit the update message to the storage management module; when the updated subgraph changes, the data calculation module is also used to transmit the subgraph to the data release module; when the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph to the disk; when the subgraph is written to the disk, the storage management module is used to set the state of the subgraph to convergence. By applying the subgraph-centric computing model to a single-machine system and establishing a unique pipelined processing architecture, this architecture can overlap data I / O and CPU operations, thereby reducing the I / O cost of the traditional vertex-centric computing model while improving CPU utilization and promoting sequential access to the disk. In addition, this architecture uses a shared memory data structure for message passing and efficient synchronization, which can separate computation from memory management and scheduling, thus providing new opportunities for optimization. Attached Figure Description

[0026] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram illustrating the calculation process of connected components on the vertex center model and the subgraph center model;

[0028] Figure 2 This is a schematic diagram of the processing architecture of a large-scale graph data processing system provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of the state management and optimization strategy of a large-scale graph data processing system provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, features, and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0031] The inventors, through analysis of existing technologies, discovered that vertex-centric computational models undoubtedly incur additional communication or I / O overhead due to message passing between vertices, such as... Figure 1 As shown, the computation of connected components on the input graph G using a computation model based on subgraph centers (which allows information to be freely transferred within the subgraph during the computation process) has significantly fewer computational steps than using a computation model based on vertex centers.

[0032] However, traditional subgraph-centric computational models are designed for multi-machine systems, and no work has yet introduced them into single-machine environments. Therefore, some questions remain unanswered. For example, when communication costs are potentially converted into I / O costs, can introducing a subgraph-centric computational model systematically reduce the I / O cost of off-core graph systems and improve multi-core parallelism? Traditional subgraph-centric computational models require finer-grained graph partitioning to improve parallelism, but this comes at the cost of more redundant control information, such as the mapping from global vertex IDs to local vertex IDs. This problem can be solved in a distributed environment by allocating sufficient memory to each compute node, but in a single-machine multi-core environment, finer-grained graph partitioning consumes already precious memory resources.

[0033] The inventors believe that extending the subgraph-centric computing model to a single-machine system will face the following improvement requirements: When the input graph exceeds the memory capacity, the single-machine system needs to use auxiliary storage (such as hard disks, SSDs, etc.) as memory expansion for computation, thus requiring reasonable management of graph scheduling between memory and disk; Traditional subgraph-centric computing models use computer networks to pass messages between computing units for synchronization, but in the case of shared memory, the synchronization logic of the single-machine system changes, thus requiring more efficient message synchronization; Subgraph-centric computing models only utilize data partition parallelism, which may lead to insufficient CPU core utilization or excessive graph fragmentation when memory capacity is limited, thus requiring a balance between parallel computing between subgraphs and parallel computing within subgraphs; Due to the shared memory architecture of the single-machine system, the cost of migrating work between cores of a single machine is very low, thus requiring flexible resource scheduling to improve system performance.

[0034] This embodiment provides a large-scale graph data processing system based on a single machine, including a data loading module, a data calculation module, a data release module, a storage management module, and a disk; the disk stores large-scale graph data composed of several subgraphs; the storage management module stores state information corresponding to each subgraph; initially, the subgraph is active.

[0035] The data loading module is used to obtain the subgraph that is active from the disk and transmit the subgraph to the data calculation module;

[0036] The data calculation module is used to update the subgraph and transmit the update message to the storage management module;

[0037] When the updated subgraph changes, the data calculation module is also used to transmit the subgraph to the data release module;

[0038] When the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph to the disk;

[0039] When the subgraph is written to the disk, the storage management module sets the state of the subgraph to convergence.

[0040] In the embodiments of this application, in contrast to the high communication or I / O overhead of existing large-scale graph processing systems based on single machines, this application applies a subgraph center-based computational model to a single-machine system and establishes a pipelined processing architecture. (Refer to...) Figure 2Given a large graph G (which is initially stored on disk), the pipelined processing architecture uses subgraphs {F0, F1, F2, F3, ..., Fn-1} of graph G as the smallest input / output unit and iteratively updates the large graph G in a pipelined manner. Specifically, the architecture decomposes the out-of-core processing of subgraph Fi into three consecutive stages: reading Fi into memory, computing and updating Fi, and writing the updated Fi back to external storage if necessary. These stages are accomplished by three modules: a data loading module, a data computing module, and a data release module. These modules work asynchronously in the pipelined processing architecture through two task queues: an "input queue" and an "output queue".

[0041] The pipelined processing architecture effectively overlaps subgraph I / O and CPU operations, performing computations on in-memory subgraphs while loading suspended subgraphs from disk. This reduces the I / O cost of traditional vertex-centric computation models, improves CPU utilization by reducing idle wait times, and enables continuous disk access. Furthermore, the architecture employs shared-memory data structures for message passing and efficient synchronization, separating computation from memory management and scheduling, thus providing new opportunities for optimization.

[0042] The following will further describe a large-scale graph data processing system based on a single machine in this exemplary embodiment.

[0043] In this embodiment, the system employs APIs based on a hybrid computing model. These APIs use a unified PIE+ interface, which integrates vertex-centric and subgraph-centric programming models. Users can not only parallelize sequential graph algorithms under the subgraph-centric computing model to simplify parallel programming (inter-subgraph parallelism), but also further explore the intra-subgraph parallelism of the vertex-centric computing model through the new interface. It should be noted that the hybrid model simultaneously supports inter-subgraph parallelism of the "subgraph-centric computing model" and intra-subgraph parallelism of the "vertex-centric computing model." With limited memory, it can better utilize multi-core resources and avoid fragmentation of the input graph. Furthermore, it provides a unified interface from which users can select the interface most suitable for their application and graph.

[0044] In this embodiment, the system further includes a scheduler. The scheduler tracks and allocates threads in a thread pool, where each thread corresponds to a physical CPU core. It decides whether to allocate physical threads to virtual worker threads for (parallel) computation on a subgraph. It also makes proactive adjustments to support two levels of parallelism: when a thread becomes available, the scheduler allocates it to a new computational unit by consuming an "input queue" to accelerate parallelism between subgraphs, or improves parallelism within a subgraph by utilizing running worker cores.

[0045] In this embodiment, the storage management module includes a message storage unit; the data computing module is used to transmit updated messages to the message storage unit. The message storage unit is used to realize message synchronization between parallel computing units. Specifically, the message storage unit is implemented as an in-memory data structure; to improve space efficiency, it can be implemented as a compact variable-length array. It should be noted that the space complexity of the message storage unit is closely related to the partitioning strategy; the more boundary vertices / edges there are, the more space the message storage unit consumes. Compared with the message passing strategy of multi-machine systems, using a message storage unit is more efficient in a shared memory environment.

[0046] In this embodiment, the storage management module further includes a state management unit; the state management unit stores the state information and can update the state information at specific times. The state management unit is used to maintain a state machine to model the state of the subgraph. Specifically, the state management unit is implemented as a lightweight data structure, maintaining only a few states for each subgraph, and the memory space it occupies is negligible.

[0047] Using the state management unit to record the state information of subgraphs is a low-cost convergence detection method. It only requires a tag list M to help track message exchanges between computing units and a lightweight state machine to model the work progress of each computing unit. Specifically, the state management unit constructs a tag list M, with each subgraph corresponding to a tag to indicate whether it has received any messages in the current iteration. If a subgraph has at least one pending update to be retrieved from the message storage unit, its corresponding M[i] is true; otherwise, M[i] is false. In practice, a finite state machine can be used to model the progress of each subgraph, and the flag M[i] can be used to trigger the state transition of the subgraph.

[0048] like Figure 3As shown, the subgraph has five states: "active", "waiting to be computed", "compiling", "releasing", and "converging". At any given time, the subgraph is in one of these five states. The first two states indicate that the subgraph is on the disk, and the remaining states indicate that the subgraph is in memory. Each subgraph is initially in an "active" state, meaning it is waiting for the data loading module to load it into memory. When the subgraph is acquired by the data loading module, the state management unit sets its state to "waiting to compute," meaning it is already residing in memory and awaiting allocation to a processing core. When the subgraph is transmitted to the data computation module, the state management unit sets its state to "compute," meaning it is being processed by a processing core. When the subgraph is transmitted to the data release module (i.e., the subgraph generates a message that needs to be sent to other subgraphs during the current update round), the state management unit sets its state to "releasing." When the subgraph is written to the disk, the state management unit sets its state to "converging." When the current update round ends, the state management unit also sets the state of subgraphs participating in the next update round to active, so that these subgraphs can begin the next update round. The entire system stops updating only when the current update round ends and there is no message cache in the message storage unit.

[0049] Under certain conditions, the system can skip certain states in a round of calculation without affecting correctness. In other words, it can take some "shortcuts" in state transitions and reduce unnecessary calculations and I / O.

[0050] like Figure 3 As shown, in this embodiment, when the current round of updates ends, the storage management module sets the state of the subgraph that received the message to "active" ("Shortcut A"). To begin a new round of incremental computation, the state of the subgraph that is "converged" needs to be reset to "active". If M[i] corresponding to a subgraph is true at this time, the subgraph can remain in the "converged" state, preventing it from participating in the next round of updates. This allows for the complete skipping of processing of subgraphs that do not require updating without affecting the correctness of the program. "Shortcut A" is often used when the input graph is not well connected and a certain subgraph is "isolated," effectively reducing I / O costs.

[0051] In this embodiment, when the subgraph is the last one in the current round of updates, the data release module is used to transfer the subgraph to the data loading module ("Shortcut B"). A new round of updates will begin after all subgraphs have completed the current round of updates. If a subgraph is still in the "releasing" state, meaning it hasn't been fully saved to disk, its state can be directly set to "awaiting computation," thus allowing a new round of updates to the subgraph to be started without going through the disk. "Shortcut B" can be utilized at the end of each round, effectively reducing I / O costs.

[0052] In this embodiment, when the updated subgraph remains unchanged, the data calculation module writes the subgraph to the disk ("Shortcut C"). When a subgraph is updated, if it has not changed compared to before the calculation, the "Release in Progress" state can be skipped, and it can be directly set to "Convergence", thereby effectively reducing redundant disk writes.

[0053] In this embodiment, the data calculation module includes an aggregation calculation unit. When the current round of updates ends and there is no message cache in the message storage unit, the aggregation calculation unit is used to call a preset aggregation function to aggregate all the subgraphs to obtain updated large-scale graph data.

[0054] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0055] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0056] The above provides a detailed description of a large-scale graph data processing system based on a single machine provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A large-scale graph data processing system based on a single machine, characterized in that, It includes a data loading module, a data calculation module, a data release module, a storage management module, and a disk; the disk stores large-scale graph data composed of several subgraphs; the storage management module stores state information corresponding to each subgraph; initially, the subgraph is active. The data loading module is used to obtain the subgraph that is active from the disk and transmit the subgraph to the data calculation module; The data calculation module is used to update the subgraph and transmit the update message to the storage management module; When the updated subgraph changes, the data calculation module is also used to transmit the subgraph to the data release module; When the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph to the disk; When the subgraph is written to the disk, the storage management module sets the state of the subgraph to convergence.

2. The system according to claim 1, characterized in that, When the updated subgraph remains unchanged, the data calculation module is also used to write the subgraph to the disk.

3. The system according to claim 1, characterized in that, When the subgraph is the last one in the current round of updates, the data release module is also used to transfer the subgraph to the data loading module.

4. The system according to claim 1, characterized in that, When the current round of updates ends, the storage management module is also used to set the state of the subgraph that received the message to active.

5. The system according to claim 1, characterized in that, When the current round of updates ends and there is no message cache in the storage management module, the data calculation module is also used to aggregate all the subgraphs to obtain updated large-scale graph data.

6. The system according to claim 1, characterized in that, The storage management module includes a message storage unit and a status management unit; the status management unit stores the status information. The data calculation module is used to transmit the updated message to the message storage unit; When the current round of updates ends, the state management unit sets the state of the subgraph that received the message to active.

7. The system according to claim 6, characterized in that, The data calculation module includes an aggregation calculation unit; When the current round of updates ends and there is no message cache in the message storage unit, the aggregation calculation unit is used to aggregate all the subgraphs to obtain updated large-scale graph data.

8. The system according to claim 1, characterized in that, When the subgraph is acquired by the data loading module, the storage management module is also used to set the state of the subgraph to wait for calculation.

9. The system according to claim 1, characterized in that, When the subgraph is transmitted to the data calculation module, the storage management module is also used to set the state of the subgraph to be being calculated.

10. The system according to claim 1, characterized in that, When the subgraph is transmitted to the data release module, the storage management module is also used to set the state of the subgraph to be released.

Images