Single unit-based large-scale graph data processing system

Abstract

The present application provides a single unit-based large-scale graph data processing system, comprising a data loading module, a data computing module, a data release module, a storage management module and a disk; the data loading module is used for acquiring sub-graphs in an active state from the disk, and transmitting the sub-graphs to the data computing module; the data computing module is used for updating the sub-graphs, and transmitting messages generated by the updating to the storage management module; the data computing module is further used for transmitting the sub-graphs to the data release module; the data release module is used for writing the sub-graphs into the disk; when the sub-graphs are written into the disk, the storage management module is used for setting the states of the sub-graphs to be convergent. Applying a sub-graph center-based computing model to a single unit system and establishing a set of unique pipeline processing architecture capable of overlapping data I / O and CPU operations reduce I / O costs of traditional vertex center computing models while increasing CPU utilization rate, and promote sequential access to the disk.

Description

A large-scale graph data processing system based on a single machine Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a large-scale graph data processing system based on a single machine. Background Art

[0002] In recent years, graph data has become a highly sought-after topic in data science and engineering due to its ease of abstracting real-world entities and relationships. It has been widely applied in fields such as social network analysis, recommender systems, financial fraud detection, and drug discovery. Furthermore, graph data offers a high degree of flexibility, allowing many problems previously modeled using matrices, relations, or other data structures to be converted to graph data processing, further highlighting its importance. With the proliferation of social media and mobile internet applications, the scale of abstract graph data generated or collected by computer systems is rapidly increasing. This massive growth poses significant challenges to the large-scale data storage, analysis, and mining capabilities of modern computer systems.

[0003] Traditional large-scale graph computing systems use a data partitioning parallelization approach, integrating the resources of multiple computers to complete graph computing tasks. Although these computing systems play an important role in large-scale graph processing, the high maintenance and construction costs restrict large-scale graph computing to only a few companies with large-scale computer clusters. Furthermore, distributed computing systems are often based on the assumption that using more computing nodes will reduce computation time. In reality, this assumption does not always hold true. Adding computing nodes may result in increased communication costs, thus failing to significantly improve system performance.

[0004] To address the practical needs of large-scale graph analysis and resource-constrained scenarios, a series of single-machine, large-scale graph processing systems have been proposed. These systems utilize external memory as an extension of memory to process large graphs and employ a vertex-centric computing model (which confines information transfer between nodes during the computation process) to improve data locality and simplify user experience. Although vertex-centric computing models are simple and easy to understand, they also suffer from high communication and I / O overhead.

[0005] Summary of the Invention

[0006] In view of the above problems, the present application is proposed to provide a large-scale graph data processing system based on a single machine to overcome the problems or at least partially solve the problems, including:

[0007] A large-scale graph data processing system based on a single machine includes a data loading module, a data calculation module, a data release module, a storage management module, and a disk; the disk stores large-scale graph data consisting of a plurality of subgraphs; the storage management module stores status information corresponding to each subgraph; in an initial state, the subgraph is active;

[0008] The data loading module is used to obtain the subgraph in the active state from the disk and transmit the subgraph to the data calculation module;

[0009] The data calculation module is used to update the subgraph and transmit the updated message to the storage management module;

[0010] When there is a change in the updated subgraph, the data calculation module is further configured to transmit the subgraph to the data release module;

[0011] When the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph into the disk;

[0012] When the subgraph is written to the disk, the storage management module is configured to set the state of the subgraph to converged.

[0013] Preferably, when there is no change in the updated subgraph, the data calculation module is further configured to write the subgraph into the disk.

[0014] Preferably, when the subgraph is the last one in the current round of updates, the data releasing module is further configured to transmit the subgraph to the data loading module.

[0015] Preferably, when the current round of updating is finished, the storage management module is further configured to set the state of the subgraph that has received the message to active.

[0016] Preferably, when the current round of updates is completed and there is no message cache in the storage management module, the data calculation module is further used to aggregate all the subgraphs to obtain updated large-scale graph data.

[0017] Preferably, the storage management module includes a message storage unit and a state management unit; the state management unit stores the state information;

[0018] The data calculation module is used to transmit the updated message to the message storage unit;

[0019] When the current round of updating is finished, the state management unit is used to set the state of the subgraph that has received the message to active.

[0020] Preferably, the data calculation module includes an aggregation calculation unit;

[0021] When the current round of update is completed and there is no message cache in the message storage unit, the aggregation calculation unit is used to aggregate all the subgraphs to obtain updated large-scale graph data.

[0022] Preferably, when the subgraph is acquired by the data loading module, the storage management module is further configured to set the state of the subgraph to waiting for calculation.

[0023] Preferably, when the subgraph is transmitted to the data calculation module, the storage management module is further configured to set the state of the subgraph to being calculated.

[0024] Preferably, when the sub-graph is transmitted to the data release module, the storage management module is further configured to set the state of the sub-graph to being released.

[0025] This application has the following advantages:

[0026] In the embodiments of the present application, relative to the problem of high communication overhead or I / O overhead of existing large-scale graph processing systems based on a single machine, the present application provides a solution for applying a subgraph-centric computing model to a single machine system and establishing a pipeline processing architecture, specifically: "A large-scale graph data processing system based on a single machine, comprising a data loading module, a data computing module, a data releasing module, a storage management module and a disk; the disk stores large-scale graph data consisting of a plurality of subgraphs; the storage management module stores status information corresponding to each of the subgraphs; in an initial state, the subgraph is active; The data loading module is used to obtain the active subgraph from the disk and transmit the subgraph to the data computing module; the data computing module is used to update the subgraph and transmit the update message to the storage management module; when the updated subgraph has changed, the data computing module is also used to transmit the subgraph to the data release module; when the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph to the disk; when the subgraph is written to the disk, the storage management module is used to set the state of the subgraph to convergence. By applying the subgraph-centric computing model to a single-machine system and establishing a unique pipeline processing architecture, this architecture can overlap data I / O and CPU operations, thereby reducing the I / O cost of the traditional vertex-centric computing model while improving CPU utilization and promoting sequential access to the disk. In addition, this architecture uses shared memory data structures for message passing and efficient synchronization, which can separate computing from memory management and scheduling, thereby providing new opportunities for optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0028] Figure 1 is a schematic diagram of the connected component calculation process on the vertex center model and the subgraph center model;

[0029] FIG2 is a schematic diagram of a processing architecture of a large-scale graph data processing system provided by an embodiment of the present application;

[0030] FIG3 is a schematic diagram of a state management and optimization strategy for a large-scale graph data processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.

[0032] By analyzing the existing technology, the inventors found that the vertex-centric computing model will undoubtedly bring additional communication overhead or I / O overhead due to the transmission of messages between vertices. As shown in Figure 1, when completing the calculation of connected components on the input graph G, the subgraph-centric computing model (which allows information in the calculation process to be freely transmitted within the subgraph) has significantly fewer computing steps than the vertex-centric computing model.

[0033] However, traditional subgraph-centric computing models were designed for multi-machine systems, and no work has yet introduced subgraph-centric computing models into a single-machine environment. Therefore, some questions remain unanswered. For example, when communication costs potentially translate into I / O costs, can introducing a subgraph-centric computing model systematically reduce the I / O cost of out-of-core graph systems and improve multi-core parallelism? Traditional subgraph-centric computing models require finer-grained partitioning of the graph to improve parallelism, but this comes at the cost of more redundant control information, such as the mapping of global vertex IDs to local vertex IDs. This problem can be solved in a distributed environment by allocating sufficient memory to each computing node, but in a single-machine multi-core environment, finer-grained graph partitioning consumes already precious memory resources.

[0034] The inventors believe that extending the subgraph-centric computing model to a stand-alone system will face the following improvement requirements: when the input graph exceeds the memory capacity, the stand-alone system needs to use auxiliary storage (such as hard disk, SSD, etc.) as memory extension for calculation, so it is necessary to reasonably manage the scheduling of the graph between memory and disk; the traditional subgraph-centric computing model transmits messages between computing units through a computer network for synchronization, but in the case of shared memory, the synchronization logic of the stand-alone system has changed, so it is necessary to implement message synchronization more efficiently; the subgraph-centric computing model only utilizes data partition parallelism, which may lead to insufficient CPU core utilization or excessive graph fragmentation when memory capacity is limited. Therefore, it is necessary to balance the parallel computing between subgraphs and the parallel computing within the subgraph; due to the shared memory architecture of the stand-alone system, the cost of work migration between the cores of a single machine is very low, so flexible resource scheduling is needed to improve system performance.

[0035] In this embodiment, a large-scale graph data processing system based on a single machine is provided, comprising a data loading module, a data calculation module, a data release module, a storage management module, and a disk; the disk stores large-scale graph data consisting of a plurality of subgraphs; the storage management module stores status information corresponding to each of the subgraphs; in an initial state, the subgraph is active;

[0036] The data loading module is used to obtain the subgraph in the active state from the disk and transmit the subgraph to the data calculation module;

[0037] The data calculation module is used to update the subgraph and transmit the updated message to the storage management module;

[0038] When there is a change in the updated subgraph, the data calculation module is further configured to transmit the subgraph to the data release module;

[0039] When the subgraph is not the last one in the current round of updates, the data release module is used to write the subgraph into the disk;

[0040] When the subgraph is written to the disk, the storage management module is configured to set the state of the subgraph to converged.

[0041] In an embodiment of the present application, relative to the problem of high communication overhead or I / O overhead of existing large-scale graph processing systems based on a single machine, the present application applies a subgraph-centric computing model to a single-machine system and establishes a pipeline processing architecture. Referring to Figure 2, given a large graph G (large graph G is initially stored on disk), the pipeline processing architecture uses the subgraph {F0, F1, F2, F3, ..., Fn-1} of graph G as the minimum input and output unit, and updates the large graph G iteratively using a pipeline. Specifically, the architecture decomposes the out-of-core processing of the subgraph Fi into three consecutive stages: reading Fi into memory, calculating and updating Fi, and, if necessary, writing the updated Fi back to external memory. These stages are completed through three modules, namely a data loading module, a data calculation module, and a data release module. These modules work asynchronously in the pipeline processing architecture through two task queues, "input queue" and "output queue".

[0042] The pipeline processing architecture effectively overlaps subgraph I / O and CPU operations, performing computations on in-memory subgraphs while loading suspended subgraphs from disk, which can reduce the I / O cost of traditional vertex-centric computing models while improving CPU utilization by reducing idle waits and enabling continuous access to disk. In addition, the architecture uses shared memory data structures for message passing and efficient synchronization, which can separate computation from memory management and scheduling, thereby providing new opportunities for optimization.

[0043] Next, a large-scale graph data processing system based on a single machine in this exemplary embodiment will be further described.

[0044] In this embodiment, the system adopts APIs based on a hybrid computing model. The APIs adopt a unified PIE+ interface, which integrates the vertex-centric and subgraph-centric programming models. Users can not only parallelize sequential graph algorithms under the subgraph-centric computing model to simplify parallel programming (inter-subgraph parallelism), but also further explore the parallelism within the subgraph based on the vertex-centric computing model through the new interface. It should be noted that the hybrid model supports both the inter-subgraph parallelism of the "subgraph-centric computing model" and the intra-subgraph parallelism of the "vertex-centric computing model". Under limited memory, it can better utilize multi-core resources and avoid fragmentation of the input graph. In addition, it provides a unified interface from which users can choose the interface that best suits their applications and graphics.

[0045] In this embodiment, the system further includes a scheduler. The scheduler is used to track and allocate threads in a thread pool, where each thread corresponds to a physical CPU core. The scheduler determines the allocation of physical threads to virtual worker threads to perform (parallel) computations on subgraphs. The scheduler also actively adjusts to support two levels of parallelism: when threads become available, the scheduler allocates them to new computational units by consuming the "input queue" to accelerate inter-subgraph parallelism, or improves intra-subgraph parallelism by using running worker cores.

[0046] In this embodiment, the storage management module includes a message storage unit; the data calculation module is used to transmit the updated messages to the message storage unit. The message storage unit is used to realize message synchronization between parallel computing units. Specifically, the message storage unit is implemented as a data structure in memory. In order to improve space efficiency, it can be implemented as a compact variable-length array. It should be noted that the space complexity of the message storage unit is closely related to the partitioning strategy. The more boundary vertices / edges there are, the more space the message storage unit consumes. Compared with the message passing strategy of a multi-machine system, the use of a message storage unit in a shared memory environment is more efficient.

[0047] In this embodiment, the storage management module also includes a state management unit; the state management unit stores the state information and can update the state information at specific times. The state management unit is used to maintain a state machine to model the state of the subgraph. Specifically, the state management unit is implemented as a lightweight data structure, maintaining only a few states for each subgraph, and the memory space occupied is negligible.

[0048] Using the state management unit to record the state information of subgraphs is a low-cost convergence detection method. It only requires a marker list M to help track the message exchange between computing units and a lightweight state machine to model the work progress of each computing unit. Specifically, the state management unit constructs a marker list M, with each subgraph corresponding to a marker to indicate whether it has received any messages in this iteration. If a subgraph has at least one pending update to be extracted from the message storage unit, its corresponding M[i] is true; otherwise, M[i] is false. In actual operation, a finite state machine can be used to model the progress of each subgraph, and the flag M[i] can be used to trigger the state transition of the subgraph.

[0049] As shown in Figure 3, the states of the subgraph include "active", "waiting for calculation", "calculating", "releasing" and "converging". At any time, the subgraph is in one of the five states, among which the first two states indicate that the subgraph is on the disk, and the remaining states indicate that the subgraph is in memory. The initial state of each subgraph is "active", which means that the subgraph is waiting to be loaded into the memory by the data loading module. When the subgraph is obtained by the data loading module, the state management unit is used to set the state of the subgraph to "waiting for calculation", which means that the subgraph is already resident in the memory and is waiting to be assigned a processing core. When the subgraph is transferred to the data calculation module, the state management unit is used to set the state of the subgraph to "calculating", which means that the subgraph is being calculated by the processing core. When the subgraph is transferred to the data release module (that is, the subgraph generates a message that needs to be sent to other subgraphs in the current round of update), the state management unit is used to set the state of the subgraph to "releasing". When the subgraph is written to the disk, the state management unit is used to set the state of the subgraph to "converging". When the current round of update ends, the state management unit is also used to set the state of the subgraph participating in the next round of update to active, so that these subgraphs can start the next round of update. If and only if the current round of update ends and there is no message cache in the message storage unit, the entire system stops updating.

[0050] Under certain conditions, the system can skip certain states in a round of computation without affecting correctness, that is, it can take some “shortcuts” in state transitions and reduce unnecessary computation and I / O.

[0051] As shown in FIG3 , in this embodiment, when the current round of updates is completed, the storage management module is used to set the state of the subgraph that received the message to “active” (“shortcut A”). In order to start a new round of incremental calculations, the state of the subgraph that is in the “convergence” state needs to be reset to “active”. If the M[i] corresponding to a subgraph is true at this time, the subgraph can be kept in the “convergence” state so that it does not participate in the next round of updates. In this way, the processing of subgraphs that do not need to be updated can be completely skipped without affecting the correctness of the program. “Shortcut A” is often used when the input graph is not well connected and a subgraph is “isolated”, and effectively reduces the I / O cost.

[0052] In this embodiment, when the subgraph is the last in the current update round, the data release module is used to transfer the subgraph to the data loading module ("shortcut B"). After all subgraphs have completed the current update round, a new update round begins. If a subgraph is still in the "releasing" state, meaning it has not yet been fully saved to disk, its status can be directly set to "waiting for calculation", thereby initiating a new update round for that subgraph without going through the disk. "Shortcut B" can be used at the end of each update round, effectively reducing I / O costs.

[0053] In this embodiment, when the updated subgraph does not change, the data calculation module is configured to write the subgraph to the disk ("shortcut C"). When a subgraph update is completed, if it has not changed since before the calculation, the "releasing" state can be skipped and the subgraph can be directly set to "converging", thereby effectively reducing redundant disk writes.

[0054] In this embodiment, the data calculation module includes an aggregation calculation unit. When the current round of update is completed and there is no message cache in the message storage unit, the aggregation calculation unit is used to call a preset aggregation function to aggregate all the subgraphs to obtain updated large-scale graph data.

[0055] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0056] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0057] The above is a detailed introduction to a large-scale graph data processing system based on a single machine provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A large-scale graph data processing system based on a single machine, characterized in that: It includes a data loading module, a data calculation module, a data release module, a storage management module and a disk; the disk stores large-scale graph data consisting of a number of subgraphs; the storage management module stores state information corresponding to each of the subgraphs; in the initial state, the state of the subgraph is active; The data loading module is used for acquiring the sub-graph in the active state from the disk, and transmitting the sub-graph to the data calculation module; The data calculation module is used to update the subgraph and transmit the message generated by the update to the storage management module; When there is a change in the updated sub-graph, the data calculation module is further used to transmit the sub-graph to the data release module; When the sub-graph is not the last one in the current round of update, the data release module is used to write the sub-graph into the disk; When the subgraph is written to the disk, the storage management module is used to set the state of the subgraph to converged.

2. The system according to claim 1, characterized in that When there is no change in the updated sub-graph, the data calculation module is further used to write the sub-graph into the disk.

3. The system according to claim 1, characterized in that When the sub-graph is the last one in the current round of updating, the data releasing module is further used to transmit the sub-graph to the data loading module.

4. The system according to claim 1, characterized in that When the current round of updating is finished, the storage management module is further used to set the state of the subgraph that has received the message to active.

5. The system according to claim 1, characterized in that When the current round of updates is completed and there is no message cache in the storage management module, the data calculation module is also used to aggregate all the sub-graphs to obtain updated large-scale graph data.

6. The system according to claim 1, characterized in that The storage management module includes a message storage unit and a state management unit; the state management unit stores the state information; The data calculation module is used to transmit the updated message to the message storage unit; When the current round of update is finished, the state management unit is used to update the subgraph received the message Set the status to Active.

7. The system according to claim 6, characterized in that The data calculation module includes an aggregation calculation unit; When the current round of updates is completed and there is no message cache in the message storage unit, the aggregation calculation unit is used to aggregate all the subgraphs to obtain updated large-scale graph data.

8. The system according to claim 1, characterized in that When the subgraph is acquired by the data loading module, the storage management module is further used to set the state of the subgraph to wait for calculation.

9. The system according to claim 1, characterized in that When the subgraph is transmitted to the data calculation module, the storage management module is further configured to set the state of the subgraph to being calculated.

10. The system according to claim 1, characterized in that When the sub-graph is transmitted to the data release module, the storage management module is further configured to set the state of the sub-graph to being released.

Images