A multi-task external memory schema graph processing method based on I/O scheduling

Abstract

The invention discloses a multi-task external memory mode diagram processing method based on I / O scheduling, includes streaming partitioning graph data to obtain graph partition, evenly placing graphpartition in multiple external storage devices, selecting target external storage devices from multiple external storage devices based on I / O scheduling, and taking graph partition in the target external storage device that has not been accessed by graph processing task as designated partition; Judging whether the synchronization field of the designated partition is not mapped into the memory according to the synchronization field of the designated partition, if so, mapping the designated partition from the external storage device into the memory, and updating the synchronization field of thedesignated partition; Otherwise, the graph partition data is accessed directly through the address information mapped to memory by the specified partition. Through I / O scheduling, the invention selects the external storage device with the least number of tasks to access, thereby controlling the sequence of accessing the data of the external storage diagram partition and balancing the I / O pressure.By setting the synchronization field to realize the data sharing of graph partition, the repeated loading of the same graph partition is reduced, so as to reduce the total I / O bandwidth and improve I / O efficiency.

Description

technical field [0001] The invention belongs to the field of external storage mode graph processing, and more specifically relates to a multi-task external storage mode graph processing method based on I / O scheduling. Background technique [0002] With the continuous improvement of the degree of social informatization, the scale of graph data is growing at an explosive rate. In order to better analyze and process large-scale graph data, a number of specialized graph processing systems have been produced in recent years. Among them, the external memory mode graph processing system stores graph data in cheap and easily expandable The access speed of graph data improves the execution efficiency of graph algorithms, and can effectively handle graph computing problems with billions of edges in a stand-alone environment. [0003] When more and more graph processing tasks are executed on the same graph processing platform, time overlap will inevitably occur, that is, task-level pa...

AI Technical Summary

Problems solved by technology

However, these graph processing systems are not suitable for multi-tasking scenarios. This is because each graph algorithm has different I / O characteristics and different access speeds of external graph data. Therefore, when multiple graph processing tasks are executed in parallel, the Affect or even destroy the existing external memory I / O optimization method, resulting in serious external memory random access overhead

In addition, multiple graph algorithms initiate I / O requests to external memory separately, which will compete for limited I / O bandwidth, and serious I / O conflicts

Images