A method of i/o management based on multiple external memory devices and multiple queues

Abstract

The invention discloses an I / O management method based on multiple external storage devices and multiple queues. In the file; use the above striping method to perform address mapping on the original I / O request; determine whether the original I / O request needs to be decomposed, and if so, decompose it into multiple new I / Os aligned to the stripe unit boundary request; otherwise, directly map it into a new I / O request of equal length; perform prefetching and merging on the new I / O request; assign the new I / O request to the corresponding I / O task queue of the corresponding external storage device. The present invention ensures that each I / O request is only served by one external storage device through I / O management based on multiple peripherals and multiple queues, and each application I / O thread is dedicated to one external storage device. The prefetching and merging of requests combines multiple small requests into one large I / O request, which effectively reduces the number of accesses to external memory devices and enables the multi-external memory I / O performance to be fully utilized.

Description

technical field [0001] The invention belongs to the technical field of external storage graph processing, and more specifically relates to an I / O management method based on multiple external storage devices and multiple queues. Background technique [0002] With the increasing scale of graph data, the graph processing system using external storage has become a popular solution to replace the distributed memory mode graph processing system due to its advantages of low cost and easy expansion. performance bottleneck. When processing large-scale graph datasets where I / O throughput is required, using multi-peripheral devices becomes a natural and common solution. However, the multi-external-memory I / O management method adopted by the current graph processing system using external memory has problems such as insufficient utilization of sequential bandwidth, limited parallel I / O capability, and high management overhead, which makes the performance of multi-external-memory I / O uns...

AI Technical Summary

Problems solved by technology

It is inefficient for the operating system to perform sorting and merging in such a plug-list, and at fine-grained stripe depths, concurrent I / O threads may cause sequential small I / O requests on each external memory device. Unpredictable out-of-order arrivals make the merge invalid

Such a thread pool model also leads to contention for multiple application I / O threads in the application layer to lock on a single shared I / O task queue

Images