An I/O management method based on multiple external storage devices and multiple queues

Abstract

The invention discloses an I / O management method based on multiple queues of multiple external storage devices. The method comprises the following steps: dividing edge block files into stripe units with equal size; striping the stripe units into striped files in an increasing order; striping the striped units into striped files in an increasing order; and striping the striped units into striped files. The address mapping of the original I / O request is carried out by adopting the striping mode; Determining whether the original I / O request needs to be decomposed, and if so, decomposing the original I / O request into a plurality of new I / O requests aligned with stripe cell boundaries; Otherwise, it is mapped directly to a new I / O request of equal length; Prefetching consolidation of new I / O requests; dispatching A new I / O request to the corresponding I / O task queue for the corresponding external memory device. Through I / O management based on multiple peripheral equipments and multiple queues, Ensuring that each I / O request is serviced by only one external device, dedicating Each application I / O thread to one external storage device. By combining the prefetching and merging of new I / O requests, a plurality of small requests are merged into one large I / O request, which effectively reduces the access times of external storage devices and makes the performance of multi-external storageI / O fully play.

Description

technical field [0001] The invention belongs to the technical field of external storage graph processing, and more specifically relates to an I / O management method based on multiple external storage devices and multiple queues. Background technique [0002] With the increasing scale of graph data, the graph processing system using external storage has become a popular solution to replace the distributed memory mode graph processing system due to its advantages of low cost and easy expansion. performance bottleneck. When processing large-scale graph datasets where I / O throughput is required, using multi-peripheral devices becomes a natural and common solution. However, the multi-external-memory I / O management method adopted by the current graph processing system using external memory has problems such as insufficient utilization of sequential bandwidth, limited parallel I / O capability, and high management overhead, which makes the performance of multi-external-memory I / O uns...

AI Technical Summary

Problems solved by technology

It is inefficient for the operating system to perform sorting and merging in such a plug-list, and at fine-grained stripe depths, concurrent I / O threads may cause sequential small I / O requests on each external memory device. Unpredictable out-of-order arrivals make the merge invalid

Such a thread pool model also leads to contention for multiple application I / O threads in the application layer to lock on a single shared I / O task queue

Images