Disk caching method and device in a parallel computing system

Abstract

The invention provides a disk cache method and device in parallel computer system. The method comprises pre-assigning the respective occupied rate of record data and information data in a data processing memory area; caching part of the record data in disk space with a Hash bucket as the unit when the capacity of the record data in the data processing memory area is about to exceed the pre-assigned rate during data loading; during calculating of task traversal access of the recorded data, caching the accessed Hash buckets in the data processing memory area in the disk space one by one till the released space can load the Hash buckets required to be accessed if the Hash buckets to be accessed is in the disk space and the surplus record data space in the data processing memory area is insufficient to load the Hash buckets to be accessed. By means of the method and the device, automatic caching of data to the disk can be achieved in a BSP model based parallel iterative computation system.

Description

technical field [0001] The present invention relates to the technical field of a Bulk Synchronous Parallel Computing Model (BSP) computing model, in particular to a disk cache method and device in a parallel computing system based on the BSP computing model. Background technique [0002] The BSP computing model, also known as the large synchronization model or the BSP model, was proposed by Viliant of Harvard University and BillMcColl of Oxford University (1990). A BSP model includes multiple processing units interconnected by a network. Each processing unit has a fast local memory and can launch multiple threads of computing tasks. A BSP calculation process includes a series of iterative processes of global super steps. Such as figure 1 As shown, a superstep consists of the following three sequential phases: [0003] (1) Concurrent local computing: Several computing tasks are performed on each processing unit. Each computing process only uses data stored in local memor...

AI Technical Summary

Problems solved by technology

[0023] From the above, it can be seen that the existing parallel iterative processing systems based on the BSP model mainly include: Google's Pregel system, Apache's open source HAMA system and Giraph system. Both data and message data are completely resident in memory and do not have the ability to be cached to external storage. When running a job whose data volume exceeds the upper limit of memory in a cluster system with limited memory resources, it cannot be processed.

Images