Shuffle data caching method based on mapping-reduction calculation model

Abstract

The invention discloses a shuffle data caching method based on a mapping-reduction calculation model. The shuffle data caching method comprises the following steps of: a mapping-reduction calculation frame sends the division, which takes tasks as a unit, of one piece of mapping-reduction work to a shuffle caching host through an interface, after the shuffle caching host receives task division data, a timestamp is added to the task division data, and then, the task division data with the timestamp is stored in local memory; and the shuffle caching host adopts a random algorithm to carry out one-to-three mapping on reduction task in the task division data and each node in a cluster, and the reduction task and each node of the cluster are stored in the memory of the shuffle caching host in a form of a hash table. By use of the method, the calculation performance of a distributed calculation frame based on the mapping-reduction model can be improved, the low-efficiency manual setting of checking points by users can be avoided, and the robustness of the distributed calculation frame is improved.

Description

technical field [0001] The invention relates to the field of computer distributed systems and distributed computing frameworks. Specifically, it mainly provides a memory-based distributed shuffle data cache for the Map-Reduce computing model, thereby improving the performance and robustness of the computing framework. Background technique [0002] The map-reduce computing model and the distributed computing system designed based on this model are currently mainstream big data distributed systems, such as Spark and Hadoop. The calculation based on this model has a shuffling (Shuffle) between the mapping and reduction stages, which isolates the mapping and reduction. All current designs use shuffled data to be written to disk for persistent processing, and then transmitted. However, the performance of the disk is far inferior to that of the memory, so it brings this large performance overhead to the computing system. [0003] At the same time, this type of computing framewo...

AI Technical Summary

Problems solved by technology

Because these fault-tolerant mechanisms overlap with the calculation logic, not only do not make full use of the existing hardware features, but also interspersed in the calculation process greatly affects the performance of the calculation itself.

[0004] Although there are some memory-based distributed file systems, they are mainly for the data block itself, and the volume of the data block itself is often much larger than the shuffled data, so a large amount of memory is required as a support

Images