A shuffle method for non-volatile memory

Abstract

The invention relates to a Shuffle method aiming at a nonvolatile memory. The Shuffle method includes following steps: utilizing a partition ID to write output data of a Map task into a persistent buffer zone; pulling data in the persistent buffer zone corresponding to a Reduce task.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a Shuffle method for non-volatile memory. Background technique [0002] With the development of science and technology, the world today has entered the era of big data. MapReduce is a popular programming model for large-scale data parallel computing. How to optimize the performance of MapReduce has always been a hot topic in the industry. [0003] Shuffle is a specific stage between the Map stage and the Reduce stage in the MapReduce framework. figure 1 is a schematic diagram of the MapReduce process, such as figure 1 As shown, Shuffle refers to the process that when the output result of Map is to be used by Reduce, the output result is hashed according to the key and distributed to each Reduce. Shuffle involves disk reading and writing and network transmission, so Shuffle The level of performance directly affects the operating efficiency of the entire program. [...

AI Technical Summary

Problems solved by technology

[0005] Themis published an article on Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC), 2012, proposing to use a dynamic memory allocation strategy in the Shuffle stage to store data in the process, that is, during the process of processing data, data from The number of disk reads and writes is only twice, and the rest of the process will not interact with the disk; SpongeFiles published an article on the Proceedings of the 2014 ACM SIGMOD international conference on Management of data, proposing to share the unused memory space in the Task, the above two methods Acceleration is only through memory, which requires high memory performance;

[0006] In addition, Sailfish published an article on Proceedings of the 3rd ACM Symposium on CloudComputing (SoCC), 2012, proposing that when writing Shuffle data, the data of each partition corresponding to the Map Task is gathered, and the distributed file system is used to store the corresponding data; Hadoop-A published an article on Proceedings of the 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, which proposes to use the characteristics of high-speed network (RDMA) and use the Network-Levitated Merge algorithm to execute the Shuffle stage, but the above two The disadvantage of this method is that it is too dependent on network performance, and the time overhead for data access in the form of a file system is relatively large

Images