Memory data set replacing system and replacing method for big data processing system

Abstract

The invention discloses a memory data set replacing system for a big data processing system. The system comprises an analysis module, an information monitoring module and a decision making module, wherein the analysis module is used for performing logic analysis on an upper layer user program to obtain the operation step set during the generation of the memory data set in each operation stage; the information monitoring module is used for monitoring the user program in the operation process, collecting information during the memory data set generation and submitting the information to the decision making module; the decision making module is used for analyzing and sequencing the collected information, judging whether the memory data set in the system needs to be replaced or not in the current stage, determining the memory data set to be removed when the replacement is needed by the system, and notifying the system of replacing the new memory data set. In the work process, the complete transparency is realized on the upper layer user layer program; the replacing scheme of the memory data set can be flexibly decided according to the real-time condition of the system. Meanwhile, compared with a conventional replacing method, the method has the advantage that the influence on the system performance is well reduced.

Description

technical field [0001] The invention belongs to the field of big data distributed computing, and more specifically relates to a memory data set replacement system and a replacement method for a big data processing system. Background technique [0002] In recent years, big data distributed processing systems have received extensive attention. Several big data distributed processing systems have emerged in industry and academia: Spark, Flink, Dryad, GraphX, etc. Developers use the APIs provided by such systems to describe big data applications. Usually, the amount of data processed by such applications far exceeds the computing power of a single machine. Therefore, in practice, such systems are usually deployed in clusters containing multiple computing nodes. [0003] The input data processed by the big data application is usually located in the storage shared by the cluster (such as HDFS, etc.), and the big data processing system divides the program into multiple operation ...

AI Technical Summary

Problems solved by technology

When the data that has been removed is used later in the program, the processing system will run necessary operations to generate the data again. The operation steps in this process are the same as when the data was first generated, and the time required is also close to the time spent on the first generation. Computational overhead introduced by the calculation affects the overall time required for the program to run

[0007] Most of the current big data processing systems use the LRU (Least Recently Used) algorithm to decide how to replace the memory data set. The LRU algorithm will remove the memory data set that has not been used for the longest time since the current time. It does not consider the overhead of data recalculation, resulting in high The data for recalculation overhead is replaced out of memory, thus increasing the overall running time of the system

Other common replacement algorithms such as the FIFO (FirstInFirstOut) algorithm make decisions based on the generation time of the memory data set, and the LFU (Least Frequently Used) algorithm makes decisions based on the access frequency of the data set, which also cannot guarantee the minimum overhead introduced when recalculating the data set

At present, there is a lack of replacement methods for memory dataset replacement in big data distributed processing systems with low recalculation overhead

Images