File merging method and device for big data platform

Abstract

The present invention provides a method and device for merging files on a big data platform. The method includes: monitoring the directory changes of the big data platform, and judging whether the number of files in the changed directory changes; When the number of files changes, group files with similar characteristics under the changed directory; judge whether there are small files with a set number of smaller than an integer multiple of the set data block size in the files of the same group; If the small files exist in the files of the group, the small files of the same group are acquired, and the small files of the same group are merged. Through the above solution, small files can be reduced, the memory usage of namenode can be optimized, and the big data platform can accommodate more files.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a file merging method and device for a big data platform. Background technique [0002] In a big data platform, such as a Hadoop cluster, when performing data analysis, there are often a large number of small files in the data directory. The existence of these small files puts a lot of pressure on the namenode, resulting in a decrease in the computing efficiency of the cluster by several times or even dozens of times. times. In the prior art, it is necessary to develop functional components for each group of data directories or each type of target data to merge files. [0003] However, existing file merging schemes can only allocate schedules based on time. This maintenance mode has many disadvantages: first, the development content is relatively trivial and the development cost is high; second, the scheduling plan cannot be arranged according to the actual data situation, an...

AI Technical Summary

Problems solved by technology

[0003] However, existing file merging schemes can only allocate schedules based on time

This maintenance mode has many disadvantages: first, the development content is relatively trivial and the development cost is high; second, the scheduling plan cannot be arranged according to the actual data situation, and there may not be many small files when the task is started, which wastes cluster computing resources, or the task is executed. There are new files written into the directory, and the problem of the number of small files has not been well resolved; the third is that the computing resources for each file processing cannot be dynamically applied according to the actual situation

Images