A file merging optimization method and terminal

Abstract

The application discloses a file merging optimization method and a terminal. Files smaller than a preset byte amount in distributed nodes are screened and sorted according to byte amounts. Information extraction is performed on the screened files, and file information and sorting information are written into an array structure. Two files adjacent in sorting in the array structure are merged. The application sorts files smaller than a preset byte amount, stores file information and sorting information by using an array structure, and merges two files with similar byte amounts two by two. By using the principle that files with similar byte amounts have similar processing time, the file merging time is saved, and the phenomenon that the byte amounts of two files are too different when randomly combined and the processing time is based on the processing process of a large file is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a file merging optimization method and terminal. BACKGROUND

[0002] At present, there are many needs of big data processing tasks, so there are many related big data processing software. The most famous one is the hadoop related distributed file storage system, the main principle of which is to deploy hadoop nodes on different server nodes, and realize the distributed file storage function of multiple nodes of hadoop through multiple ordinary machine resources.

[0003] Among them, the bottom layer of hadoop uses small byte files for data storage, and each file needs to have corresponding metadata record information. The function of metadata information is to query the location of data in the node according to the metadata information. Because it needs to be queried quickly, the metadata information is generally stored in the cache, and the cache resource is expensive. In order to reduce the occupation of cache, it is necessary to reduce the amount of metadata, and generally a plurality of small byte files will be merged to generate a large byte file and new metadata information. However, the current random file merging method has a long merging time. SUMMARY

[0004] The technical problem to be solved by the present application is to provide a file merging optimization method and terminal to solve the problem of slow merging speed and long time consumption.

[0005] In order to solve the above technical problems, the technical scheme adopted by the present application is:

[0006] A file merging optimization method, comprising the steps of:

[0007] S1, screening files smaller than a preset size in a distributed node and sorting the files according to file size;

[0008] S2, extracting information from the screened files, and writing file information and sorting information into an array structure;

[0009] S3, grouping and merging each file by taking two adjacent files in the array structure as a group.

[0010] In order to solve the above technical problems, another technical scheme adopted by the present application is:

[0011] A file merging optimization terminal, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to complete the following steps:

[0012] S1, screen files less than a preset size in a distributed node and sort the files according to file sizes;

[0013] S2, extract information from the screened files, and write the file information and sorting information into an array structure;

[0014] S3, group and merge the files by taking two files adjacent in the array structure as a group.

[0015] The present application has the advantages that a file merging optimization method and terminal are provided, files less than a preset byte amount are sorted, file information and sorting information are stored by using an array structure, two files with similar file sizes are merged two by two, the principle that files with similar sizes have similar processing times is used, file merging time is saved, and the phenomenon that two files have a large size difference and the processing time is based on the processing of a large file is avoided. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 A flowchart of a file merging optimization method in an embodiment of the present application;

[0017] Figure 2 A schematic diagram of a file merging optimization terminal in an embodiment of the present application;

[0018] Label explanation:

[0019] 1, terminal; 2, memory; 3, processor. DETAILED DESCRIPTION

[0020] To describe the technical content, purposes and effects of the present application in detail, the following describes the present application in conjunction with the embodiments and the accompanying drawings.

[0021] Please refer to Figure 1 and Figure 2 A file merging optimization method, comprising the steps of:

[0022] S1, screen files less than a preset size in a distributed node and sort the files according to file sizes;

[0023] S2, extract information from the screened files, and write the file information and sorting information into an array structure;

[0024] S3, group and merge the files by taking two files adjacent in the array structure as a group.

[0025] It should be noted that, in the hadoop underlying file merging process, the inventor has found through repeated experiments that, when randomly combining files for merging, if the byte amount difference of the files to be merged is too large, the small file processing needs to wait for the large file processing to be completed before the merging can be performed, which results in that the processing time of the combination of the large and small files is usually determined by the processing time of the large file, the processing module is occupied for a long time in the file merging process, and resources are wasted.

[0026] From the above description, it can be known that the beneficial effects of the present application are that: by sorting the files smaller than the preset byte amount, storing the file information and sorting information by using the array structure, and merging two files with similar byte amounts two by two, the file merging time is saved by using the principle that the processing time of the files with similar byte amounts is approximate, and the phenomenon that the processing time is based on the processing process of the large file when the byte amount difference of the two files is too large in the random combination is avoided.

[0027] In the embodiment of the present application, step S3 further comprises the steps of:

[0028] After the merging of each group of files in the array structure is completed, returning to step S1.

[0029] From the above description, it can be known that the files selected and sorted are merged two by two according to the proximity principle, that is, after the selected files are sorted according to the byte amount, all the files are merged two by two, since the file sizes of the adjacent files are similar, the merging process time is relatively uniform, and the resource waste phenomenon does not occur, for example, if there are 10 files sorted in the array structure, the files 【1, 2】, 【3, 4】, 【5, 6】, 【7, 8】 and 【9, 10】 are selected and merged; in addition, after the files in the array structure are all merged, the files are selected again according to the preset byte amount, and the already merged files and new files are merged again, further reducing the occupation of the cache.

[0030] In the embodiment of the present application, the step S3 is specifically:

[0031] The two files adjacent in the array structure are extracted and grouped, and the threads control the merging of the two files in each group.

[0032] From the above description, it can be known that after the files in the array are grouped two by two, the threads are used to merge the files in each group, and the merging efficiency is improved; preferably, multiple threads can be used to simultaneously merge multiple groups of files, and the merging efficiency is improved.

[0033] In the embodiment of the present application, the file is a file file.

[0034] As can be seen from the above description, since the specific type of the file file can be many, such as text files, image files, audio files, video files, etc., merging the file type file can maximize the reduction of cache occupancy.

[0035] In the embodiment of the application, the array structure stores sorting information in an array.

[0036] A file merging optimization terminal, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to complete the steps in the file merging optimization method.

[0037] As can be seen from the above description, the execution carrier of the file merging optimization method is provided, and when the steps in the above method are executed, the files smaller than the preset byte amount are sorted, the file information and sorting information are stored in the array structure, and two files with similar byte amounts are merged two by two, the principle of similar processing time of files with similar byte amounts is used to save the file merging time, and the phenomenon that the byte amounts of two files are too different when randomly combined and the processing time is based on the processing process of large files is avoided.

[0038] The application provides a file merging optimization method and terminal, mainly applied to file merging under the hadoop distributed architecture, reducing cache occupancy while improving file merging efficiency, which will be specifically described below in conjunction with embodiments.

[0039] Please refer to Figure 1 The embodiment one of the application is a file merging optimization method, comprising the steps of:

[0040] S1, screening files smaller than a preset size in a distributed node and sorting the files according to file size;

[0041] S2, extracting information from the screened files, and writing the file information and sorting information into an array structure;

[0042] S3, grouping and merging each file in the array structure with two files adjacent in sorting as a group;

[0043] After each group of files in the array structure is merged, return to step S1.

[0044] By sorting files smaller than a preset byte amount, storing file information and sorting information using an array structure, and merging two files with similar byte amounts two by two, the processing time of files with similar byte amounts is approximated, the file merging time is saved, and the phenomenon that the processing time is based on the processing of large files when the two files are randomly combined and the byte amounts of the two files differ greatly is avoided. In addition, after the files in the array structure are all merged, the files are again screened according to the preset byte amount, and the merged files and new files are again merged, further reducing the occupation of the cache.

[0045] Specifically, in step S3:

[0046] The two files adjacent in the array structure are extracted and grouped, and the threads control the merging of the two files in each group.

[0047] That is, in this embodiment, the files in the array are grouped two by two, and the threads are used to merge the files in each group to improve the merging efficiency; preferably, multiple threads can be used to simultaneously merge multiple groups of files to improve the merging efficiency.

[0048] Embodiment two of the present application is as follows:

[0049] 1. Small file scanning and sorting:

[0050] First, scan each distributed node to identify all log files smaller than 1MB. At the same time, in order to improve the processing speed, the number of files selected for each operation is 10,000.

[0051] 2. File sorting and array storage:

[0052] Sort all selected small files to be merged in ascending or descending order according to their size, and store the references or paths of these files in the array structure, for example, array[1] represents the first file, and array

[10000] represents the 10,000th file.

[0053] 3. Single or multi-thread merging:

[0054] Use single or multi-thread technology to process different groups of file merging operations, merge two adjacent files each time, for example, array[1] and array[2], array

[4999] and array

[5000] , until all files in the array structure are merged.

[0055] That is, in this embodiment, the system can significantly reduce the I / O overhead generated by processing a large number of small files, and improve the overall efficiency of log processing and analysis.

[0056] Reference will now be made to the drawings Figure 2 Embodiment three of the present application is a file merging optimization terminal 1, comprising a memory 2, a processor 3, and a computer program stored in the memory 2 and executable on the processor 3, wherein the processor 3 executes the computer program to complete the steps in the file merging optimization method in embodiments one to two.

[0057] In summary, the present application provides a file merging optimization method and terminal, which sorts files smaller than a preset byte amount, stores file information and sorting information using an array structure, and merges two files with similar byte amounts two by two, thereby saving file merging time and avoiding the phenomenon that the processing time is based on the processing of large files when the byte amounts of two files differ greatly.

[0058] The above description is merely an embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent transformation or direct or indirect application in related technical fields using the content of the present application specification and drawings is also included in the patent protection scope of the present application.

Claims

1. A file merging optimization method, characterized in that: Including the following steps: S1. Filter files smaller than the preset size from the distributed nodes and sort them by file size; S2. Extract information from the filtered files and write the file information and sorting information into an array structure; S3. Group and merge files into one group, taking two adjacent files in the array structure as a group; Step S3 also includes the following steps: After merging all the files in the array structure, return to step S1; Step S3 specifically involves: Extract two adjacent files from the array structure and group them. Then, control the thread to merge the two files in each group.

2. The file merging optimization method according to claim 1, characterized in that: The file in question is a file named "file".

3. The file merging optimization method according to claim 1, characterized in that: The array structure uses an array to store sorting information.

4. A file merging optimization terminal, characterized in that: Includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the following steps: S1. Filter files smaller than the preset size from the distributed nodes and sort them by file size; S2. Extract information from the filtered files and write the file information and sorting information into an array structure; S3. Group and merge files into one group, taking two adjacent files in the array structure as a group; Step S3 also includes the following steps: After merging all the files in the array structure, return to step S1; Step S3 specifically involves: Extract two adjacent files from the array structure and group them. Then, control the thread to merge the two files in each group.

5. A file merging optimization terminal according to claim 4, characterized in that: The file in question is a file named "file".

6. A file merging optimization terminal according to claim 4, characterized in that: The array structure uses an array to store sorting information.

Images