File merging method and device

Abstract

The invention provides a file merging method and device, and relates to the technical field of big data, and the method comprises the steps: obtaining a namespace mirror image file of a metadata node in a distributed file system at a preset moment; analyzing the namespace mirror image file to obtain storage information of each file; wherein the storage information comprises a storage path and a file size; importing the storage information of each file into a first file storage table in a database; the files with the file sizes smaller than a first preset threshold value in the first file storage table are determined as files to be merged; according to the storage path of the to-be-merged file, determining a directory hierarchy of storage of the to-be-merged file, and writing storage information of the to-be-merged file into a second file storage table in the database according to the directory hierarchy; and merging the to-be-merged files according to the directory hierarchy of the second file storage table. According to the method, the files in the distributed file system can be automatically merged, and the file merging efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a file merging method and device. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.

[0003] Files in HDFS (distributed file system) are stored in blocks. The storage size of a block is 64MB or 128MB. If the file size is smaller than a block, it also needs to occupy a block. Each block will occupy 150 bytes of memory in the metadata node. If there are too many small files, a large amount of memory space will be occupied, reducing the performance of the metadata node.

[0004] The existing solution is to manually search for small files and merge small files based on the directory where the small files are located. However, manual search for small files leads to low file merging efficiency. The manual search method makes it difficult to find the files that need to be merged in time, and still wastes too much memory space. Summary of the invention

[0005] The embodiment of the present invention proposes a file merging method for automatically merging files in a distributed file system, improving the efficiency of file merging, and avoiding the problem of wasting memory space due to untimely file merging, including:

[0006] At a preset time, obtain the namespace image file of the metadata node in the distributed file system;

[0007] Parse the namespace image file to obtain the storage information of each file; wherein the storage information includes the storage path and the file size;

[0008] Importing the storage information of each file into the first file storage table in the database;

[0009] Determine the files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged;

[0010] Determine the directory level at which the files to be merged are stored according to the storage path of the files to be merged, and write the storage information of the files to be merged into the second file storage table in the database according to the directory level;

[0011] The files to be merged are merged according to the directory level of the second file storage table.

[0012] The embodiment of the present invention provides a file automatic merging device for automatically merging files in a distributed file system, improving the efficiency of file merging, and avoiding the problem of memory space waste caused by untimely file merging, including:

[0013] The acquisition module is used to obtain the namespace image file of the metadata node in the distributed file system at a preset time;

[0014] A parsing module, used to parse the namespace image file and obtain the storage information of each file; wherein the storage information includes the storage path and the file size;

[0015] An import module, used for importing the storage information of each file into the first file storage table in the database;

[0016] A screening module, used for determining files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged;

[0017] A level determination module, used to determine the directory level at which the files to be merged are stored according to the storage paths of the files to be merged, and write the storage information of the files to be merged into the second file storage table in the database according to the directory level;

[0018] The merging module is used to merge the files to be merged according to the directory level of the second file storage table.

[0019] In an embodiment of the present invention, a computer device is proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a file merging method when executing the computer program.

[0020] In an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, a file merging method is implemented.

[0021] A computer program product is provided in an embodiment of the present invention. The computer program product includes a computer program. When the computer program is executed by a processor, a file merging method is implemented.

[0022] The file merging method and device proposed in the embodiment of the present invention can solve the problems of low efficiency of file merging caused by manual search for small files in the prior art, and waste of memory space caused by untimely file merging; the embodiment of the present invention obtains the namespace mirror file of the metadata node in the distributed file system at a preset time; parses the namespace mirror file to obtain the storage information of each file; wherein the storage information includes the storage path and the file size; imports the storage information of each file into the first file storage table in the database; determines the file whose file size is less than the first preset threshold in the first file storage table as the file to be merged; determines the directory level of the file to be merged according to the storage path of the file to be merged, writes the storage information of the file to be merged into the second file storage table in the database according to the directory level; merges the files to be merged according to the directory level of the second file storage table. The embodiment of the present invention does not need to manually search for small files to be merged, can automatically merge files under the distributed file system, improves the efficiency of file merging, obtains the namespace mirror file at the preset time, merges the files to be merged, avoids the problem of waste of memory space caused by untimely file merging, and reasonably uses the memory space. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flowchart of a file merging method in an embodiment of the present invention;

[0025] Figure 2 is a specific example diagram of the file merging method in an embodiment of the present invention;

[0026] Figure 3 is a specific example diagram of the file merging method in an embodiment of the present invention;

[0027] Figure 4 is a schematic diagram of a file merging device in an embodiment of the present invention;

[0028] Figure 5 is a specific example diagram of a file merging device in an embodiment of the present invention;

[0029] Figure 6 Schematic diagram of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0031] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0032] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.

[0033] The terms of the present invention are explained as follows:

[0034] Hadoop: is an open source framework for efficiently storing and processing large data sets ranging from GB to PB.

[0035] HDFS (Distributed File System): is Hadoop’s distributed file system;

[0036] NameNode (metadata node): NameNode is called metadata node in HDFS architecture. NameNode is used to store metadata, such as the number of data blocks and the number of replicas. These metadata are stored in the memory of the metadata node. In order to obtain data as quickly as possible, NameNode is responsible for maintaining and managing slave nodes and assigning tasks to slave nodes.

[0037] Files in a distributed file system are stored in blocks, with a block size of 64MB or 128MB. If the file size is smaller than a block, it also needs to occupy a block. Each block will occupy 150 bytes of memory in the metadata node. If there are too many small files, a lot of memory will be occupied, reducing the performance of the metadata node. If you search for small files through shell commands, the underlying API interface of Hadoop is called, which will put a lot of pressure on the Hadoop cluster during scanning; therefore, there is an urgent need for a file merging method that can automatically merge files in a distributed file system and improve the efficiency of file merging.

[0038] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0039] Figure 1 FIG. 1 is a flowchart of a method for merging files according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0040] Step 101, at a preset time, obtain a namespace image file of a metadata node in a distributed file system;

[0041] Step 102, parsing the namespace image file to obtain storage information of each file; wherein the storage information includes storage path and file size;

[0042] Step 103, importing the storage information of each file into the first file storage table in the database;

[0043] Step 104, determining the files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged;

[0044] Step 105, determining the directory level at which the files to be merged are stored according to the storage paths of the files to be merged, and writing the storage information of the files to be merged into the second file storage table in the database according to the directory level;

[0045] Step 106: merge the files to be merged according to the directory level of the second file storage table.

[0046] Depend on Figure 1It can be known from the process shown that the embodiment of the present invention obtains the namespace mirror file of the metadata node in the distributed file system at a preset time; parses the namespace mirror file to obtain the storage information of each file; wherein the storage information includes the storage path and the file size; imports the storage information of each file into the first file storage table in the database; determines the files in the first file storage table whose file size is less than the first preset threshold as the files to be merged; determines the directory level of the files to be merged according to the storage path of the files to be merged, and writes the storage information of the files to be merged into the second file storage table in the database according to the directory level; merges the files to be merged according to the directory level of the second file storage table. The embodiment of the present invention does not need to manually search for small files to be merged, and can automatically merge files under the distributed file system, thereby improving the efficiency of file merging. By obtaining the namespace mirror file at the preset time and merging the files to be merged, the problem of memory space waste caused by untimely file merging is avoided, and the memory space is used reasonably.

[0047] In one embodiment of the present invention, before obtaining the namespace image file of the metadata node in the distributed file system at a preset time, the method further includes:

[0048] Copy the namespace image file to the preset storage space.

[0049] During specific implementation, the namespace image file (fsimage) of the standby metadata node in the distributed file system is copied to other directories to prevent the namespace image file from being moved or modified.

[0050] In one embodiment of the present invention, with respect to step 102, the namespace image file is parsed to obtain storage information of each file; specifically, the namespace image file is parsed by the HDFS oiv command to obtain a csv file after parsing, and the csv file stores the storage information of each file in the namespace image file, including: storage path (Path), number of copies (Replication), modification time (Modification Time), access time (Access Time), number of block data blocks (Blocks Count), and file size (FileSize).

[0051] In one embodiment of the present invention, the database is: a data warehouse of a distributed file system; specifically, in an embodiment of the present invention, a data warehouse hive of a distributed file system is used, and a first file storage table and a second file storage table are established in the data warehouse hive; wherein the fields of the first file storage table are the same as the csv file header; and the second file storage table includes directory hierarchy, file name, and file size fields.

[0052] In one embodiment of the present invention, with respect to step 104, determining the files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged includes:

[0053] The files whose file sizes are smaller than a first preset threshold value in the first file storage table are screened out by using SQL statements, and the files whose file sizes are smaller than the first preset threshold value in the first file storage table are determined as files to be merged.

[0054] In the specific implementation, the storage information of each file is imported into the first file storage table in the database, and the storage information of each file includes: storage path, number of copies, modification time, access time, number of block data blocks, and file size; SQL statements are used to filter out files with a file size less than 50MB in the first file storage table, and files in the first file storage table with a file size less than a first preset threshold are determined as files to be merged, and then the files to be merged are segmented according to the first-level directory, second-level directory and other directory levels according to the storage path of the files to be merged, and the directory level where the files to be merged are stored is determined, and the storage information of the files to be merged is written into the second file storage table in the database according to the segmented directory level.

[0055] Figure 2 It is a specific example diagram of the file merging method in an embodiment of the present invention.

[0056] In one embodiment of the present invention, reference Figure 2 , with respect to step 105, merging the files to be merged according to the directory level of the second file storage table includes:

[0057] Step 201, sorting the file directories contained in the directory hierarchy of the second file storage table according to the number of files;

[0058] Step 202: merge the files to be merged in the sorted file directory in sequence; wherein the size of the merged files does not exceed a second preset threshold.

[0059] Specifically, SQL statements are used to sort the files in the second file storage table according to the number of small files in the file directory, and a scheduling tool is used to automatically merge the files to be merged in the sorted file directory; wherein the size of the merged file does not exceed 50MB; the scheduling tools that can be used include:

[0060] Figure 3 It is a specific example diagram of the file merging method in an embodiment of the present invention.

[0061] In one embodiment of the present invention, reference Figure 3, copy the namespace image file (fsimage) of the standby metadata node (NameNode) in the distributed file system (HDFS) to other directories to prevent the namespace image file from being moved or modified; and create the first file storage table and the second file storage table in the data warehouse hive; through HDFS The oiv command parses the namespace image file to obtain a csv file after parsing. The csv file stores the storage information of each file in the namespace image file, including: storage path, number of copies, modification time, access time, number of block data blocks, and file size; the storage information of each file in the csv file is imported into the first file storage table, and the files whose file sizes in the first file storage table are smaller than the first preset threshold are screened out using SQL statements, and the files whose file sizes in the first file storage table are smaller than the first preset threshold are determined as files to be merged; the directory level at which the files to be merged are stored is determined according to the storage path of the files to be merged, and the storage information of the files to be merged is written into the second file storage table in the database according to the directory level; wherein the storage fields of the second file storage table include: directory level (determined according to the file storage path), file name, and file size; in the second file storage table, the files are sorted according to the number of small files in the file directory, and small file merging is performed on the files to be merged in the sorted file directory.

[0062] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0063] The implementation of the file merging device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" or "unit" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0064] Based on the same inventive concept, the present invention also proposes a file merging device, such as Figure 4 As shown, the device comprises:

[0065] The acquisition module 401 is used to acquire the namespace image file of the metadata node in the distributed file system at a preset time;

[0066] The parsing module 402 is used to parse the namespace image file to obtain the storage information of each file; wherein the storage information includes the storage path and the file size;

[0067] Import module 403, used to import the storage information of each file into the first file storage table in the database;

[0068] A screening module 404, configured to determine files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged;

[0069] A level determination module 405 is used to determine the directory level at which the files to be merged are stored according to the storage paths of the files to be merged, and write the storage information of the files to be merged into the second file storage table in the database according to the directory level;

[0070] The merging module 406 is used to merge the files to be merged according to the directory level of the second file storage table.

[0071] Figure 5 FIG. 1 is a specific example diagram of a file merging device in an embodiment of the present invention. Figure 5 As shown, in one embodiment of the present invention, Figure 4 The file merging device also includes:

[0072] The copy module 501 is used to copy the namespace image file to a preset storage space.

[0073] In one embodiment of the present invention, the database is: a data warehouse of a distributed file system.

[0074] In one embodiment of the present invention, the screening module 404 is specifically used for:

[0075] The files whose file sizes are smaller than a first preset threshold value in the first file storage table are screened out by using SQL statements, and the files whose file sizes are smaller than the first preset threshold value in the first file storage table are determined as files to be merged.

[0076] In one embodiment of the present invention, the merging module 406 is specifically used for:

[0077] According to the number of files, sort the file directories contained in the directory hierarchy of the second file storage table;

[0078] The files to be merged in the sorted file directory are merged in sequence; wherein the size of the merged files does not exceed a second preset threshold.

[0079] It should be noted that although several modules of the file merging device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules to be embodied.

[0080] Based on the above invention concept, Figure 6 As shown, the present invention also proposes a computer device 600, including a memory 601, a processor 602, and a computer program 603 stored in the memory 601 and executable on the processor 602, wherein the processor 602 implements the aforementioned file merging method when executing the computer program 603.

[0081] Based on the aforementioned inventive concept, the present invention proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned file merging method is implemented.

[0082] Based on the aforementioned inventive concept, the present invention proposes a computer program product, wherein the computer program product comprises a computer program, and the computer program implements a file merging method when executed by a processor.

[0083] The file merging method and device proposed in the embodiment of the present invention can solve the problems of low efficiency of file merging caused by manual search for small files in the prior art, and waste of memory space caused by untimely file merging; the embodiment of the present invention obtains the namespace mirror file of the metadata node in the distributed file system at a preset time; parses the namespace mirror file to obtain the storage information of each file; wherein the storage information includes the storage path and the file size; imports the storage information of each file into the first file storage table in the database; determines the file whose file size is less than the first preset threshold in the first file storage table as the file to be merged; determines the directory level of the file to be merged according to the storage path of the file to be merged, writes the storage information of the file to be merged into the second file storage table in the database according to the directory level; merges the files to be merged according to the directory level of the second file storage table. The embodiment of the present invention does not need to manually search for small files to be merged, can automatically merge files under the distributed file system, improves the efficiency of file merging, obtains the namespace mirror file at the preset time, merges the files to be merged, avoids the problem of waste of memory space caused by untimely file merging, and reasonably uses the memory space.

[0084] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0086] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0087] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0088] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A file merging method, characterized in that: include: At a preset time, obtain the namespace image file of the metadata node in the distributed file system; Parse the namespace image file to obtain the storage information of each file; wherein the storage information includes the storage path and the file size; Importing the storage information of each file into the first file storage table in the database; Determine the files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged; Determine the directory level at which the files to be merged are stored according to the storage paths of the files to be merged, and write the storage information of the files to be merged into the second file storage table in the database according to the directory level; The files to be merged are merged according to the directory level of the second file storage table.

2. The method according to claim 1, characterized in that Before obtaining the namespace image file of the metadata node in the distributed file system at the preset time, the following steps are also included: Copy the namespace image file to the preset storage space.

3. The method according to claim 1, characterized in that The database is: a data warehouse of a distributed file system.

4. The method according to claim 1, characterized in that: Determining the files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged includes: The files whose file sizes are smaller than a first preset threshold value in the first file storage table are screened out by using SQL statements, and the files whose file sizes are smaller than the first preset threshold value in the first file storage table are determined as files to be merged.

5. The method according to claim 1, characterized in that According to the directory level of the second file storage table, the files to be merged are merged, including: Sort the file directories contained in the directory hierarchy of the second file storage table according to the number of files; The files to be merged in the sorted file directory are merged in sequence; wherein the size of the merged files does not exceed a second preset threshold.

6. A file merging device, characterized in that: include: The acquisition module is used to obtain the namespace image file of the metadata node in the distributed file system at a preset time; A parsing module, used to parse the namespace image file and obtain the storage information of each file; wherein the storage information includes the storage path and the file size; An import module, used for importing the storage information of each file into the first file storage table in the database; A screening module, used for determining files in the first file storage table whose file sizes are smaller than a first preset threshold as files to be merged; A level determination module, used to determine the directory level at which the files to be merged are stored according to the storage paths of the files to be merged, and write the storage information of the files to be merged into the second file storage table in the database according to the directory level; The merging module is used to merge the files to be merged according to the directory level of the second file storage table.

7. The device according to claim 6, characterized in that Also includes: The copy module is used to copy the namespace image file to the preset storage space.

8. The device according to claim 6, characterized in that The database is: a data warehouse of a distributed file system.

9. The device according to claim 6, characterized in that The screening module is specifically used for: The files whose file sizes are smaller than a first preset threshold value in the first file storage table are screened out by using SQL statements, and the files whose file sizes are smaller than the first preset threshold value in the first file storage table are determined as files to be merged.

10. The device according to claim 6, characterized in that The merge module is specifically used for: Sort the file directories contained in the directory hierarchy of the second file storage table according to the number of files; The files to be merged in the sorted file directory are merged in sequence; wherein the size of the merged files does not exceed a second preset threshold.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images