Small file merging method and device, equipment and medium

Abstract

The invention relates to the technical field of data processing, and discloses a small file merging method and device, equipment and a medium, and the method comprises the steps: analyzing a merging instruction to obtain a preset configuration table under the control of the merging instruction for merging small files; and according to the preset configuration table, obtaining a to-be-merged table, and according to the file type of the to-be-merged table, selecting an adaptive preset method to perform file merging. Small files can be automatically merged, and the file merging efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a small file merging method, device, equipment and medium. Background Art

[0002] Currently, most structured data processed by big data tasks will be stored in distributed file storage systems. However, due to the complexity and diversity of structured data processed by big data tasks, more small files will be generated. These small files are stored in distributed file storage systems. As time goes by, there are more and more small files in the distributed file storage system, which will reduce the performance of the distributed storage system. Therefore, in order to ensure the performance of the distributed storage system, small files need to be merged. The commonly used merging method is that staff manually screen small files and merge files.

[0003] The inventors realize that manually merging small files is not only time-consuming but also prone to errors and omissions, resulting in low efficiency. Summary of the invention

[0004] The present invention provides a small file merging method, device, computer equipment and medium to solve the technical problem of low efficiency of the existing small file merging method.

[0005] In a first aspect, a small file merging method is provided, comprising:

[0006] Controlled by a merging instruction for merging small files, parsing the merging instruction to obtain a preset configuration table;

[0007] The table to be merged is obtained according to the preset configuration table, and a suitable preset method is selected to merge the files according to the file type of the table to be merged.

[0008] In a second aspect, a small file merging device is provided, comprising:

[0009] A parsing module, which is controlled by a merging instruction for merging small files, and parses the merging instruction to obtain a preset configuration table;

[0010] The merging module is used to obtain the table to be merged according to the preset configuration table, and select a suitable preset method to merge the files according to the file type of the table to be merged.

[0011] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned small file merging method when executing the computer program.

[0012] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned small file merging method are implemented.

[0013] In the scheme implemented by the above-mentioned small file merging method, device, computer equipment and storage medium, when a merge instruction for merging small files is received, the merge instruction can be parsed to obtain a preset configuration table, and the table to be merged can be obtained through the preset configuration table. After obtaining the table to be merged, the file type of the table to be merged is confirmed. Different types of tables to be merged correspond to different merging methods. After determining the file type of the table to be merged, a matching preset method is selected to merge the tables to be merged, so that small files can be automatically merged, thereby improving the efficiency of merging small files. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0015] Figure 1 is a schematic diagram of an application environment of a small file merging method in an embodiment of the present invention;

[0016] Figure 2 is a flowchart of a method for merging small files in one embodiment of the present invention;

[0017] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S20;

[0018] Figure 4 yes Figure 2 Another specific implementation flow diagram of step S20;

[0019] Figure 5 yes Figure 4 A schematic flow chart of a specific implementation of step S23;

[0020] Figure 6 yes Figure 4 A schematic flow chart of a specific implementation of step S24;

[0021] Figure 7 It is another flowchart of a method for merging small files in one embodiment of the present invention;

[0022] Figure 8 is a structural schematic diagram of a small file merging device in one embodiment of the present invention;

[0023] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0024] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] The small file merging method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, when it is necessary to sort the small files in the distributed system, that is, when a merge instruction is received, the merge instruction is parsed to obtain a preset configuration table, and then the table to be merged is obtained according to the preset configuration table, and a matching preset method is selected according to the file type of the table to be merged to merge the files. In the present invention, various types of small files in the distributed system can be automatically merged to improve the speed of merging small files. The computer device can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.

[0027] See also Figure 2 As shown, Figure 2 A flowchart of a method for merging small files provided in an embodiment of the present invention includes the following steps:

[0028] S10: Under the control of a merging instruction for merging small files, the merging instruction is parsed to obtain a preset configuration table.

[0029] The present invention provides a small file merging method which can be applied to a distributed file system (Hadoop Distributed File System, HDFS for short) and is used to merge small files in a distributed file system. HDFS is a distributed file system designed for big data. An HDFS consists of a management node (NameNode) and multiple data nodes (DataNode). NameNode is a metadata node that manages the metadata of the file system. NameNode stores the metadata of the file system in memory. DataNode is a data node that stores actual file data. Metadata storage is generally small file storage. Too many small files will cause the namenode metadata to be particularly large, thereby occupying too much memory and seriously affecting the performance of HDFS.

[0030] Hive table is the core component of Hive, a data warehouse tool based on Hadoop. It maps structured data files to a database table and provides SQL-like query functions. Hive table is also the basic storage unit in the Hive data warehouse, which allows users to store, query and analyze data in the form of tables.

[0031] The present invention can write various Hive tables into corresponding tables in advance, so as to summarize various data in the Hive table, and the obtained table is the preset configuration table. After obtaining the preset configuration table, the preset configuration table is stored and waits for calling. When a merge instruction is received, the merge instruction is parsed to call the preset configuration table, so as to obtain the information in the preset configuration table.

[0032] For example, there are two Hive tables in a Hadoop cluster, namely example_table1 and example_tabl2. The information of example_table1 and example_table2 can be written into a preset configuration table, which at least includes relevant information such as file names.

[0033] S20: Obtain a table to be merged according to the preset configuration table, and select a suitable preset method to merge files according to the file type of the table to be merged.

[0034] After obtaining the preset configuration table, confirm the table to be merged that needs to be merged from the preset configuration table, then confirm its file type based on the table to be merged, and select a matching preset method to merge the files according to the file type.

[0035] For example, for a preset configuration table, which includes information of two Hive tables, example_table1 and example_table2, the preset configuration table includes at least the file names of the two Hive tables, namely example_table1 and example_table2. When reading the preset configuration table, the two file names, example_table1 and example_table2, can be read, and then based on the two file names, they are connected to the file names in the Hadoop cluster that match the above two file names, and then the files in the Hadoop cluster that contain the two file names, example_table1 and example_table2, are merged.

[0036] It should be understood that the preset quota configuration table is mainly used to confirm the small files to be merged. Figure 3As shown, in step S20, that is, obtaining the table to be merged according to the preset configuration table, includes the following steps:

[0037] S21: traverse the preset configuration table to confirm the target field, and confirm the file name recorded in the target field;

[0038] S22: Confirm the hive table that needs to be merged according to the file name and set it as the table to be merged.

[0039] For steps S21-S22, after obtaining the preset configuration table, you can traverse the preset configuration table to confirm the target field, then confirm the file name recorded in the target field from the target field, and confirm the hive table that needs to be merged according to the file name and set it as the table to be merged.

[0040] For example, the preset configuration table may include fields such as table_name, merge_threshold, last_merge_time, and description, where table_name is used to store the file name, merge_threshold is used to set the merge threshold for merging small files, last_merge_time is used to record the time of the last merge of small files, and merge_threshold is used to store the notes of the preset configuration table. When traversing the above preset configuration table, table_name is the target field, and the file names recorded in table_name can be obtained, such as example_table1 and example_table2, and then example_table1 and example_table2 are set as the tables to be merged.

[0041] It should be understood that the preset quota configuration table is mainly used to confirm the small files to be merged. Figure 4 As shown, in step S20, that is, selecting a preset method suitable for merging files according to the file type of the table to be merged, includes the following steps:

[0042] S23: If the file type of the table to be merged is a partition file, merge the files of the table to be merged according to a first preset merging method.

[0043] S24: If the file type of the table to be merged is a non-partitioned file, merge the files of the table to be merged according to a second preset merging method, wherein the preset method includes the first preset merging method and the second preset merging method.

[0044] For steps S23-S24, the first preset merging method and the second preset merging method are both pre-configured merging methods for different types of files. The file types of the table to be merged can be mainly divided into partitioned files and non-partitioned files, specifically referring to whether the corresponding Hive table is a partitioned table or a non-partitioned table. For the table to be merged that is a partitioned file, the first preset merging method is used to merge the files, and for the table to be merged that is a non-partitioned file, the second preset merging method is used to merge the files.

[0045] For example, for example_table1 and example_table2, if example_table1 is a partitioned table, files of example_table1 are merged using a first preset merge method, and if example_table2 is a non-partitioned table, files of example_table2 are merged using a second preset merge method.

[0046] Among them, Figure 5 As shown, in step S23, that is, merging the files of the table to be merged according to the first preset merging method, the following steps are included:

[0047] S231: Confirm the first mean value of each sub-file included in the table to be merged.

[0048] S232: Merge the sub-files whose first mean values ​​are smaller than a preset threshold.

[0049] S233: Remove the subfiles whose first mean value is greater than or equal to the preset threshold from the subfiles to be merged.

[0050] When merging files, the small files that need to be merged can be selected based on the file size. After confirming the file type of the table to be merged, further confirm the file size of the table itself or the small files it contains. The first mean value refers to the mean value obtained by dividing the total size of a file by the number of all files contained in the file. When making a specific judgment, a preset threshold value can be set, and the table to be merged with the first mean value less than the preset threshold value will be merged, and the table to be merged with the first mean value greater than or equal to the preset threshold value will not be merged.

[0051] For example, the preset configuration table may include a merge_threshold field, which records a preset threshold value, and the recorded preset threshold value may be 268435456 bytes, that is, 256MB. When the file size is greater than or equal to 256MB, the file does not need to be merged, and when the file size is less than 256MB, the file needs to be merged. The preset threshold value is an empirical value, which can be adjusted according to the specific application scenario, including but not limited to 128MB and 512MB.

[0052] In actual use, you can connect to the Hadoop cluster according to the file name, and then obtain the file number, file size, and metadata information of the corresponding file on the Hadoop cluster according to the file name. For partitioned files, you can calculate the file size of each sub-file under the partitioned file and calculate the first mean of each sub-file. If the first mean is greater than or equal to the preset threshold, the file will not be merged. If

[0053] If the first mean is less than the preset threshold, the corresponding sub-files are merged. When merging files, multiple small files whose file sizes are less than the preset threshold can be merged, and small files whose file sizes are greater than the preset threshold can be split into multiple small files whose file sizes are less than the preset threshold, thereby ensuring that the sizes of the merged small files are close to the preset threshold.

[0054] Among them, Figure 6 As shown, in step S24, that is, merging the files of the table to be merged according to the second preset merging method, the following steps are included:

[0055] S241: Confirm the second mean value of the table to be merged.

[0056] S242: If the second mean value is smaller than a preset threshold, merging files of the table to be merged.

[0057] The second mean can be obtained by dividing the total size of the table to be merged by the number of files of all files included in the table to be merged. When the second mean is less than a preset threshold, the table to be merged is merged to ensure that the file size of each file in the table to be merged is close to the preset threshold.

[0058] Among them, Figure 7 As shown, Figure 7 A schematic diagram of a sub-flow diagram of a small file merging method provided in an embodiment of the present invention includes the following steps:

[0059] S30: Acquire a first file parameter of the table to be merged before the files are merged and a second file parameter of the table to be merged after the files are merged.

[0060] S40: Generate a log record table according to the first file parameter and the second file parameter.

[0061] The first file parameter refers to the parameter of the table to be merged before the file merge is performed, and the second file parameter refers to the parameter of the table to be merged after the file merge is performed. For example, the first file parameter may include the number of files before the merge and the mean value of the files before the merge, and the second file parameter may include the number of files after the merge, the mean value of the files after the merge, and the merge time. After obtaining the first file parameter and the second file parameter, a log record table is generated according to the first file parameter and the second file parameter to facilitate monitoring and maintenance by developers.

[0062] It can be seen that in the above scheme, for small files in the distributed file system, a matching preset method can be selected according to different file types to merge files, without the need for manual merging by staff, thereby improving the efficiency of file merging.

[0063] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0064] In one embodiment, a small file merging device is provided, and the small file merging device corresponds to the small file merging method in the above embodiment. Figure 8 As shown, the small file merging device includes a parsing module 101 and a merging module 102. Each functional module is described in detail as follows:

[0065] The parsing module 101 is used to be controlled by a merging instruction for merging small files, and parse the merging instruction to obtain a preset configuration table;

[0066] The merging module 102 is used to obtain a table to be merged according to the preset configuration table, and select a suitable preset method to merge files according to the file type of the table to be merged.

[0067] In one embodiment, the merging module 102 is specifically configured to:

[0068] Traversing the preset configuration table to confirm the target field, and confirming the file name recorded in the target field;

[0069] The hive table that needs to be merged is confirmed according to the file name and is set as the table to be merged.

[0070] In one embodiment, the merging module 102 is further configured to:

[0071] If the file type of the table to be merged is a partition file, merging the files of the table to be merged according to a first preset merging method;

[0072] If the file type of the table to be merged is a non-partitioned file, the table to be merged is merged according to a second preset merging method, wherein the preset method includes the first preset merging method and the second preset merging method.

[0073] In one embodiment, the merging module 102 is specifically configured to:

[0074] Determine the first mean value of each sub-file included in the table to be merged;

[0075] Merging subfiles whose first mean value is less than a preset threshold;

[0076] The subfiles whose first mean value is greater than or equal to the preset threshold are removed from the subfiles to be merged.

[0077] In one embodiment, the merging module 102 is further configured to:

[0078] Confirming the second mean of the table to be merged;

[0079] If the second mean value is smaller than a preset threshold, the table to be merged is subjected to file merging.

[0080] In one embodiment, the small file merging device further includes an acquisition module and a generation module. The functional modules are described in detail as follows:

[0081] An acquisition module, used for acquiring a first file parameter of the table to be merged before the files are merged and a second file parameter after the files are merged;

[0082] A generating module is used to generate a log record table according to the first file parameter and the second file parameter.

[0083] The present invention provides a small file merging device, which can select matching preset methods according to different file types to merge small files in a distributed file system, eliminating the need for manual merging by staff, thereby improving the efficiency of file merging.

[0084] For the specific definition of the small file merging device, please refer to the definition of the small file merging method above, which will not be repeated here. The various modules in the above-mentioned small file merging device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0085] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a small file merging method server side.

[0086] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a small file merging method

[0087] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0088] Controlled by a merging instruction for merging small files, parsing the merging instruction to obtain a preset configuration table;

[0089] The table to be merged is obtained according to the preset configuration table, and a suitable preset method is selected to merge the files according to the file type of the table to be merged.

[0090] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0091] Controlled by a merging instruction for merging small files, parsing the merging instruction to obtain a preset configuration table;

[0092] The table to be merged is obtained according to the preset configuration table, and a suitable preset method is selected to merge the files according to the file type of the table to be merged.

[0093] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0094] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0095] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0096] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A small file merging method, characterized in that: include: Controlled by a merging instruction for merging small files, parsing the merging instruction to obtain a preset configuration table; The table to be merged is obtained according to the preset configuration table, and a suitable preset method is selected to merge the files according to the file type of the table to be merged.

2. The method according to claim 1, characterized in that The obtaining the table to be merged according to the preset configuration table includes: Traversing the preset configuration table to confirm the target field, and confirming the file name recorded in the target field; The hive table that needs to be merged is confirmed according to the file name and is set as the table to be merged.

3. The method according to claim 1, characterized in that The step of selecting a preset method suitable for merging files according to the file type of the table to be merged includes: If the file type of the table to be merged is a partition file, merging the files of the table to be merged according to a first preset merging method; If the file type of the table to be merged is a non-partitioned file, the table to be merged is merged according to a second preset merging method, wherein the preset method includes the first preset merging method and the second preset merging method.

4. The method according to claim 3, characterized in that The step of merging the files of the table to be merged according to the first preset merging method includes: Determine the first mean value of each sub-file included in the table to be merged; The sub-files whose first mean value is less than a preset threshold are merged.

5. The method according to claim 4, characterized in that The step of merging the files of the table to be merged according to the first preset merging method further includes: The subfiles whose first mean value is greater than or equal to the preset threshold are removed from the subfiles to be merged.

6. The method according to claim 3, characterized in that The step of merging the files of the table to be merged according to the second preset merging method includes: Confirming the second mean of the table to be merged; If the second mean value is smaller than a preset threshold, the table to be merged is subjected to file merging.

7. The method according to claim 1, characterized in that The method further comprises: Obtain a first file parameter of the table to be merged before the files are merged and a second file parameter of the table to be merged after the files are merged; A log record table is generated according to the first file parameter and the second file parameter.

8. A small file merging device, characterized in that: include: A parsing module, which is controlled by a merging instruction for merging small files, and parses the merging instruction to obtain a preset configuration table; The merging module is used to obtain the table to be merged according to the preset configuration table, and select a suitable preset method to merge the files according to the file type of the table to be merged.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the small file merging method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the small file merging method as claimed in any one of claims 1 to 7 are implemented.

Images