Method and apparatus for processing small files based on HDFS

Abstract

The application provides a small file processing method and device based on HDFS, which can be used in the financial field or other fields. The method comprises the following steps: obtaining a target text file, the target text file comprising: a plurality of data table records corresponding to a target database cluster, each data table record comprising: a database name and a data table name of a uniquely corresponding data table, different data table records corresponding to different data tables, and at least one small file contained in an HDFS corresponding to each data table; and obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, a database name and a data table name corresponding to each data table. The application can relieve the storage pressure of HDFS, improve the efficiency of small file merging, and thus improve the operation efficiency of the database cluster.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a small file processing method and device based on HDFS. BACKGROUND

[0002] Hive is a database warehouse tool based on Hadoop, which is used for data extraction, transformation and loading; Hadoop Distributed File System (HDFS) is a distributed file system suitable for running on commodity hardware, which is used to store all files generated by Hive.

[0003] In actual application, a large number of small files are generated due to the mechanism of Hive, and storing a large number of small files will cause the memory pressure of the Namenode server in HDFS, thereby affecting the read-write speed of the small files. SUMMARY

[0004] In view of the problems in the prior art, the present application provides a small file processing method and device based on HDFS, which can relieve the storage pressure of HDFS, improve the efficiency of small file merging, and further improve the operation efficiency of a database cluster.

[0005] In order to solve the above technical problems, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a small file processing method based on HDFS, comprising:

[0007] obtaining a target text file, wherein the target text file comprises a plurality of data table records corresponding to a target database cluster, each data table record comprises a database name and a data table name of a uniquely corresponding data table, different data table records correspond to different data tables, and each data table corresponds to at least one small file in an HDFS;

[0008] obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, the database name and the data table name of each data table;

[0009] applying the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table.

[0010] Further, before the step of obtaining the small file merging statement corresponding to each data table according to the preset HQL statement template, the database name and the data table name of each data table, the method further comprises:

[0011] obtaining a storage format of each data table;

[0012] According to the storage format of each data table, filter out the to-be-filtered data table record in the target text file, which is the data table record corresponding to the data table in non-RC and non-ORC storage format.

[0013] Further, the application of each data table corresponding to the small file merge statement, the execution of the small file merge operation corresponding to the data table, comprising:

[0014] The application of each data table corresponding to the small file merge statement, concurrent execution of the small file merge operation corresponding to different partitions in the same data table, each data table comprising: at least one partition.

[0015] Further, before the application of each data table corresponding to the small file merge statement, concurrent execution of the small file merge operation corresponding to different partitions in the same data table, further comprising:

[0016] According to the preset standard partition key, the partition key of each data table and the partition key data, the partition of the data table is filtered.

[0017] Further, the target text file is obtained, comprising:

[0018] Obtain the data table list file corresponding to the target database cluster;

[0019] The data table list file is divided into a plurality of target text files, and the data table records in the plurality of target text files constitute a plurality of data table records corresponding to the target database cluster;

[0020] Correspondingly, the application of each data table corresponding to the small file merge statement, the execution of the small file merge operation corresponding to the data table, comprising:

[0021] The application of each data table corresponding to the small file merge statement, concurrent execution of the small file merge operation corresponding to different target text files.

[0022] Secondly, the application provides a small file processing device based on HDFS, comprising:

[0023] The acquisition module acquires the target text file, which includes a plurality of data table records corresponding to the target database cluster, each data table record including the database name and the data table name of the corresponding data table, and the data tables corresponding to different data table records being different, and at least one small file being contained in the HDFS corresponding to each data table;

[0024] The generation module is configured to obtain the small file merge statement corresponding to each data table according to the preset HQL statement template, the database name and the data table name corresponding to each data table;

[0025] an execution module, configured to apply the small file merging statement corresponding to each data table to perform small file merging operation corresponding to the data table.

[0026] Further, the small file processing device based on HDFS further comprises:

[0027] a storage format acquisition module, configured to acquire the storage format of each data table;

[0028] a filtering module, configured to filter out the data table record to be filtered in the target text file according to the storage format of each data table, the data table record to be filtered being the data table record corresponding to the data table in non-RC and non-ORC storage format.

[0029] Further, the execution module comprises:

[0030] a concurrent execution unit, configured to apply the small file merging statement corresponding to each data table to concurrently perform small file merging operation corresponding to each partition in the same data table, each data table comprising at least one partition.

[0031] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the small file processing method based on HDFS when executing the program.

[0032] In a fourth aspect, the present application provides a computer readable storage medium, having computer instructions stored thereon, wherein the instructions are executed to implement the small file processing method based on HDFS.

[0033] As can be seen from the above technical solutions, the present application provides a small file processing method and device based on HDFS. The method comprises: acquiring a target text file, the target text file comprising: a plurality of data table records corresponding to a target database cluster, each data table record comprising: a database name and a data table name of a corresponding data table, different data table records corresponding to different data tables, and at least one small file contained in the HDFS corresponding to each data table; obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, the database name and the data table name of the data table; and applying the small file merging statement corresponding to each data table to perform small file merging operation corresponding to the data table. This can relieve the storage pressure of HDFS, improve the efficiency of small file merging, and further improve the operation efficiency of the database cluster. Specifically, it can solve the problem of excessive number of Hadoop bottom layer files, and improve the operation efficiency of the database cluster. The method has the advantages of high merging efficiency, fast speed, complete transparency to upper layer applications, stable and reliable execution process, and the like. The merging method is flexible, has low coupling degree with external systems, is simple to configure, and is easy to maintain. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the small file processing method based on HDFS in an embodiment of this application;

[0036] Figure 2 This is a flowchart illustrating a small file processing method based on HDFS according to another embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating a small file processing method based on HDFS in another embodiment of this application;

[0038] Figure 4 This is a flowchart illustrating the small file processing method based on HDFS in an application example of this application;

[0039] Figure 5 This is a schematic diagram of the structure of the HDFS-based small file processing device in the embodiments of this application;

[0040] Figure 6 This is a logical diagram of the small file processing system based on HDFS in the application example of this application;

[0041] Figure 7 This is a schematic block diagram illustrating the system configuration of an electronic device according to an embodiment of this application. Detailed Implementation

[0042] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0043] In the data warehouse, the higher the data table is, the higher the degree of aggregation is, and the smaller the data quantity is; these data tables can be partitioned according to the date, and the number of files in the HDFS will gradually increase over time. In the prior art, there are two ways to merge small files in the HDFS: (1) inventory merging, querying each partition of each data table, and sequentially completing merging for each partition in the data table; (2) incremental merging, setting a file data threshold, and after the Map and Reduce programs are executed, the number of files is counted, and if the number of files reaches the file data threshold, the small file merging is triggered to realize incremental small file merging.

[0044] The above two ways have the following pain points respectively: (1) for inventory small file merging, each time only a single table can be serially completed for each partition, and when the number of partitions in the data table is too large, the time for completing inventory merging for a single table is too long; (2) for incremental merging, by setting a file data threshold, the effect of realizing incremental merging is not obvious from the actual situation, because some jobs will shield some parameter settings of the system layer.

[0045] In order to solve the problems in the prior art, the present application considers changing the existing small file merging method, and provides a small file processing method and device based on HDFS, relating to the Hive service of the Hadoop distributed database and big data technology, fully utilizing the maximum number of processes and server resources of the system, solving the problem of slow serial merging of small files, and realizing high-speed merging of single-table small files by using single-table high concurrency.

[0046] It should be noted that the small file processing method and device based on HDFS disclosed in the present application can be used in the field of financial technology, and can also be used in any field other than the field of financial technology, and the application field of the small file processing method and device based on HDFS disclosed in the present application is not limited.

[0047] The embodiments are specifically described as follows.

[0048] In order to alleviate the storage pressure of the HDFS and improve the efficiency of small file merging, and thus improve the running efficiency of the database cluster, the present embodiment provides a small file processing method based on HDFS, and the execution subject is a small file processing device based on HDFS, which includes but is not limited to a server, such as Figure 1 As shown in the figure, the method specifically includes the following contents:

[0049] Step 100: obtaining a target text file, the target text file comprising: a plurality of data table records corresponding to a target database cluster, each data table record comprising: a database name and a data table name of a uniquely corresponding data table, different data table records corresponding to different data tables, and at least one small file contained in an HDFS corresponding to each data table.

[0050] Specifically, the target database cluster can be a Hadoop distributed database corresponding to a financial transaction system. For example, when data is inserted into a dynamic partition table in the Hadoop distributed database, and the number of reduce type data in the data table is large, the HDFS generates corresponding small files while processing the reduce type data. The small file in the present scheme can represent a file whose size is less than the size of a block in the HDFS.

[0051] Step 200: obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, a database name and a data table name corresponding to each data table.

[0052] Specifically, the data tables in the target database cluster can be traversed according to the database name and the data table name corresponding to each data table to determine the partition key corresponding to each data table and the partition data in each partition. A small file merging statement corresponding to each data table can be obtained according to the data table name and the partition key corresponding to each data table, the partition data, and a preset HQL statement template.

[0053] The preset HQL statement template can be: alter table table name partition(partition name) concatenate. For example, the database name corresponding to a data table A is abc, the partition key is pt_dt, and the partition data corresponding to the partition key in the data table A comprises: 2020-10-1, 2020-10-2, and 2020-10-3, then it is determined that the data table A comprises three partitions. The obtained small file merging statement comprises: “set mapred.min.split.size.per.node=100000000; set mapred.min.split.size.per.rack=100000000; alter table abc partition(pt_dt='2020-10-1') concatenate; alter table abc partition(pt_dt='2020-10-2') concatenate; alter table abc partition(pt_dt='2020-10-3') concatenate”.

[0054] Step 300: apply a small file merge statement corresponding to each data table to perform a small file merge operation corresponding to the data table.

[0055] To further improve the efficiency of small file processing, in an embodiment of the present application, referring to Figure 2 Before step 200, further comprising:

[0056] Step 021: obtain the storage format of each data table.

[0057] Step 022: filter out the to-be-filtered data table records in the target text file according to the storage format of each data table, the to-be-filtered data table records being the data table records corresponding to the data tables in non-RC (Record Columnar) and non-ORC (Optimized Row Columnar) storage format.

[0058] Specifically, the data table records corresponding to the data tables in RC and ORC storage format can be retained.

[0059] To further improve the efficiency of small file processing, in an embodiment of the present application, referring to

[0060] Step 301: apply a small file merge statement corresponding to each data table to perform a small file merge operation corresponding to each partition in the same data table, each data table comprising at least one partition.

[0061] For example, if one partition in a data table originally corresponds to 256 small files of 10 MB in size in HDFS, after the small file merge operation, the small files corresponding to the partition can be merged into 10 files of 256 MB in size, and the number of data files in the partition is greatly reduced.

[0062] To further improve the efficiency of small file processing, in an embodiment of the present application, before step 301, further comprising:

[0063] Step 003: filter the partitions of the data table according to the preset standard partition key, the partition key of each data table, and the partition key data.

[0064] Specifically, the preset standard partition key can be set according to actual conditions, and the application does not limit this, for example, a timestamp; the partition key of the partition in which the data record can be kept as the preset standard partition key is the same; for example, the data table can be filtered out to eliminate temporary partitions (such as partitions with a timestamp of 9999-01-01), future partitions (such as partitions with a timestamp of 3000-01-01), partitions of the last 7 days (such as partitions with a timestamp within seven days before the program runs), partitions with a partition key other than a timestamp, and partitions with an illegal timestamp (such as partitions with a timestamp of ABCD-EF-GH).

[0065] In order to improve the efficiency of the small file merging operation corresponding to the plurality of text files, see Figure 3 In an embodiment of the application, step 100 comprises:

[0066] Step 110: Obtain a data table list file corresponding to the target database cluster.

[0067] Step 120: Divide the data table list file into a plurality of target text files, and the data table records in the plurality of target text files constitute a plurality of data table records corresponding to the target database cluster.

[0068] Specifically, the split command line program can be applied to divide the data table list file into a specified number of target text files, and the access mode is simple and the execution process is stable.

[0069] Correspondingly, step 300 comprises: step 310: applying a small file merging statement corresponding to each data table, and concurrently executing a small file merging operation corresponding to different target text files.

[0070] Specifically, the data table records of data table A and data table B are stored in different target text files, and the small file merging statements corresponding to data table A and data table B can be concurrently executed.

[0071] Further, the availability of the HiveServer server in the database cluster can be monitored in real time, and if it is not available, the current operation is terminated until the HiveServer server is normal.

[0072] In order to further illustrate the present scheme, see Figure 4 The application provides an application example of a small file processing method based on HDFS, which is specifically described as follows:

[0073] Step 1: Write the database list expected to be merged in advance into a first text file; the database names in the first text file are separated by a carriage return line.

[0074] Step 2: Legitimacy detection is performed on the database name in the first text file, and the legitimacy detection includes: judging whether the database name contains special characters, if not and the database name in the first text file does not belong to the database cluster, an error is displayed in the log.

[0075] Step 3: A plurality of second text files are generated; according to the database name in the first text file, all data tables in the database are queried, and data table records are written into the second text file with the corresponding database name as the file name, each data table record includes: a unique corresponding database name, a data table name and storage format information; the data table records in the same second text file correspond to the same database name, and the data table records in different second text files correspond to different database names.

[0076] Specifically, the show tables command can be applied to view the data table, and the data table viewed by the show tables command can contain views.

[0077] Step 4: A third text file is generated and is divided into a plurality of fourth text files; all second text files are combined into a third text file, which can contain: respective data table records of each data table in the database cluster, which can be equivalent to the above-mentioned data table list file, the number of rows of the third text file is detected, and it is automatically divided into a plurality of fourth text files, and a concurrent query operation is performed on each fourth text file, and the fourth text file can be equivalent to the above-mentioned target text file.

[0078] Step 5: Filtering is performed on each fourth text file; data table records corresponding to data tables of non-RC and non-ORC storage formats are filtered out.

[0079] Step 6: The fourth text file is filtered again; according to the filtered fourth text file, a plurality of fifth text files are generated; the file name of the fifth text file includes: a database name and a data table name; each fifth text file corresponds to a unique data table, and different fifth text files correspond to different data tables; each fifth text file contains a unique record, which includes: a database name, a data table name, beeline prompt information and a partition table identifier.

[0080] Step 7: Filtering is performed on the fifth text file; if there is a fifth text file with a partition table identifier of error, the fifth text file is filtered out.

[0081] Specifically, the beeline prompt information can be some connection information of the beeline command line client tool connected to the hiveserver, when the show partitions table_name command is applied to view the partition information of the data table, if the viewed data table is a view, the beeline command line client tool will automatically return an error, the error can indicate that the table is a view, when the fifth text file is formatted, the records containing the error are filtered out, and at this time the file size of the fifth text file is 0; when a non-partition table is encountered, the beeline command line client tool will automatically return an error, the error can indicate that the table is not partitioned, that is, a non-partition table, and the records containing the error are filtered out, and at this time the file size of the fifth text file is 0.

[0082] Step 8: According to the filtered fifth text file, batch generate HQL small file merging statement; at the same time, eliminate the temporary partition, future partition and the partition of the last 7 days in the data table corresponding to the fifth text file, and eliminate the partition that is not a timestamp and the partition that is not a legal timestamp, to obtain a total task list, the total task list corresponds to all data tables corresponding to the filtered fifth text file, and the total task list contains the data table name and database name of all data tables that need to be merged.

[0083] Step 9: According to the HQL small file merging statement, implement the small file merging operation line by line on the total task list.

[0084] Specifically, while performing the current table corresponding to the small file merging operation, it is detected whether the next data table corresponding to the total task list is called by the database cluster, if yes, the data table name and database name of the next data table are recorded, and the next data table is skipped and waits to be automatically called at the idle time in the evening, if no, the small file merging operation is performed according to the total task list.

[0085] According to the different number of partitions of each table, concurrent sharding is automatically completed, a background startup script is spliced, and the script is started to implement batch small file merging in the background; that is, the current table is automatically sharded into multiple concurrent processes, it is detected whether the small file merging corresponding to the current data table has been completed, if yes, the merging small file operation corresponding to the next data table is performed.

[0086] Step 01: Verify the small file merging of each data table, and count the number of files before and after the small file merging of each table.

[0087] From the software level, in order to alleviate the storage pressure of HDFS, improve the efficiency of small file merging, and then improve the running efficiency of the database cluster, an embodiment of a small file processing device based on HDFS for implementing all or part of the contents of the small file processing method based on HDFS is provided, see Figure 5 The small file processing device based on HDFS specifically includes the following contents:

[0088] The acquisition module 10 acquires a target text file, which includes a plurality of data table records corresponding to a target database cluster, each data table record including a database name and a data table name of a uniquely corresponding data table, different data table records corresponding to different data tables, and at least one small file contained in an HDFS corresponding to each data table;

[0089] The generation module 20 is configured to obtain a small file merging statement corresponding to each data table according to a preset HQL statement template, a database name and a data table name corresponding to each data table;

[0090] The execution module 30 is configured to apply the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table.

[0091] In an embodiment of the present application, the small file processing device based on HDFS further includes:

[0092] The storage format acquisition module is configured to acquire a storage format of each data table;

[0093] The filtering module is configured to filter out a to-be-filtered data table record in the target text file according to the storage format of each data table, the to-be-filtered data table record being a data table record corresponding to a data table in a non-RC and non-ORC storage format.

[0094] In an embodiment of the present application, the execution module includes:

[0095] The concurrent execution unit is configured to apply the small file merging statement corresponding to each data table to concurrently perform a small file merging operation corresponding to each partition in the same data table, each data table including at least one partition.

[0096] The embodiment of the small file processing device based on HDFS provided in the specification can be specifically used to execute the processing flow of the above-described embodiment of the small file processing method based on HDFS, and the functions thereof will not be repeated here, and can be referred to the detailed description of the above-described embodiment of the small file processing method based on HDFS.

[0097] In order to further illustrate the present application, an application example of a small file processing system based on HDFS is provided, see Figure 6, specifically comprising the following contents:

[0098] A Hadoop database management system, a small file processing device and an application server.

[0099] The Hadoop database management system comprises a plurality of HiveServer servers and a HDFS distributed file system; in a typical scenario, the plurality of HiveServer servers and the HDFS distributed file system constitute main components of the Hadoop database management system, the HiveServer servers parse and execute Hive SQL statements and store the results in the HDFS distributed file system; wherein the functions implemented by the Hadoop database management system can be equivalent to the functions implemented by the above-mentioned target database cluster.

[0100] The application server, on which an application program can be deployed, is installed with a Hadoop cluster client, and the small file processing device can submit corresponding tasks to the database cluster by calling commands in the Hadoop cluster client.

[0101] The small file processing device can communicate with the HiveServer servers through the Hadoop cluster client in a port communication manner, submit small file merging statements to the HiveServer servers, and detect the availability of the HiveServer servers; the Hadoop cluster client and the HiveServer servers communicate through an SSH protocol or an RPC service, submit small file merging statements of a single table one by one according to a program, and control concurrent processes according to the number of partitions of each table to realize high-speed merging of partitions of a single table.

[0102] The small file processing device can comprise an arbitration master node, an arbitration slave node, an execution master node, an execution backup node and an execution slave node; the arbitration master node is communicatively connected with the arbitration slave node, the execution master node, the execution backup node and the execution slave node, the arbitration slave node is communicatively connected with the execution master node, the execution backup node and the execution slave node, and the arbitration master node, the arbitration slave node, the execution master node, the execution backup node and the execution slave node can represent different servers, which can be deployed at different locations.

[0103] Before starting the small file processing based on HDFS, the arbitration master node is connected, the arbitration backup node is connected if the arbitration master node cannot be connected, the delay of the execution master node, the execution backup node and the execution slave node is detected after the arbitration master node is successfully connected, and the execution master node, the execution backup node and the execution slave node are deployed in three parks to realize high availability.

[0104] The arbitration node connected successfully preferentially connects the execution master node, and when the delay detection is more than 1500 milliseconds, it can be considered as an exception, and the exception scenario includes: about to be down and network instantaneous interruption, etc., if the exception, the delay conditions of the execution standby node and the execution slave node are detected, a node with lower delay is selected to be connected, the execution file in the execution master node can be synchronized to the execution standby node and the execution slave node in real time, and the consistency of data in the execution master node, the execution standby node and the execution slave node is ensured.

[0105] From the above description, the small file processing method and device based on HDFS provided by the application can relieve the HDFS storage pressure, improve the efficiency of small file merging, and further improve the database cluster operation efficiency; specifically, the problem of too many Hadoop bottom layer files can be solved, and the database cluster operation efficiency can be improved; it has the advantages of high merging efficiency, fast speed, complete transparency to upper layer application, stable and reliable execution process, etc.; the merging mode is flexible, the coupling degree with external system is low, the configuration is simple, and the maintenance is convenient; single table intelligent concurrent merging, transparent merging to upper layer application and failure retry merging can be realized.

[0106] From the hardware level, in order to relieve the HDFS storage pressure, improve the efficiency of small file merging, and further improve the database cluster operation efficiency, an embodiment of an electronic device for implementing all or part of the contents of the small file processing method based on HDFS is provided, and the electronic device specifically includes the following contents:

[0107] A processor, a memory, a communications interface and a bus; wherein the processor, the memory and the communications interface complete the communication among each other through the bus; the communications interface is used for realizing the information transmission between the small file processing device based on HDFS and related devices such as user terminals; the electronic device can be a desktop computer, a tablet computer and a mobile terminal, etc., and the embodiment is not limited thereto. In the embodiment, the electronic device can be implemented by referring to the embodiment for implementing the small file processing method based on HDFS and the embodiment for implementing the small file processing device based on HDFS, the contents of which are incorporated herein, and the repeated parts will not be described again.

[0108] Figure 7 The schematic block diagram of the system structure of the electronic device 9600 of the embodiment of the application is shown in FIG. 9. Figure 7 As shown in the figure, the electronic device 9600 can include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. It is worth noting that the Figure 7are exemplary; other types of structures can also be used to supplement or replace the structure to implement telecommunication functions or other functions.

[0109] In one or more embodiments of the present application, the HDFS-based small file processing function can be integrated into the central processor 9100. Wherein the central processor 9100 can be configured to control as follows:

[0110] Step 100: Obtain a target text file, the target text file comprising: a plurality of data table records corresponding to a target database cluster, each data table record comprising: a database name and a data table name of a uniquely corresponding data table, different data table records corresponding to different data tables, and at least one small file contained in an HDFS corresponding to each data table;

[0111] Step 200: According to the preset HQL statement template, the database name and the data table name corresponding to each data table, obtain the small file merging statement corresponding to the data table;

[0112] Step 300: Apply the small file merging statement corresponding to each data table to execute the small file merging operation corresponding to the data table.

[0113] From the above description, it can be seen that the electronic device provided by the embodiments of the present application can relieve the HDFS storage pressure, improve the efficiency of small file merging, and further improve the operation efficiency of the database cluster.

[0114] In another embodiment, the HDFS-based small file processing device can be configured separately from the central processor 9100, for example, the HDFS-based small file processing device can be configured as a chip connected with the central processor 9100, and the HDFS-based small file processing function is realized through the control of the central processor.

[0115] As shown in Figure 7 , the electronic device 9600 can also include a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily include all the components shown in Figure 7 ; in addition, the electronic device 9600 can also include components not shown in Figure 7 , which can refer to prior art.

[0116] As shown in Figure 7 , the central processor 9100, also known as a controller or operation control, can include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of each component of the electronic device 9600.

[0117] The memory 9140, for example, can be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The above-mentioned information related to failure can be stored, and in addition, a program for executing the information related to failure can be stored. The central processing unit 9100 can execute the program stored in the memory 9140 to achieve information storage or processing, and the like.

[0118] The input unit 9120 provides input to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and characters. The display can be, for example, an LCD display, but is not limited thereto.

[0119] The memory 9140 can be a solid state memory such as a read only memory (ROM), a random access memory (RAM), a SIM card, and the like. It can also be a memory that retains information even when power is off, can be selectively erased, and is provided with more data, and examples of such a memory are sometimes referred to as an EPROM, and the like. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 can include an application / function storage section 9142 for storing application programs and function programs or for storing a flow for executing operations of the electronic device 9600 by the central processing unit 9100.

[0120] The memory 9140 can also include a data storage section 9143 for storing data such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. A driver storage section 9144 of the memory 9140 can include various drivers of the electronic device for a communication function and / or for executing other functions of the electronic device such as a messaging application, an address book application, and the like.

[0121] The communication module 9110 is a transmitter / receiver 9110 that transmits and receives signals via an antenna 9111. The communication module (transmitter / receiver) 9110 is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0122] Based on different communication technologies, multiple communication modules 9110, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc., can be provided in the same electronic device. The communication module (transmitter / receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132, thereby enabling typical telecommunication functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers, etc. In addition, the audio processor 9130 is also coupled to the central processor 9100, thereby enabling the recording of audio on the local device via the microphone 9132 and enabling the playing of stored audio on the local device via the speaker 9131.

[0123] As described above, the electronic device provided by the embodiments of the present application can relieve the HDFS storage pressure, improve the efficiency of small file merging, and further improve the operation efficiency of the database cluster.

[0124] The embodiments of the present application further provide a computer readable storage medium capable of implementing all steps of the HDFS-based small file processing method in the above embodiments. The computer readable storage medium stores a computer program. When the computer program is executed by a processor, all steps of the HDFS-based small file processing method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0125] Step 100: obtaining a target text file, the target text file including: a plurality of data table records corresponding to a target database cluster, each data table record including: a database name and a data table name of a uniquely corresponding data table, different data table records corresponding to different data tables, and at least one small file contained in an HDFS corresponding to each data table;

[0126] Step 200: obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, a database name and a data table name corresponding to each data table;

[0127] Step 300: applying the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table.

[0128] As described above, the computer readable storage medium provided by the embodiments of the present application can relieve the HDFS storage pressure, improve the efficiency of small file merging, and further improve the operation efficiency of the database cluster.

[0129] The above-described methods are each described in the present application in progressive manner, and the same or similar parts among the embodiments can be mutually referred to, and each embodiment mainly describes the difference from other embodiments. The related parts can be referred to the part of the method embodiment.

[0130] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-readable program code.

[0131] The present application is described herein with reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0132] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0133] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 Figure 1 means for performing each of the functions specified in the flowchart illustrations and / or block diagrams.

[0134] The principles and implementation manners of the present application are described in the specific embodiments in the present application, and the above embodiment descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for the general technical personnel in the art, according to the idea of the present application, the specific implementation manners and application ranges will be changed, and the above is not understood as the limitation of the present application.

Claims

1. A small file processing method based on HDFS, characterized in that, The method comprises the following steps: obtaining a target text file, wherein the target text file comprises a plurality of data table records corresponding to a target database cluster, each data table record comprises a database name and a data table name of a data table corresponding thereto, and different data table records correspond to different data tables, and at least one small file is contained in an HDFS corresponding to each data table; obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, the database name and the data table name corresponding to each data table; applying the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table; wherein, when performing the merging operation, each data table to be processed is detected and scheduled according to a preset standard, and concurrent sharding is automatically completed according to the number of partitions of each data table; wherein, before performing the merging operation, the method further comprises filtering the partitions of the data table according to a preset standard partition key, the partition key and the partition key data of each data table.

2. The HDFS-based small file processing method of claim 1, wherein, Before the step of obtaining a small file merging statement corresponding to each data table according to a preset HQL statement template, the database name and the data table name corresponding to each data table, the method further comprises: obtaining a storage format of each data table; filtering out a data table record to be filtered in the target text file according to the storage format of each data table, wherein the data table record to be filtered is a data table record corresponding to a data table in a non-RC and non-ORC storage format.

3. The HDFS-based small file processing method of claim 1, wherein, The step of applying the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table comprises: applying the small file merging statement corresponding to each data table to concurrently perform a small file merging operation corresponding to each partition in the same data table, and each data table comprises at least one partition.

4. The HDFS-based small file processing method of claim 1, wherein, The step of obtaining a target text file comprises: obtaining a data table list file corresponding to a target database cluster; dividing the data table list file into a plurality of target text files, and data table records in the plurality of target text files form a plurality of data table records corresponding to the target database cluster; correspondingly, the step of applying the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table comprises: applying the small file merging statement corresponding to each data table to concurrently perform a small file merging operation corresponding to different target text files.

5. A small file processing apparatus based on HDFS, characterized by, The method comprises the following steps: an obtaining module is configured to obtain a target text file, wherein the target text file comprises a plurality of data table records corresponding to a target database cluster, each data table record comprises a database name and a data table name of a data table corresponding thereto, and different data table records correspond to different data tables, and at least one small file is contained in an HDFS corresponding to each data table; a generating module is configured to obtain a small file merging statement corresponding to each data table according to a preset HQL statement template, the database name and the data table name corresponding to each data table; an executing module is configured to apply the small file merging statement corresponding to each data table to perform a small file merging operation corresponding to the data table. The device is further configured to detect and schedule each data table to be processed according to a preset standard when performing the merging operation, and automatically complete concurrent sharding according to the number of partitions of each data table. Before performing the merging operation, the device is further configured to filter the partitions of the data table according to a preset standard partition key, the partition key of each data table, and the partition key data.

6. The HDFS-based small file processing apparatus according to claim 5, wherein, Further comprising: An acquisition storage format module configured to acquire the storage format of each data table. A filtering module configured to filter out, according to the storage format of each data table, the to-be-filtered data table records in the target text file, the to-be-filtered data table records being the data table records corresponding to the data tables in non-RC and non-ORC storage formats.

7. The HDFS-based small file processing apparatus according to claim 5, wherein, The execution module comprises: A concurrent execution unit configured to apply the small file merging statements corresponding to each data table, and concurrently execute the small file merging operations corresponding to different partitions in the same data table, each data table comprising at least one partition.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the HDFS-based small file processing method of any one of claims 1 to 4.

9. A computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed to implement the HDFS-based small file processing method of any one of claims 1 to 4.

Images