A spark-hbase batch data rapid warehousing implementation method and system based on parameter configuration

Abstract

This invention discloses a method and system for rapid batch data import into HBase based on parameter configuration, relating to the field of database technology. The method includes: configuring a data import and merging strategy for HBase tables in scenarios where batch data calculated by the Spark engine is imported into an HBase database; performing calculations on business data using the Spark engine, writing the generated dataset to HDFS and recording it in an information table; continuously adding HDFS file directories according to the file generation time order of the information table, and submitting an MapReduce task to generate an HFile file when the requirements of the data import and merging strategy are met; and importing the HFile file into the HBase database. This invention avoids the problem of too many small files in HBase, improves the read and write performance of the HBase database, and reduces the time spent on business calculations.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method for rapid batch data import into Spark-HBase based on parameter configuration, and a system for rapid batch data import into Spark-HBase based on parameter configuration. Background Technology

[0002] Currently, in Spark processing of batch offline and small batch real-time streaming data, when the data processing results need to be stored in an HBase database, the spark-bulkload method is generally used.

[0003] like Figure 1 As shown, spark-bulkload is the assembly of HBase datasets after Spark has completed computation and produced the resulting RDD dataset.<rowkey,cell> The RDD dataset is formatted, sorted, and written to HDFS (Hadoop Distributed File System) to generate an hfile file. Finally, the hfile file is loaded into the database using HBase's bulk load method.

[0004] The advantage of using this method to first generate an hfile and then load it via bulkload is that it enables file-level batch operations, avoiding the access pressure on the HBase data server regionserver and the significant performance overhead of I / O and CPU caused by large-scale single-row or batch put operations.

[0005] While spark-bulkload enables file-level batch data import into HBase, HBase tables are typically designed with pre-partitioning, resulting in 20-200 partitions depending on the cluster size. This process presents two main problems:

[0006] (1) Spark's RDD dataset is not generated according to the partition of the HBase table by default. This will result in the start and end rowkeys of the generated hfile files not being in a certain partition. When inserting data into the HBase table, the hfile will be frequently split and smaller hfile files will be generated, which will affect the data insertion efficiency and generate more small hfile files.

[0007] (2) Although HFile splitting can be avoided by customizing Spark RDD data partitioning to HBase data partitioning, if the data processing volume of the current task batch is small, distributing it across partitions of, for example, 200 HBase tables will still generate a large number of small HFile files.

[0008] Small files are defined as files stored on HDFS that are significantly smaller than data blocks (default 128MB), such as 10MB files. Firstly, HDFS itself is not suitable for storing a large number of small files. Too many small files will result in excessively large NameNode metadata, consuming too much memory and severely impacting HDFS performance.

[0009] Secondly, for HBase, too many small files will affect query performance and will also cause HBase to frequently perform automatic hfile merging operations, which will seriously affect the read and write performance of the HBase database. Summary of the Invention

[0010] To address the aforementioned issues, this invention provides a method and system for rapid batch data import into Spark-HBase based on parameter configuration. By determining the data import and merging strategy through parameter configuration, Spark no longer directly generates HFile files or imports data into HBase. Instead, it dynamically calculates and reduces the number of partitions before writing to HDFS. The MapReduce task continuously adds source data until the strategy is met, merging and generating HFiles before importing them into HBase. This effectively utilizes Spark's in-memory computing and MapReduce's massive data processing capabilities, avoiding the problem of excessive small files in HBase, improving the read and write performance of the HBase database, and reducing business computation time.

[0011] To achieve the above objectives, this invention provides a method for rapid batch data import into Spark-HBase based on parameter configuration, including:

[0012] Configure the data merging strategy for HBase tables to handle scenarios where batch data computed by the Spark engine is imported into an HBase database.

[0013] The Spark engine is used to perform calculations on business data, and the resulting dataset is written to HDFS and recorded in an information table.

[0014] Based on the file generation time sequence of the information table, the HDFS file directory is continuously added. When the requirements of the data entry and merging strategy are met, the MR task is submitted to generate the hfile file.

[0015] The hfile file is imported into the hbase database.

[0016] In the above technical solution, preferably, the method for rapid batch data import into Spark-HBase based on parameter configuration also includes:

[0017] After the hfile file is imported into the hbase database, the result dataset of the hdfs is cleared.

[0018] In the above technical solution, preferably, the configuration information of the data entry and merging strategy includes the HBase table name, the amount of data in each partition when writing the HDFS file, the minimum amount of merged data to generate the HFile file, the maximum amount of merged data to generate the HFile file, the maximum allowable delay time for data entry, and the scan interval.

[0019] In the above technical solution, preferably, the specific process of writing the generated result dataset into HDFS and recording it in the information table includes:

[0020] The result dataset does not need to be assembled into a preset format. The number of partitions of the result dataset is dynamically calculated and reduced. Based on the number of partitions, the result dataset is directly written to HDFS without sorting and recorded in the information table of this batch. The information table includes transaction history, table name, HDFS file directory, data volume, status and generation time.

[0021] In the above technical solution, preferably, an asynchronous scanning application is started, the information table is scanned according to the data entry and merging strategy, and the HDFS file directory to be processed is continuously added according to the file generation time order of the information table until the requirements of the data entry and merging strategy are met. Then, the MR task is submitted. If not, the next polling is performed.

[0022] The MR task merges multiple data sources to generate an hfile file, and loads the hfile file into the hbase database using the bulkload method.

[0023] This invention also proposes a system for rapid batch data import into Spark-HBase based on parameter configuration, applying the parameter-configured method for rapid batch data import into Spark-HBase disclosed in any of the above technical solutions, including:

[0024] The data ingestion strategy configuration module is used to configure the data ingestion and merging strategy for HBase tables in scenarios where batch data calculated by the Spark engine is ingested into the HBase database.

[0025] The data writing and recording module is used to perform calculations on business data using the Spark engine, write the generated dataset to HDFS, and record it in an information table.

[0026] The file merging and generation module is used to continuously add HDFS file directories according to the file generation time order of the information table, and submit MR tasks to generate hfile files when the requirements of the data entry and merging strategy are met.

[0027] The file import module is used to import the hfile file into the hbase database.

[0028] In the above technical solution, preferably, the Spark-HBase batch data fast import system based on parameter configuration also includes an intermediate data clearing module, which is used to clear the result dataset of HDFS after the file import module imports the hfile file into the HBase database.

[0029] In the above technical solution, preferably, the configuration information of the data entry and merging strategy includes the HBase table name, the amount of data in each partition when writing the HDFS file, the minimum amount of merged data to generate the HFile file, the maximum amount of merged data to generate the HFile file, the maximum allowable delay time for data entry, and the scan interval.

[0030] In the above technical solution, preferably, the data writing and recording module is specifically used for:

[0031] The result dataset does not need to be assembled into a preset format. The number of partitions of the result dataset is dynamically calculated and reduced. Based on the number of partitions, the result dataset is directly written to HDFS without sorting and recorded in the information table of this batch. The information table includes transaction history, table name, HDFS file directory, data volume, status and generation time.

[0032] In the above technical solution, preferably, the file merging and generation module is specifically used for:

[0033] Start the asynchronous scanning application, scan the information table according to the data entry and merging strategy, and continuously add the HDFS file directory to be processed according to the file generation time order of the information table until the requirements of the data entry and merging strategy are met. Then submit the MR task. If not, perform the next polling. The MR task merges multiple data sources to generate an hfile file.

[0034] The file loading module is specifically used to load the hfile file into the hbase database using the bulkload method.

[0035] Compared with existing technologies, the beneficial effects of this invention are as follows: By determining the data entry and merging strategy through parameter configuration, Spark no longer directly generates HFile files and enters them into HBase. Before writing to HDFS, the number of partitions is dynamically calculated and reduced. The MapReduce task merges and generates HFiles and enters them into HBase by continuously adding source data until the strategy is met. This effectively utilizes the characteristics of Spark's in-memory computing and MapReduce's massive data processing, avoids the problem of too many small files in HBase, improves the read and write performance of the HBase database, and reduces the time spent on business calculations. Attached Figure Description

[0036] Figure 1 A flowchart illustrating the existing Spark-HBase batch data import implementation method;

[0037] Figure 2 This is a flowchart illustrating a method for rapid batch data import into Spark-HBase based on parameter configuration, as disclosed in one embodiment of the present invention.

[0038] Figure 3 This is a schematic diagram of the Spark dynamic partition shrinking and writing to HDFS process disclosed in one embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram illustrating the process of generating an hfile file and writing it to hbase in an MR task according to an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram of a system for rapid batch data import into Spark-HBase based on parameter configuration, as disclosed in one embodiment of the present invention.

[0041] In the diagram, the correspondence between the components and the reference numerals is as follows:

[0042] 1. Data import strategy configuration module, 2. Data writing record module, 3. File merging and generation module, 4. File import module, 5. Intermediate data clearing module. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] The present invention will now be described in further detail with reference to the accompanying drawings:

[0045] like Figure 2 As shown, according to the present invention, a method for rapid batch data import into Spark-HBase based on parameter configuration includes:

[0046] Configure the data merging strategy for HBase tables to handle scenarios where batch data computed by the Spark engine is imported into an HBase database.

[0047] The Spark engine is used to perform calculations on business data, and the resulting dataset is written to HDFS and recorded in an information table.

[0048] Based on the file generation time sequence of the information table, the HDFS file directory is continuously added. When the requirements of the data entry and merging strategy are met, the MR task is submitted to generate the hfile file.

[0049] The hfile file is imported into the hbase database.

[0050] In this implementation, the data merging strategy is determined through parameter configuration. Spark no longer directly generates HFile files and imports them into HBase. Before writing to HDFS, the number of partitions is dynamically calculated and reduced. The MapReduce task continuously adds source data until the strategy is met, then merges and generates HFile files and imports them into HBase. This effectively utilizes the features of Spark's in-memory computing and MapReduce's massive data processing capabilities, avoids the problem of too many small files in HBase, improves the read and write performance of the HBase database, and reduces the time spent on business computations.

[0051] Specifically, the main steps include: configuring the data import and merging strategy for the HBase database; writing the result dataset generated by the business to HDFS without sorting and recording relevant information in the information table; scanning the configuration of the strategy and sorting it according to the file generation time, continuously adding pending records until the strategy requirements are met, then submitting the MR task to generate the hfile file and importing it into HBase; otherwise, performing the next round of polling.

[0052] In the above embodiments, preferably, the method for rapid batch data import into Spark-HBase based on parameter configuration further includes:

[0053] After importing the hfile file into the hbase database, clear the result dataset from hdfs.

[0054] In the above implementation, preferably, the configuration information of the data entry and merging strategy includes the HBase table name, the amount of data in each partition when writing the HDFS file, the minimum amount of merged data to generate the HFile file, the maximum amount of merged data to generate the HFile file, the maximum allowable delay time for data entry, and the scan interval.

[0055] like Figure 3 As shown, in the above embodiment, preferably, the specific process of writing the generated result dataset into HDFS and recording it in the information table includes:

[0056] The resulting dataset RDD does not need to be assembled into a...<rowkey,cell> The format involves writing the result dataset directly to HDFS without sorting, based on the number of partitions. The batch information table is recorded in the following format: Transaction history, table name, HDFS file directory, data volume, status (0 - Unprocessed, 1 - Processing, 2 - Completed), and generation time. During the HDFS write process, to reduce the number of HDFS files, increase the size of individual files, and reduce the number of mappers used for downstream reads, the number of partitions in the result dataset RDD needs to be dynamically calculated and reduced based on the batch's data volume and strategy. For example, in Spark business calculations, with 100 partitions, a total data volume of 1 million records, and a strategy configured to assign 200,000 records per partition, the number of partitions is reduced to 1 million / 200,000 = 5 before writing to HDFS, resulting in 5 HDFS files being generated.

[0057] like Figure 4 As shown, in the above implementation, preferably, an asynchronous scanning application is started, and the information table is scanned according to the data entry and merging strategy. The HDFS file directory to be processed is continuously added according to the file generation time order of the information table until the requirements of the data entry and merging strategy are met. Then, the MR (yarn-mapreduce) task is submitted. If not, the next round of polling is performed.

[0058] The MapReduce task merges multiple data sources to generate an hfile file, and then loads the hfile file into the HBase database using the bulkload method.

[0059] MapReduce merges multiple data sources to generate HFile files, which are then loaded into HBase via bulk load. For example, consider 10 tasks: each task has 1 million data entries, a merge strategy of 10 million entries, 100 table partitions, and evenly distributed data. Without the merge strategy, each task would generate 100 HFile files, totaling 1000 HFile files. With the merge strategy, the number of HFile files generated after merging is reduced to 100, a 10-fold reduction. If each batch of data is smaller, more batches of source data will be merged, significantly reducing the number of small HFile files.

[0060] Based on the parameter configuration-based method for rapid batch data import into Spark-HBase disclosed in the above embodiments, the following specific examples will illustrate the method in detail.

[0061] Example 1

[0062] 1) Configure the HBase table merging strategy as follows:

[0063]

[0064] 2) Spark performs computations and generates HDFS files as follows:

[0065]

[0066] 3) Scan according to the merge strategy, read the configuration of table clzx_opr:RM_OEDER_INFO_1017, that is, submit the task if the data volume is between 10,000,000 and 15,000,000 or the first record exceeds the maximum allowed delay time of 30 minutes. Based on the records generated by HDFS as described above, for each new record to be processed, determine whether the data volume meets the strategy requirements or whether the time exceeds the maximum allowed delay time. The process is as follows:

[0067] Search for pending tasks by table name and sort them by time record:

[0068] Add the first pending task:

[0069] Total data volume: 1,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:43:58; Strategy not met, continue adding or polling.

[0070] Add the first and second pending tasks:

[0071] Total data volume: 3,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:44:58; Strategy not met, continue adding or polling.

[0072] Add the 1st, 2nd, and 3rd pending tasks:

[0073] Total data volume: 5,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:45:58; Strategy not met, continue adding or polling. ...

[0075] Add the 1st, 2nd, and 3rd pending tasks:

[0076] Total data volume: 5,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-29 00:12:58; Strategy satisfied (exceeded the maximum allowed delay time of 30 minutes); Task submitted, processing status updated to "processing," and the next round of iterations continued.

[0077] This merge involves three batches of source data and 100 partitions of HBase tables. Before the merge, each batch would generate 100 hfile files, for a total of 300 hfile files. After the merge, a total of 100 hfile files will be generated.

[0078] Example 2

[0079] 1) The merging strategy for the configuration tables is as follows:

[0080]

[0081] 2) Spark performs computations and generates HDFS files as follows:

[0082]

[0083] 3) Scan according to the merge strategy, read the configuration of table clzx_opr:RM_OEDER_INFO_1017, that is, submit the task if the data volume is between 10,000,000 and 15,000,000 or the first record exceeds the maximum allowed delay time of 30 minutes. Based on the records generated by HDFS as described above, for each new record to be processed, determine whether the data volume meets the strategy requirements or whether the time exceeds the maximum allowed delay time. The process is as follows:

[0084] Search for pending tasks by table name and sort them by time record:

[0085] Add the first pending task:

[0086] Total data volume: 1,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:43:58; Strategy not met, continue adding or polling.

[0087] Add the first and second pending tasks:

[0088] Total data volume: 7,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:44:58; Strategy not met, continue adding or polling.

[0089] Add the 1st, 2nd, and 3rd pending tasks:

[0090] Total data volume: 11,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:45:58; meets the strategy (total data volume greater than 10,000,000); submit task, update processing status to "processing," and continue to the next round of iteration.

[0091] This merge involves three batches of source data from 100 HBase tables across 100 partitions. Before the merge, each batch generated 100 HFile files, totaling 300 HFile files. After the merge, a total of 100 HFile files will be generated.

[0092] Example 3

[0093] The merging strategy for the configuration tables is as follows:

[0094]

[0095] 2) Spark performs computations and generates HDFS files as follows:

[0096]

[0097]

[0098] 3) Scan according to the merge strategy, read the configuration of table clzx_opr:RM_OEDER_INFO_1017, i.e., if the data volume is between 10,000,000 and 15,000,000, or if the first record exceeds the maximum allowed delay time of 30 minutes, submit the task. Based on the records generated by HDFS as described above, for each added record to be processed, determine whether the data volume meets the strategy requirements or whether the time exceeds the maximum allowed delay time of the strategy. The process is as follows:

[0099] Search for pending tasks by table name and sort them by time record:

[0100] Add the first pending task:

[0101] Total data volume: 7,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:43:58; Strategy not met, proceed to the next polling cycle.

[0102] Add the first and second pending tasks:

[0103] Total data volume: 16,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:44:58. The policy is not met; the data volume exceeds the maximum, so this record is discarded and the next record is searched for.

[0104] Add the first and third pending tasks:

[0105] Total data volume: 11,000,000; Earliest time: 2021-09-28 23:42:51; Current time: 2021-09-28 23:45:58; meets the strategy (total data volume greater than 10,000,000 and less than 15,000,000); submit the task, update the status of records 1 and 3 to "processing," and continue to the next round of iteration.

[0106] This merge involves two batches of source data and 100 partitions of HBase tables. Before the merge, each batch would generate 100 hfile files, for a total of 200 hfile files. After the merge, a total of 100 hfile files will be generated.

[0107] like Figure 5 As shown, this invention also proposes a system for rapid batch data import into Spark-HBase based on parameter configuration, applying the parameter-configured method for rapid batch data import into Spark-HBase disclosed in any of the above embodiments, including:

[0108] Data Ingestion Strategy Configuration Module 1 is used to configure the data ingestion and merging strategy for HBase tables in scenarios where batch data calculated by the Spark engine is ingested into the HBase database.

[0109] Data writing and recording module 2 is used to perform calculations on business data using the Spark engine, write the generated result dataset to HDFS and record it in the information table;

[0110] File merging and generation module 3 is used to continuously add HDFS file directories according to the file generation time order of the information table. When the requirements of the data entry and merging strategy are met, MR task is submitted to generate hfile files.

[0111] File import module 4 is used to import hfile files into the hbase database.

[0112] In the above implementation, preferably, the Spark-HBase batch data fast import system based on parameter configuration also includes an intermediate data clearing module 5, which is used to clear the result dataset of HDFS after the file import module 4 imports the hfile file into the HBase database.

[0113] In the above implementation, preferably, the configuration information of the data entry and merging strategy includes the HBase table name, the amount of data in each partition when writing the HDFS file, the minimum amount of merged data to generate the HFile file, the maximum amount of merged data to generate the HFile file, the maximum allowable delay time for data entry, and the scan interval.

[0114] In the above embodiment, preferably, the data writing and recording module 2 is specifically used for:

[0115] The result dataset does not need to be assembled into a preset format. The number of partitions in the result dataset is dynamically calculated and reduced. Based on the number of partitions, the result dataset is directly written to HDFS without sorting and recorded in the information table of this batch. The information table includes the transaction history, table name, HDFS file directory, data volume, status, and generation time.

[0116] In the above embodiments, preferably, the file merging and generation module 3 is specifically used for:

[0117] Start the asynchronous scanning application, scan the information table according to the data entry and merging strategy, and continuously add the HDFS file directory to be processed according to the file generation time order of the information table until the requirements of the data entry and merging strategy are met. If the requirements are not met, submit the MR task. If not, perform the next polling and merge multiple data sources into an hfile file through the MR task.

[0118] The file import module 4 is specifically used to load hfile files into the hbase database using the bulkload method.

[0119] According to the parameter-configured Spark-HBase batch data fast import system disclosed in the above embodiments, the functions to be implemented by each module are consistent with the steps in the above parameter-configured Spark-HBase batch data fast import method. For the specific implementation process, please refer to the above embodiments, which will not be repeated here.

[0120] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for rapid batch data import into Spark-HBase based on parameter configuration, characterized in that, include: For scenarios where batch data computed by the Spark engine is written to an HBase database, a data entry and merging strategy for the HBase table is configured. The configuration information for the data entry and merging strategy includes the HBase table name, the amount of data in each partition when writing the HDFS file, the minimum amount of data to be merged when generating the HFile file, the maximum amount of data to be merged when generating the HFile file, the maximum allowed delay time for data entry, and the scan interval. The Spark engine is used to perform calculations on business data, and the resulting dataset is written to HDFS and recorded in an information table. The specific process includes: The result dataset does not need to be assembled into a preset format. The number of partitions of the result dataset is dynamically calculated and reduced. Based on the number of partitions, the result dataset is directly written to HDFS without sorting and recorded in the information table of this batch. The information table includes transaction history, table name, HDFS file directory, data volume, status and generation time. Based on the file generation time sequence of the information table, the HDFS file directory is continuously added. When the requirements of the data entry and merging strategy are met, the MR task is submitted to generate the hfile file. The hfile file is imported into the hbase database.

2. The method for rapid batch data import into Spark-HBase based on parameter configuration according to claim 1, characterized in that, Also includes: After the hfile file is imported into the hbase database, the result dataset of the hdfs is cleared.

3. The method for rapid batch data import into Spark-HBase based on parameter configuration according to claim 1, characterized in that: Start the asynchronous scanning application, scan the information table according to the data entry and merging strategy, and continuously add the HDFS file directory to be processed according to the file generation time order of the information table until the requirements of the data entry and merging strategy are met. Then submit the MR task. If not, perform the next polling. The MR task merges multiple data sources to generate an hfile file, and loads the hfile file into the hbase database using the bulkload method.

4. A system for rapid batch data import into Spark-HBase based on parameter configuration, characterized in that, The method for rapid batch data import into Spark-HBase based on parameter configuration, as described in any one of claims 1 to 3, includes: The data ingestion strategy configuration module is used to configure the data ingestion and merging strategy for HBase tables in scenarios where batch data calculated by the Spark engine is ingested into the HBase database. The data writing and recording module is used to perform calculations on business data using the Spark engine, write the generated dataset to HDFS, and record it in an information table. The file merging and generation module is used to continuously add HDFS file directories according to the file generation time order of the information table, and submit MR tasks to generate hfile files when the requirements of the data entry and merging strategy are met. The file import module is used to import the hfile file into the hbase database.

5. The system for rapid batch data import into Spark-HBase based on parameter configuration as described in claim 4, characterized in that, It also includes an intermediate data clearing module, which is used to clear the result dataset of HDFS after the file entry module has entered the hfile into the hbase database.

6. The system for rapid batch data import into Spark-HBase based on parameter configuration as described in claim 5, characterized in that, The configuration information for the data import and merging strategy includes the HBase table name, the amount of data in each partition when writing the HDFS file, the minimum amount of data to be merged when generating the HFile file, the maximum amount of data to be merged when generating the HFile file, the maximum allowed delay time for data import, and the scan interval.

7. The system for rapid batch data import into Spark-HBase based on parameter configuration as described in claim 6, characterized in that, The data writing and recording module is specifically used for: The result dataset does not need to be assembled into a preset format. The number of partitions of the result dataset is dynamically calculated and reduced. Based on the number of partitions, the result dataset is directly written to HDFS without sorting and recorded in the information table of this batch. The information table includes transaction history, table name, HDFS file directory, data volume, status and generation time.

8. The system for rapid batch data import into Spark-HBase based on parameter configuration as described in claim 6, characterized in that, The file merging and generation module is specifically used for: Start the asynchronous scanning application, scan the information table according to the data entry and merging strategy, and continuously add the HDFS file directory to be processed according to the file generation time order of the information table until the requirements of the data entry and merging strategy are met. Then submit the MR task. If not, perform the next polling. The MR task merges multiple data sources to generate an hfile file. The file loading module is specifically used to load the hfile file into the hbase database using the bulkload method.

Images