A HBase offline data import optimization method based on key value pre-partitioning

Abstract

The application discloses a kind of HBase offline data import optimization method based on key value pre-partitioning, including, the metadata of the target HBase table is inquired, the key value range of all Region is obtained;Based on the key value range of the Region obtained, the source data is handled through a MapReduce task, in the Reduce stage of task, according to Region boundary, data stream is dynamically split and parallelly generates multiple HFile, and these boundary-aligned HFile are directly loaded into its corresponding Region by the batch loading tool of HBase;The application fundamentally eliminates the need for HFile splitting when loading by HBase server, thereby reducing I / O and CPU overhead, while improving the efficiency, timeliness and stability of large-scale data offline import.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data storage and distributed computing, and particularly relates to a HBase offline data import optimization method based on key value pre-partitioning. BACKGROUND

[0002] With the evolution of data scale to PB or even EB level, distributed NoSQL database HBase has become a key component for real-time read and write of massive data. For data filling of HBase, generating HFile based on MapReduce framework and performing bulk load (BulkLoad) is the mainstream offline import solution in the industry. The technical evolution of this solution aims to balance the concurrent overhead and data processing efficiency of the import task. The early solution starts an independent Reduce task for each HBase data partition (Region), which can ensure accurate correspondence between the generated file and the partition, but when the number of Regions is large, the time and resource overhead caused by the start and scheduling of Reduce tasks is huge, which limits the overall import performance. To solve this problem, the subsequent optimization introduces a merging mechanism, which allows multiple Regions to process data in a single Reduce task through a ratio parameter (ratio), significantly reducing the total number of Reduce tasks and reducing the task start overhead.

[0003] However, the above-mentioned merging optimization exposes a new performance bottleneck when processing super large scale data sets (such as tens of TB level). Since the data key value (RowKey) range processed by a single Reduce task may cross the boundaries of multiple Regions, the single HFile generated by it must be split (Split) by the HBase server when loaded into HBase, which is split and distributed to the correct Region. This splitting process involves complex data block migration, metadata update and storage space reallocation, which constitutes a high-load I / O intensive operation. This process not only consumes a large amount of cluster disk I / O and CPU resources, but also causes the Region to be temporarily unavailable during splitting, which seriously affects the timeliness of data import and may cause cluster load jitter, becoming a key obstacle to the efficiency of HBase offline data import.

[0004] CN117130561A discloses a data storage method, device and business system across Hbase cluster, which comprises, in the case of receiving data upload message, obtaining the position information, first time information and first identification information of the data to be stored, encoding the position information, first time information and first identification information to obtain the index number, reversing the index number to obtain the row key value of the data to be stored, storing the data to be stored into the first target partition in the target server according to the row key value, and the first target partition is the storage partition corresponding to the byte of the corresponding first identification information in the row key value. Although this scheme mentions key value partitioning, it does not consider the alignment of Region boundary HFile and the splitting process of HFile.

[0005] CN120277045A discloses a data processing method, device and electronic equipment based on a distributed file model, which comprises, obtaining source data, determining the current job type, configuring the target parameters of the data warehouse and the parameters of the memory computing engine, filtering the source data to obtain data fields, establishing the mapping relationship between the HBase data table and the data fields, and then processing the data file to form the original distributed data set, obtaining the storage unit interval in the HBase data table, and re-partitioning the original distributed data set according to the storage unit interval to obtain the partitioned distributed data set, mapping the key value pairs in the partitioned distributed data set to the data storage basic unit of HBase, inputting the source data into the data storage basic unit, marking the data storage basic unit to obtain the marked storage basic unit, and executing the write or delete operation on the source data. This scheme mainly solves the problem of high response delay caused by low system throughput when the data volume is large, and does not achieve the effect of offline import and shortening query delay.

[0006] Therefore, there is an urgent need for an import optimization method that can fundamentally avoid HFile splitting while taking into account task concurrency efficiency. SUMMARY

[0007] This section aims to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the specification to avoid obscuring the purpose of this section, abstract and title, and such simplifications or omissions cannot be used to limit the scope of the present application.

[0008] In view of the above-mentioned existing problems, the present application is proposed. Therefore, the present application provides an HBase offline data import optimization method based on key value pre-partitioning, which is used to solve the problem of resource waste and low efficiency caused by HFile splitting when loading due to the mismatch between HFile and Region boundary in the prior art.

[0009] To solve the above technical problems, the application provides the following technical scheme: a HBase offline data import optimization method based on key value pre-partitioning, comprising:

[0010] querying metadata of a target HBase table to obtain key value ranges of all regions of the table;

[0011] based on the obtained key value ranges of the regions, processing source data and generating a plurality of HFiles, so that all data key values in each generated HFile are completely within the key value range of a corresponding target region;

[0012] by calling a batch loading tool of the HBase, the generated plurality of HFiles are loaded into their respective corresponding regions.

[0013] As a preferred scheme of the HBase offline data import optimization method based on key value pre-partitioning, wherein: the metadata of the target HBase table is obtained by accessing a system table storing region distribution information.

[0014] As a preferred scheme of the HBase offline data import optimization method based on key value pre-partitioning, wherein: the process of processing the source data is completed by executing a MapReduce task.

[0015] As a preferred scheme of the HBase offline data import optimization method based on key value pre-partitioning, wherein: the MapReduce task includes a Map phase, through which the source data is read and globally sorted according to HBase row keys determined from each source data record.

[0016] As a preferred scheme of the HBase offline data import optimization method based on key value pre-partitioning, wherein: the MapReduce task includes a Reduce phase, in which a single Reduce task receives a sorted data stream and creates and generates an independent HFile for each target region covered by the data stream of the task in parallel.

[0017] As a preferred scheme of the HBase offline data import optimization method based on key value pre-partitioning, wherein: the Reduce phase comprises:

[0018] parallel generation of HFiles is realized by maintaining a mapping relationship between a region identifier and an HFile writer;

[0019] For each piece of received data, the Region identifier corresponding to the row key of the data is determined, and then the HFile writer associated with the Region identifier in the mapping relationship is used for writing;

[0020] If the corresponding HFile writer does not exist in the mapping relationship, the writer is created first and then writing is performed.

[0021] As a preferred scheme of the HBase offline data import optimization method based on key-value pre-partitioning, the number of Reduce tasks started in the MapReduce task is determined according to the total number of Regions of the target HBase table and a preset proportion parameter.

[0022] As a preferred scheme of the HBase offline data import optimization method based on key-value pre-partitioning, the MapReduce task further performs a partitioning step between the Map phase and the Reduce phase, and the partitioning step sends the row key continuous data to the same Reduce task.

[0023] As a preferred scheme of the HBase offline data import optimization method based on key-value pre-partitioning, the generated HFiles are loaded into the respective corresponding Regions, and the loading process is performed by bypassing the pre-write log path of HBase.

[0024] As a preferred scheme of the HBase offline data import optimization method based on key-value pre-partitioning, the method further includes: constructing a data structure supporting interval query according to the obtained key value range of the Region, and then quickly locating the Region to which the row key belongs.

[0025] Compared with the prior art, the application has the following beneficial effects:

[0026] 1. By pre-generating the HFile aligned with the Region boundary in the Reduce phase of MapReduce, the HFile splitting operation necessary when loading data to the HBase server is fundamentally eliminated, the disk I / O and CPU resource consumption caused by data migration and metadata update is reduced, and the overall efficiency of large-scale data offline import is improved.

[0027] 2. The instantaneous peak of cluster load caused by the HFile splitting process is avoided, the data loading process is smoothly transitioned to a lightweight HDFS file moving operation, the running impact on the HBase cluster is reduced, the stability of the cluster service is ensured, and the overall resource utilization of the system is improved.

[0028] 3. By bypassing the Region temporary unavailable time caused by HFile split, and combining the characteristics of the batch loading tool bypassing the pre-write log path, near real-time visibility after data loading is realized, the delay from data import to query is greatly shortened, the timeliness of data import is effectively enhanced, and the needs of real-time analysis and other business scenarios can be better met. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and all other drawings obtained by those skilled in the art without creative labor should belong to the protection scope of the present application. Among them:

[0030] Figure 1 The overall flowchart of the HBase offline data import optimization method based on key value pre-partitioning according to an embodiment of the present application;

[0031] Figure 2 The offline import flowchart of the HBase offline data import optimization method based on key value pre-partitioning according to an embodiment of the present application before the existing HBase-1.2.6 version;

[0032] Figure 3 The offline import flowchart of the HBase offline data import optimization method based on key value pre-partitioning according to an embodiment of the present application after the existing HBase-1.2.6 version;

[0033] Figure 4 The data import flowchart of the HBase offline data import optimization method based on key value pre-partitioning according to an embodiment of the present application;

[0034] Figure 5 The data import flowchart of the HBase offline data import optimization method based on key value pre-partitioning according to an embodiment of the present application;

[0035] Figure 6 The scene test result chart of the HBase offline data import optimization method based on key value pre-partitioning according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the protection scope of the present application.

[0037] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, it will be apparent to one skilled in the art that the present application can be practiced without the specific details set forth in this description. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present application.

[0038] Secondly, the "one embodiment" or "an embodiment" appearing in the specification means that the specific feature, structure or characteristic can be included in at least one implementation of the present application. The "in one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an embodiment that is independent of or exclusive to other embodiments.

[0039] The present application is described in detail in conjunction with the schematic diagram. In the detailed description of the embodiments of the present application, the sectional view of the device structure is partially enlarged without the general proportion for the convenience of illustration, and the schematic diagram is only an example, which should not limit the scope of protection of the present application. In addition, the three-dimensional spatial dimensions of length, width and depth should be included in actual production.

[0040] Meanwhile, in the description of the present application, it should be noted that the terms "upper, lower, inner and outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first, second or third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0041] In the present application, unless otherwise specifically specified and limited, the terms "mounting, connection, connection" should be understood broadly, for example: it can be fixed connection, detachable connection or integral connection; it can also be mechanical connection, electrical connection or direct connection, it can also be indirectly connected through an intermediate medium, or it can be the communication between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0042] Embodiment 1

[0043] Reference Figure 1 For the first embodiment of the present application, the embodiment provides a HBase offline data import optimization method based on key value pre-partitioning, comprising:

[0044] S1, querying the metadata of the target HBase table to obtain the key value range of all Regions of the table;

[0045] Further, the HBase client application is initialized and connected to the target HBase cluster through standard HBase configuration information (such as the ZooKeeper cluster address); the connection is the entrance for all interactions with the HBase cluster, and a Table object instance representing the target data table can be obtained through the connection; from the Table object instance, a key interface instance, RegionLocator, can be obtained;

[0046] It should be explained that RegionLocator is a standard tool provided by HBase for locating Region information, which encapsulates all underlying details of accessing HBase metadata;

[0047] Specifically, the core function of the RegionLocator interface is to query the system table storing Region distribution information, i.e., the hbase:meta table; and the query process is transparent to the user, i.e., when the getAllRegionLocations() method of the RegionLocator is called, the RegionLocator interface will automatically initiate a scan operation to the hbase:meta table;

[0048] Specifically, each row of the hbase:meta table records the metadata information of a Region of the user table;

[0049] Further, in the present application scheme, all rows related to the target table can be filtered by scanning the hbase:meta table, and for each row, the following key information is extracted:

[0050] Region Name: represents the unique identifier of the Region;

[0051] Start Key: represents the lower bound of the RowKey of the data stored in the Region; it is a byte array, and for the first Region of the table, the start key is empty;

[0052] End Key: represents the upper bound of the RowKey of the data stored in the Region; it is also a byte array, which defines the open interval boundary of the Region range; for the last Region of the table, the end key is empty;

[0053] Further, by traversing all entries about the target table in the hbase:meta table, a complete list containing all Region information of the table can be obtained; each element in the list defines the key value range of a Region, i.e., ;

[0054] It should be noted that if a list containing all the Region boundaries is directly used for query, it is inefficient; and in the traditional Reduce phase, since the data flow is continuous, the Reduce task needs to frequently determine whether the current processed row key crosses the Region boundary, and if the entire list is traversed each time (i.e. the time complexity of linear scanning is , N is the number of Regions), when the number of Regions reaches thousands or even more, great performance loss will be caused; therefore, the information list of the obtained key value range of the Region is constructed into an interval query data structure in the client memory by the present application scheme;

[0055] Specifically, the preferred construction scheme is obtained by using java.util.TreeMap or a functionally similar SkipList, and the specific construction process is as follows:

[0056] A TreeMap<byte[], RegionInfo> instance is created, in which the key (Key) is the start key of each Region, and the value (Value) is an object containing the complete information (such as Start Key, End Key, etc.) of the Region, and the TreeMap will automatically sort all start keys according to the byte sequence, and when the sorting is completed, an interval query data structure is obtained;

[0057] The TreeMap becomes a high-efficiency Region "navigation map", and when the Reduce task needs to locate the Region to which any row key currentRowKey belongs, the floorEntry(currentRowKey) method of the TreeMap is called, which can quickly find the maximum start key less than or equal to the currentRowKey with logarithmic time complexity (O(logN) ), and the start key corresponds to the Region where the currentRowKey is located;

[0058] Illustratively, assuming that we have RegionA [a, c) and RegionB [c, f), when the currentRowKey is d, the floorEntry('d') will return the entry with the start key c, thereby immediately determining that the row key belongs to RegionB;

[0059] Further, the constructed Region boundary data structure (such as TreeMap) with efficient query capability is serialized and distributed to each node performing the task through the MapReduce DistributedCache mechanism or job configuration, so as to be directly used in the local memory of the Reduce task;

[0060] S2, based on the key value range of the acquired Region, processing the source data and generating a plurality of HFiles, so that all data key values in each generated HFile completely fall within the key value range of a corresponding target Region;

[0061] Further, the MapReduce task mainly includes three stages: the Map stage, the partition and sorting stage, and the Reduce stage.

[0062] Further, for the Map stage, the stage is mainly responsible for reading the source data and converting it into the key value format required by HBase, that is, converting all source data records into the KeyValue format recognizable by HBase, and taking the row key as the basis to prepare for the partition and sorting stage of the next stage.

[0063] Specifically, a Map task reads the source data shard allocated to it, and for each source data record, the Mapper performs the following operations:

[0064] According to the format of the source data (such as CSV, JSON or delimiter text), all fields required to constitute an HBase record are parsed; then from the parsed fields, the HBase row key of the record is constructed according to the business logic; the row key, column family, column qualifier, timestamp and cell value are encapsulated into one or more HBase KeyValue objects; finally, the constructed KeyValue objects are output by the Mapper as its output.

[0065] It should be noted that the HBase row key is the only basis for sorting and positioning data in HBase.

[0066] In addition, in order to utilize the sorting mechanism of MapReduce, we encourage taking the row key as the output key and the KeyValue object as the output value.

[0067] It should be noted that between the Map stage and the Reduce stage, since MapReduce performs the partition and sorting process, the application utilizes the global sorting idea similar to Hadoop TeraSort to perform global sorting according to the HBase row key determined from each source data record.

[0068] Further, in order to ensure that the data with continuous row keys can be sent to the same Reduce task, the application also adopts a partitioner based on row key range, which determines the number of Reduce tasks according to the total number of Regions of the target HBase table and a user-configurable preset ratio parameter (ratio), and then divides the value range space of the entire row key into several continuous intervals equal to the number of Reduce tasks. When a Map output key-value pair needs to be partitioned, the partitioner determines which interval the row key belongs to and directs it to the Reduce task responsible for the interval, ensuring that the input data stream received by each Reduce task is not only strictly sorted by row key internally, but also covers a globally ordered and continuous segment of row key range.

[0069] Specifically, the calculation formula of the number of Reduce tasks is: Reduce task number = Region total number * ratio.

[0070] It should be noted that since the above operation is performed, a single Reduce task receives a continuous and ordered data stream, and the row key range of the data stream may fall within a Region or span multiple Region boundaries. The goal of the Reduce task is to accurately divide and write the data stream into HFiles corresponding to each Region. The process of the Reduce task is as follows:

[0071] At the beginning of the Reduce task, load and deserialize the Region boundary query data structure (such as TreeMap) constructed in step S1 from the distributed cache, and initialize a data structure for maintaining the mapping relationship between Regions and their corresponding HFile writers (HFile.Writer), such as a HashMap<RegionID, HFile.Writer>.

[0072] The Reduce task processes each KeyValue object in its input data stream in order. For each KeyValue object, the following logic is executed:

[0073] Extract the row key of the current KeyValue, and use the in-memory Region boundary query data structure (TreeMap) to quickly determine the target Region identifier to which the row key belongs through floorEntry() and other query methods with logarithmic time complexity.

[0074] Use the obtained Region identifier as the key to find the corresponding HFile.Writer instance in the HashMap.

[0075] If the writer exists, it means that the current data belongs to the same Region as the last data, so the writer is used directly. If the writer does not exist, it means that the input data stream has crossed a Region boundary, so the Reduce task first checks and closes the writer of the last Region (if it exists), then creates a new and independent HFile.Writer instance for the current new target Region, and specifies a unique output path for the HFile on the HDFS that is related to the target Region (and its column family), and stores the newly created writer in the HashMap;

[0076] The current KeyValue object is appended to the corresponding HFile using the obtained or newly created writer.

[0077] When the Reduce task has processed all the input data streams, all the remaining HFile writers in the HashMap are traversed, and the close() method of each writer is called to ensure that all generated HFiles are correctly closed and persisted to the HDFS.

[0078] It should be noted that through the execution of each stage of the above MapReduce task, even if the data processed by a single Reduce task spans multiple Regions, multiple independent HFiles can be intelligently generated, and the data in each HFile strictly belongs to the key value range of a corresponding target Region, thereby achieving the pre-alignment of HFiles and Region boundaries.

[0079] S3, the generated multiple HFiles are loaded into their respective corresponding Regions by calling the bulk loading tool of HBase.

[0080] It should be noted that this step aims to import the multiple HFiles aligned with the above Region boundaries into the target HBase table so that the data is visible to users. The loading process is completed by calling the bulk loading (BulkLoad) tool provided by HBase, using the LoadIncrementalHFiles class, which can be invoked through the command line or programmatically.

[0081] Specifically, the specific implementation steps of the loading process are as follows:

[0082] When the MapReduce task of S2 step is successfully executed, all generated HFiles are stored in a specified output directory of HDFS, at this time, the loading process is started, and two core parameters need to be provided to the LoadIncrementalHFiles tool;

[0083] Specifically, the core parameters include the HFile source path, i.e., the path of the output directory of the MapReduce task on HDFS; and the target HBase table name, i.e., the name of the HBase table to which the data is imported;

[0084] When the batch loading tool is called, it does not directly perform data writing, but first connects to the HBase cluster and communicates with the HBase Master node. The loading tool traverses all HFiles (i.e., HFile file list) under the source path and submits the HFile file list to the Master;

[0085] The Master node determines which Region on which RegionServer should "adopt" each HFile according to the column family information of each HFile and the Region distribution information in the hbase: meta table managed by the Master node, and then the Master node distributes the loading instruction to each related RegionServer;

[0086] The RegionServer receiving the instruction performs the core operation of loading, which is not a traditional data writing, but a HDFS file moving operation. Specifically, the RegionServer moves the HFile assigned to it from the temporary output directory of HDFS to the corresponding column family storage directory under the HBase table directory of the Region. Because this process is a metadata operation at the file system level, its speed is much faster than transmitting data over the network and writing it piece by piece. Moreover, since it has been ensured in the S2 step that the data of each HFile belongs to a single Region completely, the moving operation does not need any data-level check or splitting and can be completed directly;

[0087] Once the file moving is successful, the RegionServer updates the metadata of the Region and includes the corresponding new HFile in the data file list managed by it, and thus the data in the HFile is immediately visible to the read request of the client;

[0088] In addition, the entire loading process is performed by bypassing the pre-write log path of HBase, because in a standard HBase write operation (such as Put), data is first written into MemStore in memory, and a log record is appended to the pre-write log at the same time to ensure that data is not lost, and through batch loading, a complete and persistent HFile file can be directly managed by a Region, and since the data already exists on HDFS, it is not necessary to protect the data through the pre-write log again;

[0089] It should be noted that by bypassing the pre-write log, a large amount of disk I / O overhead is avoided, the throughput of data import is improved, and the write pressure on the RegionServer is reduced, which not only makes the loading process faster, but also makes the load of the entire HBase cluster more stable during data import, and the interference to online services is minimized.

[0090] Embodiment 2

[0091] Reference Figures 2 to 6 For the second embodiment of the present application, the embodiment provides an HBase offline data import optimization method based on key value pre-partitioning, comprising: comparing the present application scheme with the offline schemes before and after the existing HBase-1.2.6 version;

[0092] In the version before HBase-1.2.6, HBase data import relies on MapReduce, and each Region starts a Map stage, and completes data preprocessing (sorting, reorganization, etc.) in the Map stage; subsequently, the processed data is handed over to Reduce for processing, however, if a table has many Regions, many Reduces will be started to process the data when the data is imported, and Reduce needs to consume a lot of time in the startup stage (mainly class loading time-consuming and resource application time-consuming), so the total time of data import will also be greatly increased, as shown in Figure 2 ;

[0093] In the version after HBase-1.2.6, in view of the performance problem that too many Reduces may cause, optimization is performed at the HBase source code level, a ratio parameter is added in the Importtsv tool class, so that the result data of Map is pre-merged before Reduce, and bucketing is introduced, so that the number of Reduces is reduced; for example, if a table has 10 Regions, and the ratio is set to 0.5, then the entire MapReduce task starts 10 Maps and 5 Reduces, and generates 5 HFile files, and then imports them into the corresponding 10 Regions, so that the number of Reduces is reduced by half, and the corresponding startup loss is also reduced by half, as shown inFigure 3 The design can optimize frequent starting of Reduce caused by excessive partition of HBase table, but since the HFile file does not match the number of regions, when processing a large amount of data (tens of TB, hundreds of TB), the HFile file needs to be split into the same number of regions, so the HFile splitting process also takes a long time (since the splitting of the region is essentially writing, it also occupies the disk IO resource, and the bottleneck of HBase is mainly on the disk IO);

[0094] The scheme of the application divides the source data according to the preset number of regions in the Map stage, and globally sorts the RowKey; the mapping relationship between RowKey and Region is established; the Region distribution information in the hbase: meta table is obtained in real time by using the RegionLocator interface of HBase, the key value range of each Region is extracted, and the skip list is used to realize the interval query of the complexity According to the ratio setting, the Map can transmit data to the corresponding Reduce according to the mapping relationship; the Reduce divides the data into multiple groups according to the obtained Region boundary, and generates an independent HFile for each group (for example, if the data crosses 2 regions, 2 HFiles are output), and the reference Figure 4 and Figure 5 ;

[0095] In addition, for the versions before HBase-1.2.6, HBase-1.2.6 and the improved version of the scheme of the application, the ratio of the above-mentioned versions is set to 0.5, and the time interval of the test results is as shown in Figure 6 From Figure 6 , it can be clearly seen that when the data amount is small (100GB), the performance of the application method is close to that of HBase-1.2.6, and is better than that of the version before HBase-1.2.6, which reflects the advantage of reducing the startup overhead of Reduce; as the data amount increases to 200GB and 500GB, the performance bottleneck of HBase-1.2.6 is highlighted, and the time consumption increases sharply, especially in the 500GB scenario, since the large-scale HFile splitting produces a huge I / O overhead, the time consumption even exceeds that of the version before HBase-1.2.6; in contrast, the application method shows the best performance and good linear scalability under all data amount levels, since the HFile splitting is fundamentally avoided, the larger the data amount, the more significant the performance advantage of the application method relative to HBase-1.2.6;

[0096] The test results fully prove that the method of the present application effectively solves the performance bottleneck problem caused by HFile splitting under large data volume in the prior art, and significantly improves the data import efficiency.

[0097] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0098] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0099] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0101] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that such additions and modifications be included within the scope of the application. It is the following claims, including any amendments thereto, which define the scope of the application.

[0102] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. An optimized method for offline data import into HBase based on key-value pre-partitioning, characterized in that, include: Query the metadata of the target HBase table to obtain the key-value range of all Regions in the table; Based on the obtained key value range of the Region, the source data is processed and multiple HFiles are generated, so that all data key values ​​in each generated HFile fall completely within the key value range of a corresponding target Region. The process of processing the source data is accomplished by executing a MapReduce task; The MapReduce task includes a Reduce phase, in which a single Reduce task receives a sorted data stream and, within the single Reduce task, creates and generates an independent HFile in parallel for each target Region covered by its data stream. By calling HBase's batch loading tool, the generated HFiles are loaded into their respective Regions. The generated HFiles are loaded into their respective Regions. This loading process is performed by bypassing HBase's write-ahead log path.

2. The HBase offline data import optimization method based on key-value pre-partitioning as described in claim 1, characterized in that, The metadata of the target HBase table is obtained by accessing the system table that stores Region distribution information.

3. The HBase offline data import optimization method based on key-value pre-partitioning as described in claim 1, characterized in that, The MapReduce task includes a Map phase, through which the source data is read and globally sorted based on the HBase row keys determined from each source data record.

4. The HBase offline data import optimization method based on key-value pre-partitioning as described in claim 1, characterized in that, The Reduce phase includes: Parallel generation of HFiles is achieved by maintaining a mapping between a Region identifier and an HFile writer; For each piece of data received, first determine the Region identifier corresponding to its row key, and then use the HFile writer associated with the Region identifier in the mapping relationship to write it; If the corresponding HFile writer does not exist in the mapping relationship, the writer will be created first and then the writing will be performed.

5. The HBase offline data import optimization method based on key-value pre-partitioning as described in claim 1, characterized in that, The number of Reduce tasks launched in the MapReduce task is determined by the total number of Regions in the target HBase table and a preset ratio parameter.

6. The HBase offline data import optimization method based on key-value pre-partitioning as described in claim 1, characterized in that, The MapReduce task also performs a partitioning step between the Map and Reduce phases, which sends row-key-contiguous data to the same Reduce task.

7. The HBase offline data import optimization method based on key-value pre-partitioning as described in claim 1, characterized in that, The method also includes: constructing a data structure that supports range queries based on the obtained key value range of the Region, thereby quickly locating the Region to which the row key belongs.

Images