HBase data processing method and device

Abstract

The present application relates to the field of big data, and particularly relates to an HBase data processing method and device. The method comprises: screening to-be-processed data from an HBase initial table according to a screening condition; storing the to-be-processed data screened out in an isolated storage area which is isolated from stock data in the HBase initial table; combining at least one HFile according to the to-be-processed data in the isolated storage area; loading the HFile into an HBase target table; and processing the to-be-processed data in the HBase target table after receiving a processing instruction. The present application screens to-be-processed data in batches, directly generates an underlying HFile, avoids sending a large number of requests to a RegionServer, reduces the write pressure on the RegionServer caused by a large number of write requests, reduces the consumption of memory resources and bandwidth resources, and ensures the stability of a cluster.

Description

Technical Field

[0001] This article relates to the field of big data, and in particular to an HBase data processing method and device. Background Technology

[0002] HBase is a distributed, column-oriented, non-relational distributed database storage system built on top of HDFS, used to store loosely structured and unstructured data. In actual business operations, as data (especially temporary data) is continuously written, the available space on the cluster disk becomes increasingly limited. When unlimited expansion of the cluster is not possible, it is necessary to perform batch cleanup of some temporary or garbage data.

[0003] Currently, HBase's native cleanup strategy is Time To Live (TTL) cleanup, which supports setting TTL at the table or row level. Users can specify the TTL when creating a table or set the TTL for a specific row using the `setTTL` interface of the `Put` class. However, setting the TTL at the table level is clearly not suitable when users only need to clean up a portion of the table's data.

[0004] In addition, HBase provides the `delete` method to delete data, allowing you to delete a specific field or column family by calling the `delete` interface. However, due to HBase's underlying reliance on HDFS storage, it needs to wait for the delete command before physically deleting the data.

[0005] Whether setting TTL at the row level or calling the delete interface, the essence is that the client implements this by calling different interfaces when writing data. Taking a write operation as an example, when the RegionServer (a service running on each worker node of the HBase cluster) processes a data write request, the data is first written to the write-ahead log (WAL). Simultaneously, the HBase client determines the data label and encapsulates the data based on the request type. When the data in the Memstore reaches its capacity and time limits, the data is flushed to the HFile. When the number of HFiles reaches a threshold, multiple HFiles are merged into one HFile, and the data is processed according to the labels in the HFile. When there is too much data to delete in batches, a large number of write requests reach the RegionServer, putting significant write pressure on it and consuming substantial CPU, memory, and bandwidth resources. This can also lead to resource contention with other services and affect cluster stability. Summary of the Invention

[0006] To address the problem of high data write requests putting pressure on the RegionServer in the prior art, this embodiment provides an HBase data processing method and apparatus.

[0007] This embodiment provides an HBase data processing method, which includes: filtering data to be processed from an initial HBase table according to filtering conditions; storing the filtered data to be processed in an isolated storage area that is isolated from the existing data in the initial HBase table; generating at least one HFile based on the data to be processed in the isolated storage area; loading the HFile into a target HBase table respectively; and processing the data to be processed in the target HBase table according to the processing instructions after receiving processing instructions.

[0008] According to one aspect of the embodiments herein, filtering data to be processed from an HBase initial table according to filtering criteria includes: establishing a Hive external table based on the HBase initial table, the Hive external table pointing to all data in the HBase initial table; and using HQL instructions to filter data to be processed from the Hive external table according to filtering criteria.

[0009] According to one aspect of the embodiments herein, the step of storing the filtered data to be processed in isolation from the existing data in the HBase initial table includes: inserting the filtered data to be processed into a Hive internal table, wherein the HDFS storage area corresponding to the Hive internal table is isolated from the storage area of ​​the existing data in the HBase initial table.

[0010] According to one aspect of the embodiments herein, generating at least one HFile based on the data to be processed in an isolated storage area includes: reading the data to be processed from the isolated storage area; determining the value corresponding to the primary key from an HBase initial table based on the primary key of the data to be processed to form key-value pairs of the data to be processed; and integrating the data to be processed into an HFile with a corresponding primary key based on each key-value pair of the data to be processed.

[0011] According to one aspect of the embodiments herein, loading the HFile into the HBase target table includes: determining the primary key of the data to be processed contained in each HFile; mapping each HFile to a corresponding partition according to the primary key, wherein different partitions have a corresponding relationship with different primary keys; when the HFile is moved to the storage area to which the corresponding partition belongs, notifying the RegionServer corresponding to each partition to load the corresponding HFile, thereby loading the HFile into the HBase target table.

[0012] According to one aspect of the embodiments herein, the method further includes: setting different types of processing labels for the data to be processed based on the Type attribute of the data to be processed; and performing corresponding processing on the data to be processed in the HBase target table according to the processing labels and the processing instructions.

[0013] This embodiment provides an HBase data processing apparatus, comprising: a filtering unit for filtering data to be processed from an initial HBase table according to filtering conditions; an isolation unit for storing the filtered data to be processed in an isolated storage area isolated from the existing data in the initial HBase table; an HFile generation unit for generating at least one HFile based on the data to be processed in the isolated storage area; and a processing unit for loading the HFile into an HBase target table and, upon receiving a processing instruction, processing the data to be processed in the HBase target table accordingly.

[0014] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the HBase data processing method.

[0015] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the HBase data processing method.

[0016] This embodiment also provides a computer program product, which includes a computer program that, when executed by a processor, implements the HBase data processing method.

[0017] This article describes how to batch filter out the data to be processed, process the underlying HBase data, and directly generate the underlying HFile files using MapReduce. This avoids sending a large number of write requests directly to the RegionServer for processing, thereby reducing the write pressure on the RegionServer caused by a large number of write requests, reducing the additional consumption of memory and bandwidth resources, and ensuring cluster stability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The diagram shown is a flowchart of an HBase data processing method according to an embodiment of this article;

[0020] Figure 2 The diagram shown is a flowchart of a method for filtering data to be processed according to an embodiment of this article;

[0021] Figure 3 The diagram shown is a flowchart of a method for generating an HFile according to an embodiment of this article;

[0022] Figure 4 The diagram shown is a flowchart of a method for loading an HFile according to an embodiment of this article;

[0023] Figure 5 The diagram shown is a flowchart of a method for processing data to be processed according to an embodiment of this article;

[0024] Figure 6 The diagram shown is a structural schematic of an HBase data processing device according to an embodiment of this paper.

[0025] Figure 7 The diagram shown is a schematic representation of the specific structure of the device in the embodiment of this article;

[0026] Figure 8 The diagram shown is a structural schematic of a computer device according to an embodiment of this article.

[0027] Explanation of the filter unit number for the attached figure:

[0028] 601. Filtering Unit;

[0029] 6011. External appearance creation module;

[0030] 602. Isolation Unit;

[0031] 603, HFile generation unit;

[0032] 6031. Data reading module;

[0033] 6032. Key-value pair determination module;

[0034] 6033, Integration Module;

[0035] 604. Processing Unit;

[0036] 6041. Label setting module;

[0037] 802. Computer equipment;

[0038] 804, Processor;

[0039] 806. Memory;

[0040] 808. Drive mechanism;

[0041] 810. Input / Output Module;

[0042] 812. Input devices;

[0043] 814. Output devices;

[0044] 816. Presentation equipment;

[0045] 818. Graphical User Interface;

[0046] 820. Network interface;

[0047] 822. Communication link;

[0048] 824. Communication bus. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments herein will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments herein, and not all of the embodiments. Based on the embodiments herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0050] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0051] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or device products, the methods shown in the embodiments or accompanying drawings can be executed sequentially or in parallel. The acquisition, storage, use, and processing of data in the technical solutions of this application all comply with the relevant provisions of national laws and regulations.

[0052] It should be noted that the HBase data processing method described in this article can be used in the fields of big data and fintech. This article does not limit the application areas of the HBase data processing method and device.

[0053] Figure 1 The diagram shown is a flowchart of an HBase data processing method according to an embodiment of this article, which specifically includes the following steps:

[0054] Step 101: Filter the data to be processed from the initial HBase table according to the filtering criteria. In this step, the HBase table records all the data. Based on business needs or development and testing requirements, the data to be processed needs to be filtered from the initial HBase table. The initial HBase table is the cluster storing the data. The data to be processed includes data to be deleted, added, or modified. In this step, HQL statements can be used to filter the data to be processed from the HBase table, including but not limited to: one or more records of data to be processed, one or more column families of data to be processed, and batches of data to be processed.

[0055] Step 102 involves storing the filtered data to be processed in an isolated storage area, separate from the existing data in the initial HBase table. In this step, the filtered data to be processed is inserted into another table. The data in this table resides in a different storage area than the existing data in the initial HBase table; that is, the data to be processed is stored isolated from the existing data in the initial HBase table, and they do not affect each other. This application stores the data to be processed and the existing data independently, facilitating the separate retrieval and tagging of the data to be processed, eliminating the need for the RegionServer to tag the data to be processed.

[0056] Step 103: Generate at least one HFile based on the data to be processed in the isolated storage area. In this step, the data to be processed selected from the initial HBase table can be in text format. This application converts the format of the data to be processed into key-value pair format, and then further aggregates the data to be processed into HFiles, which are finally loaded into the target HBase table. Here, HFile is the file format for data storage.

[0057] Step 104: Load the HFiles into the target HBase table respectively. After receiving the processing instruction, process the data to be processed in the target HBase table according to the processing instruction. Figure 2 The diagram shown is a flowchart of a method for filtering data to be processed according to an embodiment of this paper, which specifically includes the following steps:

[0058] Step 201: Based on the HBase initial table, create a Hive external table. The Hive external table is a mapping of the HBase initial table and points to all the data in the HBase initial table. In some embodiments of this specification, assuming the HBase initial table `test_table` has a column family `A` with three columns, then four fields need to be created in the Hive external table. The first field corresponds to the row key of the HBase initial table, and the remaining fields correspond one-to-one with other fields in the HBase initial table.

[0059] Step 202: Based on the filtering criteria, use HQL commands to filter the data to be processed from the Hive table. In this step, HQL statements can be used to select the statements to be processed. For example, by specifying certain fields in an HQL command or setting filtering criteria and expressing them in the form of HQL commands, the data to be processed can be filtered out from the Hive table.

[0060] For example, a cluster has an HBase table A that stores mobile banking transaction details. As the amount of mobile banking transaction data increases, the efficiency of querying data from table A slows down. Therefore, the query scenario can be optimized. This can be achieved by creating a new table B, splitting the data stored in HBase table A. The goal is to use the split table A to store transaction data for the most recent 7 days, and table B to store transaction data older than 7 days.

[0061] Therefore, a Hive external table is constructed, containing all the data from HBase table A. Based on this, data to be processed is selected from the Hive table according to certain filtering criteria. The data to be processed selected from the Hive external table is then inserted into an internal Hive table, forming a new table A. Table B is created to store the remaining data after the data to be processed has been deleted from the Hive external table.

[0062] In some embodiments of this specification, data earlier than a certain date can be filtered based on the transaction time period (e.g., the detail_time field) in the original HBase table; this is historical data. For example, to delete historical data older than 7 days from HBase table A, if it is necessary to delete data before March 1, 2023, then the data corresponding to the transaction field before March 1, 2023 needs to be deleted: detail_time < '20230301'. An exemplary HQL statement is shown below:

[0063] CREATE TABLE hive_tmp_table_name

[0064] The HQL statement `ROW FORMAT DELIMITED FIELDS TERMINATED BY '\27' STORED AS TEXTFILE AS select rowkey,column_1,column_2,column_3 from hive_external_table_name where detail_time<'20230301'` creates a new table based on data from an external table that occurred before March 1, 2023. In this application, different filtering conditions can be set based on different databases and data volumes; however, this application does not limit the specific filtering conditions.

[0065] In some embodiments of this specification, storing the filtered data to be processed in isolation from the existing data in the HBase initial table includes: inserting the data to be processed from the Hive external table into the Hive internal table, wherein the HDFS storage area corresponding to the Hive internal table is isolated from the storage area of ​​the existing data in the HBase initial table. In this application, after the data to be processed is filtered from the Hive external table, it is inserted into the Hive internal table. The Hive internal table is a temporarily constructed table, and the data in the Hive internal table resides on HDFS, and its storage location is independent of the storage area of ​​other existing data in the HBase initial table besides the data to be processed. That is, the storage location of the data to be processed is not affected by the storage location of the existing data in the HBase initial table.

[0066] Figure 3 The diagram shown is a flowchart of a method for generating an HFile according to an embodiment of this paper, which specifically includes the following steps:

[0067] Step 301: Read the data to be processed from the isolated storage area. In this step, by running a distributed computing system (MapReduce) task, the MapReduce mapper reads the data to be processed from the isolated storage area and assembles it into a composite KeyValue format. The primary key of the data to be processed is the rowkey of the data to be processed recorded in the HBase initial table. Specifically, the rowkey in the HBase initial table is used to uniquely identify a row of data in the HBase initial table. Based on the primary key of the data to be processed, the rowkey in the HBase initial table can be determined, and a row of data in HBase containing that rowkey can be read.

[0068] Step 302: determining, according to the primary key of the data to be processed, a value corresponding to the primary key from an initial HBase table to form a key-value pair of the data to be processed. In this step, in addition to recording the rowkey of the data to be processed, the initial HBase table further includes the value corresponding to the rowkey of the data to be processed. Therefore, the Value corresponding to the primary key key of the data to be processed can be determined from the value corresponding to the primary key of the data to be processed in the initial HBase table.

[0069] Specifically, a row of data is read from an internal Hive table, and the format of a row of data can be expressed as follows:

[0070] id□column_1□column_2□column_3, wherein □ represents a delimiter. For example: 1□Xiaoming□20230221□30.5.

[0071] The data is parsed according to the delimiter, and it can be determined that the rowkey of the row of data in the internal Hive table is 1, so that corresponding value values can be parsed from HBase field by field. Further, if the row of data needs to be deleted, the row of data only needs to be determined according to the rowkey, and there is no need to parse field values. Therefore, the following code can be used to construct a KeyValue object.

[0072]

[0073]

[0074] Step 303: integrating the data to be processed into HFiles with corresponding primary keys according to the key-value pair of each piece of data to be processed. Specifically, according to the primary keys of a plurality of pieces of data to be processed, the data to be processed with different primary keys are respectively aggregated into different HFiles. The HFiles are distinguished according to the difference of the primary keys of the data to be processed comprised therein. For example, if there are currently 100 pieces of data to be processed in the KeyValue form, and these data to be processed need to be aggregated into 10 HFiles respectively, the corresponding HFile is determined according to the primary key of the data to be processed. The primary keys of the data to be processed comprised in the HFile named A are

[0075] Figure 4 shown in the flow chart of a method for loading an HFile in an embodiment of the present application, which specifically comprises the following steps:

[0076] Step 401: Determine the primary key of the data to be processed contained in each HFile. Each HFile has a different primary key or primary key range. For example, the first HFile has primary key 1, the second HFile has primary key 2, and the third HFile has primary key 3. Alternatively, the first HFile has a primary key range from primary key 1 to primary key 5, and the second HFile has a primary key range from primary key 6 to primary key 10.

[0077] Step 402: Based on the primary key, map each HFile to its corresponding partition, wherein different partitions correspond to different primary keys. In some embodiments of this specification, a partition is a data shard of a data table in HBase, and each partition is responsible for accessing and storing data corresponding to a certain range of primary keys. For example, the first HFile has a primary key range of primary key 1 to primary key 5, and the second HFile has a primary key range of primary key 6 to primary key 10.

[0078] The primary key range for partition_K is primary key 1 to primary key 5, and the primary key range for partition_M is primary key 6 to primary key 10. Based on the primary key range contained in each HFile, the first HFile is mapped to partition_K, and the second HFile is mapped to partition_M.

[0079] Step 403: When the HFile is moved to the storage area corresponding to the partition, the RegionServer corresponding to each partition is notified to load the corresponding HFile, thus loading the HFile into the HBase target table. Further, the HFile is moved to the HDFS file directory where the corresponding partition is located. Once a partition has a corresponding HFile, the RegionServer to which the partition belongs is notified to load the HFile for each partition. In this application, RegionServer is a service running on each worker node of the HBase cluster. RegionServer is used to maintain the status of partitions and the correspondence between partitions and primary key rowkeys, providing management and services for partitions. Specifically, a RegionServer can include one or more partitions. A RegionServer is responsible for reading and writing data to multiple partitions. Therefore, when a RegionServer receives notification that a partition already has an HFile, it loads the HFile corresponding to each partition into the HBase target table corresponding to the RegionServer. Thus, the RegionServer only performs the HFile loading action after receiving a notification from the partition. Compared to the traditional process, the number of events processed by the RegionServer is greatly reduced, and the processing pressure is significantly lowered, alleviating the workload of the RegionServer.

[0080] Figure 5The diagram shown is a flowchart of a method for processing data according to an embodiment of this paper, which specifically includes the following steps:

[0081] Step 501: Based on the Type attribute of the data to be processed, set different types of processing tags for the data to be processed.

[0082] In the aforementioned steps of this application, after determining the data to be processed based on the filtering criteria, the data to be processed is inserted into an internal Hive table and converted into an HFile file. After MapReduce converts the data to be deleted into an HFile file, different types of data processing tags can be set for the HFile file during the MapReduce stage, including but not limited to: delete tags, modify tags, and add tags. Specifically, processing tags can be set directly for the HFile containing a batch of data to be processed, or a corresponding tag can be set for each piece of data to be processed.

[0083] For example, if a batch of data needs to be deleted based on filtering criteria, after converting the batch of data to be deleted into HFiles, MapReduce directly sets the deletion tags on the HFiles. In this application, different processing tags are set for the data to be processed based on the Type attribute of the data to be processed, replacing the work of setting tags for data by the conventional RegionServer, thus reducing the pressure on the RegionServer.

[0084] Step 502: Process the data to be processed in the HBase target table according to the processing tag and the processing instruction. After the HFile containing the data to be processed is loaded into the HBase target table, process the data to be processed from the HBase target table according to the processing tag of the HFile or the processing tag of the data to be processed. After triggering the major compact instruction, batch deletion, modification or addition of data is realized.

[0085] This paper utilizes HQL provided by Hive as the query language to batch filter out the data to be processed, and directly generates the underlying HFile file through MapReduce. This avoids sending a large number of write requests directly to the RegionServer for processing, which can reduce the write pressure on the RegionServer caused by a large number of write requests, reduce the additional consumption of memory and bandwidth resources, and ensure the stability of the cluster.

[0086] like Figure 6The diagram shown is a structural schematic of an HBase data processing device according to an embodiment of this paper. The basic structure of the HBase data processing device is illustrated in this diagram. The functional units and modules can be implemented in software, or HBase data processing can be implemented using general-purpose chips or specific chips. Specifically, the HBase data processing device includes:

[0087] The filtering unit 601 is used to filter data to be processed from the initial HBase table according to the filtering conditions;

[0088] Isolation unit 602 is used to store the filtered data to be processed in an isolated storage area that is isolated from the existing data in the HBase initial table;

[0089] HFile generation unit 603 is used to generate at least one HFile based on the data to be processed in the isolated storage area;

[0090] The processing unit 604 is used to load the HFile into the HBase target table, and after receiving the processing instruction, to process the data to be processed in the HBase target table accordingly.

[0091] As one embodiment of this article, reference may also be made to, for example, Figure 7 The diagram shown is a schematic representation of the specific structure of an HBase data processing device according to this embodiment.

[0092] As one embodiment of this document, the filtering unit 601 further includes:

[0093] The external table creation module 6011 is used to create a Hive external table based on the HBase initial table, wherein the Hive external table points to the full data in the HBase initial table;

[0094] As one embodiment of this document, the HFile generation unit 603 further includes:

[0095] The data to be processed reading module 6031 is used to read data to be processed from the isolated storage area;

[0096] The key-value pair determination module 6032 is used to determine the value corresponding to the primary key from the HBase initial table based on the primary key of the data to be processed, so as to form a key-value pair of the data to be processed;

[0097] The integration module 6033 is used to integrate the data to be processed into an HFile with a corresponding primary key based on the key-value pair of each data to be processed.

[0098] As one embodiment of this document, the processing unit 604 further includes:

[0099] The processing label setting module 6041 is used to set different types of processing labels for the data to be processed according to the Type attribute of the data to be processed.

[0100] like Figure 8 As shown, a computer device is provided in this embodiment, and the HBase data processing method can be executed by the computer device. The computer device 802 may include one or more processors 804, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. The computer device 802 may also include any memory 806 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, the memory 806 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 802. In one case, when the processor 804 executes associated instructions stored in any memory or combination of memories, the computer device 802 can perform any operation of the associated instructions. The computer device 802 also includes one or more drive mechanisms 808 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0101] Computer device 802 may also include an input / output module 810 (I / O) for receiving various inputs (via input device 812) and providing various outputs (via output device 814). A specific output mechanism may include a presentation device 816 and an associated graphical user interface (GUI) 818. In other embodiments, the input / output module 810 (I / O), input device 812, and output device 814 may be omitted, and the device may function solely as a computer device within a network. Computer device 802 may also include one or more network interfaces 820 for exchanging data with other devices via one or more communication links 822. One or more communication buses 824 couple the components described above together.

[0102] Communication link 822 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 822 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0103] Corresponding to Figures 1 to 5In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0104] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figures 1 to 5 The method shown.

[0105] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0106] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0107] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0108] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0109] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0111] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. An HBase data processing method, characterized in that, The method includes: Based on the filtering criteria, filter the data to be processed from the initial HBase table; The filtered data to be processed is stored in an isolated storage area that is isolated from the existing data in the initial HBase table; Based on the data to be processed in the isolated storage area, generate at least one HFile; The HFiles are loaded into the target HBase table respectively. After receiving the processing instruction, the data to be processed in the target HBase table is processed according to the processing instruction. The process of filtering data to be processed from the initial HBase table according to the filtering criteria includes: Based on the initial HBase table, create a Hive external table, which points to all the data in the initial HBase table; Based on the filtering criteria, HQL commands are used to filter the data to be processed from the Hive table. The step of generating at least one HFile based on the data to be processed in the isolated storage area includes: Read the data to be processed from the isolated storage area; Based on the primary key of the data to be processed, determine the value corresponding to the primary key from the HBase initial table to form key-value pairs of the data to be processed. Based on the key-value pairs of each data to be processed, the data to be processed is integrated into an HFile with a corresponding primary key; wherein, the HFiles are distinguished according to the different primary keys containing the data to be processed. The method further includes: Based on the Type attribute of the data to be processed, different types of processing tags are set for the data to be processed; wherein, the processing tags include: delete tags, modify tags, and add tags; According to the processing label and the processing instruction, the data to be processed in the target table of HBase is processed accordingly.

2. The HBase data processing method according to claim 1, characterized in that, The step of isolating and storing the filtered data to be processed from the existing data in the initial HBase table includes: The filtered data to be processed is inserted into a Hive internal table, wherein the HDFS storage area corresponding to the Hive internal table is isolated from the storage area of ​​the existing data in the HBase initial table.

3. The HBase data processing method according to claim 1, characterized in that, Loading the HFile into the target HBase table includes: Determine the primary key of the data to be processed contained in each HFile file; Based on the primary key, each HFile is mapped to a corresponding partition, wherein different partitions have a corresponding relationship with different primary keys; When the HFile is moved to the storage area of ​​the corresponding partition, the RegionServer corresponding to each partition is notified to load the corresponding HFile, thereby loading the HFile into the HBase target table.

4. An HBase data processing device, characterized in that, The device includes: The filtering unit is used to filter data to be processed from the initial HBase table according to the filtering criteria; An isolation unit is used to store the filtered data to be processed in an isolated storage area that is isolated from the existing data in the initial HBase table; The HFile generation unit is used to generate at least one HFile based on the data to be processed in the isolated storage area. The processing unit is used to load the HFile into the HBase target table, and after receiving the processing instruction, to process the data to be processed in the HBase target table accordingly. The filtering unit further includes: The external table creation module is used to create a Hive external table based on the HBase initial table. The Hive external table points to the full data in the HBase initial table. Based on the filtering conditions, HQL commands are used to filter the data to be processed from the Hive external table. The HFile generation unit includes: a data to be processed reading module, used to read data to be processed from the isolated storage area; The key-value pair determination module is used to determine the value corresponding to the primary key from the HBase initial table based on the primary key of the data to be processed, so as to form key-value pairs of the data to be processed. An integration module is used to integrate the data to be processed into an HFile with a corresponding primary key based on the key-value pair of each data to be processed; wherein the HFiles are distinguished based on the different primary keys containing the data to be processed. The processing unit further includes: The processing tag setting module is used to set different types of processing tags for the data to be processed according to the Type attribute of the data to be processed; wherein, the processing tags include: delete tags, modify tags, and add tags; The processing unit is specifically used for: According to the processing label and the processing instruction, the data to be processed in the target table of HBase is processed accordingly.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 3.

7. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 3.

Images