HBase real-time major compact method, system and device

Abstract

The present invention relates to the technical field of data merging, and more specifically, to a real-time HBase Major Compact method, system, and device. The method obtains a pre-set merge start time. If the merge start time is reached, a pre-established real-time Major Compact plan is obtained. Based on the real-time Major Compact plan, a region server and a merge trigger time are assigned to each region in an HBase cluster to perform the merge. By assigning a region server and a merge trigger time to each region in the HBase cluster, the present application enables real-time HBase cluster merging, avoiding the problem of existing Major Compact methods that concentrate all merge tasks at night, resulting in high system resource consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of data merging, and in particular to an HBase real-time Major Compact method, system and device. Background Art

[0002] Major compaction is an HBase self-optimization operation. Conventional technology manually sets the time for major compactions, triggering them during off-peak hours. Typically, evenings and early mornings are off-peak hours. For example, for table A, a major compaction can be triggered every Monday evening at 8:00 PM. Small tables under 1 TB are typically completed within an hour, with relatively minimal impact on business reads and writes. However, for large tables of 20 to 30 TB, a major compaction typically lasts 10 hours, significantly impacting business reads and writes. Even when executed at night during off-peak hours, this can have a significant impact. This is because major compactions consume a significant amount of system resources, including I / O, memory, CPU, and network resources. In severe cases, this can lead to system service unavailability and often cause system failures. Summary of the Invention

[0003] In view of this, the object of the present invention is to provide an HBase real-time Major Compact method, system and device to overcome the current problem that Major Compact consumes a lot of system resources.

[0004] To achieve the above objectives, the present invention adopts the following technical solutions:

[0005] In a first aspect, the present application provides an HBase real-time Major Compact method, comprising:

[0006] Get the preset start merge time;

[0007] If the merging start time is reached, a pre-established real-time Major Compact plan is obtained;

[0008] According to the real-time Major Compact plan, each region in the HBase cluster is assigned a regionserver and a merge trigger time, and then merged.

[0009] Furthermore, the above method, before obtaining the preset merging start time, further includes:

[0010] Get the regions of all tables in the HBase cluster, calculate their number, and number the regions;

[0011] Set the merge period and the concurrent number rs_concurrent of major compactions for each region server in the HBase cluster;

[0012] Get the number of region servers in the hbase cluster;

[0013] Calculate the number of concurrent operations that the HBase cluster can perform at the same time based on the number of region servers and the concurrent number rs_concurrent of Major Compact performed by each region server;

[0014] According to the number of regions, the region numbers, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, a region server and a merge trigger time are allocated to each region in the HBase cluster, and a real-time Major Compact plan is established.

[0015] Furthermore, the method described above, wherein the method assigns a region server and a merge trigger time to each region in the HBase cluster based on the number of regions, the region numbers, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, and establishes a real-time Major Compact plan, includes:

[0016] Based on the number of regions, the region numbers, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, a real-time Major Compact plan is established by allocating region servers and merging trigger times to each region in the HBase cluster in minutes.

[0017] Furthermore, the above method further includes:

[0018] Get the current time;

[0019] Calculate the time difference between the current time and the start merging time;

[0020] Determine whether the time difference reaches the merging period. If so, reassign region servers and merging trigger times to each region in the HBase cluster according to the real-time Major Compact plan, and perform merging.

[0021] In a second aspect, the present application provides an HBase real-time Major Compact system, including:

[0022] Time detection module, used to determine whether the merging start time has arrived;

[0023] The merging module is used to obtain a pre-established real-time Major Compact plan if the start merging time is reached, and assign a region server and a merging trigger time to each region in the HBase cluster according to the real-time Major Compact plan to perform the merging.

[0024] In a third aspect, the present application provides an HBase real-time Major Compact device, comprising a processor and a memory, wherein the processor is connected to the memory:

[0025] The processor is configured to call and execute the program stored in the memory;

[0026] The memory is used to store the program, and the program is at least used to execute any one of the methods described above.

[0027] The beneficial effects of the present invention are:

[0028] This application obtains a pre-set merge start time. If the merge start time is reached, it obtains a pre-established real-time Major Compact plan. Based on the real-time Major Compact plan, it assigns a region server and a merge trigger time to each region in the HBase cluster to perform the merge. This application implements real-time HBase cluster merges by assigning a region server and a merge trigger time to each region in the HBase cluster. This avoids the problem of existing Major Compact that concentrates all merge tasks at night, resulting in a large consumption of system resources by Major Compact. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0030] Figure 1 This is a flowchart provided by an embodiment of an HBase real-time Major Compact method of the present invention;

[0031] Figure 2 This is a structural diagram of an HBase real-time Major Compact system provided by an embodiment of the present invention;

[0032] Figure 3 This is a structural diagram provided by an embodiment of an HBase real-time Major Compact device of the present invention. DETAILED DESCRIPTION

[0033] To make the objectives, technical solutions, and advantages of the present invention more apparent, the technical solutions of the present invention will be described in detail below. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other implementations obtained by those of ordinary skill in the art without inventive effort are within the scope of protection of the present invention.

[0034] Major compaction is an HBase self-optimization operation. Conventional technology manually sets the time for major compactions, triggering them during off-peak hours. Typically, evenings and early mornings are off-peak hours. For example, for table A, a major compaction can be triggered every Monday evening at 8:00 PM. Small tables under 1 TB are typically completed within an hour, with relatively minimal impact on business reads and writes. However, for large tables of 20 to 30 TB, a major compaction typically lasts 10 hours, significantly impacting business reads and writes. Even when executed at night during off-peak hours, this can have a significant impact. This is because major compactions consume a significant amount of system resources, including I / O, memory, CPU, and network resources. In severe cases, this can lead to system service unavailability and often cause system failures.

[0035] In view of this, the object of the present invention is to provide an HBase real-time Major Compact method, system and device to overcome the current problem that Major Compact consumes a lot of system resources.

[0036] Figure 1 This is a flowchart provided by an embodiment of the HBase real-time Major Compact method of the present invention. Figure 1 , this embodiment may include the following steps:

[0037] S1. Obtain a preset merging start time.

[0038] S2. If the merge start time is reached, obtain the pre-established real-time Major Compact plan.

[0039] S3: According to the real-time Major Compact plan, each region in the HBase cluster is assigned a region server and a merge trigger time, and then merged.

[0040] It should be noted that Minor Compact is used to select some small, adjacent Store Files and merge them into a larger one. This process will not process deleted or expired cells.

[0041] Major Compact: Merges all Store Files in the HStore within a Region into a single file. This process cleans up three types of meaningless data: deleted data, data with expired time-to-live (TTL) times, and data with a version number exceeding the specified value. This operation takes a long time, consumes significant CPU and I / O resources, and significantly impacts upper-layer services. Therefore, this operation is typically disabled online and manually triggered during off-peak hours.

[0042] As you can understand, this embodiment obtains a pre-set merge start time. If the merge start time is reached, it retrieves a pre-established real-time Major Compact plan. Based on the real-time Major Compact plan, it assigns a region server and a merge trigger time to each region in the HBase cluster, and then performs the merge. This embodiment implements real-time HBase cluster merges by assigning a region server and a merge trigger time to each region in the HBase cluster. This avoids the problem of existing Major Compact methods that concentrate all merge tasks at night, resulting in high system resource consumption.

[0043] Preferably, before step S1, the method further includes:

[0044] Get the regions of all tables in the HBase cluster, calculate their number, and number the regions;

[0045] Set the merge cycle and the concurrent number of Major Compacts (rs_concurrent) for each region server in the HBase cluster;

[0046] Get the number of region servers in the HBase cluster;

[0047] Based on the number of region servers and the concurrent number of Major Compacts performed by each region server (rs_concurrent), calculate the number of concurrent operations that the HBase cluster can perform at the same time.

[0048] Based on the number of regions, region numbers, number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, a region server and merge trigger time are assigned to each region in the HBase cluster to establish a real-time Major Compact plan.

[0049] Preferably, a real-time Major Compact plan is established based on the number of regions, region numbers, number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, by assigning a region server and merging trigger times to each region in the HBase cluster, including:

[0050] Based on the number of regions, region numbers, number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, a real-time Major Compact plan is established by assigning region servers and merging trigger times to each region in the HBase cluster in minutes.

[0051] Preferably, it also includes:

[0052] Get the current time;

[0053] Calculate the time difference between the current time and the start time of merging;

[0054] Determine whether the time difference reaches the merge period. If so, reassign the region server and merge trigger time to each region in the HBase cluster according to the real-time Major Compact plan and merge.

[0055] In practice, select a compact start time, compact_start_time, with a time granularity of minutes. For example, here we select 2022-08-16 12:10 as compact_start_time. Specify the compact merge period, compact_peroid, for example, 10 days. Specify the concurrent number rs_concurrent for each region server to perform Major Compact, for example, 2. Obtain the regions of all tables in the HBase cluster, calculate their total number, total_rs_num, for example, 3000, and number each region 1, 2, 3, .... Calculate the total merge time (in minutes), for example, if compact_start_time is 2022-08-16 12:10, the merge period is 10 days, and the total time is 14400. First, obtain the number of region servers in the HBase cluster, which is 30, and number each region server rs1, rs2, ..., rs30. Calculate the number of concurrent merges that can be triggered at the same time. Step (6) gives the number of cluster machines, 30. Multiply this by the rs_concurrent value of 2 specified in step (3), resulting in a concurrency of 60. Loop through all regions in the HBase cluster, assigning each region a corresponding machine and a trigger merge time. For example, regions 1-60 are scheduled to trigger a merge at 2022-08-16 12:10. Regions 1 and 3 will be executed on machine rs1, and regions 4 and 5 will be executed on machine rs2, and so on. Regions 61-120 are scheduled to trigger a merge at 2022-08-16 12:11, and so on, until all regions are assigned. At the fixed time, the corresponding region will be triggered to merge.

[0056] The present invention also provides an HBase real-time Major Compact system for implementing the above method embodiment. Figure 2 This is a structural diagram of an embodiment of an HBase real-time Major Compact system provided by the present invention. Figure 2 Shown, including:

[0057] Time detection module 1, used to determine whether the merging start time has arrived;

[0058] Merge module 2 is used to obtain the pre-established real-time Major Compact plan if the merge start time is reached, and assign a region server and merge trigger time to each region in the HBase cluster according to the real-time Major Compact plan for merging.

[0059] Preferably, a real-time Major Compact plan establishment module is also included, specifically including:

[0060] The region information update unit is responsible for collecting HBase region information and saving each region's name, size, region server, table, and other information to the database.

[0061] The Compact configuration unit is responsible for collecting user-set information such as "Compact start time", "table information not participating in Compact", "merge cycle", "number of concurrent requests for each region server", and "Compact plan update time", and saves the results to the database.

[0062] The Compact planning unit implements a specific planning algorithm and generates the final Region Compact plan. First, information such as the "Compact start time," "Merge period," and "Number of concurrent requests per region server" is retrieved from the database. Regions for all tables in the HBase cluster are then obtained, numbered 1, 2, or 3. The total merge time and the number of concurrent requests that can trigger a merge at the same time are calculated. All regions in the HBase cluster are then looped through, assigning each region a specific machine and a triggering merge time. After this process is complete, a Region Compact plan is generated and saved in the database.

[0063] The Compact execution unit sends a Major Compact request to a specific regionserger machine in the HBase cluster for each region according to the region compaction plan, the triggering time and the triggering machine of each region, and enables the regionserger to execute Major Compact on the region.

[0064] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0065] The present invention also provides an HBase real-time Major Compact device for implementing the above method embodiment. Figure 3 This is a structural diagram of an embodiment of an HBase real-time Major Compact device provided by the present invention. Figure 3As shown, an HBase real-time Major Compact device in this embodiment includes a processor 21 and a memory 22, wherein the processor 21 is connected to the memory 22. The processor 21 is used to call and execute the program stored in the memory 22; the memory 22 is used to store the program, which is used to at least execute the method in the above embodiment.

[0066] The specific implementation scheme of an HBase real-time Major Compact device provided in the embodiment of the present application can refer to the implementation scheme of an HBase real-time Major Compact method in any of the above embodiments, which will not be repeated here.

[0067] It can be understood that the same or similar parts of the above embodiments can be referenced to each other, and the contents not described in detail in some embodiments can refer to the same or similar contents in other embodiments.

[0068] It should be noted that, in the description of the present invention, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, the meaning of "plurality" is at least two.

[0069] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

[0070] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0071] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0072] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

[0073] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0074] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0075] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A real-time major compaction method for HBase, characterized in that: include: Get the regions of all tables in the HBase cluster, calculate their number, and number the regions; Set the merge period and the concurrent number rs_concurrent of major compactions for each region server in the HBase cluster; Get the number of region servers in the hbase cluster; Calculate the number of concurrent operations that the HBase cluster can perform at the same time based on the number of region servers and the concurrent number rs_concurrent of Major Compact performed by each region server; Assign a region server and merge trigger times to each region in the HBase cluster based on the number of regions, the region ID, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, and establish a real-time Major Compact plan; Get the preset start merge time; If the merging start time is reached, a pre-established real-time Major Compact plan is obtained; According to the real-time Major Compact plan, each region in the HBase cluster is assigned a region server and a merge trigger time, and then merged.

2. The method according to claim 1, characterized in that The method allocates a region server and merges trigger times to each region in the HBase cluster based on the number of regions, the region numbers, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, and establishes a real-time Major Compact plan, including: Based on the number of regions, the region numbers, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, a real-time Major Compact plan is established by allocating region servers and merging trigger times to each region in the HBase cluster in minutes.

3. The method according to claim 2, characterized in that Also includes: Get the current time; Calculate the time difference between the current time and the start merging time; Determine whether the time difference reaches the merging period. If so, reassign region servers and merging trigger times to each region in the HBase cluster according to the real-time Major Compact plan, and perform merging.

4. An HBase real-time Major Compact system, characterized in that: include: A real-time Major Compact plan building module is used to obtain the regions of all tables in the HBase cluster, calculate their number, and number the regions; Set the merge period and the concurrent number rs_concurrent of major compactions for each region server in the HBase cluster; Get the number of region servers in the hbase cluster; Calculate the number of concurrent operations that the HBase cluster can perform at the same time based on the number of region servers and the concurrent number rs_concurrent of Major Compact performed by each region server; Assign a region server and merge trigger times to each region in the HBase cluster based on the number of regions, the region ID, the number of region servers, and the number of concurrent operations that can be performed by the HBase cluster at the same time, and establish a real-time Major Compact plan; Time detection module, used to determine whether the merging start time has arrived; The merging module is used to obtain a pre-established real-time Major Compact plan if the start merging time is reached, and assign a region server and a merging trigger time to each region in the HBase cluster according to the real-time Major Compact plan to perform the merging.

5. An HBase real-time Major Compact device, characterized in that: The device comprises a processor and a memory, wherein the processor is connected to the memory: The processor is configured to call and execute the program stored in the memory; The memory is used to store the program, and the program is at least used to execute the method according to any one of claims 1 to 3.

Images