Node processing method and apparatus in hbase cluster

Abstract

The application relates to a node processing method and device in an hbase cluster. The method comprises the following steps: receiving a node grouping instruction for grouping nodes in the hbase cluster; grouping the hbase cluster to obtain a first node group without load balancing and a second node group with load balancing; deploying nodes in the first node group according to a metadata table in the hbase cluster; and using a preset load balancing strategy to evenly distribute other data tables in the hbase cluster except the metadata table to nodes in the second node group. The method can access nodes without data transfer based on the metadata table, and then respond to customer requests for the nodes.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and apparatus for node processing in an HBase cluster. Background Technology

[0002] HBase is a distributed, column-oriented, open-source database. When managing tables in an HBase cluster, all tables are partitioned and divided into several regions (storage units), which are then evenly distributed across the nodes in the HBase cluster. These tables can be further divided into metadata tables and other tables.

[0003] In related technologies, when a client wants to access data from other data tables in a certain storage unit, it needs to first access the metadata table, obtain the node address of the storage unit to be accessed through the metadata table, and then initiate a data request based on the node address to obtain the data stored in the storage unit.

[0004] In practice, HBase clusters periodically migrate certain storage units from one node to another based on node load and failure status to balance node load. This can cause storage units to become unavailable during the migration. If the migrated storage unit corresponds to a metadata table, all storage units will become unavailable during the migration, causing the HBase cluster to be unable to respond to all client requests in the short term. Therefore, a node handling method is urgently needed in HBase clusters. Summary of the Invention

[0005] This application provides a node processing method and apparatus in an HBase cluster, which can access nodes that do not perform data transfer based on metadata tables, and then respond to client requests for those nodes.

[0006] Firstly, this application provides a method for node processing in an HBase cluster, the method comprising:

[0007] Receive node grouping instructions that group nodes in the HBase cluster;

[0008] The HBase cluster is divided into two groups: a first group of nodes that do not require load balancing and a second group of nodes that require load balancing.

[0009] Based on the metadata tables in the HBase cluster, deploy the nodes in the first node group;

[0010] Using a preset load balancing strategy, the data tables in the HBase cluster, excluding the metadata table, are evenly distributed to the nodes in the second node group.

[0011] Optionally, grouping the HBase cluster to obtain a first group of nodes that do not require load balancing and a second group of nodes that require load balancing includes:

[0012] Retrieve the metadata tables in the HBase cluster;

[0013] The metadata tables in the HBase cluster are split to obtain at least one first storage unit;

[0014] Based on the number of the first storage units, at least one node is selected in the HBase cluster, wherein the number of nodes is greater than or equal to the number of the first storage units.

[0015] Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0016] Optionally, grouping the HBase cluster to obtain a first group of nodes that do not require load balancing and a second group of nodes that require load balancing includes:

[0017] In the HBase cluster, a preset number of nodes are selected;

[0018] Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0019] Optionally, the method further includes:

[0020] Detect whether a faulty node appears in the first node group;

[0021] When the faulty node is detected, a backup node is selected from the second node group according to the number of faulty nodes.

[0022] The first storage unit in the faulty node is transferred to the backup node.

[0023] Optionally, after selecting a backup node, the method further includes:

[0024] Detect whether a second storage unit is deployed in the backup node, the second storage unit being obtained based on the other data tables;

[0025] When the backup node deploys the second storage unit, a data receiving node is determined in the second node group;

[0026] The second storage unit is then transferred to the data receiving node.

[0027] Optionally, after grouping the HBase cluster to obtain a first group of nodes that do not require load balancing and a second group of nodes that require load balancing, the method further includes:

[0028] Detect whether the load balancing strategy is being used;

[0029] When the use of the load balancing strategy is detected, the use of the load balancing strategy is suspended, and it is checked whether there is a node in the first node group that has deployed the second storage unit, which is obtained based on the other data tables;

[0030] When a node deploying a second storage unit is detected in the first node group, a data receiving node is identified in the second node group.

[0031] The second storage unit is then transferred to the data receiving node.

[0032] Optionally, the method further includes:

[0033] Check if a group cancellation command has been received for canceling a group in the HBase cluster;

[0034] When the group cancellation instruction is received, the nodes in the first node group are obtained and added to the second node group, so as to use a load balancing strategy to perform load balancing on all nodes in the second node group.

[0035] Secondly, this application provides a node processing device in an HBase cluster, the device comprising:

[0036] The receiving unit is used to receive node grouping instructions that group nodes in the HBase cluster.

[0037] A grouping unit is used to group the HBase cluster into a first node group that does not require load balancing and a second node group that requires load balancing.

[0038] The first deployment unit is used to deploy nodes in the first node group according to the metadata table in the HBase cluster;

[0039] The second deployment unit is used to use a preset load balancing strategy to evenly distribute other data tables in the HBase cluster, excluding the metadata table, so as to deploy the other data tables to the nodes in the second node group.

[0040] Optional, grouping units, used for:

[0041] Retrieve the metadata tables in the HBase cluster;

[0042] The metadata tables in the HBase cluster are split to obtain at least one first storage unit;

[0043] Based on the number of the first storage units, at least one node is selected in the HBase cluster, wherein the number of nodes is greater than or equal to the number of the first storage units.

[0044] Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0045] Optional, grouping units, used for:

[0046] In the HBase cluster, a preset number of nodes are selected;

[0047] Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0048] Optionally, the device further includes a first detection unit, which is used for:

[0049] Detect whether a faulty node appears in the first node group;

[0050] When the faulty node is detected, a backup node is selected from the second node group according to the number of faulty nodes.

[0051] The first storage unit in the faulty node is transferred to the backup node.

[0052] Optionally, the device further includes a second detection unit, which is used for:

[0053] Detect whether a second storage unit is deployed in the backup node, the second storage unit being obtained based on the other data tables;

[0054] When the backup node deploys the second storage unit, a data receiving node is determined in the second node group;

[0055] The second storage unit is then transferred to the data receiving node.

[0056] Optionally, the device further includes a third detection unit, which is used for:

[0057] Detect whether the load balancing strategy is being used;

[0058] When the use of the load balancing strategy is detected, the use of the load balancing strategy is suspended, and it is checked whether there is a node in the first node group that has deployed the second storage unit, which is obtained based on the other data tables;

[0059] When a node deploying a second storage unit is detected in the first node group, a data receiving node is identified in the second node group.

[0060] The second storage unit is then transferred to the data receiving node.

[0061] Optionally, the device further includes a fourth detection unit, which is used for:

[0062] Check if a group cancellation command has been received for canceling a group in the HBase cluster;

[0063] When the group cancellation instruction is received, the nodes in the first node group are obtained and added to the second node group, so as to use a load balancing strategy to perform load balancing on all nodes in the second node group.

[0064] Thirdly, this application provides a node processing device in an HBase cluster, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to:

[0065] Receive node grouping instructions that group nodes in the HBase cluster;

[0066] The HBase cluster is divided into two groups: a first group of nodes that do not require load balancing and a second group of nodes that require load balancing.

[0067] Based on the metadata tables in the HBase cluster, deploy the nodes in the first node group;

[0068] Using a preset load balancing strategy, the data tables in the HBase cluster, excluding the metadata table, are evenly distributed to the nodes in the second node group.

[0069] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described node processing method in an HBase cluster.

[0070] Compared with the prior art, the technical solution provided in this application has the following advantages: In this application embodiment, a node grouping instruction is received to group nodes in the HBase cluster, and the HBase cluster is grouped to obtain a first node group that does not require load balancing and a second node group that requires load balancing. Then, based on the metadata table in the HBase cluster, nodes in the first node group are deployed, and a preset load balancing strategy is used to evenly distribute other data tables in the HBase cluster, excluding the metadata table, to the nodes in the second node group. It is evident that when using the load balancing function, this application only performs load balancing on the nodes in the second node group, not on the nodes in the first node group, which stores the metadata table. In other words, using the load balancing function does not cause the metadata table to move, avoiding the problem of the metadata table becoming unavailable due to movement. Since the metadata table is available, nodes that do not undergo data transfer can still be accessed based on the metadata table, thereby responding to client requests for these nodes and avoiding the problem of the HBase cluster being unable to respond to all client requests in the short term. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0072] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0073] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0074] Figure 1 A flowchart illustrating a node processing method in an HBase cluster, provided as an embodiment of this application;

[0075] Figure 2 A flowchart illustrating a node group determination method provided in an embodiment of this application;

[0076] Figure 3 A flowchart illustrating a data transfer method provided in an embodiment of this application;

[0077] Figure 4 A flowchart illustrating yet another data transfer method provided in an embodiment of this application;

[0078] Figure 5 A flowchart illustrating another data transfer method provided in this application embodiment;

[0079] Figure 6 This application provides a flowchart illustrating a node processing device in an HBase cluster.

[0080] Figure 7 This is a schematic diagram of a node processing device in an HBase cluster, provided as an embodiment of this application. Detailed Implementation

[0081] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0082] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0083] First, let's briefly introduce the HBase cluster involved in this application, as follows:

[0084] HBase is a distributed, column-oriented, open-source database implemented using Hadoop and ZooKeeper. It provides real-time processing solutions for large, wide tables on top of Hadoop and utilizes ZooKeeper for service discovery. Unlike typical relational databases, HBase is well-suited for storing unstructured data. An HBase cluster contains two types of nodes: master and regionservers. The master is the cluster's management node, responsible for managing the status of all regionserver nodes and handling cluster-wide changes. Regionservers are data nodes responsible for data storage and responding to external client requests.

[0085] An HBase cluster also contains several inherent metadata tables that record the basic information of an HBase cluster. These tables are automatically created when the HBase cluster is deployed and started. The main metadata tables include the following: the meta table, which records the server addresses of all storage units; the namespace table, which records detailed information about the namespace; and the ACL table, which records all user authorization information.

[0086] Secondly, HBase clusters generally use load balancing strategies to manage all nodes. However, this management process can sometimes cause metadata tables to become unavailable. To address this, the HBase cluster can be divided into two groups: one group that doesn't require load balancing (meaning the data deployed on its nodes won't be moved), and the other group that does require load balancing. By deploying the metadata tables on nodes that don't require load balancing, the metadata tables won't be moved when load balancing is used, thus avoiding the problem of metadata table unavailability due to data movement. Because the metadata tables are available, nodes that haven't undergone data migration can still be accessed based on them, allowing for responses to client requests directed to those nodes. This prevents the HBase cluster from being unable to respond to all client requests in the short term.

[0087] In summary, this application provides a node processing method in an HBase cluster. This method can access nodes that do not perform data transfer based on metadata tables, and then respond to client requests for those nodes, such as... Figure 1 As shown, the specific steps include:

[0088] Step 101: Receive node grouping instructions to group nodes in the HBase cluster.

[0089] The node grouping command is used to instruct nodes in the HBase cluster to be grouped.

[0090] In this step, when a user wants to pin a metadata table to certain nodes, they can click the trigger button on the control terminal to group nodes in the HBase cluster. When the terminal receives the trigger command from this button, it obtains the node grouping instruction and sends it to the management node in the HBase cluster. When the management node receives the instruction, it groups the HBase cluster to pin the metadata table to a specific group of nodes.

[0091] Step 102: Group the HBase cluster to obtain the first node group that does not require load balancing and the second node group that requires load balancing.

[0092] In this step, technicians use a control terminal to group the HBase cluster. For example, the control terminal displays all the nodes in the HBase cluster. When the control terminal receives a selection command for certain nodes, it sends these selected nodes to the management node so that the management node can group the HBase cluster based on these nodes.

[0093] Of course, technicians can also pre-set grouping rules and store them on the management node in the HBase cluster. When the management node receives a node grouping instruction, it can retrieve the pre-stored grouping rules and automatically group the HBase cluster. The grouping rules can be: select a preset number of nodes from the HBase cluster and form a first node group, then form a second node group with the remaining nodes. For example, in general, since the metadata table occupies very little space, it only needs to be divided into one storage unit, so the preset number can be set to 1. Alternatively, based on the above, more nodes can be set in the first node group as backup nodes, thus setting the preset number to 3.

[0094] Furthermore, with technological advancements, the number of data tables stored in an HBase cluster will increase, leading to a larger metadata table. If the first node group includes too many nodes, it will result in resource waste; conversely, if it includes too few nodes, some data will have nowhere to be deployed. Therefore, the grouping rule can be set as follows: based on the actual needs of the metadata table, select nodes in the HBase cluster to obtain the first and second node groups. Specifically: determine the number of storage units required by the metadata table; based on this number, select a certain number of nodes in the HBase cluster; form the first node group with the selected nodes; and form the remaining nodes into the second node group.

[0095] In this step, other grouping methods can also be used to group the HBase cluster; there are no limitations here.

[0096] Step 103: Deploy the nodes in the first node group according to the metadata tables in the HBase cluster.

[0097] In this step, generally speaking, the metadata table corresponds to only one storage unit. Therefore, in this case, it is only necessary to deploy the storage unit corresponding to the metadata table to any node in the first node group. Specifically, when the metadata table corresponds to only one storage unit, the management node only needs to deploy the storage unit corresponding to the metadata table to any node in the first node group.

[0098] Furthermore, with technological advancements, the number of data tables stored in an HBase cluster will increase, leading to a larger metadata table. In this situation, when the management node partitions the metadata table, it will obtain multiple storage units. The management node can then select an unselected node from the first node group and deploy one of the storage units to that node. The management node then repeats this process until all storage units are deployed.

[0099] In this step, other deployment methods can also be used to deploy the metadata table; there are no limitations here.

[0100] Step 104: Using a preset load balancing strategy, distribute the data tables in the HBase cluster, excluding the metadata table, evenly to deploy the other data tables to the nodes in the second node group.

[0101] In this step, the management node stores a load balancing strategy. When the management node needs to execute step 104, it can retrieve the pre-stored load balancing strategy and other data tables in the HBase cluster besides the metadata table. It then splits these other data tables into multiple storage units and uses the load balancing strategy to evenly distribute these storage units across the nodes in the second node group. For each node assigned a storage unit, the storage unit is deployed to that node.

[0102] In this embodiment, upon receiving a node grouping instruction to group nodes in the HBase cluster, the HBase cluster is grouped into a first node group that does not require load balancing and a second node group that requires load balancing. Then, based on the metadata tables in the HBase cluster, nodes in the first node group are deployed, and a preset load balancing strategy is used to evenly distribute other data tables in the HBase cluster (excluding the metadata tables) to nodes in the second node group. It is evident that this application only performs load balancing on nodes in the second node group, not on nodes in the first node group, which stores the metadata tables. This means that using load balancing does not cause the metadata tables to move, avoiding the problem of metadata table unavailability due to movement. Since the metadata tables are available, nodes that do not undergo data transfer can still be accessed based on the metadata tables, and customer requests to these nodes can be responded to, preventing the HBase cluster from being unable to respond to all customer requests in the short term.

[0103] With the development of technology, the number of data tables stored in HBase clusters will increase, leading to a larger and larger metadata table. If the first node group includes too many nodes, it will result in a certain degree of resource waste; if the first node group includes too few nodes, some data will have nowhere to be deployed. Therefore, this application embodiment can use the metadata table to select nodes, avoiding an excessively large or small number of nodes in the first node group. Therefore, this application embodiment provides a node group determination method, which is as follows: Figure 2 As shown, the specific steps include:

[0104] Step 201: Obtain the metadata tables in the HBase cluster.

[0105] In this step, the management node can retrieve the metadata tables in the HBase cluster.

[0106] Step 202: The metadata table is split to obtain at least one first storage unit.

[0107] The storage unit contains data of a preset size, and the first storage unit is a storage unit obtained by splitting the metadata table.

[0108] In this step, after the management node obtains the metadata table, it can divide the metadata table into at least one storage unit according to the preset size, and these storage units are determined as the first storage unit.

[0109] Step 203: Select at least one node in the HBase cluster based on the number of the first storage units.

[0110] To ensure that each first storage unit has a corresponding storage node, the number of nodes selected can be greater than or equal to the number of first storage units. Preferably, to avoid redundant nodes in the first group, which would lead to resource waste, the number of nodes selected can be equal to the number of first storage units.

[0111] In this step, the management node can randomly select a number of nodes in the HBase cluster that is greater than or equal to the number of the first storage units.

[0112] Furthermore, the node selection method can be determined based on the state of the HBase cluster. Specifically, the state of the HBase cluster is checked. When the HBase cluster is in its initial state, the required number of nodes are randomly selected from the cluster. When the HBase cluster is not in its initial state, the nodes that have not yet deployed data are identified, and it is determined whether their number is greater than or equal to the required number. If so, the required number of nodes are randomly selected from these nodes; otherwise, these nodes are considered as selected nodes, and then some nodes are randomly selected from the nodes that have deployed data.

[0113] In implementation, after the HBase cluster is started, a trigger button for whether to use a load balancing strategy will be displayed on the control terminal. Users can control whether to use a load balancing strategy through this trigger button. When the terminal receives a trigger command to not use a load balancing strategy, it can be determined that the HBase cluster is in its initial state. When the terminal receives a trigger command to use a load balancing strategy, it indicates that the HBase cluster has likely already used a load balancing strategy, and at this time, the HBase cluster is determined to be in a non-initial state. When the HBase cluster is in its initial state, that is, the HBase cluster has just started but has not yet used a load balancing strategy to allocate data to the nodes in the HBase cluster, since each node in the HBase cluster has not yet loaded data, the required number of nodes can be randomly selected from the HBase cluster. When the HBase cluster is in a non-initial state, that is, the load balancing strategy has already been used to allocate data to the nodes in the HBase cluster, it is necessary to first pause the use of the load balancing strategy to avoid conflicts between the method described in this application and the load balancing strategy, which could lead to data processing errors. Afterwards, to avoid frequent data transfers when using the method described in this application, nodes without deployed data are given priority; when the number of such nodes is insufficient, nodes are randomly selected from other nodes.

[0114] Preferably, since metadata tables typically correspond to a single storage unit, and to avoid node waste, the required number is generally set to 1. Based on this, when the HBase cluster is in its initial state, a node can be randomly selected from the cluster. When the HBase cluster is not in its initial state, the nodes that have not yet deployed data are identified. If these nodes can be identified, a node can be randomly selected directly from among them. If no such nodes can be identified, a node can be randomly selected from among the nodes that have already been allocated data.

[0115] Furthermore, to prevent the HBase cluster from entering a non-initial state, the method described in this application can be run immediately after the HBase cluster starts up, and before using the load balancing strategy, to randomly select a node in the HBase cluster. The specific steps are as follows: immediately after the HBase cluster starts up, a trigger button will be displayed on the control terminal asking whether to run the method described in this application. When a trigger command to run the method described in this application is received, the method can be executed, and a node will be randomly selected in the HBase cluster during this step. When a trigger command not to run the method described in this application is received, it is not necessary to execute the method described in this application.

[0116] In addition, after receiving a trigger instruction to not run the method described in this application, the control terminal may also display a trigger button for whether to use a load balancing strategy, so as to determine whether to trigger the use of a load balancing strategy.

[0117] Step 204: Based on the selected nodes, determine the first node group that does not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second node group that requires load balancing.

[0118] In this step, the management node can select the nodes as the first node group to obtain the first node group, and select the remaining nodes in the HBase cluster as the second node group to obtain the second node group.

[0119] In this embodiment, to save computing resources, the management node can directly select a preset number of nodes in the HBase cluster to obtain the first node group. Therefore, this embodiment provides a method for determining a node group, the specific steps of which are as follows:

[0120] In the HBase cluster, select a preset number of nodes. Based on the selected nodes, determine the first group of nodes that do not require load balancing. Based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0121] The preset number is set based on the experience of technical personnel, and is generally set to 1. The node selection method in this step is similar to the node selection method involved in step 203, and will not be described in detail here.

[0122] Preferably, in the HBase cluster, the management node selects one node and sets it as the first node group, and sets the remaining nodes as nodes in the second node group, thus obtaining the second node group.

[0123] according to Figure 2In the method described in the embodiments, the number of nodes in the first node group is greater than or equal to the number of storage units corresponding to the metadata table, that is, the number of nodes in the first node group is greater than or equal to the number of first storage units. When the number of nodes in the first node group is greater than the number of first storage units, nodes without deployed first storage units can be used as backup nodes, and then, when other nodes fail, the first storage units in other nodes can be transferred to the backup nodes. However, when the number of nodes in the first node group is equal to the number of first storage units, data transfer needs to be performed based on nodes in the second node group. Therefore, the embodiments of this application provide a data transfer method, which is as follows: Figure 3 As shown, it specifically includes:

[0124] Step 301: Detect whether a faulty node appears in the first node group.

[0125] In this step, each node in the first node group periodically sends its relevant data to the fault detection node, allowing the fault detection node to determine whether the sending node has failed. The fault detection node can be the management node, any node in the HBase cluster, or a node outside the HBase cluster. When the fault detection node is not the management node, it needs to send the detection results to the management node so that the management node can determine whether a faulty node has appeared in the first node group.

[0126] Alternatively, each node in the first node group can detect whether it has failed and send the detection result to the management node so that the management node can determine whether there is a faulty node in the first node group.

[0127] Step 302: When a faulty node is detected, a backup node is selected from the second node group based on the number of faulty nodes.

[0128] In this step, when a faulty node is detected, the management node can select a certain number of nodes from the second node group based on the number of faulty nodes and use these nodes as backup nodes.

[0129] It should be noted that the node selection method involved in this step can be either to randomly select several nodes from the second node group, or to find nodes without deployed data within the second node group and obtain backup nodes based on these nodes. The specific execution steps are the same as... Figure 2 The node selection methods involved in the embodiments are similar, and will not be described in detail here.

[0130] Step 303: Transfer the first storage unit in the faulty node to the standby node.

[0131] In this step, when there is only one faulty node, the management node only needs to transfer the first storage unit from the faulty node to the standby node. When there are multiple faulty nodes, for each faulty node, the management node can randomly select a node from the standby nodes and designate it as the node to receive the first storage unit from that faulty node, and then transfer the first storage unit to that standby node.

[0132] It should be noted that if the standby node is still in the second node group, the standby node will still be adjusted when using the load balancing strategy. Therefore, to avoid this situation, the standby node can be added to the first node group.

[0133] In this embodiment, if the backup node is randomly selected from the second node group, the selected node may already have a second storage unit deployed. In this case, the second storage unit needs to be transferred to other nodes in the second node group. Therefore, this embodiment provides a data transfer method, as follows: Figure 4 As shown, the specific steps include:

[0134] Step 401: Check whether a second storage unit is deployed in the standby node.

[0135] The second storage unit is obtained based on other data tables.

[0136] In this step, the management node stores the data deployment information for each node. Based on this information, the management node can then determine whether a second storage unit should be deployed in the standby node.

[0137] Step 402: When the deployment of the second storage unit on the backup node is detected, the data receiving node is determined in the second node group.

[0138] In this step, when the deployment of the second storage unit on the standby node is detected, the node in the second node group that has not deployed data can be identified as the data receiving node, or a node can be randomly selected and identified as the data receiving node.

[0139] Step 403: Transfer the second storage unit to the data receiving node.

[0140] In this step, the management node transfers the second storage unit to the data receiving node.

[0141] After obtaining the grouping results, using a load balancing strategy might lead to data processing errors. To avoid this, after obtaining the grouping results, it is necessary to periodically check whether a load balancing strategy has been used. When a load balancing strategy is used, it means that the data allocation was performed using a load balancing strategy when grouping the HBase cluster. Nodes in the resulting first node group may have second storage units. For such nodes, the second storage units need to be moved to other nodes. Therefore, this application provides a data transfer method, such as... Figure 5 As shown, it specifically includes:

[0142] Step 501: Check whether a load balancing strategy is used.

[0143] In this step, the management node can determine whether to use a load balancing strategy to distribute data evenly.

[0144] Step 502: When the use of a load balancing strategy is detected, the use of the load balancing strategy is paused, and the presence of a node with a second storage unit deployed in the first node group is checked.

[0145] The second storage unit is obtained based on other data tables.

[0146] In this step, when a load balancing strategy is detected, the management node suspends the use of the load balancing strategy and checks whether there are nodes in the first node group that have deployed the second storage unit, based on the data deployment status of each node.

[0147] It should be noted that since the load balancing strategy is suspended in this step, it can be automatically resumed when needed, for example, in step 104.

[0148] Step 503: When a node deploying the second storage unit is detected in the first node group, a data receiving node is determined in the second node group.

[0149] In this step, when a node with a second storage unit is detected in the first node group, the second storage unit is obtained. In the second node group, a node without deployed data can be identified as a data receiving node, or a node can be randomly selected and identified as a data receiving node.

[0150] Step 504: Transfer the second storage unit to the data receiving node.

[0151] In this embodiment of the application, the user can also determine whether to cancel the group settings according to their own needs. Therefore, this embodiment of the application provides a group cancellation method, which specifically includes:

[0152] Check if a group cancellation command for canceling a group has been received in the HBase cluster; when a group cancellation command is received, retrieve the nodes in the first node group and add the nodes in the first node group to the second node group, so as to use the load balancing strategy to perform load balancing on all nodes in the second node group.

[0153] In this step, a button to cancel the grouping operation can be displayed on the terminal. When the user triggers this button, the terminal sends a group cancellation command to the HBase cluster. When the management node receives the group cancellation command to cancel the grouping of the HBase cluster, it can add the nodes in the first node group to the second node group, and then use load balancing strategies to load balance all nodes in the second node group.

[0154] In this application embodiment, a node processing device in an HBase cluster is also provided. This device corresponds to the method embodiment, and is specifically as follows: Figure 6 As shown, it includes:

[0155] The receiving unit 601 is used to receive node grouping instructions that group nodes in the HBase cluster.

[0156] Grouping unit 602 is used to group the HBase cluster into a first node group that does not require load balancing and a second node group that requires load balancing.

[0157] The first deployment unit 603 is used to deploy nodes in the first node group according to the metadata table in the HBase cluster;

[0158] The second deployment unit 604 is used to distribute the data tables in the HBase cluster, excluding the metadata table, evenly using a preset load balancing strategy, so as to deploy the other data tables to the nodes in the second node group.

[0159] Optionally, grouping unit 602 is used for:

[0160] Retrieve the metadata tables in the HBase cluster;

[0161] The metadata tables in the HBase cluster are split to obtain at least one first storage unit;

[0162] Based on the number of the first storage units, at least one node is selected in the HBase cluster, wherein the number of nodes is greater than or equal to the number of the first storage units.

[0163] Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0164] Optionally, grouping unit 602 is used for:

[0165] In the HBase cluster, a preset number of nodes are selected;

[0166] Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

[0167] Optionally, the device further includes a first detection unit 605, which is used for:

[0168] Detect whether a faulty node appears in the first node group;

[0169] When the faulty node is detected, a backup node is selected from the second node group according to the number of faulty nodes.

[0170] The first storage unit in the faulty node is transferred to the backup node.

[0171] Optionally, the device further includes a second detection unit 606, which is used for:

[0172] Detect whether a second storage unit is deployed in the backup node, the second storage unit being obtained based on the other data tables;

[0173] When the backup node deploys the second storage unit, a data receiving node is determined in the second node group;

[0174] The second storage unit is then transferred to the data receiving node.

[0175] Optionally, the device further includes a third detection unit 607, which is used for:

[0176] Detect whether the load balancing strategy is being used;

[0177] When the use of the load balancing strategy is detected, the use of the load balancing strategy is suspended, and it is checked whether there is a node in the first node group that has deployed the second storage unit, which is obtained based on the other data tables;

[0178] When a node deploying a second storage unit is detected in the first node group, a data receiving node is identified in the second node group.

[0179] The second storage unit is then transferred to the data receiving node.

[0180] Optionally, the device further includes a fourth detection unit 608, which is used for:

[0181] Check if a group cancellation command has been received for canceling a group in the HBase cluster;

[0182] When the group cancellation instruction is received, the nodes in the first node group are obtained and added to the second node group, so as to use a load balancing strategy to perform load balancing on all nodes in the second node group.

[0183] like Figure 7 As shown in the figure, this application embodiment provides a node processing device in an HBase cluster, including a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0184] Memory 703 is used to store computer programs;

[0185] In one embodiment of this application, when the processor 701 executes a program stored in the memory 703, it implements the node processing method in the HBase cluster provided in any of the foregoing method embodiments, including:

[0186] Receive node grouping instructions that group nodes in the HBase cluster;

[0187] The HBase cluster is divided into two groups: a first group of nodes that do not require load balancing and a second group of nodes that require load balancing.

[0188] Based on the metadata tables in the HBase cluster, deploy the nodes in the first node group;

[0189] Using a preset load balancing strategy, the data tables in the HBase cluster, excluding the metadata table, are evenly distributed to the nodes in the second node group.

[0190] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the node processing method in the HBase cluster provided in any of the foregoing method embodiments.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0193] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0194] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A node processing method in an HBase cluster, characterized in that, The method includes: Receive node grouping instructions that group nodes in the HBase cluster; The HBase cluster is divided into two groups: a first group of nodes that do not require load balancing and a second group of nodes that require load balancing. Based on the metadata tables in the HBase cluster, deploy the nodes in the first node group; Using a preset load balancing strategy, the data tables in the HBase cluster, excluding the metadata table, are evenly distributed to the nodes in the second node group.

2. The method according to claim 1, characterized in that, The process of grouping the HBase cluster to obtain a first group of nodes that do not require load balancing and a second group of nodes that require load balancing includes: Retrieve the metadata tables in the HBase cluster; The metadata tables in the HBase cluster are split to obtain at least one first storage unit; Based on the number of the first storage units, at least one node is selected in the HBase cluster, wherein the number of nodes is greater than or equal to the number of the first storage units. Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

3. The method according to claim 1, characterized in that, The process of grouping the HBase cluster to obtain a first group of nodes that do not require load balancing and a second group of nodes that require load balancing includes: In the HBase cluster, a preset number of nodes are selected; Based on the selected nodes, determine the first group of nodes that do not require load balancing, and based on the remaining nodes in the HBase cluster, determine the second group of nodes that require load balancing.

4. The method according to claim 2, characterized in that, The method further includes: Detect whether a faulty node appears in the first node group; When the faulty node is detected, a backup node is selected from the second node group according to the number of faulty nodes. The first storage unit in the faulty node is transferred to the backup node.

5. The method according to claim 4, characterized in that, After selecting a backup node, the method further includes: Detect whether a second storage unit is deployed in the backup node, the second storage unit being obtained based on the other data tables; When the backup node deploys the second storage unit, a data receiving node is determined in the second node group; The second storage unit is then transferred to the data receiving node.

6. The method according to claim 1, characterized in that, After grouping the HBase cluster into a first group of nodes that do not require load balancing and a second group of nodes that require load balancing, the method further includes: Detect whether the load balancing strategy is being used; When the use of the load balancing strategy is detected, the use of the load balancing strategy is suspended, and it is checked whether there is a node in the first node group that has deployed the second storage unit, which is obtained based on the other data tables; When a node deploying a second storage unit is detected in the first node group, a data receiving node is identified in the second node group. The second storage unit is then transferred to the data receiving node.

7. The method according to claim 1, characterized in that, The method further includes: Check if a group cancellation command has been received for canceling a group in the HBase cluster; When the group cancellation instruction is received, the nodes in the first node group are obtained and added to the second node group, so as to use a load balancing strategy to perform load balancing on all nodes in the second node group.

8. A node processing device in an HBase cluster, characterized in that, The device includes: The receiving unit is used to receive node grouping instructions that group nodes in the HBase cluster. A grouping unit is used to group the HBase cluster into a first node group that does not require load balancing and a second node group that requires load balancing. The first deployment unit is used to deploy nodes in the first node group according to the metadata table in the HBase cluster; The second deployment unit is used to use a preset load balancing strategy to evenly distribute other data tables in the HBase cluster, excluding the metadata table, so as to deploy the other data tables to the nodes in the second node group.

9. A node processing device in an HBase cluster, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to: Receive node grouping instructions that group nodes in the HBase cluster; The HBase cluster is divided into two groups: a first group of nodes that do not require load balancing and a second group of nodes that require load balancing. Based on the metadata tables in the HBase cluster, deploy the nodes in the first node group; Using a preset load balancing strategy, the data tables in the HBase cluster, excluding the metadata table, are evenly distributed to the nodes in the second node group.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the node processing method in the HBase cluster as described in any one of claims 1 to 7.

Images